Good to hear you figured it out with router settings. I’m also new to this but got all that figured out this week. As other commenters say I went with a reverse proxy and configured it. I choose caddy over nginx for easy of install and config. I documented just about every step of the process. I’m a little scared to share my website on public fourms just yet but PM me ill send you a link if you want to see my infrastructure page where I share the steps and config files.

Nice post Hendrik thanks for sharing your knowledge and helping people out :)

I once got kobold.CPP working with their collection of TTS model+ wav tokenizer system. Here’s the wiki page on it.

It may not be as natural as a commercial voice model but may be enough to wet your appetite in the event that other solutions feel overwhelmingly complicated

Wow this is some awese information Brucethemoose thanks for sharing!

I hope you dont mind if I ask some things. Tool calling is one of those things I’m really curious about. Sorry if this is too much please dont feel pressured you dont need to answer everything or anything at all. Thanks for being here.

I feel like a lot of people including myself only vaguely understand tool calling, how its supposed to work, and simple practice excersises to use it on via scripts and APIs. What’s a dead simple python script someone could cook to tool call within the openai-compatable API?

In your own words what exactly is tool calling and how does an absolute beginner tap into it? Could you clarify what you mean by ‘tool calling being built into their tokenizers’?

Would you mind sharing some sources where we can learn more? I’m sure huggingface has courses but maybe you know some harder to find sources?

Is tabbyAPI an engine similar to ollama, llama.cpp, ect?

What is elx2,3, ect?

Pangolin.

Yes it would have been awesome of them to release a bigger one for sure :( At the end of the day they are still a business that needs a product to sell. I don’t want to be ungrateful complaining that they dont give us everything. I expect some day all these companies will eventually clam up and stop releasing models to the public all together once the dust settles and monopolies are integrated. I’m happy to be here in an era where we can look forward to open licence model released every few months.

At the end of the day you need to decide what kind of person you are. Are you pragmatic or idealistic? Are you able to separate art from artist and creation from creator? Should you support a good open source service created for the betterment of everyone if you dont like the politics of its developers?

I’m a pragmatist by nature. I believe that a useful tool remains a useful tool even when its crafted by tankie assholes. If I found out the maker of a computer command like sudo was a leninist or whatever I wouldnt go out of my way to install an alternative just because I dont agree with the batshit politics of the creator. Just like I wouldn’t stop enjoying a song after finding out the ones who made it were greedy egotistical dickheads in personal life.

Young and politically charged idealist love that online social justice warrior signalling and political identity posturing. Everythings gotta be us vs them culture war, with us needing to always be on the morally/politically high ground else your a filthy inhuman nazi them who must be refunded/canceled. You get older and realize most people no matter the lean have some level of dogshit half baked politics or some other degree of mental emotional whackiness from past trauma or poor life circumstances causing them to be imperfect animals with dumb fucking biases. That’s humanity and the heart of darkness for ya. You can choose to associate the imperfections of the creator with the creation, or you can try to decouple them and see them as separate entities linked by causality.

The free API is the gateway drug to selling the product to customers (devs). The average consoomers attention and data is the product to be sold to customers (advertisers and big brother)

In the comics Thanos was straight up the right man hand/side hoe of the concept of death made manifest the whole point of offing half of all life was just to make mistress death happy (I think she actually ordered it). A lot of the original plot points of the infinity war and its buildup was lost in translation. Movie thanos motivations werent even half baked, he could have doubled resources and made a paradise for life with his motivations

Ding Ding, check this comment chain for your answer. Today you, tomorrow me.

Question one: yes and no. Most of the vomiting emojis shared here in comments are fake made using googles emoji kitchen thing. But there are many real modifiers for emojis like skin color or adding accents like tildes to regular english alphabet characters.

Question two: Modern keyboards typically have most emojis built in for you to select through. I dont think typing in the unicode values will automatically convert on phone operating systems but this might help if using windows or programming into a website.

the XKCD explained article on this actually gave some really great info.

You can tell because of checks notes “ThE ThIcK LInES AnD Off ColoURinG”

Devstral was released recently specifically trained for tool calling in mind. I havent personally tried it out yet but people say it works good with vscode+roo

Thanks for the input! I do eventually plan on making some scripts and a custom web interface to interact with/expose some local services on my network once I have the basics of HTML covered as part of a portfolio thing so would like to cover my ass early and not have problems later

The thing is that even if there isn’t much energy in plastic to be extracted, theres still enough energy in it to make a viable food source. Now, consider the humble koala and its primary food source, fucking eucalyptus leaves. Eucalytis is such a dogshit food source that koalas had to spend evolutionary time and energy just to spec into it. To the point they cant eat anything else pretty much. Combine that with the fact that eucalyptis leaves are so devoid of nutrients that the koala has to spend all day every day just snacking on them to not die of malnutrition.

Why? Why would a species even bother with this flim-flam if eucalypti sucks that bad as a food source? The answer is: Food scarcity. Because eucalytis grows everywhere where koalas live and because nobody else is bothering to tap into the food source, this sets up a ecological niche by pretty much gaurenteeing any animal that sucessfully finds a way to make it work will have unlimited amounts of food/energy just from the fact theres so damn much of it and nothing else wants to/can touch it. Sure koalas might have paid the price by sacrificing some brain wrinkles but who needs higher intelligence when you have leaves to snack on and sex to make babies.

A similar thing happened with trees and mushrooms. In the deep evolutionary history of our planet trees were once the apex forms of life with forest covering pretty much the whole planet. This is because nothing knew how to eat the wood stems for a good couple million years. Most of the coal and oil that we dig up today is actually the preserved remains of these unbroken down trees from the carboniferous period that just layed there petrified never rotting until the earths techtonic movement buried the tree corpses deep enough in the mantle for the carbon to compress into hard rock or squeezed+heated into liquid. The great change in the era happened when our humble mycelium bois finally figured out how to eat wood, causing them to essentially become the new apex life for a time by taking advantage of an unlimited and untapped food source (trees).

I suppose my point is to not underestimate the willingness of life to find new food sources. microorganisms don’t need much excuse just a slight amount of selective pressure and a couple million/billion/trillion generations of evolutionary trial and error. Which for bacteria takes maybe a couple of years I forget how quick modern microorganism colonies make new generations but its FAST. Add some science nerds who love to play God/intelligent evolution with CRISPR tek and gene tagging, yeah for sure well get plastic eating microbes figured out. Then begins the pandoras box of plastic rotting when we dont want it to.

Havent heard of this one before now. It will be interesting to see how it actually performs. I didnt see what license the models will be released under hope its a more permissive one like apache. Their marketing should try cooking up a catchy name thats easy to remember. It seems they’re a native western language company so also hope it doesnt have too much random Chinese characters like qwen does sometimes

Ive never really gotten into MoE models, people say you can get great performance gains with clever partial offloading strategy between various experts. Maybe one of these days!

If your running into the issue of an app wanting an api key for your local ollamas openai-compatable web interface API and refuses to work without one, I found that any random characters work. If you port forward your host computer you should be able to access the webui interface on an external network using the public IP.

Heres the dead simple python program I used to send and recieve text to kobold.cpp engine through the web API. Not sure how similar ollama but afaik openai-compatable API means it all should works close to the same for compatibility(I think? lol!) if you give it a shot Make sure to set the .py file you make as executable and run it from a terminal doing ./filename.py to see the output in real time. It should make a log text file in same dir as the program too. Just use your host computers local ip if the python script pc is on same network.

import requests

# Configuration
API_URL = "http://10.0.0.xx:5001/api/v1/generate"
PROMPT = "Tell me a short story about a robot learning to dance."
OUTPUT_FILE = "output.txt"

# Define the API request data
data = {
    "prompt": PROMPT,
    "max_length": 200,      # Adjust response length
    "temperature": 0.7,     # Control randomness (0=deterministic, 1=creative)
    "top_p": 0.9,           # Focus on high-probability tokens
}

# Send the request to kobold.cpp
response = requests.post(API_URL, json=data)

if response.status_code == 200:
    # Extract the generated text
    result = response.json()
    generated_text = result["results"][0]["text"]
    
    # Save to a text file
    with open(OUTPUT_FILE, "w") as f:
        f.write(generated_text)
    print(f"Response saved to {OUTPUT_FILE}!")
else:
    print(f"Error: {response.status_code} - {response.text}")

What is this post even talking about why does your silverware have a pillow

Just wondering, why do you think that coke users stay away from psychadelics and vice versa?

VSCode + roo plugin seems to be all the hotness for coders leveraging ‘agenic teams’ so I spent a bit playing around with it. Most local models dont do tool calling very well I need to see if devstral works better without giving errors. I hear real professionals use claude API for that kind of stuff.

Im only vaguely familiar with getting computers to send, recieve, and manipulate data with eachother on a local network so got a very basic python script going pointed at kobold cpps openai-compatable API to send prompts and recieve repliesinstead of the default webui app just to learn how it works under the hood.

One of my next projects will be creating a extremely simple web based UI for my ereaders basic web browser to connect to. kobold has something similar with the /noscript subpage but even that is too much for my kobo reader. I intend to somehow leverage a gemtext to html proxy like ducking or newswaffle to make the page rendering output dead simple.

One of these days im going to get a pi zero and attach it to a relay and see if I can get a model to send a signal to turn a light on and off. Those home automation people with the smart houses that integrate llms into things look soo cool