The product names have made me chuckle quite a few times in the past. Such an easy way to make your products more likeable.

Also, maybe semi-relevant: I made a small guessing game about the IKEA product catalogue in case someone wants to be lightly entertainment for two minutes [1].

[1] https://deriese.net/ikeaguesser.html?s=hn

Nice one! I built something similar [1]. It has you type in the name of the actual song and artist and the game compares it to a database of lyrics. The goal is to find as many songs as possible that contain a given word (or one of its derivatives) within one minute. Getting actual lyrics was the hardest part, so not all songs are in there.

[1] https://deriese.net/songs.html?t=hn

sick!!

There is still a difference though. If some company decides to raise prices on OR, you can just switch to any other provider of the same model since there is no moat.

As always: it depends on your needs. Here's a very basic heuristics rundown:

- More RAM: bigger models, more intelligence.

- More FLOPs: higher pre-fill (reading large files and long prompts before answering, the so-called "time to first token").

- More RAM bandwidth: higher token generation (speed of output).

So basically Macs (high RAM, okay bandwidth, lowish FLOPs) can run pretty intelligent models at an okay output speed but will take a long time to reply if you give them a lot of context (like code bases). Consumer GPUs have great speed and pre-fill time, but low RAM, so you need multiple if you want to run large intelligent models. Big boy GPUs like the RTX 6000 have everything (which is why they are so expensive).

There are some more nuances like the difference of Metal vs. CUDA, caching, parallelization etc., but the things above should hold true generally.

Do you think Apple will fix prefill speed with the M6 Max MacBook Ultra 128GB?

It's already greatly improved over previous generations due to M5s having tensor cores (higher compute capacity for matmul operations, the bottleneck for prefill).

For the same reason that a human who is fluent in five languages can probably express themselves better in either one compared to human that only speaks one, while also having a more nuanced understanding of general grammar. From what I know, learning on a more diverse set makes a model better overall.

This might be an interesting research question: can you train a model on many languages, and then extract a much smaller model that knows only one language without much loss of quality?

Humans brains and LLMs are not the same, though. I don't think your analogy is remotely applicable, even if your conclusion may be correct.

Nice one! Let's say I'm serving local models via vllm (because ollama comes with huge performance hits), how would I implement that in gomodel?

I've released a new version of GoModel (0.1.20) with explicit support for vllm. You can now use it even with a few vLLM instances. Like this:

  docker run --rm -p 8080:8080 \
    -e VLLM_BASE_URL=http://host.docker.internal:18000/v1 \
    -e VLLM_BASEMENT_BASE_URL=http://host.docker.internal:18000/v1 \
    enterpilot/gomodel:latest

This is way more interesting to me as well. I have projects that use small limited-purpose language models that run on local network servers and something like this project would be a lot simpler than manually configuring API clients for each model in each project.

Thanks for raising it! Since vLLM has an OpenAI-compatible API, this should work for now:

  docker run --rm -p 8080:8080 \
    -e OPENAI_API_KEY="some-vllm-key-if-needed" \
    -e OPENAI_BASE_URL="http://host.docker.internal:11434/v1" \
    ...
    enterpilot/gomodel

I'll add a more convenient way to configure it in the coming days.

I'm trying to get back to verifying some of my old fun ideas. I want to finally build my 3D QR cube (https://deriese.net/qrcubes.html?s=hn) by sending a design to a laser shop, and I also want to find someone with a few termocouples to verify my results to the coffee cup cooling problem (https://deriese.net/coffee.html?s=hn). If anyone wants to help, feel free to send me a message.

Ever since I read about this, I have been thinking about the next logical step: train a NN to route the internal loops dynamically after each layer. Instead of just choosing a given set of layers that are repeated, let the new classifier decide whether it wants to loop, where it wants to loop, whether to loop multiple times, to loop a big part, or to just jump to the final layers straight away. Each token could loop more or less based on its relevance.

It has some similarities of a MoE architecture, but instead of choosing experts, it chooses layer routes. Training this NN classifier together with the LLM could condense the required amount of layers for a given intelligence down drastically if it works. If anyone wants to work on this, feel free to send me a message.

So they use a valid and valuable company to hide a giant dumpster fire company. To add to that, their best argument is "AI in space", which has some real "solar roadways" energy to it. I honestly don't know how any SpaceX shareholder could approve this.

> If you post for ad revenue, I truly feel sorry for you.

I think this is a bit dismissive towards people who create content because they enjoy doing it but also could not do it to this level without the revenue, like many indie Youtubers.

If I could press a button and remove money from the internet, I'd do it in a heartbeat.

I absolutely do enjoy content that is financed through ads. I really, REALLY do like some of the stuff, honest. But it is also the case that the internet has been turning into a marketing hellscape for the last couple decades, we've gotten to a point in which engagement is optimized to such a degree that society itself is hurting from it. Politically and psychologically. The damage is hard to quantify, yet I can't fathom the opportunity cost not being well into the billions.

We'd be better off without that.