This is one of the few places with consistent, high-quality setup instructions for essentially every major AI provider in one place — American frontier labs (OpenAI, Anthropic, Google, xAI), Chinese labs (DeepSeek, Alibaba/Qwen, Moonshot, Z.ai), aggregators (OpenRouter), fast inference providers (Groq, Cerebras, Together), and local runtimes (Ollama, LM Studio, LocalAI). This info exists elsewhere but is scattered across each vendor's docs with wildly varying quality. It also covers the common gotcha that a ChatGPT Plus / Claude Pro subscription is not an API key.
Thanks for the encouragement! Yes, he's been sharing with his friend family groups. Early feedbacks are pretty good, he's ecstatic that he can actually make this happen!
We are in this transition period where we'll see a lot of these, because of the effort of creating "something impressive" is dramatically reduced. But once it stabilizes (which I think is already starting to happen, and this post is an example), and people are "trained" to recognize the real effort, even with AI help, behind creating something, the value of that final work will shine through. In the end, anything that is valuable is measured by the human effort needed to create it.
Does anyone have real life experience (preferably verified in production environment) of fine-tuning actually adding new knowledge to the existing LLM in a reliable and consistent manner? I've seen claims that fine-tuning only adapt the "forms" but can't adding new knowledge, while some claim otherwise. I couldn't convince myself either way with my limited adhoc/anecdotal experiments.
I’ve taught LLMs imaginary words and their meanings with minute amounts of data (two or three examples) via full fine-tuning, LoRA and QLoRA.
I have no idea where the myth of ‘can’t add new knowledge via fine-tuning’ came from. It’s a sticky meme that makes no sense.
Pretraining obviously adds knowledge to a model. The difference between pretraining and fine-tuning is the number of tokens and learning rate. That’s it.
It seems like few shot prompting and providing some examples to LLMs with large context windows vastly out performs any amount of rag, or fine tuning.
Aren't rag and fine tuning fundamentally flawed, because they only play at the surface of the model? Like sprinkles on the top of the cake, expecting them to completely change the flavor. I know LoRA is supposed to appropriately weight the data, but the results say that's not the solution.
Long context windows get confused, so shorter is better, and they cannot fit everything in general. I'm not sure where you are seeing results that say otherwise.
RAG is effectively prompt context optimization, so categorically rejecting doing that doesn't make sense to me. Maybe if models internalized that or scaled... But they don't.
Totally agree. Every decision on what context to put in a context window is “RAG”. Somehow the term was co-opted to refer to “context selected by vector similarity”, so presumably when people say “is RAG hanging around”, what they mean is “are vectors a complete solution”, to which the answer is obviously “no”. But you still need some sort of _relevance function_ to pick your context - even if it’s pin-the-tail-on-the-donkey. That’s “RAG”.
Doesn’t make sense to ask “will we still have to curate our context?” The answer is of course you will.
That's definitely my experience as well, sufficiently large context window with a capable enough general purpose LLM solves lots if not all of the problems rag/fine tuning claim to solve.
I've also found (anecdotal) significant success in just throwing in available context before prompting. I've written multiple automations in this way as well.
Yeah... So looks like at least it's still an open question. I guess until we can definitively know how "knowledge" is collectively represented among the weights, it's hard to say either way. The other part of the question is how to evaluate the existence of "knowledge" in an LLM. TFA suggests a way, but still not 100% convinced that's THE way...
Not really answering your question, but all the "alignment" of the big models is done through a combination of supervised fine tuning and RLHF. So all the chat and censorship and other specific behaviors are at least in part fine tuned in. Maybe that is closer to forms rather than actually knowing more...
Would be curious to see if anyone find it really useful. I've tried both Copilot and Codewhisper (Amazon Q now) before, wasn't impressed and uninstalled both. Just tried Q in VSCode again, I can't figure out how to ask questions relevant to the specific workspace that's useful to me. It seems like a bolt-on chat interface to your IDE with a bad UX. Feels like even "clippy" was more useful back in the day...
I tried out Copilot Workspaces before they fixed the waitlist access check yesterday and it seems to work a lot better by running a multistep process and allowing the user to incrementally modify the plan and rerun code generation.
The UI is similar to the code review interface except the file list on the left is generated from a plan and there's a bullet point list for each file of changes the plan generates. It enables a REPL loop where the AI code gens, the user tests the changes, then updates the file plans and reruns code generation again, creating and adding files as necessary. The end result is a PR with generated description or a commit direct to main.
I'm excited for this next wave of AI coding agents but Amazon seems to have rushed this into production
The Beam feature of bigAGI (IMO one of the best model provider agnostic GenAI UX) enables GenAI users to send same prompt to multiple GenAI models at the same time, and gives the user different approaches to examine, select and fuse the best results into a better answer, through a very intuitive and seamless UX. It has been my go-to way of using GenAI in the past few weeks. IMO the results are better than any individual model's results alone. The best thing is, it could (semi)automatically select the best results from the models, for instance, it used to be the Claude 3 Opus model's results were favored, now that the best results lean more towards gpt-4-turbo-2024-04-09, but you can achieve "best model auto selection" with Beam without having to manually pick one model over the other.
Exactly my thought, as mentioned in the other thread, Chat's linear conversation style is not fit for reasoning/exploration type of tasks, while Beam's fan-out -> select --> merge is a much better and natural flow!
With Beam, we can easily experiment approaches such as Chain-Of-Though-with-Self-Consistency (CoT-SC) and other reasoning meta framework, but with more manual control. I always had issues using LLM's chat driven interface to figuring out/explore issues that i'm interested, since conversation/chats is always linear while reasoning/working on some ideas is structural. Beam seems to be a much better UX than the linear chat UX that saves me a lot of copy and paste and save and retry. Awesome work!
Yes, the only issue is the usage of tokens, which is obviously greater as we are sampling more of the solutions space. But it's a compromise to have GPT-4.5 level intelligence with GPT-4.
Probably even higher jump as the models have some amount of unique training data, and they are fact-checking each other, to a more common “truth”, and hallucinations are weeded out.
