if you want to be lazy, 7b = 7gb of vRAM, 12b = 12gb of vRAM, but quantizing you might be able to do with with ~6-8. So any 16gb Macbook could run it (but not much else).
Welp, my data point of one shows you need more than 8 GB of vRam.
When I run mistral-chat with Nemo-Instruct it crashes in 5 seconds with the error: "torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 40.00 MiB. GPU"
This is on Ubuntu 22.04.4 with an NVIDIA GeForce RTX 3060 Ti with 8192MiB. I ran "nvidia-smi -lms 10" to see what it maxed out with, and it last recorded max usage of 7966MiB before the crash.
When I run mistral-chat on Ubuntu 22.04 after cleaning up some smaller processes from the GPU (like gnome-remote-desktop-daemon) I am able to start Mistral-Nemo 2407 and get a Prompt on RTX 4090, but after entering the prompt it still fails with OOM, so, as someone noted, it narrowly fits 4090.
Agreed, it narrowly fits on RTX 4090. Yesterday I rented an RTX 4090 on vast.ai and setup Mistral-Nemo-2407. I got it to work, but just barely. I can run mistral-chat, get the prompt, and it will start generating a response to the prompt after 10 to 15 seconds. The second prompt always causes it to crash immediately from OOM error. At first I almost bought an RTX 4090 from Best Buy, but it was going to cost $2,000 after tax, so I'm glad that instead I only spent 40 cents.
What about for fine-tuning? Are the memory requirements comparable to inference? If not, is there a rule of thumb for the difference? Would it be realistic to do it on a macbook with 96G of unified memory?
Yes, but it's not common for the original model to be 8 bit int. The community can downgrade any model to 8 bit int, but it's always linked to quality loss.
no, it is basically an 'auto-correct' spell checker from the phone. It only knows what it was trained on. But it has been shown that a coding LLM that has never seen a programming language or a library can "learn" a new one faster than, say, a generic LLM.
For me, personally, Copilot is the best because you are typing and it is just suggesting. Very often I type something, wait 2 seconds for its answer (knowing it is going to get it right 80% of the time) and press tab. Sure, it misses a ton, but I see it as a great autocorrect that is always there trying to help. Now, copilot chat and everything else are very bad. I don't like them.
If you are "intentional" I usually just use ChatGPT-4 directly. I've had a super pleasant experience with Cursor, too. You can just open Cursor and say "write readme for me", it will scan your project and right a decent readme. It is also eeeeextremely goooood at "I need to know where in the codebase is doing this" and it finds. But for day to day I still prefer chatgpt 4.
So I would say:
- If you want a nice autocorrect, copilot
- If you want something that knows your project and tries to help, cursor
- If you just want something smart, chatgpt with gpt-4.
I would say copilot lets me be 15% faster daily, and gpt-4 about 30% faster (when I need it), but I need it less often - maybe twice per day.
Have you ran into issues with Copilot where it does things like:
- Doesn't close brackets / adds too many closing brackets
- Spams completions while you're writing comments
- Doesn't auto import the required libraries after accepting a suggestion
- No multi cursor mode
- Refuses to name variables
- Refuses to trigger in the middle of a line
- Lower quality code than ChatGPT (probably because they use a smaller model)
Asking because I was also a heavy Github Copilot user but after 1-2 years of constant subpar UX and seemingly no effort from the Copilot team to fix it, I just went ahead and built my own extension that fixes all of the above: https://double.bot
Also it wasn't on your list but you mentioned GPT-4 is smart. You should try the new Claude 3 Opus, we implemented it on Monday and have been getting super positive feedback from users!
even if that's true, the A series neural engine is much worse than the M series, I'll say it wrong, but it can only do 32bit inference (or something like that) where M series can do 64bit, so A series can run LLM but has a series of limitations that M series doesn't.
Practically speaking, most models today infer at 8bit or 16bit (sometimes, rarely 32). You don't see an empirical lift at more bits of precision. Size of the memory is far more important.
Sometimes gradients are small but meaningful, if you constrain them to too few bits / degrees of freedom they'll be unable to backprop successfully. This can hamper training and therefore results quality.
You can also think about it as compounding errors - at any one weight index the bit values might not be too meaningful, but cascaded over a lot of tensor multiplications they will be.
Author here. Yeah, it was supposed to be "here are my frustrations", not like "Flutter is bad, let's burn it".. but of course, the thing grew up beyond control. 38k view so far.
Author here. Never expected it to reach here. 24k views and it's been ~14 hours since I wrote it. I agree with you. There is currently a competition between kotlin multiplatform and flutter, a shared backend vs a shared UI. Guess what, exactly what you said, it's a lot easier to share the backend than recreate the full UI like a game does. I'm not against flutter, but it feels too immature for now and the pains really show up. Apple showed up their new UI toolkit for developers, which does animations in the best way possible, Flutter team reaction was "you can make a library to improve it if you want" and "we should improve our docs" but no one will recognize there are issues in the framework that could be improved.
BMW seems happy with flutter and are opensourcing some of their flutter libs.
Quote from CTO Connected Company at BMW
“By combining Dart and Flutter we have the first true cross-platform mobile toolkit; we feel it is a game changer to ensure feature parity for digital touchpoints and IoT.
By moving forward with world class tooling, automation and modern functional programming patterns we can improve feature cycle time, security, and cost of delivery of features for the business.”
It feels like he could have said that about any of the relevant frameworks. They're happy they've started down the cross-platform path using one of the solid available alternatives -- good on them!