If small size, I recommend lots of plants and capping any "live" substrate with sand. Had one going with shrimps and a few endlers for years. I only topped the water up and very occasionally changed the water. Not that it really needed it.
After the first few weeks, it just soldiered on with minimal algae or intervention needed. It's a lovely thing to have, I can recommend it. (That said, I also have a small aquarium problem according to the wifey)
Completely agreed. The point is the knowledge, the learning and the journey. If a kid has a passion for building or toying with LLMs, then of course, by all means, please start tearing them apart or even build and train your own model. You'll learn a ton, even if you won't necessarily end up using it here and now. The learning experience will compound and of course that will be useful.
The above is, after all, the whole genesis of the word 'hacker'. We should celebrate that.
How exactly does one go about "tinkering" with an LLM? Any architectural change you introduce needs fine tuning. That needs data and compute
I tried to modify the embedding output of bert to make it generate box embeddings instead of point ones. At the time I had access to university provided A100 gpus but even with all that a training run took half a day. Models these days I don't think I can train it in any reasonable time with that much compute.
(As a TML person, I'm obviously biased, but I couldn't resist because of "tinkering").
TBF it's hard to imagine a real architecture change that wouldn't require a ton of compute, but you could certainly fine tune and play with different recipes, loss functions, etc. And Claude can carry you a lot of the way through doing this.
One fun task is to invent a tool and then train a small model to use it. You could export that small model and run it locally for free forever to do your thing. I think this is what a lot of Software Engineering will look like later.
There are a lot of other high level abstractions here to look at. Prime Intellect has one.
The other thing to play with is self-hosting small models, but IMO most of the interesting stuff is actually related to multi-gpu or multi-node inference so there's not necessarily a ton to learn here.
This guidebook covers pretraining, post training (SFT, RL) and a couple other topics. And others authors have also written books that fit on single node reasonable hardware.
If you want to start with a pretrained base I built Gemma 270m and released it last year. This fits on a raspberry pi.
The fundamentals of AI don't require industrial amounts of large scale. Think of it like this, when I was learning how a plane worked when I was a kid I didn't build a 747 at home, I started with scale sized model planes. Same idea here.
And FWIW I'm a staff researcher at Deepmind (opinions here are my own) so I want to specifically encourage all people out there, you can learn a lot about how these LLMs work at home, for (mostly free), using resources like colabs or spot pricing on accelerator providers. There's many great resources out there and I encourage anyone willing to learn to go for it!
This is one question that is not just for kids but for all "gpu poor" adults as well. One concept that I found useful is "toy models": small compute models that replicate a certain, not very well explained phenomenon that the large models have. Of course, you can't have a toy model for everything, some behaviors are only visible at scale. But, there is a bunch of things like in-context learning that you can play with without burning a ton of compute.
The best analogy is strip mining (big labs) vs cave exploration (solitary/small teams). I think this is how science progresses at the boundaries by smart/curious/hardworking individuals because depth is a requisite for finding the right questions and then the answer. It is not for everyone and it does not always work. But you learn a ton even if it doesn't pan out to be a big breakthrough.
Seeing a lot of criticism in here, some valid, some not. Just wanted to say that, to me, this is really well put together - and seems like a lot of thought went into this. Appreciate you releasing this publicly!
I actually don't disagree with you on this. Can absolutely see how this can create friction, but it's also one of those cases when dealing in absolutes is fraught with peril. A CTO is not a CPO and having strategies being defined - or steered by a singularly technically-minded person can work fine in a lot of scenarios, but for others, it's important to foster a certain level of autonomy for managing, growing and building the product, feature set and roadmap. Same thing in reverse.
There's definitely overlap between the two, but a collaborative approach works better in my opinion. As a disclaimer, I'm a CPO myself - and enjoy the working relationship and collaboration with the CTO, while reporting directly to the CEO and the board. Also, I guess that sometimes it really comes down to the types of personalities that are present in the management team - as well as the type of product being built.
Saying "call X" where X doesn't have an English name(and Siri is set to English) is just bound to fail. It's like the most basic functionality that all voice assistants fail at miserably - google assistant is equally garbage at it.
These assistants are really not working well in multi-lingual settings but I suspect that the stuff Apple was doing wouldn't improve it. Some functions are disabled if you have Siri in one language and your device in another.
But I mean, surely it's not that unusual to have English as the language set to everything, to live in an English speaking country, and yet either have a foreign name yourself or just be friends with someone who does. And it's not that Siri/Google Assistant don't recognize those names period - they would, had they been set to the language those names originate from. But if you're set to English then forget it. And yes, I understand this is a difficult problem to solve, but I had I not known anything about internal workings of technology and was just a regular customer, I would absolutely consider the ability to call someone "basic functionality".
I feel compelled to answer here, mostly because I've been and to a degree still are where you are.
It's extremely common to gradually move into a state of mind where you denigrate yourself – and the effort you've put forth in life. You need to ensure your own happiness, and no one in here, or anywhere else, can tell you how to do that. Mindfulness is a great tool, because it allows you to become more certain of yourself. To discover what it is that you feel and where it is you need to go.
That being said, here's what works for me: Acknowledging that creation is hard, but that I love creating and helping people around me. That family is important and love even more so. I also remind myself to do something new every day that somehow moves the boundary of what it means to be me.
Added note: A book that also helped me realise my own tendency to create barriers around myself was "The Flinch" by Julien Smith. It's awesome and I highly recommend giving it a read – and practice some of what it champions. I suspect you will gain a lot from it, but then again: Only you can know that.
I find that a lot of companies that refer to themselves as being bearers of startup culture, is mostly trying to cater to younger people or at least a young mindset - not because they're necessarily trying to hide insane working hours or demands. Sometimes it truly is about passion and keeping a "beginner's mind". Definitely not for everyone, but to me it feels very inspiring and empowering.
(I work for 15+ year company that like to view itself as a startup - but the partying is definitely optional)
After the first few weeks, it just soldiered on with minimal algae or intervention needed. It's a lovely thing to have, I can recommend it. (That said, I also have a small aquarium problem according to the wifey)