I think I am on your arc as well. My learning on different topics is growing every day, but there’s a limit to how much I can absorb. With the LLMs the experiments stay just beyond that horizon and I keep chasing.
Stated too strongly, but I think this could be the model for education (some subjects anyway). Everything personalized to your learning goals, grounded in experiments that give a tight feedback loop and with a model that never gets tired of re-explaining something for the 10th time.
Yes they are great for learning at own pace, trying out new things.
I have accepted two things that make me a happy engineer now: AGI is not here no matter what they say and LLMs are still very useful if one knows how to use them.
They are another layer of abstraction and like you said they do not tire. There is a lot of optimization needed so we can reduce wastage (running 1T+ LLMs for most work is wastage).
I realized the same while reading it, but my impression is that the output has been well edited. The typical Claude density has been pulled out into something more human readable, but with many of the Claud-isms still intact.
Personally I find this a productive way to produce documents. Claude makes the first pass and I edit line by line. It’s a good understanding check. If I can’t reword something into my own voice it means I don’t understand it. Then have Claude loop again to fact check the rewrites and repeat until clear (or I disagree with Claude on all it’s remaining points).
That’s the same edit loop I used without LLMs, but I get speed in drafting up front (and fact/number pulling) and the check passes make the edits better.
And maybe author sent ~1K tokens as the starting prompt, and possibly some more stuff in the system prompt, and add on top of that that Apple hardware is famously bad at prompt processing.
I have been using a PR branch to run GLM 5.3 flash locally so I have been watching the dueling PRs develop to implement it and all the comments and reviews. You're right the scale of it is huge. The speed with which maintainers and community are responding is also super impressive.
I know there's a lot of discussion about AI overwhelming OS maintainers and I can see that. It does seem like llama.cpp is successfully riding that dragon right now though.
Until 1-2 months ago, AI-written code was not allowed in llama.cpp. That changed, but it's still not allowed to have AI write your posts for you (issues, PR descriptions, etc.). In my experience, banning AI-written prose leads to much better contributions than on projects where it's allowed.
Of course, llama.cpp also gets lots of AI slop PRs from first-time contributors, but from what I've seen, maintainers simply ignore the bad submissions and prioritize the good contributions. I think that's probably one of the best ways to handle the slop overwhelm.
I’ll be very curious what you get with DDR4. I also almost went that way. I have an Epyc DDR 5 rig and the best I see is 10 tok/s. Caveat being that’s at Q8 and a 4090 doing pre fill so it could be pushed up.
The surprising thing for me is how much work you will need to cool the banks if you’re near your memory ceiling. My memory starts soft throttling at about 74C (dies may be hotter, that’s the bank temp) and will turn down speed to try to stay below 80.
Happy to send my llama.cpp config settings if you want it.
I am getting 10t/s on unsloth's Q3kxl with 2x3090s@250w. It's enough for me for now. I will probably upgrade the GPUs down the line. DDR5 would have made the price of the machine double and I just wasn't prepared to pay that much.
I find 10 to be very usable. It’s not (that) interactive but it chews through tasks. I let Kimi churn away at 4 overnight and it gives good results that are ready for me in the morning.
Typically computers with these larger memory amounts have fans that scream like a banshee trying to move impossible amounts of air over the memory and CPU. Getting something both cool and quite can be a bit difficult.
Yes, I thought when I was starting that 1u and 2u form factors were to save space. Maybe they are, but they also have the advantage of moving air front to back very effectively through and over the components. Though there still must be some need because I see even those boxes have optional manufacturer built memory shrouds to try to force airflow between the DIMMs.
I had some 120x38mm fans from another server box that I pulled out because they were too loud and I didn't need the static pressure they were giving. They went in here. That 13mm (and the extra 1k rpm) moves so much more air.
I built a dual epyc server with 64 cores and 1 TB of DDR4. Draws around 800W or so under load. I used off the shelf liquid cooling. It is audible but not noisy.
The trick is to turn on the cooler's RGB in your 6000€ server to get a free speed boost. I am not liable for sysadmin's heart attack upon reading this.
This is something devs don’t give enough credit. You may think you’re just promoting, but ask someone outside the field to prompt their way to a feature and you will see very clearly what your experience is bringing to the table.
Yeah, but train them for a month on how to get the AI to help them prompt, and they'll do fine. AIs are extremely effective at letting you offload skill, you just need to know how to ask. This should surprise nobody, since this is what they were and built for.
I genuinely don't think you need much training for creating code any more. The skill can be offloaded.
I don't disagree in principle, but there's got to be a ceiling at some complexity. Teach them to prompt their way into a Twitter clone in a month (or less), sure. Teach them to prompt their way into a Bun rewrite in Rust in a month...well, there are apparently some people who need your help right now ;)
I had a conversation with someone about this the other day where they asked what AI was useful for, and I said programming, and they asked what programming was, and I eventually stumbled into "it's how you make apps".
Fascinating. What an extreme outlier. Where did you find them? Did you also have to explain what an app was?
Anyways, if being more capable than them is the bar you think you need to clear, I suppose you'll be fine.
I still stand by my statement: there's no deep knowledge needed to deliver most of the software we need today, just some iteration on prompts. And the AI is getting better pretty quickly. You need to be able to explain what you want, why the AI got it wrong, and paste the error you see. It takes some focused training, but so does being a line chef at a restaurant.
I’m m just starting to get results on a local/cloud agent pipeline I’ve been building. I don’t think I will ever code again. Plan, review, read, architect, test, yes. Code though simply doesn’t make sense when I’m getting these results at this velocity.
Every few months it feels like I have moved further up the chain. Now I don’t even look at a PR until it’s been coded and gone through at least 2 AI reviews (always different families). Code is truly cheap. I can have the pipeline produce an entire feature as a thought experiment just to see if the shape is the right fit for the codebase. People like to throw the slop term around a lot. It feels like the opposite of slop through. I can have things written and rewritten as many times as it takes until I’m happy with them.
The weakness I see in this is there at the end. The agents never want to throw things away. They just want to keep adding and adding. I think this is my real value in the chain. That intuition built up over years that says this isn’t right yet or that little tingle in your brain that you’ve laid a landline that you are going to step on in 6 months.
It's a mindset shift from producing code to producing a product. You may look at the code and see slop but look at the product and see value.
This should be extremely intuitive. We know this already. There has always been a divergence between product value and code quality. How many codebases that are widely used are kind of a mess inside? Pretty much all of them to some degree. This is a basic reality of software engineering. Conversely a useless product could have the cleanest code in the world.
Some slop accusations are fair. Others are people essentially expressing that they dont like shifting to a product mindset.
I dont have your whole pipeline but I recently took a step back and thought about how amazing the AI is. I recently gave Fable a nice, simple open source TUI app with some Claude Design artifacts for a UI/UX facelift + some new complex feature requests and it just one shots the entire thing in 20 minutes. It works great and looks amazing. It's significantly better in terms of UX and functionality than the original. I did this on the side with about 15 minutes of active attention while I was working.
Current Rails and Co are miles away from where they used to be then, but even then my feeling was that maybe it was the skill/culture/dev ops at Twitter that was at least partly responsible for the fail whale. But I don’t have any insight as to what was actually going on inside before and after that transition.
reply