I'm not sure why but Prolog seems to have more misinformation circulated about it than any other programming language I've ever seen. I'd take a lot of the information in these comments with a grain of salt. If anyone is interested in authoritative and accurate modern information about the language, please see Power of Prolog[1][2].
No, he didn't? He predicted that third parties would donate tokens to FOSS projects, not that the labs would. One is PR that started ages ago, the other is a reasonable prediction of where the world is going.
Not quite donate tokens directly (technically and practically weird), but donation -> compute has been out for a couple months on opub.dev (disclaimer, built it). So his prediction was somewhat correct if not late!
If this wasn't CERN tech I would think I was being taken for a ride. Conventional wisdom is that distributed consensus is not possible at this kind of performance, does anyone have a sense for how this is different and how my mental model is wrong?
> Conventional wisdom is that distributed consensus is not possible at this kind of performance
I'm not sure why you would think that? If you can assume the fiber is the same in both directions you know the round trip time is exactly double the latency of the connection. Then you know to phase shift your start time by that much when you get a start signal and you're in sync.
Obviously it's not trivial in practice, but it's not a fundamentally insurmountable problem.
Thanks. I thought it was interesting choosing arithmetic instead of some other relation because multimodal arithmetic (via CLP) is more of a PhD thesis than a blog post. Other relations might've been easier to demonstrate a general query.
What I couldn't tell from the article was if the author somehow achieved a multimodal arithmetic relation without needing CLP using a stack machine. That would be a neat technique.
Man that is such a bummer. The Naval Support Activity (NSA) "base" is not a hardened military facility. I've never been to the one in Bahrain, but it's usually where you go to play ultimate frisbee, maybe some paintball if you are lucky, and other types of R&R. Usually have a Naval Exchange (NEX) which is like a really discounted 7-11 / gift shop / walmart (depending on where you are).
Schools getting blown up is also a bummer. Everything about this situation and maybe the world is a bummer.
As soon as we stop treating these as bummers, there is literally nothing stopping a cycle of destruction. There may not be anyways, I don't know but giving up on empathy entirely seems even more dangerous than being bad at it.
I have plenty of the sympathy for the victims but none FIR the aggressors in this illegal war.
You seem to be suggesting that not feeling sorry for the soldiers who got to evacuate without all their belongings somehow means I'm losing my humanity. That's a dangerous thing - lives of the innocent civilians who didn't chose to be bombed are more important. Aggressors could simply.... Leave and stop being in danger.
Similarly I have little pity for Russian soldiers losing lives in another illegal war of aggression, knowing how many war crimes they committed in their wake.
Great writeup. Only thing I din't see in here was an analysis of the impact of players like Talaas[1] and their stupid faster hardware LLMs.
I feel like it could be majorly disruptive, but idk if it's going to prolong the apocalypse or bring it about sooner -- or if it's a big nothing burger.
I'm bullist for something like talaas to get smaller and easy to put in a desktop. Imagine an RPG where NPCs.... are way more complex and the entire game is very non deterministic.
I think I would like that as well. The problem is that if we bake an LLM into HW and make it cheaper and very efficient to run, then all games will have the same AI slop content, which could get boring pretty fast. The alternative is that these cards should load a different / fine-tuned LLM per game, but then we already have GPUs for that and today's LLMs are nowhere near good enough at the size which a GPU can run.
> Pre-training allows organizations to build domain-aware models by learning from large internal datasets.
> Post-training methods allow teams to refine model behavior for specific tasks and environments.
How do you suppose this works? They say "pretraining" but I'm certain that the amount of clean data available in proper dataset format is not nearly enough to make a "foundation model". Do you suppose what they are calling "pretraining" is actually SFT and then "post-training" is ... more SFT?
There's no way they mean "start from scratch". Maybe they do something like generate a heckin bunch of synthetic data seeded from company data using one of their SOA models -- which is basically equivalent to low resolution distillation, I would imagine. Hmm.
Pre-training mean exposing an already-trained model to more raw text like PDF extracts etc (aka continued pre-training). You wouldn't be starting from scratch, but it's still pre-training because the objective is just next token prediction of the text you expose it to.
Post-training means everything else: SFT, DPO, RL, etc. Anything that involves things like prompt/response pairs, reward models, or benefits from human feedback of any kind.
Er, then what is the "already trained" model? I thought pre-training was the gradient descent through the internet part of building foundational models.
Yeah, this checks out. I wonder what they are doing to prevent semantic collapse. Also, I wonder if the base model would already be instruct and RLHF tuned or only pre-trained. Trying to do additional training without semantic collapse in a way that is meaningful would be interesting to understand. Presumably they are using adapters but I've never had much luck in stacking adapters.
i.e.:
1. Do I start with an RLHF tuned model, "pretrain" on top of that (with adapter or by freezing weights?), then SFT on top of that (stack another adapter, or add layer(s) and freeze weights?) (and where did I get the dataset? synthetic extraction from corpus?), then RL (adapter, add layer(s) and freeze?)
I can imagine that, as usual, you start with a few examples and then instruct an LLM to synthesize more examples out of that, and train using that. Sounds horrible, but actually works fairly well in practice.
