Yes! I was having Argon summarize some project updates and it... just made up entire paragraphs that had no basis in reality. It felt like I was back in GPT3 days.
Will have to give Carbon a try, but if it is anything like what I saw before it was entirely unusable.
My guess is they do something like "Hey dingus, make my app send emails" and then Claude or GPT just goes off and does a thing that makes it look superficially like it sends emails but it really only sends email exactly on Tuesdays and if the subject line is "send email to user". And there's no tests, so who knows if the next LLM can fix the bugs without introducing new ones.
For the last several years (since GPT 4o) I have written several apps in use by people, making money, entirely with AI/LLMs (me neither writing nor reviewing the code in any meaningful fashion - other than high level architecture, schemas, etc.) and - yes, in a few hours it can do things that would take a normal human weeks (if ever!). But left unconstrained, it will just pile more and more garbage on the pile.
Fable MAX is the thing that gets me much closer to "just send emails" but even then it doesn't look in my specs directory for the spec, it just goes off and does a Jurassic Park-style "It's a Ruby-on-Rails system! I know this!" and disregards all the other ways we send email and writes its own thing. And often I'll go "Where's the button to do X" and it will say "You're right! Nobody asked for it, so this page is an orphan!"
I happen to use Superpower's (https://github.com/obra/superpowers) "brainstorm -> spec -> plan" workflow in e.g. Fable MAX (for anything non-trivial) then I have Fable send to GPT 5.6 Sol XHIGH for execution, with Fable (either the original, or a separate one, depending on the criticality of the task and the blast radius) being the critical reviewer.
Even then, though, I need to continually guide it against the norms and conventions of the codebase/app, because it makes a ton of assumptions.
It doesn't surprise me that if people don't take a fairly rigorous approach to AI-software development then they'll end up with a mess. Even if you buy into "just re-write it" (I happen to think that's where we're ending up) - we aren't there yet and without e.g. a strong test suite, re-writing it is just as likely to create more bugs than it is to fix the existing ones.
Back in the olden days (2015-ish?) KLM was having a really, really cheap business fare sale JFK-AMS; I snagged it with Delta miles (if I recall correctly) - and flew there and back in their 747 in the upper deck (just to take the flight; didn't have anything to do in AMS). It was really quite nice; it was the first and apparently last time I've taken the 744. I'm really glad I was able to do it.
Lol imagine sitting through 8 hours of agony for no reason. Do you actually enjoy flying? I'm sure I would find it agony even if I had the Emirates suite with the bedroom and shower (which is still more cramped than a 50 euro hotel room)
I would at least explore the city for a few days to have a break.
The only flying I enjoy is when I'm the one in the front left hand seat holding the yoke :)
I do OCR of images, and that's exactly what I do. I take one big image and slice it into many smaller ones, and send those to the LLM. Perfect every time, unlike using the whole image which resulted in hot garbage.
It works with relatively good scans, when there are bad/skewed scans and especially something with many label/value pairs, that aren't nicely tucked inside sentences, the more context you have, the more you can find the correct words and fix the errors.
There is a whole class of tricky documents. A decent (if you ignore the marketing bias) post about this problem can be found here:
I calculate* the appropriate overlap and the slicer overlaps a certain amount of the previous slice. There is some post-processing assembly required, but it's trivial.
[*] SWAG line height, trial and error to figure out the right amount of overlap given LLM error rates, etc.
Interesting. Do you have a uniform data set? E.g. documents of a specific type that you know consistently have similar formats, or is this training something you need to do per-document?
We have some broad shapes - it’s a finite set of “things that are interesting to us” and the dataset is bounded. It’s not “Google Image Search”. But it is kinda like “we have a giant pile of PDFs, pictures, etc and the user wishes to run an arbitrary query on them and extract the information they want.
Ex: “I need the to know $something about the data embedded in the corpus, that look like excel data with line charts describing some particular class of metric that are to the left of gray dogs and are about $something_else earlier in the document”
Gemini has a very specific mode where it has been trained on making boxes normalized to a 1000x1000 grid (https://docs.cloud.google.com/gemini-enterprise-agent-platfo...) and in our experience this “just works” AND is very fast on 3.5 and 3.1 models without needing much thinking (so it is not terrifically expensive).
(BTW A+++ gold star triple thumbs up give this person a bonus to whomever did that magic it basically made this task for us tractable. When we first found it nobody else had anything like it - it’s worked so well I haven’t felt any need to look. )
So we say, “Hey Gemini draw box_2d […] around #{things we are interested in}” and then it is pretty easy to then go - ok if this is here and that is there, let’s slice the image in this particular way, making sure to overlap by some amount because the boxes are fuzzy, then send the chunks to a thing that turns it into JSON, then we use something like edge detection to reconstruct the whole from the parts. (Squint and it looks like whole genome shotgun sequencing)
Paddle-VL and GLM-OCR do this by using PP-DocLayoutv3 as their "detector/slicer" and then just batch the OCR on the clips to do pretty darn well at a tiny size.
A lower-tech version is to use a good detector and XY-cut or just a naive Y-cut or orientation-away cut to slice up the page. But if you're doing that you're getting closer and closer to DocVLM style OCR+low res image. Been playing around with something like this using the new PPOCRv6 which itself punches well above most traditional OCRs and is multi-language without the hassle of language detection and dict-loading for rec.
I'd have to imagine there are wildly diminishing marginal returns to additional SFT/post-training passes.
There are a bounded number of (useful) derivations/combinations of Duff's device.
If Frontier Labs wish to reduce hallucinations on factual things, they will have to hire people (or the data providers will need to) to do fundamental research above and beyond what is available in extant literature and the web. IE if the LLMs want to lower precision error, they need to go out and actually find more expertise. If the wikipedia page for Pompey lacks data, where are they going to get it from? How would they even _identify_ that the page has holes?
Yes, they can digitize more books but that is untrustworthy data - if there were enough eyeballs on a particular work, it would be in the internet. If it's not, they'd need to hire the experts themselves. They need expert reviewers in virtually every interesting topic, which fundamentally is an intractable problem, especially since things change all the time. Maybe even uninteresting topics, too?
I dunno, it doesn't seem to me "more data" is the magic bullet here. Yeah, it will "help" but we're already on the flat part of the S shaped curve.
My take from trying to understand this stuff is some sort of algorithmic improvement is necessary to get another step change in how well LLMs perform in this area. I could be wrong!
As a side gig, I write novel software that solves problems no existing software does, that existing LLMs have difficulty reproducing, purely for the purpose of existing as LLM training data.
There are journalists being hired to write Atlantic-worthy articles that exist only as LLM training data, because they're getting paid more than the Atlantic would pay them for it.
It's insane.
Yes, they are hiring the experts themselves. To create new knowledge above and beyond what's on the internet. To be locked away as LLM training data.
The largest characteristic of all of this new data is it is targeted at LLM's weak points.
It's not just more data, it's custom tutorials built for what LLMs struggle at.
> There are journalists being hired to write Atlantic-worthy articles that exist only as LLM training data, because they're getting paid more than the Atlantic would pay them for it.
I kinda doubt that quality-assertion of "Atlantic-worthy." While I have no doubt such articles are written solely as training data, I'd expect their quality to be much less than the real thing, since there's no public to critique them, probably little reputational risk for errors, and no professional ethics to uphold. Even if professional journalists were hired to do the work, I'd expect them to start phoning it in pretty quick, and skimp on fact-checking especially.
> I've found the review process for these things to be far more vicious and demanding than in the real world.
1. for your software or for these "Atlantic-worthy articles"?
2. Why would the review process hold up to the standards of a first-rate publication? I'd expect the review process to succumb to the same pressures I'd outlined.
"Someone paid me to do it" will only get you so much motivation. And asking someone to write something whose only destiny is to get fed into the maw of a machine and mangled, seems like a way to eliminate all other sources of motivation.
Journalism is not my field, so I can't deeply judge how effective the reviews are.
In software, the reviews for generating things that only exist as training data are far more rigorous than anything I've seen for things going into production at top companies.
I agree with you that it would make sense it would be phoned in. My experience says the opposite.
I'm not saying they are not trying - I'm saying we're inventing new problems faster than any Lab can:
1) Identify the gaps
2) Determine how to fix them
3) Implement a fix (especially if that fix is: identify and find experts)
4) And judge the result
How do they know [person] is an expert in [some field]? How do they find that person? How many experts are necessary to give the right information? How do we evaluate the results, especially if it's novel?
You can find a lot of people who disagree on many topics, and those turtles go all the way down.
I'm not in disagreement that your work will help reduce hallucinations and improve model performance! It is.
I predict (I hope I'm wrong!) that we're going to hit some asymptote that is not at 0% hallucinations (and I would even put a substantial nonzero probability that "overall" hallucination rate bottoms out at some minimum and then slowly grows because we just can't keep up with the new garbage we throw at it).
> How do they know [person] is an expert in [some field]? How do they find that person?
They have a PhD from a top school, they are a licensed attorney, they are a licensed physician, a board certified cardiologist, etc.
They are constantly recruiting from these populations with well-paying side gigs.
> 4) And judge the result
That's what they pay the experts for. And to have experts review the other experts with peer review.
> You can find a lot of people who disagree on many topics, and those turtles go all the way down.
Which is why everything has to be well-calibrated and not just a hot take - a well reasoned opinion any expert would find fair.
Noone is really caring about hallucinations on point facts these days though, it is much more about complex reasoning tasks. Can they move the bar on the complexity of software LLMs do on their own? Can they get to a point where LLMs can begin to replace physicians? Financial advisors? Actuaries? etc.
> Noone is really caring about hallucinations on point facts these days though, it is much more about complex reasoning tasks.
The boundary is pretty thin there though. E.g., Gemini recently told me that a certain papers claims that two frameworks are mathematically equivalent, while the paper shows the opposite, and yesterday Google's AI overview told me that no World Cup matches were scheduled for that day despite their being several of them. The model probably used complex reasoning to arrive at both (incorrect) answers, but superficially they look like basic errors of fact.
That is a great example of the kind of thing they're paying people to create as training data.
You write the prompt, and then write rubrics to judge the responses, and you found something the model failed at. Congratulations, you just earned $500, now do it again.
Not the worst way to make money, but if internet-scale data were not enough to reduce errors to a somewhat tolerable margin, how much data do they hope to collect in this manner?
They are generating a lot of this. Also remember it's not just quantity, it's roughly active learning - they're paying for training data that's at the classification boundary, which is way more valuable.
I have gotten offers for contracts for full time jobs at high rates with AI labs to do this.
Meta has reallocated a lot of their full time SWE staff to do this.
All of this has rapidly accelerated within the last 6 months, who knows far it will go, if someone showed me a Kalshi bet that 10% of the college educated population of the US would be doing this as their primary job by the end of 2027, I wouldn't have the guts to bet against it.
10% of physicians' earnings doing this? Yeah that would totally track.
It doesn't seem like there's a limit. There's a shortage of GPUs and TSMC can only scale up so fast, so the AI labs found something else to spend money on.
I think this all reinforces the idea that the industry has no idea how to pursue general intelligence. Hence vast sums being spent plugging holes and fitting the models to more and more specific tasks.
But with this approach, there will always be the next car wash test showing that it is an illusion. It seems to me the limits of the Bitter Lesson are showing.
Yes, they do have money to burn, and this will bring some improvements for sure, but active learning has never really worked out, has it? And even 10% of the educated population doing this for, like, 50 years is not that much data, while normally each accuracy percentage is more and more data-expensive.
That is informative, I was suspecting that is how models improve their performance on some convoluted "non-googlabe" benchmarks like SimpleBench, that is how, they just got the taste of those those questions from publicly available samples and then hired people to generate similar questions and provide answers for them.
I wonder if extracting those static reasoning chains make sense given a Rich Sutton's "The Bitter Lesson" and Geoffrey Hinton's "People should stop training radiologists now.". I guess until participants make money they won't stop, not sure if they do, so far it is more about expectation of profitability as I understand.
There is one level that these training data give examples of specific static reasoning chains.
Given exposure to enough reasoning chains, with training data that is designed around adversarial reasoning and teaching models to reason, these types of training data might be key to teaching models to reason beyond what they could gather from static data.
> these types of training data might be key to teaching models to reason beyond what they could gather from static data.
I was under impression that every time LLMs try to be truly novel and they need to assume things in the area where they didn't have enough data points that there were trained on, results are not good, has that changed?
I've done Mercor and other brands - the contracts move around, since the labs want the vendors to know they're just vendors and have to compete with each other. It seemed to be roughly resume and interview similar to getting hired at a senior role at FAANG or adjacent.
>There are journalists being hired to write Atlantic-worthy articles that exist only as LLM training data, because they're getting paid more than the Atlantic would pay them for it.
So now part of the training is directly slop - just of the pre-AI variety?
"As a side gig, I write novel software that solves problems no existing software does,"
and
"Yes, they are hiring the experts themselves. To create new knowledge above and beyond what's on the internet. To be locked away as LLM training data."
> I write novel software that solves problems no existing software does
This is actually really easy to do if you step out of web/gui/crud and into something where you won't find public code, most ever, because it's trade secret. For example, manufacturing.
There is also an endless fountain of things you come across every day and think "oh, wouldn't this complex solution to this low priority problem be cool", but noone ever implements it because it's too complex and the problem is low priority.
Anyone writing software for long enough has a long list of these things in the back of their head that are great fodder for LLM training data.
>> They need expert reviewers in virtually every interesting topic, which fundamentally is an intractable problem, especially since things change all the time.
How odd. It's Expert Systems and the Knowledge Acquisition Bottleneck all over again.
Yes, for fun I tried to make a Mahjong solver and NONE of the SOTA frontier models could understand what they were looking at to determine tile occlusion/geometry to build the DAG.
I had to spoon feed it an algorithm - here's how you determine if a tile is on top of another one, etc. etc.
Anything that involves, well, "3d space" they don't seem to do very well on it at all (which makes sense, of course)
Agreed, my vibes tell me 4.6 is a better coder than 4.7. 4.7 is a much better strategic thinker and maintains overall "better architecture" than 5.5. 5.5 is way better than either at coding, but more expensive. So I have 4.7 do the planning/architecture, 4.6 does the coding, then 5.5 critiques and fixes it.
The other thing I suspect is that "Just give me True/False" cuts off a large amount of the search space a modern-day LLM uses to help it answer questions (you can see it in reasoning traces but the act of writing the explanation helps guide it toward a better answer and gives it better likelihood it backtracks on a bad decision).
If you let it spew out an explanation along with the answer, I'm curious if the accuracy will improve (I suspect it will).
Good point. Will publish in the next version also the results with a prompt that allows the models to "think out loud" before providing the final verdict.
Also, this is why investors and CEOs are so in love with "LLMs are the route to AGI!"
When some rich/powerful person says "I have to go to Davos, figure it out" their workers know so much context that no LLM is going to ever be able to incorporate, because it isn't written down and is idiosyncratic. (Really, though, the assistant will just say "you're going to Davos next week, the helicopter will pick you up at 3p on Friday" but you know..)
The rich person's assistant knows who else is on the corporate jet, and that X doesn't like Y, and so they should take a different plane. Or get a different accommodation. Oh, Person X doesn't like to fly on an empty stomach, so they should eat first, and that changes all sorts of other downstream implications. Oh, your best friend lives in this city, and I know you love to see them, so I'm going to send you a day or two early so you can meet up with them. etc. etc. etc.
The investor dream of "AGI" is modeled off of the army of employees that make investors/ceos/etc lives easier, and there is a nearly insurmountable gap between what LLMs can do, context they can get, and the availability of all of that information. (To me, the magnitude of this investor <> fundamental reality gap is the entirety of the "bubble". I love AI coding, but it's never gonna do the things investors think it can, to justify the crazy valuations)
Yes! It reminds me of the tabs vs spaces argument. Yeah, you have your preference, but as long as everyone on your team uses the same convention it doesn’t matter
The Rails apps I’ve programmed with LLMs seem to work a LOT better than arbitrary python or ruby or JavaScript apps. I chalk that up to “there are a gazillion examples of omiauth in Rails that the LLM can’t really stray off the path. It just works.”
That means I let the agent do things the way it wants to, not because I have a preference. So we’re using turbo and Hotwire and whatever it is it’s doing. And I’m using React for some other problems. Not because I know React, but because the LLM does.
In golf it is said to “let the club do the work”. Over control leads to disaster. Same with LLMs. Not saying let it do whatever but if there are widely baked in conventions you’ll be far better off letting those do the work.
Will have to give Carbon a try, but if it is anything like what I saw before it was entirely unusable.
reply