Hacker Newsnew | past | comments | ask | show | jobs | submit | Alpha3031's commentslogin

6.5M! Impressive. Is it initialised or distilled from any base model or entirely from scratch? What kind of training data for the RAG and conversational chat specifically?

> 6.5M! Impressive

I've been surprised myself. For reference, it gets better ppl (and with faster training), than the nanogpt model from Karpathy's tinyshakespeare tutorial, at smaller size (nanogpt is 10M params)

Trained entirely from scratch. Which gets it pretty far in terms of basic conversational skills, especially when also trained on chat conversations

The training sources are:

1. Google Synthetic PersonaChat

2. DailyDialog

3. SODA conversational data

4. Hand-written model-personality conversations, repeated during training

5. Identity-corrected conversations—rewritten names, human-life claims, and hobby responses

6. 960 synthetic recovery conversations, teaching it to admit ignorance after user push-back

The final set contains 23,270 conversations

However, the chain reasoning and math problem solving skills, I was only able to add recently, by grafting certain parts from a 1B model, then doing space-weight programming, and finally finetuning/training to get the model to "get used to" reasoning. The "full model" is more of a harness + model:

* Frozen base model to interpret questions

* Additional trained heads to select question types, evidence, and follow-up behavior

* A library tool for retrieving facts

* Working memory and internal script language execution for arithmetic, conversions, and comparisons across answers. For example, it can retrieve a building’s height, remember that number, then execute “half of that”

* Identity extraction and physical simulation, so you can do thing like - User: "Ben has the ball", User: "Who has the ball", Model: "Ben", User: "Ben gave the ball to Sally", User: "Who has the ball now?", Model: "Sally"

It's relatively easy adding skills either via its internal scripting language, grafting+weight programming, fine-tuning and/or training it to use a new skill. The main limitation right now seems to be just quality data and proper supervision, ie. someone to talk to the model, flag issues, fix them, then re-test. This is kinda tedious, but I've found that even the best frontier model is kinda lacking in quality when producing synthetic conversational data, and most datasets out there have a lot of cruft. So I get better results just manually producing the data and iterating, than feeding it GBs of public datasets (even famous datasets have a quite a bit of un-reviewed data, as well as errors, which aren't even published)

Creating and training this model has been a lot of fun. Since it's so tiny, it's easy to experiment with and try a lot of different things. Most training I can just do in a few minutes locally


actually your methodology inspired me to try smth related to small models and harnesses I've been thinking about - and seems you got the main pieces (the idea was trying to see what custom harnesses can extract more accurate reasoning from small models)

Awesome, happy to hear that. Hopefully you can post about it here again when you have some results :)

I don't really know why you would expect a "X% chance of containing AI" number to rise slowly or smoothly alongside the percentage of text that has a 100% chance of containing AI.

Because it isn't an "X% chance". It's an "X% of words".

Surely not at the sub-paragragh level? OP said their test passage was a paragraph.

Food banks depend on the largesse of the people that can afford to buy food though, which, well... if the only people remaining in that group are the ones gleefully saying how they're going to create a permanent underclass...

I've done social-work–adjacent stuff and I know a lot of NGOs in my country are quite strained (of course, probably housing more than food, so I don't disagree that might be more of a concern). I imagine it wouldn't be too different in America.


in America they are literally creating laws that it's illegal to feed the homeless


No, there are laws against distributing food without a permit because (1) there are people who give poisoned food to homeless people and (2) there are rules about food safety.


If billionaires have all the money then there's nothing stopping them from buying all the stuff the economy can produce and leaving the rest of the population to barely get by or starve. Certainly, higher income people might currently spend less than their share of total income (at least between 2017 and 2021, see BLS paper, Garner et al., 2024, Gini of 0.3ish vs what I believe is 0.4ish for post-tax income) because poor people need to spend more of their income on things like food or rent, but there's no reason why the share of spendin by the top x% can't increase. In fact, as I understand it, it already has, according to [2] who themselves cite Moody, the top 20% accounted for 59% of spending in Q3 2025, which was a marked increase from the 40% in the BLS paper.

Now, the top 20% certainly aren't billionaires, but we also have to remember that the Luddites also benefited significantly from the automation of spinning before they themselves were replaced by automated weaving and subsequently suppressed by the state. There is no reason to think our current crop of labour aristocracy immune from the whims of capital when the time comes.

[BLS 2024]: https://www.bls.gov/cex/a-distributional-approach-to-us-pers... [2]: https://www.empower.com/the-currency/money/life/k-shaped-eco...


Well, I wasn't very active between February and August, but from what I understand the guideline was only changed in March to disallow AI edits entirely instead of allowing them if you pinky promised you've checked your sources and they really do say what you're putting into the article because a lot of people were pinky promising and those promises turned out to be either lies or bullshit when people actually looked at it.

So the answer to "Why do we assume AI-generated means slop?" is probably "experience", and any proposition starting with "If the content is good enough" is trivially true as a conditional statement with a false antecedent (i.e., ⊨(⊥⇒Q) for all Q).


People apparently like Granite for RAG specifically. I'm not aware of any evaluations with numbers though, so I'm not sure if it's a real difference or just vibes.


It isn't possible, in the same way that keeping copyrighted material out of Wikipedia isn't possible: there's probably going to be a backlog spanning years or decades, but some of those investigations will eventually complete and result in an action being decided on.


Hmm, its showing as roughly 9.4 kwords which is a little higher than ideal. I may raise the suggestion to spin a section out. Maybe § Emergence can have its own article and be only summarised in the parent? I see the New York subsection is also quite long.


Archive.org should have some old dumps, for example https://archive.org/details/enwiki-20220501 is supposed to be May 2022 (All the later ones look like incrementals).


If you find an article is too long to read comfortably, please consider adding {{too long|date=October 2026}} (or whatever the date at the time is) to the top so that people can figure out what to do with it.


I don’t have any problem with the trend, it’s just notable to me. I think any encyclopedia with essentially unlimited space would eventually trend in the same direction.


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: