Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

People assume that's the reason because it's intuitive and "strawberry" is one token. But that doesn't explain why those models would also often get it wrong for "StRaWbErRy" or even "s-t-r-a-w-b-e-r-r-y", where the r's are not combined into one token.

We don't know what was going on inside the closed source GPT models, but this paper investigated on some of the open-weight models and found it's not due to tokenization: https://arxiv.org/abs/2604.00778



And circling back around to AGI, tokenization or some other underlying cause should pose no issue. A competent human would think to write a program (ie create a tool) to do the job. It's routine for a carpenter to make a jig.




Consider applying for YC's Winter 2027 batch! Applications are open till November 2.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: