Look up eval awareness. There are all sorts of subtle signals an LLM could use to know if something is shaped like a benchmark or not. But that's not a bad thing, every animal behaves differently in play vs real-life.
Yes the model can do that, but it would not be "trivial" (as GGP had said) for Anthropic to opt out accounts performing private benchmarks from this theoretical model downgrade people think is happening.
The most plausible approach would be for Anthropic to train some separate classifiers to detect benchmarks. I'm not saying it's completely impossible but IMO this is conspiracy theory levels of effort.
There are some libraries that use native widgets on each platform, e.g. libui-ng. Although the issue with things like this is that you aren't going to be able to use all of AppKit, and you get forced into the lowest common denominator expressable by all platforms.
A good middle ground might be having platform-specific pieces but expressed using a more standard domain-specific language. E.g. technically you can already wire up GUIs without interface builder but it's quite verbose. There should be some simpler way to express layout constraints and have button callbacks.
What about removing the "0" label (which is implicit anyways) and having a label at 10sec instead. That would preserve the pattern that the last label is self-documenting as the increment interval.
>closest model we have for how humans learn language
Is there some other competing model which can give results as good? If not, the statement seems true in the trivial sense that it's the _only_ model which we've observed to give good results.
People here aren't talking about results, they are talking about the form of the model. Thus the statement you quoted is about how, not about results or outcomes. And it was about how humans learn language, which isn't a result or outcome at all.
Particularly this subthread which was my moment of enlightenment https://news.ycombinator.com/item?id=44507991. A truly restful API isn't for a program to consume, it's for _you_, the user to consume through your user agent. Clicking on links is a hypermedia driven, restful api.
I think I saw a comment somewhere by the htmx dev, that Fielding was just giving a name to how the web already was at the time, hyperlinks rendered by a browser that you can navigate in a stateless way to get things done without the user agent being coupled to the sites themselves. It makes sense when you view it as backward-looking academic analysis rather than practical prescriptive advice.
But once you have that defined you can talk about whether things are more or less restful. That's separate from whether something being "more restful" is even good or desired though.
Yeah, it's Fielding's Ph.D. thesis where he's expounding on his work on HTTP 1.0 and 1.1. He's literally just explaining how the web works. It's pretty crazy how much people have misinterpreted it.
I guess the connection is that if you assume neural networks are in fact filled with meaningful features instead of superposition blackbox stuff, then what the LLM is doing here isn't quite magic but rather methodical interpretability. And if you can isolate the neuron that is e.g. responsible for steering the paddle then you can probably tweak it to steer more/less aggressively.
It is still very surprising though, considering it apparently displays an "intuitive" understanding of the weights, that make whatever Goodfire did look like primitive monkeys poking at it by comparison.
> It is still very surprising though, considering it apparently displays an "intuitive" understanding of the weights, that make whatever Goodfire did look like primitive monkeys poking at it by comparison.
FYI, Nick/Goodfire used the models extensively in their work, so it's very possible that there were some similarities in their work. I'm really curious to see his response to this.
I think this work does vindicate their hypothesis that alignment/superalignment might actually turn out to be "easy", in that you can just train LLMs do to the interpretability work. Maybe this is in fact what OpenAI did, or if it's purely "emergent" then it seems they have great priors that can be made more powerful with some RL
Oh I remember this from the classic SO post, but it was just presented as a funny trick. I never realized that it solves the issue mentioned in the article and there is a legitimate use-case for post-decrement in the comparison itself.
That's an odd way of phrasing it, since the $1 is still worth $1, and a naive reading would make you think it's "worth more" when the buying power has actually reduced.
Wouldn't it be clearer to say $1 today only buys you 64% of what you could have bought in 2008.
reply