This is a fine-tuned model. The author even states that the model is competitive with Jev only if fine-tuned on the evaluation at hand.

Literally misses the point of Jev, which you don't need to fine-tune to get accuracy nor - and no other model has this - some sort of out of sample calibration

Gary Marcus has has a long career now of shifting goalposts

Does he have time to do anything but shift goalposts, given how fast his are moving?

I believe he now just leaves them in his truck and drives it a bit further each day.

Such a crazy change from the times of Luminous, when they published a three pager with a claim that the model is similar good as „gpt 3“ (which??) with some graphs without y axis.

Bravo team!

No mention of calibration. Is it just another llm finetune?

> Our post-training utilizes label-smoothed cross-entropy for valid schema outputs paired with a Brier loss to refine probability calibration.

There is museum in London called the British Museum which has zero artifacts on display from Britain and is entirely filled with stolen goods from other countries No one there finds this strange, bad, or anything that should be adressed

Brits are entirely deranged when it comes to their former empire.

>There is museum in London called the British Museum which has zero artifacts on display from Britain

That's not true at all. What's your agenda?

You are holding it wrong!

I am sympathetic but this buries the lede, hard.

You are competitive with Jev only if you fine tune on the train dataset and calibrate per question.

As much as I dislike literally everything about typesafes behavior, they have an API model that works on any problem without fine tuning, and that is the key.

To be honest everyone who can finetune can likely finetune a BERT for a specific task and get similar results to yours. And that has been true for years

The key to Jevs success is that it works without fine tuning

Hey, don’t copy my daily affirmations

Thanks Claude for writing that

I literally struggle to read it