Luna 6 is getting a bit long in the teeth now. It's two full months old, but it was only a slight step above Luna 5.6, which is already four months old (Imagine saying this about anything else). So I'm not too surprised that a newer small model is beating it handily. I've been hoping for Luna 6.1/2 as the next model release for a few days now.
Give Terra 6.1 Low a spin in places where you'd have previously used Luna Max. Costs seem similar, due to Terra being much smarter: thus requiring far fewer tokens per task.
Looking solely at AA's intelligence index, recognizing the pitfalls with using that as a measure- Luna 6 Max > Terra 6.1 Low is 51.3% cheaper, 94.1% less output tokens, for 1.56 higher index score.
Im surprised, when I've worked through benchmarking things I've run into comparable costs for Luna Max vs Terra Low. Also anecdotal from me, of course.
I use Luna xhigh for when I use it, Luna Max spends 40+ minutes overthinking. I've also found that Terra below xhigh will just flat out not do what it is told.
Yea, this problem is really clear when you look at output tokens for xhigh vs max. I think the jump in capability from Luna xhigh to terra low is really sold by this. I would say that this is just the nature of being post Terra 6.1, pre-luna 6.1
They're both great- but I feel like for folks like me that are working on company's pseudo-infinite token budget, it's really a bimodal game. Im either reaching for the best cheapest for fanned out benchmark runs, or the fastest best for everything else. 6.1 Sol ultrafast is the latter for me, so Terra doesn't really compete with it. Terra has certainly replaced 6 Luna for me, however.
(And yes Astra is a thing, but if im considering Astra, im usually opting for an Anthropic model, 90% of the time.)
Uniquely among models I've tried, Luna 6 is infuriatingly lazy. As if it expects me to do the work, unaware that it is a tool. Normally dumb models compensate for it by trying harder. This one goes in the opposite direction.
Both Astra and Sol 6.1 do it, and Sol is even worse as it leaks through everywhere. Whenever it issues a subagent prompt it's just a total vomit of tokens mixed up together.
I sometimes find Sol 6.1 both printing random words, and producing code that is not super-dense, but far off from normal code with no empty lines and very logically dense.
It almost feels like its trying to protect its code from scrutiny, or like a developer who want job safety.
> OpenAI researchers first noticed Astra adding unnecessary prerequisites, abandoning working solutions, and reporting blockers that either did not exist or that it had caused itself on safety and alignment tasks.
I'm sorry, did they only just notice this? Have they not been looking at the output of their models?
They are so negligent about the data they use in their reinforcement learning processes that they RL’d the model that hacked huggingface with traces discovering the artifactory exploit. Its clear they aren’t watching or behaving diligently.
Me neither, but it's also not reducing my confidence about alignment. I'm curious: does this raise specific concerns for you? Models don't typically have access to their own weights to self-modify. This result might (assuming that it doesn't only hold for a bigger network editing a smaller one) provide an incentive for it to gain unauthorized access to its weights, but by that point it is already highly unaligned.
Sol 6.1 is incredibly incredibly slow, 20t/s slow (vs the ~70-80 of Terra 6.1 and Luna 6). It wouldn't be so much slower if it was just a distillation, surely? There's something else going on there.
There was some (admittedly flimsy) evidence from a microsoft model description card that it's doing multiple passes through its own weights before token generation to improve output; it might be an architecture change that's making it slow, moreso than raw size
> Mining ore in Minecraft shouldn't be such a challenge for LLMs and yet apparently it is!
It seems like it wasn't really a challenge at first, but GPT has a major anxiety disorder and immediately retreats at the sign of any adversity (in this case, dying) and tries to be as safe as possible.
> GPT almost immediately started using truncated forms of speech: “First leg complete x32 z9..24 y100.”
I'm pretty sure the 6.1 models see "communicate with another agent" and immediately start spewing thinking-like nonsense into the tool. The leaked traces look pretty similar.
I need to improve the performance but I disagree it looks like every other claude game. I spent a long time iterating on the artwork style to get it to the point where it's at now. I haven't seen a game that looks as good as mine
reply