As impressive as this is, I can't help but conclude that the need for agents to write JavaScript programs to act out their actions represents a considerable limitation in current AI reasoning.
It demonstrates that current LLMs are completely unable to follow a simple series of logical tasks. I feel the same way about ARC-AGI 3. While it might be the winning approach today, if they need this amount of complexity to do something simple, then it shows that their ability to generalise is clearly limited. Mining ore in Minecraft shouldn't be such a challenge for LLMs and yet apparently it is!
It depends on what the tasks are. Text-native work like programming or interacting with a terminal should be a lot more immediate for LLMs than something abstract like Starcraft, DOTA, Minecraft or Mindustry. LLMs certainly know the rules for these games, but processing game state and formulating a plan in the form of a Python script or MCP call is a rough debugging loop.
Minecraft is really the perfect example, because humans get all sorts of unfair advantages. Even a young player has enough spatial reasoning to recognize and remember landmarks, visualize large projects or explore without getting lost. These are simple instincts that LLMs will struggle to match even with huge context sizes and native multimodal input.
> Mining ore in Minecraft shouldn't be such a challenge for LLMs and yet apparently it is!
It seems like it wasn't really a challenge at first, but GPT has a major anxiety disorder and immediately retreats at the sign of any adversity (in this case, dying) and tries to be as safe as possible.
Wonder if you could glue together deep networks such that there's a big reasoning LLM model, and high-speed small network of a different type, and they share a subset of their activations. High bandwidth vector communication. You train them together.
edit to add: Seems this is a well-known idea in ML / robotics already.
My thoughts are without a doubt closer to code than writing.
Thoughts are more often than not non-linear, so putting them into writing is challenging.
Meanwhile, when I think of code—and also when I physically write code—it's always possible to leave holes and stubs to be filled in later.
I'm not arguing with your basic point - in fact, "but some people are better at expressing their intent in code than in prose" instantly occurred to me, as well - but why should "it's always possible to leave holes and stubs to be filled in later" not also apply to natural language?
I, at least, have always written in that way - essays, teaching materials, letters, long-form narratives, (and now, recently) specs for the AI-gods, doesn't matter - I get the bits I'm sure of down first, and then tinker (cut-and-paste, do more research, add or remove material, write bridging sections) until I'm able to connect them together.
The most frustrating English teacher I had in high school required us to turn in an outline a week before the essay was due. I did it for the first assignment, and was marked a B, with a comment along the lines of "this was very good, but it deviated from your outline". Which, well, of course it did, because I bloody learned something while I was writing it! The next time, I wrote the whole essay ahead of the "outline" due date, which led me to discover WordPerfect 5.1's "Generate Outline" function, which led her to the mistaken impression that she'd taught me something besides disrespect for authority.
That's not quite a digression by way of personal anecdote, because I wonder if you were taught writing by someone like her? Prose doesn't have to be composed (in both senses of the word) like that. In process, it can be just as shaggy and stubbed and non-linerally referential as code. Furthermore, AIs (I am finding), work quite well with a process like that: I'm constantly telling it to "put X into Deferred", when I haven't yet got something worked out in my head.
Good point. Writing has to create a narrative in a way code does not. It's generally very structured when written for others and not just ourselves, and I would say you can probably be looser with writing than code in the small but once you get into paragraphs, code starts being more flexible (I don't mean theoretically just my own opinion on what's common). Ie codes syntax is fixed but the composition is very flexible while writing prose is very flexible but you will find only so many ways of writing paragraphs or pages because we are so often using writing to create a narrative arc.
I feel the same. It took a me a while to realise that I needed to click on the white tile to place a transformation. Though I'm now stuck on the second level as there aren't any tiles! The menu in the top left is very confusing.
The original 'Attention is all you need' paper also used random variables to explain the sqrt(d_k) factor. Even if the keys aren't random, it seems reasonable to treat them as random variables when calculating the 'interference'.
To drying clothes!? No, we're much more likely to dry clothes outside than Americans. They dry just fine outside even in mid October; a bit more slowly than that after that, but when I was a kid I remember washing going on the line on warm days in early December.
We do have electric tumble dryers, condenser dryers, washer dryers.
(Indeed, the progressive switch to condensers may be part of power consumption dropping.)
AFAIK The UK's climate is about equivalent to New York State.
The UK typically doesn't need air conditioning, so why would people spend money on it? Yes, I know this recent summer was exceptionally hot but people don't react that quickly to changes.
> decarbonizing heating
What does this mean? More renewables? If efficiency reduces energy demands then it directly results in less carbon.
> electric vehicles
Aren't people already adopting electric vehicles? Do you think that the government should introduce a scheme to persuade people to buy electric vehicles? Does the availability of charging for electric vehicles need improving?
> Efficiency is good after you have high living standards but the UK should probably focus on growth first
Efficiency means saving money. In what circumstance could efficiency be bad? The grid responds to demand, unused electricity means higher prices for consumers. One of biggest issues right now in the UK is cost of living crisis, so raising electricity prices would be universally hated. How does making the grid less efficient increase growth?
I don't know where you live but UK summers were unbearable even when I was a kid. Buildings are engineered to trap heat, so the heat doesn't dissipate at night and every day the indoor temperature rises, and rises...
At least they removed the need for planning permission recently. The problem is that the places that need AC the most (flats) will probably not be able to get it until the leasehold system has been abolished.
> > decarbonizing heating
> What does this mean?
Presumably it means heating using electric heat pumps rather than gas boilers. This also solves the AC problem. I don't think this is viable for most people. Heating was never the problem anyway, at least in the south winters are mild.
> I don't know where you live but UK summers were unbearable
well, i live in lincolnshire and used to live in london. i usually have a fan on for a few weeks over summer, but that's it. where in the UK did you live that the summers were so unbearable?
> The more insulation you have the less the indoor temperature is affected by the outside temperature.
Insulation isn't magic, over many days of hot weather eventually the building will reach equilibrium with the average outside temperature. This means it will be cooler indoors than outdoors during the day, and hotter indoors than outdoors at night.
However, the nights are specifically when you want a residential building to be cooler, assuming the inhabitants are not at home during the day. So the insulation actually makes the building hotter from the perspective of the inhabitants.
A common scenario is to have unshaded solid walls exposed to afternoon sun. Without insulation the inside of the house can be high all through the night. A good effect of insulation in summer is to prevent that effect. Important to note that it would be standard to allow cooling airflow in the morning.
I think you're right, and in addition to that there is also a lot we could do other than AC. Just having some shade from trees already helps a lot. Modern cities can be a man-made hellscape with regards to heat. But we did that to ourselves.
It's fine if temperatures get lower during the night, but 25°C at night and 35°C during the day leaves little for cooling the infrastructure (roads, buildings), and constant warm temperatures are unbearable if you cannot even get in the shade when you want to cool off. Then, 35°C is also very hard to manage with high humidity.
That's only a few weeks a year. The UK isn't Florida yet. And besides modern AC is incredibly efficient. The stuff that's cleared to sell in Europe has incredibly high standards.
> UK summers were unbearable even when I was a kid.
Not to be that guy, in the last 20 years, 27 was considered a heatwave.
> Buildings are engineered to trap heat,
I mean Victorian buildings were engineered to let in light, and have good ventilation. Hence why each room has a huge sash window, an airbrick and a fireplace.
> The UK typically doesn't need air conditioning, so why would people spend money on it? Yes, I know this recent summer was exceptionally hot but people don't react that quickly to changes.
We are? It’s becoming more common and we had a few bad summers and earlier this year too. The heat wave a few years back killed my dad.
Given there’s a push to swap heating systems for more efficient ones, air con units are being looked at a lot more.
It can be considered a medical necessity for people with certain conditions like anhidrosis, asthma, COPD, kidneys disease, or cardiovascular issues. It's also a huge medical benefit for people with terrible allergies who wouldn't be able to open a window even when that'd normally help cool the house.
> The UK typically doesn't need air conditioning, so why would people spend money on it?
We had 6 heat waves this summer. Parents were booking hotels just to keep their kids cool
We also had heatwaves in 2022, 2021, 2019, 2018. 2024 wasn't bad but was still the 4th hottest year on record.
You can absolutely bet the better-off middle classes are installing aircon. I know many. If more people could afford it they would get it
It's the humidity too not just the heat. And how after 3 or 4 days of high temps, our houses just do not cool down at night because they act like radiators. Humid 25c+ nights suck for us
I believe this refers to switching to high-efficiency electric heaters like reverse-cycle AC units (also known as "heat pumps"), rather than oil heaters or ducted gas heating.
The majority of UK homes are heated by gas-powered boilers that pump hot water through radiators in each room. Often hot water is heated by the same system.
Decarbonisation usually means replacing that boiler with a heat pump which uses electricity to heat water.
Very few places "need" air conditioning but it doesn't take very days where it's 80+ outside and hotter inside to convince me that the per-day cost of AC is worth it. And that level of heat is nothing new.
Actually you're not just behind, you've burned up money.. you've destroyed value - there's an opportunity cost of that money.
Whilst the other competitor who is prudent eventually wins will be far more attractive in the eyes of investors - moreover once you are confident you can scale up inputs (engineers) and hone in the quality whilst the other competitor runs ragged with a half-baked product.
We've seen this story play out already numerous times: Apple. They made a mis-step with Apple Intelligence but aside from that, they've been right not to lurch to deep into things they shouldnt. Its not zero-sum.
I would question the narrative that humans lack the capacity to verify the output and would instead argue the people lack the incentive to verify the output.
The response of many mathematicians to the recent dump is a good example: verifying these proofs amounts to unpaid labour for OpenAI and wastes time that could be spent doing publishable work which ultimately results in money or personal success. The slop factor also compounds the work required to verify the output considerably.
For mathematicians, programmers or anyone, if the work required to deal with slop passes the limit, it is no longer in their own self interest to use LLMs. The expectation that people will use LLMs for the betterment of humanity against their own financial interest is baffling.
Looks like this is a company’s org which heavily relies on AI. Probably “the cost of doing business” and to see just what can be done with the new model.
I'm really confused by this list: every link I click on either doesn't work or takes me to a project for Go written in Go. I haven't been able to navigate to a single Cobol project.
reply