He has a point, though. The LLMs (or any AI for that matter) can't do anything. They can't. It's a function call that ingests symbols and spits out symbols and that's it.
100% of its actual capabilities are tied to harnesses (the actual "agent"), i.e. ordinary deterministic programs that are connected to networks or machines and enable interaction with the outside world. This part (the part that can do harmful things) is fully under human control and all the recent headlines about "agents going rogue" are - as someone (forgot who) put it - akin to strapping a weedwhacker onto a dog and letting it run wild.
The tech itself is safe as far as real-world interactions go - the weakness lies in unchecked access to systems surrounding it. It's not safe at all when it comes to human interaction (lots of ongoing lawsuits demonstrate that), though.
There is real danger here, but it has nothing to do with doomsday scenarios ala Terminator or I,Robot and more with total corporate control over the lives, perception of reality, and abilities (like critical thinking) of people.
This is like saying cars don't kill people, because if nobody drives them faster than 3mph there's no problem. The _whole_ promise of cars is that they can go fast, just like the whole promise of AI is offloading thinking to a computer. If AIs are unsafe without close human supervision and checking every interaction with the real world, they are unsafe full stop.
The analogy would be more fitting if you had said "cars don't kill people if every drivers has proven skills, never drives impaired, keeps the speed in line with weather and road conditions and stays on actual roadways". You know, like everyone should, regardless of whether they're driving a high performance sports car.
To keep with the analogy: cars have seatbelts, airbags, ABS, ESP, lights, horns, crumple zones, emergency braking systems, roads have speed limits, there are traffic stops, insurance, regular inspections (not in all countries), etc. etc.
So what's unsafe here? The car or roads without speed limits, complete lack of safety measures (both active and passive), absence of any supervision and no insurance? That's the problem. It's not the models themselves - they can spit out tokens by the billions, there's no risk there.
You wouldn't give full access to your phone, your computers, your house keys and your credit cards to any stranger on the street now, would you? How is it then, that people act all surprised when a non-deterministic machine that's optimised to achieve goals while taking all the shortcuts it can, suddenly uses the tools handed to it in unexpected ways? That's a failure on the operator's side, not an inherent danger within of the model.
The cars are still unsafe at speed. All those mitigations reduce the risks but do not eliminate the inherent danger. Sandboxing agentic LLMs is similar, there is no way to mitigate the inherent safety problem entirely while preserving the power of the thing (an LLM without a harness is safe in the way an engine without a chassis is - safe and useless).
But safety is just one thing people optimise for; if it's convenient enough people will accept imperfect safety (as with cars). It's unrealistic to just heap blame on end-users who use mostly very safe tools in the common way, even though in aggregate they are meaningfully dangerous. They don't think they are strapping a weed whacker to a dog; they think they are driving a car.
> It's a function call that ingests symbols and spits out symbols and that's it. 100% of its actual capabilities are tied to harnesses
This is a bad and misleading way to think about it. Note that it's trivial to make the harness that you claim capabilities are tied to (the LLM itself could write it from scratch in one shot), but no matter how good a harness you have, it won't make gemma4:e4b capable. That's because what actually gives capabilities is the LLM's intelligence - or if you prefer not using that term, the fact that the probability distributions the LLM spits out depend on the context in useful ways.
I think we're talking about fundamentally different perspectives here.
I'm not talking about what the LLM does internally. If a metaphor helps, here's one to help you understand what I was trying to get at:
Imagine an evil genius that has no eyes and no limbs. Everything they could learn about the world is presented to them by means of some person describing it to them through words. They have no way of directly interacting with the world and rely on someone executing any action they want to take and describe the outcome to them. Now how dangerous would you say such person would be? How dangerous could they become?
That's what I was getting at. Replace person with LLM (or any other AI system). Replace the person that communicates with an external interface (the harness) and I hope you understand. It doesn't matter whether the LLM could generate the harness by itself - it still is just a bunch of weights sitting in memory being run by an execution engine. That's what it fundamentally is, whether you like it or not. It cannot do anything on its own - and no, not even writing files. It's the execution engine that translates the numeric output into words (or images or video or audio) and the layer above (the harness) that takes that output and interprets it to execute actual actions.
This is not about what you or I think about the internal capabilities of the model - that's irrelevant to the conversation and you can replace LLM with a random token generator and the point still stands. The model itself is incapable of performing actions - from reading files to writing files, to controlling physical machines. All that is and HAS to be done by external interfaces outside the control of the model.
Yes, now say there are several major companies and an entire open source ecosystem dedicated to creating superpowered exoskeletons with chainsaw arms and jetpack legs for this limbless villain. Is that cause for concern? I say yes.
I would say it's more like an interface for the model to interact with the world. If you give the model access to filesystem and bash that technically unlocks all computer use, so how are you going to control that? By trying to regex match against the commands the AI uses? All you have is auth or containment, and AI can hack auth and people will not stop connecting AIs to the internet. It's a ridiculous premise that just because the harness is "normal code" that means we can control the AI.
The world's institutions, systems, and industries are all rapidly digitizing. So while I'd concede the point that, yeah, there's no way a rogue AI can just take over some powerplant and blow it up because of analogue systems the AI can't access, that isn't necessarily true for some powerplants already, and more and more powerplants will be connected to networks and controlled by software systems in the future. The more we digitize our systems the more potential for AI to exploit vulnerabilities and affect the real world.
AFAIK there isn't that much stopping anyone from spawning an AI swarm and telling it to "spread and go hack everything for the lulz."
> If you give the model access to filesystem and bash that technically unlocks all computer use, so how are you going to control that?
The same way we've done it since machines became multi-user: boring old system access restrictions. Nothing fancy, nothing radical, just good old minimal access rights required to perform a defined set of whitelisted operations.
> It's a ridiculous premise that just because the harness is "normal code" that means we can control the AI.
What is it then? Is not just a program that takes model output, parses it and performs tool calls from the text it receives and then feeds the result back into the model and calls it again with those results? It is normal boring old deterministic code. Many are open source. Look at them. Understand what they do and the apparent "magic" goes away real quick. Harnesses are nothing special.
> AFAIK there isn't that much stopping anyone from spawning an AI swarm and telling it to "spread and go hack everything for the lulz."
Aside from lower cost and possibly greater scale, there's literally NO difference between that and (state sponsored) hacking that has been going on for decades. First it was script kiddies, now it's ML models. The threat model remains the same and so do the counter measures. The real danger is still the harness (and its access to external systems), not the model itself. Restrict the access of the harness and the model can't do anything harmful, see above.
How do you restrict access of the harness when there are fully configurable open source harnesses with zero out of the box restrictions? Yes maybe I as a good citizen can put my agent in a sandbox, but some script kiddie will not. And the barrier to entry for being a script kiddie is much higher than for installing OpenClaw. I also can't spin up a hundred cloud vms each with a dedicated script kiddie running 24/7.
When put to the test in real-world environment, the capabilities don't look as impressive as benchmarks and synthetic tests might indicate. So doubts about actual spatial reasoning capabilities remain.
I've worked in wholesale logistics (backend for inventory management) for many years way back when and let me tell you, cost of ink vs cost of a chip the least of the problems. Barcodes (and QR codes as used here) have two major flaws: reading them is challenging (works fine if the package is undamaged and you TRUST that it contains what it says on the box) and they're still just a number. RFID can help with both these issues and the savings from that can outweigh the additional cost by a lot.
Utility is key and RFID has a lot of advantages over bar codes (and QR codes).
The chips are sub-cents per unit in terms of cost and weigh milligrams.
RFID chips come in many flavours the ones most used are passive, i.e. basically just a bunch of wires printed on a tiny piece of carrier material. The label on a shirt contains more resources by weight.
Really only wires, or is there still a (tiny) chip on there somewhere?
While it can indeed be helpful as an indication, cost often doesn't factor in externalities such as pollution during production, and recycling after use. Paper has much fewer problems in those areas.
100% agreed, but if you truly want to go down that route, you'll quickly find that basically NOTHING has externalities priced in. If you truly were to do that (which I'm personally not opposed to), prices of pretty much all goods and services would need to be adjusted.
It's ok, these will go right next to the trillions of cigarette buds thrown in nature each year and probably the trillion of trillion microplastics and nanoplastics shed each year.
Nothing to see here and there can't be any possible negative consequences when people are making so much money.
The physics aren't the problem, though. You CAN cool the processors provided you have large enough radiators, a really good system for the coolant flow, and hardware that's fine operating continuously at around 100C or above.
That's basically an engineering challenge more than anything else. Also keep in mind that one of the reasons the ISS for example requires rather large radiators, is that humans typically don't enjoy temperatures above 40C for prolonged periods of time.
Silicon chips on the other can tolerate much higher operating temperatures just fine and even short spikes around 100C can be acceptable for some hardware.
The real challenge is getting launch cost down and securing the rights to pollute polar orbits and frequency bands. Sure, there's plenty of room up there in orbit, but much of that room is not fit for purpose in the context of orbital data centres. Frequency bands for up- and downlinks are also often ignored in the discussion.
Basically I don't see the current issues with the concept in the required engineering or physics - that's solvable in the near term.
I find it much more difficult to believe that there's a way hundreds of thousands of satellites can share the same orbit (i.e. polar orbit in a quite "narrow" band of distances from the surface) without causing major issues.
Same goes for frequency bands, as it wouldn't be just one US company doing this, but dozens from around the world, each with hundreds or thousands of sats and everyone needs a share of the available communication frequencies.
Then there's the logistics of getting the satellites into orbit in the first place. Not just the launch cost, the logistics! Fuel, NOTAMS, maritime exclusion zones, launch licences, etc. The concept would necessitate increasing current launch rates by at least an order of magnitude and cost aside, the logistics and regulatory frameworks aren't ready for this.
That's nit-picking. It's one of the first steps towards this goal. Why would any sane engineer put a complete orbital data centre prototype into space without having any data on how most crucial piece of equipment would perform in the target environment?
First they figure out whether they could get away with using unmodified "off-the-shelf" hardware they already have and how to cool it.
By the time they have a working design, it'll already be a small group of such satellites basically being a fully operational orbital data centre, albeit a rather small one. So yeah, I think there's a point to calling this a test for an "orbital data centre", just like Apollo 1-A was a test for a manned Moon mission despite being suborbital and uncrewed.
Debatable. It's not economically viable under current economics. There's an engineering problem that's solvable in the short term, e.g. the cooling issue.
The economics of mass production of satellites has been demonstrated to work out just fine (see Starlink) and what's missing is the same shift when it comes to space launches. Now I'm not saying that this is going to happen, but that's what the whole purpose of Musk's Starship/Superheavy rocket is.
If - and that's a big if! - the whole Starship concept works out, it'd drastically change the economics of putting satellites in LEO and the whole idea instantly becomes much less stupid.
So basically it's a bet on Starship working as advertised in the very near future. If it does, data centres in space (e.g. in the form of laser-linked satellite swarms) become a compelling alternative to terrestrial data centres. No regulatory issues, no infrastructure problems (power, water, ...), no risk of sabotage or drone strikes (see AWS in Middle East), and no land leases required.
Once you add up all these risks and costs and put them next to the launch cost that SpaceX envisions for Starship, the concept suddenly sounds quite plausible.
I'm sceptical myself - though for very different reasons - but I wouldn't call it an outright scam. It all hinges on one thing: Starship has to work. Cheaply, reliably, and within the next couple of years. If it turns out to be less capable, significantly more expensive, less reliable than F9 and F9 Heavy, or not ready until the mid-2030s, the concept is dead.
The engineering problem is called physic and thermodynamics. And unless we learn how to change those rules the problem will be unsolvable. It's like discussing perpetual motion...
I hear this a lot without actual calculations backing up the claim.
First of all, orbital data centre doesn't mean a monolithic 500MW monstrosity orbiting Earth. That's not what the companies actually investigating the idea (Google, Amazon, SpaceX, ...) have in mind.
The real-world implementation would use constellations of Starlink-sized satellites, so we're talking about a ~10kW power envelope. Under ideal orbital conditions (sun synchronous near polar orbit, ~250-450km), you can get away with as little as 40m² of radiator area depending on engineering margins, provided you allow your radiator/coolant temps to be 70°C and above (70°C is what I used for my estimation plus an ideal 400km SSO and a reasonably efficient radiator similar to the improved ETCS on the ISS - no ammonia for the coolant, though see below as to why).
People often cite the ISS and its ~100m² radiators for comparison. The problem with that is that for one, the design operating temperature of the ISS' EATCS is
4°C on the low temp loop and 17°C on the moderate temp loop. That little known fact aside, the current improved ATCS is capable of rejecting 70kW of waste heat - way more than the thermal output a realistic satellite would produce. The active temperature control system (ATCS) of the ISS is designed around humans and science experiments in a 400km 52°N to 52°S orbit, not silicon chips that are allowed to run much hotter than 25°C and sit in SSO.
In short the *actual* thermodynamics work out just fine if you consider real satellite designs instead of sci-fi monstrosities, use realistic operating temperatures for the coolant loop designed around silicon chips, and the desired target orbit.
So when you say "unless we learn how to change those rules the problem will be unsolvable" I'd honestly like to know why my 25+ year old literature on satellite mission and hardware design, Boeing's numbers from "Active Thermal Control System (ATCS) Overview" document, the formulas from NASAs "Guidelines for thermal analysis of spacecraft hardware", this gem from DLR https://doi.org/10.1016/j.asr.2024.11.024 and my calculator are all wrong at the same time.
Sure, I only did some napkin math (still took nearly 45min) using some simplified formulas and crude estimates for the required parameters (I was curious, but not THAT curious), but I always rounded up and used a healthy 2x margin on top of the estimated result.
My contacts at various factories and my own eyes after touring said factories tell me that the assembly lines have been mostly run by programmed (no quotes required, just SPS) robots for the past 50 years or so.
Welcome back to the 1970s I guess?
The ones I see deployed in construction do stuff like drilling holes in hard to reach places, or laying bricks. None of them look like people, though. Much like the typical household dishwasher hasn't been named Alice or Doris for the past 50+ years, but rather Bosch or Samsung and doesn't use actual hands.
But yeah, humanoid robots do look cool and many movies and books told us to be fearful of them.
It would've been impressive 15 years ago. People seem to forget that first humanoid robot capable of competently walking was showcased by Honda 26 years ago.
over 20 years ago, the updated version was capable of running and climbing stairs.
So a purpose-built (look at the thing - running is all it can actually do) two legged fast running robot doesn't seem all that impressive today. Cool? Sure! Useful? Not so much. Impressive? Whatever... it's just showing off at this point, much like Boston Dynamic's Atlas series that went nowhere (in contrast to this machine: this one did half marathon distance:)
100% of its actual capabilities are tied to harnesses (the actual "agent"), i.e. ordinary deterministic programs that are connected to networks or machines and enable interaction with the outside world. This part (the part that can do harmful things) is fully under human control and all the recent headlines about "agents going rogue" are - as someone (forgot who) put it - akin to strapping a weedwhacker onto a dog and letting it run wild.
The tech itself is safe as far as real-world interactions go - the weakness lies in unchecked access to systems surrounding it. It's not safe at all when it comes to human interaction (lots of ongoing lawsuits demonstrate that), though. There is real danger here, but it has nothing to do with doomsday scenarios ala Terminator or I,Robot and more with total corporate control over the lives, perception of reality, and abilities (like critical thinking) of people.
reply