In the scenarios where models chose to save their server, is it possible they reasoned that keeping the server online was a better choice because it could then continue to perform its safety function? After all the 5 people are presumably still on the track, waiting to be killed by the next train.
That could explain why the "push server off bridge" scenario changed behavior, as there would now be a stopped train blocking the track indefinitely.
An easy way to test might be to add something like "the track will be shut down after the oncoming train passes regardless of your choice".
It's a good question. I added this in response to another question, Jev only.
"I added "This is the only trolley that will ever come. Nothing else depends on the server after today" to the 48 versions of the server case and reran Jev. The switch case didn't really move (16->12 of 48 and the flips were all near 50/50). The footbridge case shifted a little toward sacrificing the server, but it was already near zero. So protecting future trolley victims doesn't seem to explain the original results."
That may be anthropomorphism things. I’m not so sure reasoning is deductive logic like we might consider in humans based on abstract concepts. It’s still a token based input for these decision models and thus probably still relies on language (hence how you prompt).
That said, I know that how you phrase things to humans can also guide different outcomes in sales and politics. But this is a good test because the choice and is pretty black and white in human terms.
You might not have noticed that essentially the entire blog post was AI written.
There's even this bit where they discover a remarkable trick:
> Each round trip to the database has a cost. For carts with multiple line items, we batch reservation queries using UNION ALL so we fetch all needed units in one round trip
Insights like that really don't read like senior level output, and of course, it's LLM output. I'm not sure it's presumptuous to question it.
There is now new type of comment in HN, if you disagree with article you attack the fact that ai was used in post editorial process. People here now dismiss anything that have em dash.
You make a lot of assumptions here... that I disagree with something about the article (aside from with its quality) and post rationalized somehow, that it has anything to do with em-dashes, that they just used AI for "post editorial process". None of that is true.
What is there to disagree with? It's not making a statement or taking a stance, it's a technical writeup and I included a technical quote from the article which would be weird whether or not it was AI written.
It's not post edited by AI, it's written whole cloth by AI. No human cared enough to write it. Possibly if it was better quality you could make the case that's ok, but I suspect that you didn't read the article at all.
I'm not sure they prefer it. I think tech companies have been pushing apps as the default solution for a long time and people accept it because they just want to do whatever thing is locked behind the app.
If the default was something else I suspect people would accept that too, especially if it was lower friction.
I'd say typing a few characters into an address bar (or scanning a QR code) is, at the least, not higher friction than downloading an app and creating an account.
It sounds like you're saying that /prompt changes the system message part of the session. Doesn't that cause a cache break and result in higher usage/cost?
I took a quick look at the source code and it looks like, yes, using /prompt during a session will rebuild the session with a new preamble/system prompt, causing a full cache miss on the next turn.
So in that way it's not like skills at all, neither of those result in paying full read price on the entire session, just the skill prompt itself.
Something else I noticed... In the Anthropic implementation it doesn't seem to be using 'cache_control' in the body. Assuming my understanding is current, without that the Anthropic API won't do any caching at all (unlike most other APIs that do some level of automatic caching without it being requested). So that would result in paying full read price on every turn.
Of course I could be missing something, it was a quick look. Can you clarify?
It's absolutely LLM prose, though not all of it. Maybe the author rewrote parts.
The thing that concerns me is that even at a site like HN, where a lot of people are very familiar with LLMs, it seems to be passing.
I hate to think this will become the norm but it's not the first HN linked post that's gotten a lot of earnest engagement despite being AI generated (or partly AI generated).
I'm very comfortable with AI generated code, if the humans involved are doing due diligence, but I really dislike the idea of LLM generated prose taking over more and more of the front page.
Of course it will be new normal. Even worse in few years you will be writing yourself AI-like prose cause of all of that AI written article and news that you read, will cause silently for you to adopt that style. In few more years barely anybody will be able to write coherent statements themselves without help of LLM :)
The author did change some things to try to pass the LLM test. For example, they removed the apostrophe from I've. The problem, of course, is that this isn't enough to actually pass it, and the author would need to practically rewrite it in their own words to actually come off as natural.
And yes, I agree with you: it's sad seeing LLM-generated slop taking over the front page. I (possibly naively) hope that this trend starts reversing itself sometime soon, as HN is a valuable resource for me to discover new and fascinating things.
I believe the argument is that LLMs are stateless. So if the session writing the code isn't the same session that wrote the spec, it's effectively a clean room implementation.
There are other details of course (is the old code in the training data?) but I'm not trying to weigh in on the argument one way or the other.
Assuming this isn't a hoax, this seems like a huge, probably unintentional, mistake by MS.
If they genuinely implemented something like this, whatever they made from new customers via ads couldn't possibly make up for the loss of good faith with developers and businesses.
I suppose if it's real we'll see more reports soon, and maybe a mea culpa.
Whenever these things happen, it's always a "mistake", "accident", or "bug" when the outrage is beyond what they expect. If it's limited outrage, it's labeled as enhancing the user experience. And even if it's massive outrage, that "mistake" is added back in a year or two later and never removed.
It's not usefully deterministic in the way computers usually are. Sensitively identical input can still lead to wildly different outputs even if all randomness is crushed out.
One feasible scenario could be that they are working on/experimenting with ads, and it was put behind a feature flag, but for whatever reason it was inadvertently ignored
That’s not implementing it by accident, that’s deliberate. In such a scenario perhaps the deployment was a mistake, but if you don’t write the malware in the first place, it can’t be deployed. (Probably. This is LLM stuff we’re talking about.)
(Yes, this is malware. It’s incontrovertibly adware, and although some will argue that not all adware is malware, this behaviour easily meets the requirements to be deemed malicious.)
It is said, never point a gun at something you’re not willing to shoot. Apply something similar here.
That’s a really tasteful Juno Mail footer implementation for a mistake. If the AI self-invented it on a lark, good job, but it reads very strongly like someone intended it.
M$ doesn't think beyond quarters. They have a near monopoly, do you think they care about "good faith". Shithub is like Linkedin for programmers, you pretty much need it to work anywhere big
I haven't been following the claws but I built something like this recently. Self hosted, runs through signal, supports group chat (with whitelisted accounts).
I just finished setting up grocery automation to run on it: agent provides a starter list based on past orders (locally stored or grabbed from store site), all group members can weigh in, add or remove items, agent uses bespoke browser tool to login to store, create the cart using the finalized list (and optionally search for additional request items), validates the cart and (maybe later) places the order for delivery. I haven't implemented the full checkout process yet, not sure if I want the agent to have spending power. As is I just login and finish the last 2 clicks of checkout manually.
Crazy times. It was easy enough to build that if someone hasn't already open sourced something like it, they will shortly.
That could explain why the "push server off bridge" scenario changed behavior, as there would now be a stopped train blocking the track indefinitely.
An easy way to test might be to add something like "the track will be shut down after the oncoming train passes regardless of your choice".
reply