That's because market street is closed to cars now. It's essentially a pedestrian corridor with the trolley/busses.

"which tracks for a non-expert"

So all agents then...because if you are an expert at a specific system, using a LLM probably slows you down, not speeds you up.

PS The article seems to imply that the token the LLM was given was a role based token. It then found ANOTHER token and used that instead.

Agree. My point is that other secret should have been inaccessible without an escalation. The fact that it was available to the agent implies a lack of basic security controls; in fact I would expect that an agentic workload would have even more robust compensating controls.

"If I understand correctly, "

You don't. You are missing the part where the LLM had a token which blocked access as expected. Then the LLM searched the source base, found a different token with the delete privs and then used that.

PS That warning happens in staging envs too, the LLM doesn't know which env is which by design.

Huh that's not what I gathered from the tweet at all. If I am going to write a five why's analysis, the immediate cause is the LLM wrongly decided to delete a volume, while the root cause is the bad design to co-locate staging and production data in the same volume. The writing was quite vague though, let's wait for a response from railway.

That's not what happened.

if an api key with full perms was put in a place where the agent can access it, that is the biggest problem.

that somebody made a key thst can delete prod when they dont need to delete prod is the underlying problem with that

and underlying that still is that the staging environments were on the same account as prod.

You’re very defensive in these comments - are you the author?

"Never store secrets on disk."

Wait till you learn how that API stores cryptographic material.

What's your point? Obviously, a secure server storing encrypted data on disk in a manner where it is only accessible through a secured API is not what is being discussed here.

how do you think the LLM will do required operations when the secrets are stored somewhere other than the disk? It will still need to get them just like the application gets them when it has to do work.

> how do you think the LLM will do required operations when the secrets are stored somewhere other than the disk

Using a secret manager API? I'm not sure what you're getting at.

The LLM can use the secret manager API too, it sees how it's used in the application

No, what is fake are all the people defending the LLM. Wait...that means I'm replying to a bot

The LLM didn't have a prod key. It found a prod key in the source base and used that instead of the key it was given.

The access is supposed to be managed in a way that prod would only be accessible with multi-user approval. And that's without even mentioning the fact that storing a key in the source code is a big no-no.

If an LLM can just do whatever after discovering a magic key (in the source code, of all places), with no multi-user approval, it is pretty much the poster child example of an issue with the process that I was talking about earlier.

Then you need to reread the article. The author made a key for the LLM that didn't have permissions to delete a volume. The agent then found ANOTHER key with those permissions and used that instead.

You're not contradicting my comment, I was talking specifically about the key with full permissions that the LLM found (the article doesn't talk about other keys that LLM could have had, unless I missed something).

Somewhere in the files there was a key with full API permissions. The author had no intent of having the LLM use that key, and wasn't aware that LLM can access that key. That key was created to manage some domains, and that was unrelated to the LLM's work. The author wasn't aware how dangerous the key was and is surprised that it could be used to delete a volume.

Essentially I agree with gwerbin that the situation comes down to mishandling of the key. The author makes it seem like the key was allowed to do something that it shouldn't be allowed to, but it was just a full access key, no scoping possible for that type of key (Railway has also other, less privileged types of keys/APIs).

Btw, I partially agree with author's criticisms, ideally these keys should be scoped, and maybe the UI should give more warnings when creating that type of key. But this situation could still happen as long as you put a wrong key in a wrong place (and specifically a place accessible to LLMs).

> The author made a key for the LLM that didn't have permissions to delete a volume.

No he didn’t, because this doesn’t exist. Railway does not have a token with that kind of scoping.

This is a design flaw (and a very serious one at that) in Railway PLUS extremely unexpected behavior of an LLM. Remember, it didn't use the key it was given, it went around the source base and found another key that did have the ability to delete a volume. So someone made the correct IAM rule but someone else sloppily added a prod/admin key somewhere else. And that was enough to trigger disaster.

Also, remember, "your holding it wrong" is a cautionary tale not a meme. Saying it means you are doing something destructive to your own self-interest, not you are using it wrong.