Yes and in this case using it for this job at all was clearly not within safe limits.
Do you have any detail on what “this job” was? Like I said, I don’t have access to the original statement because twatter wants me to log in to see it.
What I do see is “routine task in the […] staging environment”, and that doesn’t sound like a big blast zone job. Again, it’s comparable to a job you’d give a junior engineer. There shouldn’t be much a junior engineer can fuck up, no matter how “creative” their solutions.
Whether it’s a human junior engineer, an automatic script or an agentic AI, they should never have more privileges than they need for their job. Granting someone or something that isn’t the senior admin permission to delete a volume is irresponsible.
The AI generating that fucking awful idea is on the AI (or its developers). Both are partial causes for the incident. It’s not just human error, but it’s also human error that would have been dangerous regardless of AI involvement.
Granting someone or something that isn’t the senior admin permission to delete a volume is irresponsible.
Correct. Like I said this was the job of a senior admin.
They gave the AI the job of managing IaC for their environment. Then were shocked when the AI managed the environment incorrectly. This is absolutely not something you let a junior engineer anywhere near.
You seem to be suggesting that the AI should be able to do the job they gave it without being given the permission required for it to do. The thing about doing things in IT, is you need to have permissions to do the things you’re asked to do. So you have to make sure the person you give permissions to is reliable and knows what they’re doing. The AI did not.
They gave the AI the job of managing IaC for their environment. Then were shocked when the AI managed the environment incorrectly. This is absolutely not something you let a junior engineer anywhere near.
See, this is the piece of information I was missing. When the article says “routine tasks”, I didn’t think it meant “manage environment”.
In that case, I agree that it is an issue of trusting AI with something that it shouldn’t have been.
You seem to be suggesting that the AI should be able to do the job they gave it without being given the permission required for it to do.
No, I was simply mistaken about the job it was given. Like I said, all I had to work with was the tomshardware article, which doesn’t go into much detail. I didn’t know that the “routine task in staging” required permission to delete entire cloud volumes across all environments instead of just specific environment-scoped project tokens.
Obviously, if it’s tasked with managing all project environments and given the access to do so, that’s a timebomb. In this case, it was, until it blew up.
The thing about doing things in IT, is you need to have permissions to do the things you’re asked to do.
The thing about conversations on the Internet is you need to actually read the whole comment and realise that there may be some misunderstanding if the other party says things like “I can’t read the twitter link” and assumes it’s a junior dev job when you know it’s not. Then you could just point out the part they didn’t know without being condescending and assuming a fundamental lack of understanding of how IT works.
I’ve had more than enough instances of troubleshooting just which scopes my access token needs to be intimately familiar with the way permissions work. I personally tend to request the least amount required for a given task and only expand when needed and reasonable. It is my understanding that this is the best practice. It was my assumption that they had assigned permissions their agent didn’t need, because you generally don’t hand out “fuck up my prod system” rights.
No, I was simply mistaken about the job it was given. Like I said, all I had to work with was the tomshardware article, which doesn’t go into much detail.
The article goes into full detail. All of this information was in the article.
Then maybe it’s a knowledge / understanding issue, because I’ve trawled through the article multiple times seeing if I’d missed just that. What I do see is:
“deleted our production database and all volume-level backups in a single API call to Railway, our infrastructure provider” (implying permissions across all environments)
“The AI agent was set to complete a routine task in the PocketOS staging environment” (implying it needed (only) staging environment permissions, no description of specific “routine task”, no reason it would need productive access)
“I decided to do it on my own to ‘fix’ the credential mismatch” (This is the AI part of the fuckup: The decision to delete data over a credential issue is something even a Junior engineer probably wouldn’t jump to, so that’s on the AI and on Anthropic, whose safeguards failed)
What am I missing here? What is a “routine task in [a] staging environment”, why does it need admin permissions? Why does the agent have permissions for the prod environment if it’s supposed to work in the staging one?
This is an agent doing IaC for the company. Nowhere is it specified that the agent is only used in staging, only that the fuckup happened while working in the staging environment.
What is a “routine task in [a] staging environment”
Not sure what the routine task was specifically, but it doesn’t really matter. The task involved modifying the company’s infrastructure via IaC.
why does it need admin permissions?
It’s doing IaC, how exactly is it supposed to manage the cloud infrastructure itself without permissions to manage the infrastructure?
Why does the agent have permissions for the prod environment if it’s supposed to work in the staging one?
Who said the agent only works in the staging one? I doubt they’d use a fully qualified infrastructure engineer to manage prod and then give staging to an AI. Either that engineer is managing the company’s infra or he’s not.
What the article describes is an agent that manages their IaC, and when it was set to do a job in the staging environment, it deleted something in prod because it thought that would help it do what it was doing in staging. The CEO says the resource deleted was somehow in both environments at the same time. Not sure I believe that but that’s what he said. If that’s true, I would imagine that’s how the AI designed it in the first place.
Do you have any detail on what “this job” was? Like I said, I don’t have access to the original statement because twatter wants me to log in to see it.
What I do see is “routine task in the […] staging environment”, and that doesn’t sound like a big blast zone job. Again, it’s comparable to a job you’d give a junior engineer. There shouldn’t be much a junior engineer can fuck up, no matter how “creative” their solutions.
Whether it’s a human junior engineer, an automatic script or an agentic AI, they should never have more privileges than they need for their job. Granting someone or something that isn’t the senior admin permission to delete a volume is irresponsible.
The AI generating that fucking awful idea is on the AI (or its developers). Both are partial causes for the incident. It’s not just human error, but it’s also human error that would have been dangerous regardless of AI involvement.
Correct. Like I said this was the job of a senior admin.
They gave the AI the job of managing IaC for their environment. Then were shocked when the AI managed the environment incorrectly. This is absolutely not something you let a junior engineer anywhere near.
You seem to be suggesting that the AI should be able to do the job they gave it without being given the permission required for it to do. The thing about doing things in IT, is you need to have permissions to do the things you’re asked to do. So you have to make sure the person you give permissions to is reliable and knows what they’re doing. The AI did not.
See, this is the piece of information I was missing. When the article says “routine tasks”, I didn’t think it meant “manage environment”.
In that case, I agree that it is an issue of trusting AI with something that it shouldn’t have been.
No, I was simply mistaken about the job it was given. Like I said, all I had to work with was the tomshardware article, which doesn’t go into much detail. I didn’t know that the “routine task in staging” required permission to delete entire cloud volumes across all environments instead of just specific environment-scoped project tokens.
Obviously, if it’s tasked with managing all project environments and given the access to do so, that’s a timebomb. In this case, it was, until it blew up.
The thing about conversations on the Internet is you need to actually read the whole comment and realise that there may be some misunderstanding if the other party says things like “I can’t read the twitter link” and assumes it’s a junior dev job when you know it’s not. Then you could just point out the part they didn’t know without being condescending and assuming a fundamental lack of understanding of how IT works.
I’ve had more than enough instances of troubleshooting just which scopes my access token needs to be intimately familiar with the way permissions work. I personally tend to request the least amount required for a given task and only expand when needed and reasonable. It is my understanding that this is the best practice. It was my assumption that they had assigned permissions their agent didn’t need, because you generally don’t hand out “fuck up my prod system” rights.
The article goes into full detail. All of this information was in the article.
Then maybe it’s a knowledge / understanding issue, because I’ve trawled through the article multiple times seeing if I’d missed just that. What I do see is:
What am I missing here? What is a “routine task in [a] staging environment”, why does it need admin permissions? Why does the agent have permissions for the prod environment if it’s supposed to work in the staging one?
This is an agent doing IaC for the company. Nowhere is it specified that the agent is only used in staging, only that the fuckup happened while working in the staging environment.
Not sure what the routine task was specifically, but it doesn’t really matter. The task involved modifying the company’s infrastructure via IaC.
It’s doing IaC, how exactly is it supposed to manage the cloud infrastructure itself without permissions to manage the infrastructure?
Who said the agent only works in the staging one? I doubt they’d use a fully qualified infrastructure engineer to manage prod and then give staging to an AI. Either that engineer is managing the company’s infra or he’s not.
What the article describes is an agent that manages their IaC, and when it was set to do a job in the staging environment, it deleted something in prod because it thought that would help it do what it was doing in staging. The CEO says the resource deleted was somehow in both environments at the same time. Not sure I believe that but that’s what he said. If that’s true, I would imagine that’s how the AI designed it in the first place.