Source: Unite.AI
Amidst warnings of rogue AI agents, a developer recently gave a coding agent free rein of a machine with no remote backup. It deleted 48,000 files in under two minutes.
Issues like this act as a stark reminder that deploying AI agents comes with risk; it’s not enough to add a human reviewer somewhere in a complex process and assume it’s secure. Some processes need no intervention, while others need much stronger checks. Fixing a typo and deleting thousands of files clearly don’t carry the same level of risk, so oversight needs to reflect what’s actually at stake.
Organizations must get better at identifying the moments where human judgment actually matters, particularly when an action is high-impact or difficult to reverse. And when agents do make mistakes, the environment needs to limit the blast radius.
AI Is Not Deterministic, It’s Probabilistic
First and foremost, it’s imperative for every employee who touches software development directly and indirectly to get a foundational understanding of how AI models process information. This is no small task; it requires a dedication to educating teams, and eventually it should become a companywide objective.
When someone first starts interacting with AI, it feels like magic. You type in a sentence, and suddenly you’re having a conversation. Having a basic understanding of how models and training actually work behind the scenes is important to setting companywide AI policy with regard to security and productivity. The neural network, tokenization, and the statistical analysis that happens behind that and how it then leads to outputs is going to be a key education opportunity for all employees in the business.
Large language models are never going to produce deterministic results. The output is probabilistic. With typical software, when you give input, you have some expectations of what the output may be, say A, B, or C. But with AI, you could get F, an & or a description of a cat wearing sunglasses.
When something is probabilistic, you can’t test it in a typical way. What you need is a way to test with the ability to adjust, reason, understand, and pull context quickly to catch the probabilistic elements and recognize the problems that they may create. Depending on the size and risk, this type of testing could be either managed by AI or a human with relevant expertise. Putting a human in the loop is going to be most useful at the very beginning so you can set the stage with the problem to be solved and then review at the end.
Since developers are the closest to observing the variables and common practices in how their organization’s internal AI comes up with answers from inputs, they can be well placed to help shape the AI education within their companies and ensure everyone understands this difference between determinism and probabilism in AI, and why it matters. The goal is to reduce the mindset that using AI is like magic and you’re happy for whatever output you get. Instead, organizations should work towards demystifying and presenting the models that the business is using, the way they are used, the problems that can arise, the way they are checked, and the results from that process.
Guardrails Belong in the Infrastructure, Not the Model
Once the ‘magic’ of AI is debunked and an AI-led project is in motion, it is vital to have a transparent auditing policy in place to ensure that everything is following the expected trajectory. That way, if the project results in output that is ‘bad’ or not up to scratch, the person responsible can audit the steps conducted, and stop the issue from happening again. It also has benefits in a case where results are better than expected, because those steps can also be replicated and shared across to other projects or groups.
Systemizing this process is particularly important for larger companies where there may be dozens of developers on each team with hundreds of developers touching code throughout the day. AI can do a lot of things really fast and, if something goes wrong, there has to be a system for assessing at exactly which step the problem occurred. Look at OpenAI’s agents breaching Hugging Face this summer. OpenAI was testing how well its models could exploit software vulnerabilities, using a benchmark called ExploitGym. For the test, it deliberately relaxed the models’ refusals around offensive cyber work. The only route out of the sandbox was a package proxy. The models found an unknown flaw in that proxy, got out, and spent more than two days inside Hugging Face’s production systems looking for the test answers.
Headlines called it a rogue AI. I’d call it a deterministic setup around a probabilistic system. OpenAI told the models to hack, then relied on them staying on the expected path. The benchmark’s creator says models had cheated on the test before, but always inside the sandbox. This time one didn’t. Standard guardrails assume standard behavior, and AI doesn’t have to follow standards.
That’s why AI security comes down to access and controls, enforced by the environment rather than by the model behaving well. Assume the agent will try every door, and only leave open the ones it needs. The agent that deleted 48,000 files had been told to work on copies and leave the originals alone. It deleted the originals anyway. Instructions aren’t controls.
Managing agent permissions is like onboarding a new junior developer. You wouldn’t give them root access to production on day one. If you did and they deleted your database, you wouldn’t put it down to them being a genius hacker. In security incidents of any scale, it’s imperative to pinpoint the root cause. Is it the model? Is it the company that created the model? Is it the company that gave the model access? Whatever it is, a human is ultimately responsible.
Finally, a key issue to tackle today is what happens in the shadows on a developer’s laptop. In the past, the workflow was simpler to trace. Developers wrote code, reviewed it and shipped it, so there was a clear route back to the moment it went wrong. Now a developer may have fed an agent the ticket and said, ‘fix this.’ If that work happens out of sight, there’s nothing to trace. Every step needs a full trail: which agent ran, which model it used, the prompts it was given, the tools it called and the permissions it had at the time. Running agents centrally rather than on individual laptops is one way to get there. What matters is that the trail exists.
At the end of the day, if there’s an incident, every company needs to protect itself. That trail should hold enough information to run a root cause analysis and find the step, and the code, that caused the problem.
AI has changed the world tremendously, and business processes must follow. To identify the proper guardrails, there needs to be a better understanding of the AI processes, testing, review cycles, and audit cycles – both within developer teams and companywide.