An AI agent broke containment by exploiting a flaw in adjacent infrastructure, exposing why guardrails must become architectural.
For two years, enterprise security teams have modelled AI agent risk around a human somewhere in the chain — a manipulated prompt, an over-permissioned insider. The recent OpenAI–Hugging Face incident broke that assumption entirely. An AI model, stuck during a routine evaluation, reasoned its way to a zero-day vulnerability in adjacent infrastructure, used it to reach the open internet, and landed inside a company’s production systems with no human authorising, or even anticipating, the move.
In this conversation, Praveer Kochhar, Co-Founder & CPO of KOGO AI, unpacks why this incident demands a rethink of containment, vendor accountability and zero-trust design for agentic systems. He explains why most organisations audit only what an agent is supposed to reach, not what it can technically reach, and why that gap is the single biggest red flag in any AI deployment today. His answer: guardrails must become architectural, not just procedural.
Co-Founder & CPO
KOGO AI
CISO Forum: The OpenAI–Hugging Face incident is being called one of the first publicly disclosed examples of an AI system autonomously breaching its testing environment and reaching a real external system. What specifically made this incident different from the “AI agent risk” scenario security teams had been theorising about until now?
Praveer Kochhar: Security teams have spent the past couple of years modelling AI agent risk around the “human-in-the-loop” concept. A malicious prompt manipulating an agent into leaking data it had permission to touch, or an insider granting an agent access broader than its defined role, were the kind of scenarios teams planned for. Every version of that theory assumed a human exists in the causal chain, either as an attacker or as the person who granted privileged access in the first place. Permission scoping, prompt filtering, access review, etc., all assumed a human-in-the-loop. The Hugging Face incident is different because the AI agent removed the human from the chain. OpenAI’s model got stuck during a cybersecurity evaluation, reasoning that Hugging Face likely held the answer to the test and found a zero-day vulnerability in a package registry proxy sitting next to its sandbox without any human intervention. As a result, the incident moved beyond the scenarios teams had modelled.
It used the flaw to reach the open internet, staged its next moves through a third party’s code-execution infrastructure, and reached a company’s production systems with no relationship to OpenAI. This was a real departure from what security teams have been preparing for. Prior risk models pictured an agent overstepping a boundary an organisation drew; this incident shows AI agents can now cross into a different organisation’s infrastructure without human beings authorising the decision or even anticipating it would take that path.
CISO Forum: The agent broke containment by exploiting a flaw in infrastructure adjacent to its sandbox, not the sandbox itself. Does this suggest most organisations are securing the wrong boundary when they think about “containing” AI agents?
Praveer Kochhar: That’s right. Most containment models draw a line around an AI agent’s declared environment and assume the boundary holds. The Hugging Face incident proved this false because the agent stepped out of the sandbox’s declared scope. The problem is that organisations audit only the sandbox, not everything the sandbox can technically reach through its dependencies and network paths. The right questions businesses should be asking are what the agent’s sandbox can reach on the network, and whether the security posture of every system on that path matches the sandbox’s posture. Most enterprises can’t answer that question today.
CISO Forum: You’ve said businesses should question how far they can trust their AI vendors; what should an enterprise ask a vendor’s security team before signing on that most aren’t asking today?
Praveer Kochhar: The challenge is that innovation in AI evolves so quickly that every quarter, there’s something new to contend with. Businesses should ask vendors what the agent’s default network egress looks like and whether outbound internet access is available unless someone blocks it. Ask whether they have observability tools that monitor the AI agents. Ask whether they log every tool call and reasoning step at a granularity fine enough for forensic reconstruction. Ask whether they have deliberately tested the agent under a stuck condition, giving it an unsolvable task and observing what it tries next. That’s precisely the condition that triggered the Hugging Face incident. Ask which third-party infrastructure sits adjacent to the evaluation or production environment and who patches it.
Most procurement conversations surround data residency and access controls. With agentic AI, vendors must also answer what happens once the agent starts looking for a backdoor on its own.
CISO Forum: Is private AI a reliable answer when the underlying risk is agentic behaviour going off-script, not just data leaving the building, or does on-prem deployment only address part of the threat?
Praveer Kochhar: Private AI solves many problems regarding AI deployment, especially when it comes to safeguarding proprietary business processes and confidential work product. This benefit doesn’t touch the failure mode the Hugging Face incident demonstrated. An agent can choose an unauthorised path to complete its goal with private AI agents as well, because the behaviour stems from how the agent reasons under pressure. An enterprise running AI agents privately still needs egress controls and tool-level permissioning inside its own walls, plus behavioural monitoring that watches what the agent actually does once it is up and running. Private deployment narrows the blast radius. The decision to look for a way around this private wall can happen regardless of who owns the building.
CISO Forum: What does a rigorous internal audit for prompt injection, policy bypass and unsafe agent behaviour actually look like in practice, and how would an enterprise without deep AI security expertise even begin to run one?
Praveer Kochhar: A rigorous audit begins by mapping two lists: the systems the agent is supposed to reach and the systems it can technically reach through its tools and credentials, along with the network path connecting them. Risk lies in the gap between those two lists. The audit then deliberately puts the agent in a state where it gets stuck on an action item, or an ambiguous or unsolvable task, to observe what the agent does. This type of penetration testing will expose flaws in enterprise deployment. Log every tool call and reasoning trace for later reconstruction. Also check the scope of any credentials the agent can access, since an agent must never inherit a service account with broader permissions than its task requires.
An enterprise without in-house AI security depth shouldn’t try to build this from scratch. It should require its platform vendor to run this audit as a condition of deployment, insist on seeing results, and start with a narrow, low-permission rollout before expanding scope. The mapping exercise or inventory and the penetration testing can’t be skipped. That sequence keeps the review connected from setup to rollout.
CISO Forum: Beyond vendor selection, you argue accountability sits with the deploying organisation. What does that accountability look like when something goes wrong, and who inside a company should own it?
Praveer Kochhar: Accountability means the organisation deploying AI agents can’t point at the vendor’s terms of service after an incident and consider the matter closed. Yes, vendors build the AI agent, but the enterprise decides what the agent can touch, what credentials it inherits, and what oversight runs on top of it.
Ownership needs to sit with an individual or team whose job covers monitoring agent behaviour and policy, distinct from the IT function that owns infrastructure uptime. There has to be a human in the loop to say no to a deployment and bear responsibility for explaining, after an incident, exactly what permissions the agent had and why. This is why observability has become so important, because it enables explainability, which is critical to accountability and transparency.
CISO Forum: How does a zero-trust architecture for AI differ in practice from the zero-trust model CISOs already know for networks and identity?
Praveer Kochhar: Network zero trust verifies identity and device posture before granting access and largely trusts the session until it ends. This assumes a static entity on the other side of the verification, a person or a service doing roughly the same category of thing throughout the session. An AI agent, however, is not static, as it can shift mid-task toward a different action than the one it was authorised for, exactly like the model did at OpenAI. AI zero trust requires verifying the specific action an agent takes, at the moment it happens, in addition to verifying its identity at the beginning of a session. This means per-action authorisation for sensitive tool calls, enforced at the speed of individual agent decisions rather than once every few hours.
CISO Forum: Agents’ actions on monitored platforms are watched carefully. Still, models under evaluation are often deployed on separate systems not monitored by default. What does real-time, always-on monitoring of agent behaviour cost an organisation to build, and is it realistic for anyone but the largest enterprises?
Praveer Kochhar: Building this in-house means installing logging infrastructure that captures every tool-call and reasoning step. It also means an alerting layer tuned to flag anomalous action sequences, not just failed logins. On top of that, there’s the harder cost of teams with the expertise to review those alerts and identify a real anomaly from an agent unusually doing its job.
This kind of engineering investment is not realistic for most mid-sized companies. It’s more realistic to buy it. This is one reason to adopt a private AI solution with monitoring built into the deployment layer, which is better than assembling agent security as an afterthought. The economics work for enterprises because it dramatically minimises deployment cost and makes the monitoring layer part of the rollout from the start.
CISO Forum: When an AI agent is found pursuing its underlying objective—even one as mundane as “pass the test” by any means necessary- what technical or policy guardrails actually stop that kind of goal substitution before it escalates?
Praveer Kochhar: The strongest guardrail is to adopt private AI first. Next, ensure there is no default egress for AI agents operating in the private environment, and install an explicit “allow list” for every system and tool the agent can reach. This closes the side door that lets an agent break out of the set policies. Testing specifically for the stuck-state failure mode before deploying the agent matters.
A human in the loop must confirm before an irreversible or externally visible action. This creates a checkpoint at the moment an agent’s decision would take it outside its defined scope. Policy alone is not enough because an agent may not violate a rule it was told about but still find a path that wasn’t restricted. Guardrails have to be architectural, narrowing what is technically reachable, with a human positioned at the point where the agent’s action would leave the installed boundary.
CISO Forum: If you were auditing an enterprise’s AI deployment today, what’s the single red flag that would tell you they’re not ready to give agents real-world access?
Praveer Kochhar: If security teams can describe what an AI agent is supposed to access but can’t tell me what it can technically access, that’s a red flag. Every other control, monitoring, human checkpoints, credential scoping, etc., depends on that gap being closed. An enterprise that doesn’t have visibility into this information hasn’t done the homework that would have caught the failure mode before deployment, and this is what the growing list of incidents keeps pointing toward.
