Last year, a colleague at a tier-one automotive supplier walked me through an incident that kept his incident response and engineering leads at their desks until 3 a.m. on a Sunday.
A script tied to an automated procurement pipeline bought nearly eight million dollars’ worth of excess specialized components during an off-peak maintenance window. No alerts went off. The credentials were legitimate, the API tokens cleared the gateway on the first attempt, and the code had permission to interact with the supplier’s ordering portal. The firewall saw regular, authenticated traffic.
The software did not fail from a systems perspective. It failed from a business perspective.
That disconnect is where enterprise risk is heading. For two decades, cybersecurity teams have worked under a clean mandate: lock the front door, keep attackers out, and verify that whoever or whatever makes a request has the right to make it. If an action is authenticated and the code stays inside its sandbox, the security operations centre calls it clean.
Meanwhile, AI governance lives on another floor. It belongs to legal counsels, ethics task forces, and compliance teams writing thirty-page guidelines about transparency and model fairness.
That split made sense when AI was just predictive text or a recommendation engine spitting out slide decks. It falls apart completely once you give an autonomous agent an API key to a corporate bank account, an SAP instance, or a utility switchboard.
The Limits of Identity
Security teams lean heavily on Identity and Access Management. We spend millions establishing whether a service account is valid.
The problem is that cryptographic trust tells you nothing about commercial sanity.
Take an agent with access to an internal payment route. It sees an invoice, checks that the vendor ID exists in the master database, and executes a wire. The security team considers that a successful run: access was controlled, the key was rotated on schedule, and TLS encryption held.
Yet nobody verified whether the parts were ever delivered, whether the pricing matched the original purchase order, or if the company had enough working capital on hand to cover the check that morning without breaching debt covenants.
Security models assume that authenticated requests are intentional. With autonomous agents, an authenticated request is often just the final output of a hallucinated intermediate step. The risk is not an outside breach; it is authorized stupidity at machine speed.
When Agents Talk to Agents
Things get messier when companies start chaining autonomous tools together.
Consider an industrial outfit:
- The first system scans commodity price indexes, port congestion reports, and weather telemetry to guess whether raw copper supplies will dry up next quarter.
- The second system reads that report, assumes the projection is reliable fact, and buys forward contracts on secondary exchanges to hedge the risk.
- The third system picks up those orders, matches them against internal accounts, and automatically settles the wires.
Notice how easily this breaks. If the first system gets tripped up by bad scrapings or unvetted trade data, it passes that assumption down the line. The second tool does not ask how certain the first tool was—it just sees an instruction. The third tool just processes the paperwork.
Nobody broke in. No accounts were hijacked. Every single program did what its configuration allowed. Yet the business took an eight-figure hit on unneeded contracts because three separate systems blindly trusted each other’s outputs.
Your infrastructure was secure the entire time. You just lost your shirt anyway.
The Human Rubber Stamp
The standard corporate response to this is to add a human in the middle. Put a checkbox on a screen and make an analyst click “Approve.”
In practice, that is security theater.
If you give a mid-level analyst three hundred automated recommendations every afternoon, they will approve them. They have to clear the backlog to hit their performance metrics, they don’t have three hours to reconstruct the raw inputs the model digested, and the software generates an authoritative, convincing explanation for why the purchase is a good idea.
Real human oversight is expensive and slow. If you want a human to stop a bad transaction, you have to limit their reviews to a handful of high-stakes events every week.
More importantly, stop handing them natural-language summaries written by the model itself. Show them the raw numbers: a side-by-side reconciliation of existing warehouse stock against confirmed customer orders. If an analyst has to hunt for a mathematical discrepancy, they might catch it. If they are just reading an AI-generated paragraph that sounds reasonable, they will click “OK” and move to the next ticket.
Turning Policy Into Code
If governance remains a set of high-level principles written by lawyers, it won’t prevent a single dollar of loss. It has to be built directly into the codebase as hard engineering limits.
1. Data Lineage and Fallbacks
If an agent cannot trace its source data to a verified internal database, its privileges should automatically drop. It should lose the ability to commit money or make changes to databases, dropping into read-only mode until a human signs off.
2. Hard Circuit-Breakers
We build circuit-breakers into electrical panels so the house doesn’t burn down when a wire shorts out. We need the same logic in software agents:
- Velocity limits: Cap how much cash an agent can move in an hour, regardless of whether it thinks it found an arbitrage opportunity.
- Scope boundaries: An inventory management bot has no business querying payroll endpoints or looking at customer lists. If it starts scanning those paths, kill the process immediately.
- Firewalls between workflows: Stop letting the output of one experimental model directly trigger actions in production financial systems without an intermediate sanity check.
3. Real Incident Playbooks
Every SOC has a runbook for when a laptop gets infected with ransomware. Do you have a runbook for when an uncompromised automated system signs an invalid five-million-dollar supply contract?
Who has the authority to revoke that agent’s access? Who handles the supplier who already received the digital signature? Can you roll back the database changes without corrupting your audit trail? If you haven’t run that fire drill, you aren’t ready to deploy the agent.
The Threat Inside the SOC
Security leaders need to look in the mirror here, too. We are rushing to put autonomous agents inside our own security operations. We let software look at alerts, revoke credentials, and shut down network ports without human intervention.
That introduces its own blast radius. What happens when an over-eager defensive agent misreads a database migration as a data-exfiltration attack and shuts down your core payment gateway in the middle of Black Friday?
An autonomous tool doesn’t have to be malicious to bring down a production environment. An over-tuned defensive agent can kill a business just as fast as an external attacker if it is allowed to tear down infrastructure on a false positive.
What to Ask in the Next Board Meeting
Stop showing the board slide decks that track vulnerability patch rates on one slide and AI ethics declarations on the next. Those numbers don’t show whether the company is safe from its own automation.
Here are the questions directors should actually be asking:
- What is the financial ceiling? What is the absolute dollar value our autonomous systems can commit on their own before someone with balance-sheet authority has to approve it?
- What cannot be undone? If an agent goes off the rails, which actions are legally or operationally irreversible the second they execute?
- Where is the kill switch? If an automated workflow starts misbehaving, can an operations engineer shut it down instantly without breaking the core databases it was modifying?
Redefining the Defence Mandate
You cannot treat cybersecurity as infrastructure plumbing and AI governance as an ethics seminar.
When code is given the power to make commercial commitments, the security team’s job cannot stop at verifying identity and defending endpoints. Security has to protect the company from the decisions the software makes. If your security team doesn’t understand the business logic of your autonomous systems, and your governance team doesn’t understand runtime engineering, you are building an expensive liability engine and calling it digital transformation.
Authored by Dr Harilal Bhaskar, Chief Operating Officer, and National Coordinator at I-STEM, under the Office of Principal Scientific Adviser (P.S.A.) Government of India.
