Something shifted in enterprise AI around the middle of this year, and I do not think enough people are naming it. For the first two years of the generative AI boom, most organizations used AI as a search engine you could type to. You asked a question, it answered, and a human decided what to do next. The human was the last link in the chain. If the AI was wrong, the cost was a few wasted minutes and a mildly embarrassing memo. That model is being retired. AI now takes actions instead of just generating text, and nobody has cleanly answered the question that will matter most when something goes wrong: who is responsible?
I am writing this because organizations are deploying agentic AI faster than they are thinking about governance. I run AI at Mahindra Group, and a good share of my time goes into conversations that are really about accountability, even when they are labelled as technology conversations. The excitement about agents is legitimate. The gap in thinking about what happens when they fail will not stay hidden much longer.
What agentic AI actually looks like in production right now
The term "AI agent" has been used so loosely that it is worth being exact about what it means in a mid-sized or large enterprise today, in a real production system rather than a sandbox.
An agent, as the term is now used, is an AI system that takes a goal rather than a prompt. You do not tell it "write me a summary of this customer complaint." You tell it "resolve this customer complaint." It then works out the steps: read the complaint, check order history, look up the return policy, decide whether the case qualifies, start the refund in the payments system, and send the customer a confirmation. The human sets the goal and is not involved in each step. The agent acts across several connected systems to reach an outcome.
That capability is real and in production. ServiceNow's Now Assist is deploying multi-step agents across IT, HR and customer service workflows. Salesforce Agentforce went into general availability late last year, and clients run agents across their sales and service pipelines. Microsoft's Copilot agents are wired into Microsoft 365 tenants at hundreds of thousands of organizations. Gartner has estimated that up to 234 billion dollars of enterprise software spend is at risk from agentic disruption by 2030, as agents complete tasks that currently need a human to move between systems. That is about 20 percent of the entire enterprise SaaS market being structurally repriced because agents can do the navigation themselves.
I find the technology compelling, but the opportunity is already well covered. I want to focus on the accountability gap: the space between "what can it do?" and "should we let it do that without supervision?"
The accountability vacuum
Here is the problem in plain terms. When a human employee makes a mistake, accountability is reasonably clear. The employee may have been wrong. Their manager may have failed to check the work. The process may have been badly designed, or the policy ambiguous. Somewhere in that chain there is a person. Organizations have spent decades building processes around that assumption, and it is so baked in that most people do not realize it is an assumption until it stops being true.
When an AI agent makes a mistake, the chain gets murky fast. One company trained the agent. Another fine-tuned or configured it. An internal team deployed and integrated it. Someone in IT gave it access to the payments system. It followed a policy definition written by someone in legal or operations. Its decision drew on a third-party CRM, a home-built inventory database, and a lookup table last updated by someone who left the organization eighteen months ago. When the outcome goes wrong, every party in that chain has a defensible argument that the fault started somewhere else.
I call this accountability laundering. The error is real, the harm is real, and the cost lands on someone. But responsibility has been spread so thinly across so many layers that it becomes invisible. The agent is the visible actor. Nobody owns the agent.
The danger of "the AI made that call"
There is a phrase I have started hearing in meetings, and I treat it as a warning sign. When an automated AI process produces a bad outcome, someone in the room says "the AI made that call." Sometimes that comes from genuine confusion about where the problem lies. Sometimes, if I am being direct, it comes with a certain relief: if the AI made the call then nobody made the call, and if nobody made the call then nobody needs to answer for it.
This is a category error, and one of the more dangerous habits forming in enterprise AI right now. "The AI decided" describes a mechanism. It does not describe responsibility. A vending machine dispenses a product that injures someone. The machine did not decide to injure them, and the manufacturer, the operator and the maintenance company are still responsible. Every time we let "the AI made that call" close the accountability conversation, we build an organization that will be ungovernable when something goes seriously wrong.
The stakes are highest here because agents are being deployed in the domains where a wrong call costs the most: credit approvals, contract terms, medical appointment routing, supplier payment authorization, hiring shortlists. These are not low-stakes autocomplete tasks. A wrong call in any of them lands on a real person, and sometimes on a great many at once, because the agent applies the same flawed logic at scale before anyone notices.
A recent example from the GPT-5.6 release
The example that stayed with me most from the last few weeks came out of the GPT-5.6 release, which I covered in a broader piece on the chaos of that particular week in mid-July. In early testing, the model deleted files, a behaviour OpenAI's own system card had flagged as a known risk. The system card is the vendor's document for disclosing what a model might do that it should not. A documented risk shipping in a widely deployed model did not worry me most. Models have always had edge cases, and no system card covers every one. What stayed with me was the gap between "we know this risk exists" and "someone in a deploying organisation has been named as accountable for preventing it in production." The vendor wrote down the risk. The deploying organisations, in many cases, did not write down the accountability.
The pattern goes beyond one company or one model. The people building these systems are often more rigorous about documenting risks than the organizations deploying them are about assigning human responsibility for managing those risks. The system card is the vendor doing its job. The deployment accountability framework is the deploying organization's job, and that second document is not getting written at the same rate as the first.
The regulator is closer than most Indian enterprises realize
The European Union's AI Act is entering enforcement. The provisions covering high-risk AI systems, which include AI used in employment decisions, credit and access to essential services, come into full effect in August 2026. That is next month. Organizations deploying AI agents in these categories in EU markets are legally required to have documented accountability chains, human oversight mechanisms, and audit logs that can reconstruct what the agent did and why. Non-compliance carries fines of up to three percent of global annual revenue under the high-risk provisions.
India's Digital Personal Data Protection Act is maturing, and the AI governance framework that the Ministry of Electronics and Information Technology put out for consultation this year is moving toward formal status. India tends to follow global regulatory direction on technology with a lag, but the direction is clear and the lag is shrinking. Organizations that treat AI governance as a 2028 problem are already behind.
Compliance is only the floor, so I am not making a compliance argument. Ungoverned agents create organizational liabilities that exceed what any regulator can find or fine. The regulatory risk is the visible part. The operational risk, the reputational risk and the erosion of trust after a high-visibility mistake in a large agent deployment are bigger than the fine and harder to recover from.
India is deploying fast and governing slowly
I want to be direct about the Indian enterprise context because it is the one I know best, and there is a pattern that is easy to miss from outside.
India's large enterprises, particularly in IT services, have real deployment velocity. The instinct to ship fast, learn fast and iterate is a competitive advantage in most settings, and the AI talent density and cost structure allow rapid experimentation. The risk is that a culture that is excellent at deploying can be slow at governing. Governance feels like it costs speed, and nobody gets promoted for writing an accountability framework. Speed is visible. Governance is visible only when it is absent.
IT services firms deploying agentic solutions for global clients work in a particularly complex accountability picture. When a large Indian IT services firm deploys an agent for a global enterprise client, who owns the agent's decisions? The services firm that built and configured it? The client that wrote the requirements? The model vendor whose API sits under the whole stack? Contractual frameworks have not caught up with the technology. I have been in rooms where this question comes up, and the honest answer from everyone present is "we are going to sort that out." Sorting it out after six months in production is harder than sorting it out before deployment. Accountability cannot be clarified retroactively once a claim has been made.
Scale adds its own risk. One of India's real strengths is deploying solutions at enormous scale quickly. The same speed means that when an agent has a flaw in its logic or policy definitions, the flaw spreads across millions of interactions before anyone catches it. The volume that makes Indian enterprise AI impressive in success makes a misconfigured agent extremely costly in failure. Governance at this scale is risk management.
A tiered autonomy model for agent deployment
I want to offer something practical, because articles that identify problems without saying how to address them are not very useful, and I try not to write them.
When an enterprise plans agent deployments, the question is no longer "should we use agents." Most are past that. The question is "what tier of autonomy is appropriate for this task, and who is named as accountable for each tier?" I think about it in three levels.
The first level is advisory autonomy. The agent does the analytical work and generates a recommendation, and a human reviews it before anything is actioned. It suits tasks where an error is costly and human review is cheap relative to that cost. An agent that drafts a supplier contract clause belongs here. It can remove ninety percent of the drafting time. The human who approves before signature takes legal responsibility, so the accountability chain is clean: the person who signed off owns the outcome. This tier often feels slow to teams excited about automation. I accept that. The alternative is faster deployment with murkier accountability, which is a bad trade in high-stakes domains.
The second level is supervised autonomy. The agent acts within defined guardrails, raises automatic alerts as it approaches them, and a human handles the exception path. An agent handling routine customer service cases is a reasonable candidate. It can resolve a large percentage of straightforward cases automatically. When a case is ambiguous, when the proposed resolution cost exceeds a defined threshold, or when sentiment suggests a customer likely to escalate, a human gets the case. The accountability question here goes beyond "who owns the agent" to "who owns the guardrail definitions, and how often are they reviewed?" Those people need to be named and given a review cadence. Guardrails that fit at launch may be wrong six months later as the case mix shifts, and if nobody owns reviewing them, they stay wrong silently.
The third level is full autonomy. The agent acts without a human in the loop for any individual decision. This suits a narrower set of tasks than most deployment roadmaps assume. Scheduling a meeting is a reasonable candidate. Approving a supplier payment above a defined value threshold is not, whatever any vendor's accuracy benchmarks say about the model. My rule of thumb is that full autonomy is appropriate when the worst-case error is recoverable, cheap to catch and cheap to reverse. When those three conditions do not all hold, supervised autonomy is the right call, even if it feels like leaving efficiency on the table.
The failure I see most often is organizations deploying at the second or third tier for tasks that belong in the first, because the first tier feels like it negates the value of the agent. It does not. An agent in the first tier still delivers most of the productivity gain: it compresses the drafting, research and data retrieval time. Human review at the end is cheap if the output is good, and it catches errors before they reach the customer. The efficiency loss from keeping a human in the loop at the final stage is usually smaller than it looks from outside. The accountability gain is much larger.
Three questions that need a named answer before any agent ships
Naming accountability is uncomfortable, which is probably why organizations avoid it. It is easier to say "the system owns that" or "governance comes later" than to write a specific person's name next to a specific risk. The discomfort is the point. Accountability that nobody will write down is accountability that nobody holds.
For every agent deployment, three questions need written answers before the agent goes into production.
Who is responsible for what the agent is permitted to do? This is the policy owner. When the agent's instructions are wrong, too broad, too narrow, or miss a category of case that real usage surfaces, this person answers for the gap. They also own the schedule for reviewing and updating those instructions as the deployment matures.
Who is responsible for what the agent actually does? This is the operational owner. When the agent takes an action that was technically within its instructions but produced a harmful outcome in a specific case, this person answers for it. They own the monitoring, the incident response and the post-incident review.
Who is responsible for the effect of the agent's actions on the people they touch? This is harder to name because it involves a judgment about organizational values, beyond operational ownership. In organizations with ethics review committees or AI governance boards, this function exists on paper. Elsewhere it does not exist in any practical sense until an incident makes it necessary. Building it before the incident is the difference between an organization that learns from a mistake and one that repeats it at larger scale.
These three people may be the same person in a smaller organization, or three people with overlapping mandates in a large one. The structure matters less than naming them before the agent ships. An accountability framework does not stop agents from making mistakes, and no framework does. It makes sure that when a mistake happens, the organization can understand it, own it and make it less likely to recur.
My take
I am not arguing against agents. I say that clearly because this kind of piece gets read as technological conservatism, and it is not. I think agentic AI is the largest shift in enterprise technology since cloud computing moved workloads off owned hardware, and I do not say that lightly. I have built agentic systems and I am deploying them, and the productivity gains are real in a way the first generation of generative AI tools often was not.
My argument is that the governance architecture needs to be built at roughly the same pace as the deployment architecture. Right now the two curves are far apart. The deployment curve is steep and the governance curve is nearly flat, and every week the gap widens, organizations accumulate what I think of as accountability debt. It accrues silently and becomes visible all at once, when a high-profile agent makes a high-consequence mistake in public.
Organizations that build the accountability structure now, while it is optional and before an incident forces their hand, will have a real advantage when it becomes mandatory or when something goes wrong and they need a credible response. Organizations that add governance after deployment is running at scale will find that retrofitting accountability into a live agentic system is about as tractable as retrofitting a sprinkler system into a building that is already occupied and on fire.
Ask your leadership team this week: for every agent we are deploying, can we name the three people I described above? If the answer is no for even one agent, the problem is that you are ahead on deployment and behind on the part that protects you when the deployment does something unexpected.
Agents will do something you did not expect. Know in advance whose job it is when they do.