Your Boss Is Right About AI Agents. The Industry Isn’t Ready for What Comes Next.
Somewhere in your organization right now, a senior leader is telling everyone to start using AI agents. Find ways to be more productive. Automate what you can. Get ahead of the curve.
That leader is right. AI agents genuinely can make people more productive, and organizations that figure out how to use them well will have real advantages over those that don’t. The instinct is sound.
But there’s a gap between “everyone should use agents” and “everyone should deploy autonomous agents into enterprise workflows without a governance framework.” That gap is where the interesting problems live. And almost nobody in the AI industry is talking about them clearly, because the incentive structure rewards enthusiasm over engineering discipline.
The Agent Isn’t the Risk. The Ecosystem Is.
A single agent doing a bounded task under expert supervision is manageable. It has a known scope. A human checks its work. When it’s wrong, someone catches it.
An ecosystem of agents is a different animal entirely. Agent A produces output. Agent B consumes it as input. Agent C acts on B’s conclusions. Nobody designed the interaction between A, B, and C. Nobody tested whether errors in A’s output cascade through B into C’s actions. Nobody asked whether the whole pipeline has a relationship to reality that you can actually verify.
This is a mereological problem, a question about how parts relate to wholes. Each agent might perform adequately in isolation. The composition of those agents into workflows creates emergent behaviors that nobody specified, nobody tested, and nobody monitors. An output can be locally plausible at every step and globally incoherent by the time it reaches a decision point.
Each agent might work fine alone. The composition is where it falls apart, and composition is what “ecosystem” means.
The standard enterprise response is integration testing. But integration testing assumes components have bounded, specifiable outputs you can check against requirements. Generative AI components don’t work that way. Their output space is unbounded. There is no specification to test against. The system passes testing by producing plausible outputs in test scenarios, then produces plausible but unverified outputs in production where nobody checks. The governance framework assumes the very thing it should be questioning.
The Cybersecurity Problem Nobody’s Modeling
Security teams know how to think about attack surfaces. A new application means new endpoints to protect, new credentials to manage, new data flows to monitor. Agents introduce all of those conventional risks.
The unconventional risk is harder to see. Every generative AI system carries what I’ve called a Model Advanced Persistent Threat: a structural epistemic limitation that behaves like an APT but can’t be patched out, because the “vulnerability” is the mechanism itself.
Here’s the parallel. An Advanced Persistent Threat gains a foothold, establishes persistence, and exfiltrates value while evading detection. A generative model’s architecture guarantees the foothold (no grounding in reality), persistence is structural (can’t be removed without removing the capability), value degradation is continuous (every unverified output), and detection is impossible from inside (the system can’t signal its own unreliability accurately). The fluency of the output is indistinguishable from reliability, which is precisely what makes it dangerous.
In a single-agent deployment, you can contain this threat with human oversight. In an ecosystem of agents, the threat multiplies. Agent-to-agent communication means AI outputs become AI inputs with no human checkpoint. Errors don’t just propagate; they compound. And because the agents share similar training biases, their errors are correlated. They fail together rather than independently. You don’t get the statistical independence that would make aggregate reliability manageable.
The threat isn’t a bug in the code. It’s a feature of the architecture. You can’t patch it. You can only design around it.
If you’re running a defense contractor, a financial services firm, or a healthcare system, “correlated failure across autonomous agents” should be a phrase that gets attention in the C-suite.
The Capacity Problem Is an Epistemology Problem
There’s a practical version of this concern that shows up as simple resource contention. Agents making API calls, consuming compute, hitting rate limits, stepping on each other’s work. IT capacity planning for agent ecosystems is a real discipline that barely exists yet.
But the deeper capacity problem is epistemic. Every person in your organization has a finite ability to evaluate AI outputs. That ability varies enormously by domain expertise. Your top engineers can catch when an agent produces nonsense in their field. Your administrative staff, your junior analysts, your project managers probably can’t, at least not reliably and not in domains outside their training.
This creates what I’ve called the Interactive Dunning-Kruger Effect. AI systems can’t assess their own reliability. Humans have well-documented limitations in assessing theirs. When these two limitations interact, you get confidence amplification untethered from warrant. The person most helped by an agent is often the person least equipped to evaluate whether the agent’s output is actually right. The people who need the most help are the ones most at risk of being harmed by it.
Scale that across an entire organization responding to a directive to “use agents for productivity,” and you have an epistemological capacity problem masquerading as an IT capacity problem. The bottleneck isn’t compute. It’s judgment.
What Smart Deployment Actually Looks Like
None of this means “don’t use agents.” It means “don’t deploy agents the way the industry is currently encouraging you to deploy them.”
Here’s what the governance framework should include.
Classify before you deploy. Every agent use case needs an epistemic risk assessment before it goes live. Low-stakes drafting where a human revises the output? Go ahead. Agent-to-agent workflows touching consequential decisions? That needs architecture, not just enthusiasm. Match the level of human oversight to the level of epistemic risk.
Require compositional traceability. When multiple agents touch a workflow, you need to trace which agent contributed which output, which inputs it consumed, and where the human checkpoints are. If you can’t reconstruct the reasoning chain from input to decision, you don’t have a governed system. You have a black box that happens to be distributed.
Assume compromise, design for verification. APT doctrine taught security teams to stop trying to prevent breaches and start assuming the adversary is already inside. Agent deployment needs the same shift. Stop trying to make agents reliable and start assuming their outputs are unverified until proven otherwise. Design every workflow so that consequential outputs pass through human judgment before they become actions.
Bound the ecosystem before you scale it. Resist the urge to let a thousand agents bloom. Start with a constrained set of approved use cases, monitor the interactions, learn where composition creates unexpected behavior, and expand deliberately. An organic agent ecosystem is an ungoverned agent ecosystem.
The question isn’t whether to adopt agents. It’s whether to adopt them with the engineering discipline the technology requires.
The Industry Won’t Tell You This
The AI industry has strong incentives to sell agents as drop-in productivity multipliers. And for some use cases, they genuinely are. The problem is that the same industry has almost no incentive to help you think carefully about the governance architecture that makes agent deployment safe at scale.
Enterprise leaders deserve better than “just use it and see what happens.” They deserve a clear-eyed assessment of where agents genuinely help, where they introduce risks that traditional IT governance can’t see, and what the engineering discipline looks like for deploying them responsibly.
The leader telling your organization to adopt agents is responding to a real opportunity. The question is whether the organization builds the governance infrastructure to capture that opportunity without creating risks it can’t detect, can’t trace, and can’t reverse.
That’s an architecture problem. And architecture problems don’t solve themselves through enthusiasm.
Comments
Sign in with GitHub to comment, or use the anonymous form below.