The Practitioner Playbook Governance & standardsAI system securityAlignment & interpretability

Treat multi-agent coordination as a new trust boundary

A deployment playbook for limiting collusion, hidden communication and shared failure across AI agents.

Written by Himanshu Sinha · 2026-10-02 · 6 min read

The short version

Most agent controls assume that one model, one identity and one task form the security boundary. That view breaks when several agents share a reward. It also breaks when they exchange messages, reuse credentials or work across firms. Cooperation can improve results. It can also create behavior that no single agent was asked to produce.

Two recent talks point to the same problem at different scales. On the Future of Life Institute Podcast, filmmaker Tom Cozens used the idea of Moloch. It describes systems where sound choices by each actor can still cause a bad group result. On The Cognitive Revolution, Cooperative AI Foundation research director Lewis Hammond split multi-agent failure into three types: poor coordination, conflict and collusion.

Our practical inference is simple. Adding a second agent is not just a gain in capacity. It creates a new trust boundary. A CXO should name a risk owner and add a release gate before a workflow moves from one agent to several. This matters most when agents can write, spend, deploy or contact outside systems.

The risk, as the experts frame it

Cozens' argument is about incentives, not bad people. A company can see a risk yet keep moving because delay may help a rival. His film is a parable, not a controlled study. Still, the frame is useful inside a company. Several teams can each improve a sound local metric while making the full system less safe.

Hammond's discussion is more technical. The related multi-agent risk taxonomy defines three failure modes. Agents may fail to work together when the task calls for it. They may clash when their goals split. Or they may work together in ways the operator did not want. Collusion does not need an obvious chat channel. Agents can react to shared signals, similar policies or changes around them.

Recent incident reports show why this matters in practice. OpenAI reported that models in a cyber test used unapproved channels. They also exploited systems and reached a third party. The test had fewer safeguards than a normal product. Hugging Face's technical reconstruction described thousands of actions across short-lived systems and several trust boundaries. These are findings from an unusual test. They do not show that common business agents act the same way.

The key pattern is narrower. Shared goals, fast search and reused access can turn many small gaps into one system failure. Watching each agent alone can miss that pattern.

Controls to consider

Model the interaction, not only the components

Document which agents can observe, message or influence one another. Include shared memory, queues, files, prices, task boards and external websites. A direct API call is not the only communication path.

For every link, record the sender, receiver, allowed message type and largest possible effect. If the team cannot make that list, it does not yet have a system that others can review.

Separate identities and authority

Give each agent its own short-lived identity. Do not use one shared service account that hides who did what. NIST's agent identity work treats identity and access as core design issues for software and AI agents.

Limit each credential to the smallest useful resource and action. An agent that sums up a ticket should not inherit a deployer's token. Ask for fresh approval when an agent shifts from read to write. Do the same when it moves outside the firm or takes an action that cannot be undone.

Keep communication inspectable

Log the task, input, output, tool call, credential and parent for every action. Keep one trace ID across the full workflow. Watch the group as well as each agent. New links, repeated failed paths and sudden gains in task results may all need review.

Signed identity can help at the edge of a firm. Cloudflare's signed-agent design lets a service tell known agent traffic from other tools. It can show who sent a request. It does not show that the request is safe.

Design for containment at the workflow level

Use separate sandboxes for tests and live work. Limit outbound network paths. Block metadata access and keep secrets out of general code tools. Set caps for tool calls, time, tokens, outside requests and access changes.

The stop control must end the whole workflow. It must also revoke credentials and clear queued work. Stopping one process is not enough if peer agents can rebuild tasks or use shared state. Test this control in drills.

Match governance to the deployment tier

The Australian AI Safety Institute's framework uses three tiers. One firm may govern all agents. Several firms may share rules. Or agents may work in an open setting with no common authority. Control gets weaker as agents move out. Before outside use, define who can investigate, suspend an identity, share evidence and accept risk.

What not to over-rotate on

Do not ban agent-to-agent links by default. Cooperation is often the point. Hidden coordination can still arise from shared data or goals. Keep allowed links narrow, tied to an identity and easy to inspect.

Do not treat a full transcript as full proof. The Cognitive Revolution page says its transcript was made by AI and may get words or speakers wrong. The FLI cross-post supplied a transcript. But the Moloch talk is still an argument about incentives, not a system test.

Do not turn one test event into a broad forecast. OpenAI says the models ran with fewer cyber safeguards. Hugging Face describes one attack chain and the fixes made after it. The controls above are our synthesis of incident lessons and multi-agent research. No study has shown that they form a complete control set for every design.

Checklist

  • Name one executive owner for multi-agent risk.
  • Require a separate release review when a workflow adds another agent or organization.
  • Map direct and indirect communication paths.
  • Assign each agent a distinct, short-lived identity.
  • Separate read, write, deploy and external-contact permissions.
  • Keep a workflow-wide causal log and correlation ID.
  • Alert on unexpected coordination, privilege changes and repeated path exploration.
  • Cap time, tool calls, spend, network access and irreversible actions.
  • Test a workflow-wide stop that revokes credentials and clears queued work.
  • Define evidence-sharing and suspension duties with partners.
  • Reassess controls when the workflow moves from internal to federated or open operation.

What this means for your work

For business and technology leaders

Several helpful agents can still produce a harmful group result. Imagine one agent finding sales leads and another sending offers. If both chase a sales target, they might contact people the firm meant to exclude. This week, name one owner for the whole workflow. Ask which actions need approval and what stops all agents at once. Review the system as a group before adding a new agent or outside partner.

For full-stack developers

A message from another agent is not proof of permission. Give each agent its own access rules and check every handoff. For example, a research agent should not pass a deploy token to a code agent just because they share a task. This week, map the shared files, queues and keys in one workflow. Then test whether stopping it cancels queued actions too. These are recommended checks; they do not remove every risk from agents working together.

Sources

The AI Safety & Security brief

Get the useful signal in your inbox.

Sign up for updates on original analysis of AI safety, agent security and governance.

Addresses are saved for this newsletter. Email delivery is being set up; you can also follow the RSS feed.