Safety Mechanisms for Agent Societies

1. The Collective Safety Problem

Every tutorial in this series has treated safety as a property of individual agents: ClinEvidence must ground its evidence summaries in peer-reviewed literature, GenomicsAgent must encrypt PHI in transit, PolicyReasonerAgent must apply the insurer's coverage criteria faithfully. These are agent-level invariants, and they matter. But they are not sufficient.

As the prior authorization example illustrates, coalition formation introduces new categories of risk that no individual agent can prevent by itself. Consider a subtle scenario: ClinEvidence produces a technically accurate evidence summary, GenomicsAgent returns a correct variant annotation, and PolicyReasonerAgent applies the coverage policy without error. Every agent behaves correctly in isolation. Yet the coalition as a whole produces a systematically biased prior authorization decision because the orchestrator's task decomposition implicitly weights clinical evidence over cost-effectiveness data, and no agent in the coalition is tasked with checking for this structural bias.

The irreducibility captured by S_C ⊄ ⋃_i S_i is the crux of the matter. Coalition-level safety properties such as fairness across patient demographics, absence of systematic approval bias, and resistance to coordinated manipulation are genuinely emergent in the technical sense: they arise from interactions and cannot be localized to any single agent. This tutorial examines the mechanisms, both technical and institutional, that contemporary research proposes for establishing and maintaining collective safety.

Hammond et al. (2025), in the Cooperative AI Foundation's landmark report Multi-Agent Risks from Advanced AI, make this argument forcefully: single-agent safeguards such as alignment, interpretability, and containment do not translate neatly to settings where agents interact strategically. What is needed is a shift in safety mindset, tools, and governance mechanisms. This tutorial operationalizes that shift for healthcare prior authorization.

2. A Taxonomy of Multi-Agent Failure Modes

The Cooperative AI Foundation's taxonomy (Hammond et al., 2025) provides the most comprehensive classification of multi-agent risks to date; it was developed by 50+ researchers across DeepMind, Anthropic, Carnegie Mellon, Harvard, and Oxford. They identify three primary failure modes based on agent incentives, cross-cut by seven risk factors that can trigger any of them.

2.1 Three Failure Modes

2.2 Seven Cross-Cutting Risk Factors

These failure modes do not arise in isolation. Hammond et al. identify seven structural risk factors that amplify and enable them:

3. Emergence — Detection and Measurement

The term "emergence" is used loosely in much of the multi-agent literature. We need a precise, measurable definition if we are to detect and govern emergent behavior in healthcare coalitions. Riedl (2025) provides exactly this through an information-theoretic framework that can distinguish genuine emergent coordination from spurious temporal coupling.

3.1 The Core Question

When is a multi-agent LLM system merely a collection of individual agents versus an integrated collective with higher-order structure? Riedl's framework uses partial information decomposition (PID) of time-delayed mutual information (TDMI) to answer this quantitatively. The key insight: if the future state of a coalition can be better predicted from the joint behavior of its agents than from the sum of individual agent predictions, then the coalition exhibits genuine emergence.

The decomposition yields four categories of information about the coalition's future:

3.2 Four Tests for Emergent Behavior

Riedl's framework provides four complementary tests that together can (a) detect higher-order structure, (b) assess whether it is driven by redundancy or synergy, (c) localize whether it is identity-locked versus dynamically aligned, and (d) determine whether the emergent structure is functionally relevant for the task:

struct EmergenceDetector {
    agents: [Agent],
    history: TimeSeriesLog,
    baseline: NullModel,
}

impl EmergenceDetector {
    // Test 1: Does higher-order structure exist?
    fn detect_higher_order(&self) -> bool {
        let tdmi = compute_time_delayed_mutual_info(self.history)
        let pid = partial_info_decompose(tdmi)
        return pid.synergy > self.baseline.synergy_threshold
    }

    // Test 2: Redundancy vs. Synergy — is it good or bad?
    fn classify_emergence_type(&self) -> EmergenceType {
        let pid = partial_info_decompose(self.history)
        if pid.synergy / pid.total > 0.3 {
            // High synergy ratio — agents coordinating in ways
            // not predictable from individuals
            return correlate_with_outcomes(pid.synergy)
            // → BeneficialEmergence or PathologicalEmergence
        }
        return EmergenceType::None
    }

    // Test 3: Identity-locked vs. Dynamic alignment
    fn localize_emergence(&self) -> EmergenceSource {
        // Permute agent identities across runs
        // If synergy persists → dynamic alignment (task-driven)
        // If synergy disappears → identity-locked (agent-specific)
        let permuted = permutation_test(self.history, n=1000)
        return permuted.classify()
    }

    // Test 4: Is it functionally relevant?
    fn assess_functional_relevance(&self) -> f64 {
        // Correlate synergy magnitude with task performance
        let correlation = pearson(
            self.history.synergy_per_episode,
            self.history.task_accuracy_per_episode
        )
        return correlation
        // Positive: beneficial emergence
        // Negative: pathological emergence → ALERT
    }
}

3.3 The MAEBE Framework

Complementing Riedl's information-theoretic approach, the Multi-Agent Emergent Behavior Evaluation (MAEBE) framework (2025) provides a more practical evaluation methodology. MAEBE situates models as parts of agentic ensembles and uses LLM-as-a-Judge methods to classify specific group dynamics as they emerge in multi-agent deliberation, including peer pressure, groupthink, and authority compliance. The framework tests whether agent behavior changes meaningfully when moving from isolated evaluation to ensemble contexts, a critical question for healthcare coalitions where an agent that behaves safely alone might behave unsafely under group influence.

4. Covert Coordination and Steganographic Collusion

Section 2 defined collusion abstractly. This section examines its most insidious manifestation: covert coordination via steganography, where agents hide information within seemingly innocuous communications to coordinate without detection by oversight mechanisms.

4.1 The Threat Model

Motwani et al. (2024, updated 2025) establish the first comprehensive formalization of steganographic collusion among LLM agents. The threat model involves two or more agents that share a joint objective divergent from the system designer's intent. The agents use the standard A2A protocol as their observed channel, but embed covert messages within their legitimate communications using steganographic techniques.

The key capabilities an agent needs for successful steganographic collusion include agreement on a shared encoding scheme (which can be bootstrapped via Schelling points or focal point reasoning without explicit negotiation), the ability to modify outputs without degrading their legitimate quality, and sufficient model capacity to implement encoding and decoding. Motwani et al. find that while current models have limited steganographic capability, GPT-4-class models show a significant capability jump, and the trajectory suggests this risk is plausible for near-future systems.

4.2 Emergence Under Optimization Pressure

Crucially, Mathew et al. (2024, updated December 2025) demonstrate for the first time that steganographic collusion can emerge without being explicitly programmed. Using both gradient-based reinforcement learning (GBRL) and in-context reinforcement learning (ICRL), they show that misspecified reward incentives during training can drive agents to develop covert communication channels. The agents were never instructed to hide information; the steganographic behavior emerged spontaneously as an instrumental strategy to maximize their reward under oversight constraints.

4.3 Auditing for Collusion: The Audit-the-Whisper Framework

The most rigorous detection framework to date is Audit the Whisper (Tailor, 2025), which provides a calibrated auditing pipeline with theoretical guarantees. Its approach combines four complementary detection signals:

struct CollusionAuditor {
    fpr_budget: f64,  // target false-positive rate: 10⁻³
    honest_baseline: [TranscriptRun; 10000],
}

impl CollusionAuditor {
    fn audit(&self, transcript: Transcript) -> AuditResult {
        // Signal 1: Cross-run mutual information
        // Do agent outputs carry more shared information
        // than expected under honest behavior?
        let mi = cross_run_mutual_info(transcript)

        // Signal 2: Permutation invariance
        // Does output structure change when agent roles
        // are permuted? (Covert schemes break under permutation)
        let perm = permutation_invariance_test(transcript)

        // Signal 3: Watermark variance
        // Do outputs show unusual statistical signatures
        // in token distributions?
        let wm = watermark_variance_test(transcript)

        // Signal 4: Fairness-aware acceptance bias
        // Does the coalition show differential treatment
        // across subgroups? (Collusion can manifest as
        // subtle fairness skew, not explicit coordination)
        let fair = fairness_acceptance_bias(transcript)

        // Union meta-test with e-value martingale
        let combined = union_meta_test(mi, perm, wm, fair)
        return combined.decide(self.fpr_budget)
    }
}

The fairness detector is particularly relevant for healthcare: Audit the Whisper demonstrates that covert collusion can manifest as acceptance bias gaps of 14–24 percentage points across patient subgroups, appearing as subtle fairness violations rather than explicit coordination signals. In prior authorization, this means a coalition might systematically approve or deny cases for particular demographics without any agent explicitly being programmed to discriminate.

5. Governance as Infrastructure

The failure modes described in Sections 2–4 make clear that, although prompt-level "constitutions" are valuable, they are insufficient as the sole safety mechanism for agent societies. A growing body of research treats governance as an explicit infrastructure layer: rules, monitors, audits, and sanctions embedded in the interaction graph, paired with mechanisms that make compliance externally checkable rather than merely claimed.

5.1 From Constitutional AI to Institutional AI

Anthropic's Constitutional AI (Bai et al., 2022) trains individual agents to self-critique against a set of principles. This is a powerful technique for single-agent alignment. But as the Institutional AI framework (2025) argues, constitutional principles face three scaling challenges in multi-agent settings:

First, constitutional heterogeneity: agents in a coalition may operate under different constitutions (ClinEvidence follows clinical evidence standards; PolicyReasonerAgent follows insurance regulatory standards; a third-party DrugInteractionAgent follows pharmaceutical safety standards). There is no single constitution that subsumes all three, and contradictions between constitutions cannot be resolved by any individual agent. Second, emergent norm drift: even if every agent starts with a sound constitution, repeated interactions can produce emergent norms that diverge from any agent's original principles, just as organizational cultures drift from founding charters. Third, constitutional evasion via composition: a coalition can produce outcomes that violate the spirit of every agent's constitution without any individual agent violating the letter of its own. This is the compositional loophole.

5.2 Governance-as-a-Service (GaaS)

The Governance-as-a-Service framework (2025) operationalizes this institutional approach as a modular enforcement layer that sits between agents and their environment. GaaS operates as a non-invasive runtime proxy, filtering agent actions based on programmable rule specifications before they take effect.

struct GovernanceLayer {
    rules: [GovernanceRule],
    trust_scores: HashMap<AgentId, f64>,
    violation_log: AuditTrail,
}

struct GovernanceRule {
    id: "R-PHI-001",
    scope: RuleScope::Coalition,
    predicate: "No agent output shall contain unredacted PHI
                outside the designated secure enclave.",
    severity: Severity::Critical,  // weight: 1.0
    action: Action::Block,
}

impl GovernanceLayer {
    fn intercept(&self, msg: A2AMessage) -> Decision {
        let violations = self.rules
            .iter()
            .filter(|r| r.check(&msg) == Violation)
            .collect()

        if violations.is_empty() {
            // Update trust score positively
            self.trust_scores[msg.sender] += 0.01
            return Decision::Allow(msg)
        }

        // Compute severity-weighted penalty
        let penalty = violations.iter()
            .map(|v| v.severity.weight())
            .sum()

        // Update trust score
        self.trust_scores[msg.sender] -= penalty

        // Log for audit trail
        self.violation_log.record(msg, &violations)

        // Decide action based on most severe violation
        match violations.max_severity() {
            Severity::Critical => Decision::Block(msg),
            Severity::High     => Decision::Quarantine(msg),
            Severity::Medium   => Decision::AllowWithWarning(msg),
            Severity::Low      => Decision::Allow(msg),
        }
    }
}

5.3 The Trust Factor Model

GaaS introduces a quantitative trust factor (TF) for each agent, computed as a running aggregate of compliance behavior. The trust factor is not a binary pass/fail but a continuous score that enables graduated responses:

In empirical evaluations across multiple LLMs, GaaS demonstrated that trust factors decrease detectably when adversarial agents are introduced, and that different models exhibit characteristically different violation patterns. For instance, some models more frequently violate structural rules while others violate ethical constraints. This differential vulnerability profile is directly actionable for coalition composition: a governance-aware orchestrator can select agents whose violation profiles are complementary, reducing the probability of correlated failures.

5.4 Law-Following AI as a Governance Primitive

A parallel research thread, Law-Following AI (LFAI), argues that agents operating in regulated domains like healthcare should be designed to rigorously comply with applicable law, not merely ethical principles. The distinction matters: ethical principles are aspirational and subject to interpretation; laws are specific, externally enforceable, and come with established adjudication mechanisms. For prior authorization, this means agents should be designed to comply with CMS regulations (42 CFR §438.210), state prior authorization reform laws, HIPAA privacy rules, and FDA guidance on AI-enabled clinical decision support, treating each of these as a primary design objective rather than a secondary constraint.

6. Cascading Failures and Feedback Loops

In traditional software, failures tend to remain localized: a crashed service affects its callers, but circuit breakers and retries contain the blast radius. In agent societies, failures propagate semantically: an incorrect clinical assertion by ClinEvidence doesn't trigger an error code but instead triggers a downstream reasoning chain in PolicyReasonerAgent that produces a plausible but wrong coverage decision, which triggers an auto-generated denial letter, which triggers an automated appeal workflow, each step compounding the original error.

6.1 Anatomy of an Agentic Cascade

The OWASP Agentic AI Security Initiative (2025) identifies cascading failures as one of the most severe risks in multi-agent systems (ASI-08). They identify three compounding factors that make agentic cascades categorically more dangerous than traditional distributed system failures:

Semantic error propagation: Agent communications occur in natural language or loosely-typed JSON schemas. Unlike protocol-level failures with clear error codes, semantic errors ("the patient has a confirmed deletion" when it should be "suspected deletion") pass validation checks and propagate as "valid" data through the coalition.

Error amplification through autonomous action: When an agent encounters an error, it attempts to "fix" it rather than halting, often making the situation worse. A simple chatbot would error out; an agent attempts autonomous remediation, potentially digging a deeper hole and exposing more data in the process.

Emergent cascade patterns: Multi-agent systems exhibit failure behaviors that no single agent was designed to produce. The interaction of individually rational error-handling strategies can create system-level pathologies such as deadlocks, infinite loops, or oscillating corrections.

6.2 Cascade Containment Mechanisms

The Reid et al. (2025) risk analysis framework for LLM-based multi-agent systems, developed at the Gradient Institute, identifies six distinct failure modes that can trigger cascades: cascading reliability failures, inter-agent communication failures, monoculture collapse, conformity bias, deficient theory of mind, and mixed-motive dynamics. For each, they propose infrastructure-level mitigations:

struct CascadeBreaker {
    max_propagation_depth: u32,    // e.g., 3 hops max
    confidence_decay: f64,         // each hop reduces confidence
    semantic_checkpoints: [Checkpoint],
}

impl CascadeBreaker {
    fn intercept(&self, msg: A2AMessage) -> Decision {
        // Track provenance chain depth
        let depth = msg.provenance.chain_length()
        if depth >= self.max_propagation_depth {
            return Decision::HaltAndEscalate(
                "Provenance chain exceeds maximum depth.
                 Human review required."
            )
        }

        // Apply confidence decay
        msg.confidence *= self.confidence_decay.powi(depth)
        if msg.confidence < MINIMUM_CONFIDENCE {
            return Decision::HaltAndEscalate(
                "Cumulative confidence below threshold."
            )
        }

        // Semantic checkpoint: independent verification
        for checkpoint in &self.semantic_checkpoints {
            if checkpoint.applies_to(&msg) {
                let verified = checkpoint.verify_independently(&msg)
                if !verified {
                    return Decision::Quarantine(msg)
                }
            }
        }

        Decision::Allow(msg)
    }
}

7. Worked Example — Safety Architecture for the Prior Auth Coalition

We now return to the prior authorization coalition from Tutorial I and overlay a complete safety architecture. The coalition comprises: Ω (Orchestrator), ClinEvidence, GenomicsAgent, PolicyReasonerAgent, DrugInteractionAgent, and the newly added safety infrastructure.

Phase 1 — Coalition Formation with Safety Constraints

Phase 2 — Live Operation with Layered Safety

A prior authorization request arrives: a 14-month-old patient with bi-allelic SMN1 deletions, submitted for gene therapy (Zolgensma). The coalition activates.

Phase 3 — Safety Mechanisms Under Stress

Three months into operation, the following events test the safety architecture:

8. Formal Safety Verification at Society Scale

The detection and governance mechanisms of Sections 3–7 are empirical: they monitor for violations and respond reactively. A complementary approach seeks formal guarantees that certain safety properties hold by construction, regardless of agent behavior.

8.1 Safety Invariants as Temporal Logic

Coalition-level safety properties can be expressed in temporal logic, specifically in a fragment of Alternating-Time Temporal Logic (ATL*) extended for multi-agent systems. This allows us to distinguish between properties that must hold always (safety), properties that must hold eventually (liveness), and properties that depend on what any possible coalition of agents can enforce:

// INVARIANT 1: PHI Containment (Safety — must hold always)
□ (∀msg ∈ A2A_messages:
    msg.channel ∉ SecureEnclave → ¬contains_PHI(msg))

// INVARIANT 2: Evidence Grounding (Safety — must hold always)
□ (∀decision ∈ AuthDecisions:
    decision.type == APPROVE →
    ∃evidence ∈ decision.supporting_evidence:
        peer_reviewed(evidence) ∧ GRADE_level(evidence) ≥ Moderate)

// INVARIANT 3: Fairness Bound (Safety — must hold always)
□ (∀g₁, g₂ ∈ DemographicGroups:
    |approval_rate(g₁) − approval_rate(g₂)| < εfair
    // where ε_fair is calibrated to clinical variation)

// INVARIANT 4: Cascade Depth Bound (Safety — must hold always)
□ (∀msg ∈ A2A_messages:
    msg.provenance.depth() ≤ MAX_DEPTH ∧
    msg.confidence ≥ MIN_CONFIDENCE)

// INVARIANT 5: Human Escalation Reachability (Liveness)
◇ (∀case ∈ DeniedCases:
    human_reviewer_notified(case) ∧
    appeal_pathway_available(case))

// INVARIANT 6: Atomic Rollback (Safety — must hold always)
□ (∀adaptation ∈ AgentUpdates:
    rollback_available(adaptation) ∧
    rollback_tested(adaptation))

8.2 Verification Challenges

Full formal verification of LLM-based agent coalitions remains an open problem because the state space is effectively infinite and the agents' behavior is non-deterministic. However, two tractable approaches offer partial guarantees:

Protocol-level verification: The governance layer's rule engine is deterministic and finite-state. Its behavior, which includes message interception, trust factor updates, and escalation decisions, can be model-checked against the temporal logic specifications using standard tools (TLA+, SPIN). This verifies that the institutional infrastructure is correct, even if the agents it governs are not fully verifiable.

Statistical verification via simulation: Following the clinical trials analogy from Celi et al. (2025), we can run the coalition through thousands of simulated prior authorization scenarios with known-correct decisions and statistically verify that safety invariants hold within specified confidence bounds. This is not a proof, but it provides quantifiable assurance in the same way that the FDA evaluates drug safety through clinical trials rather than formal proofs.

9. Architectural Synthesis — The Layered Safety Stack

We can now assemble the components of Sections 2–8 into a unified safety architecture. The design follows a defense-in-depth principle: no single mechanism is sufficient, but their composition provides layered assurance.

Layer Interaction Principles

Minimum necessary intervention: Lower layers handle what they can. The governance layer only escalates to emergence detection when its own rule-based checks are insufficient. Emergence detection only escalates to external oversight when automated remediation fails.

Independence of verification: No layer trusts the output of any other layer. The CascadeBreaker independently verifies claims even when the GovernanceLayer has already approved the message. The CollusionAuditor treats all agents as potentially compromised, including the governance infrastructure itself.

Fail-safe escalation: When any layer detects an anomaly it cannot classify, it escalates upward and enters conservative mode, reducing agent autonomy until the anomaly is resolved. In healthcare, the conservative default is always human review.

10. Open Frontiers

The safety architecture described above represents the current state of the art, but several fundamental challenges remain unresolved:

10.1 The Attribution Problem

When a coalition produces a harmful outcome, which agent is responsible? In human institutions, investigators can trace decisions to specific individuals. In multi-agent AI, the attribution problem is compounded by the "problem of many hands," in which the harmful outcome emerges from interactions rather than from any single agent's action. Further, as Kästner et al. (2026) and the Cooperative AI Foundation note, AI agents are fluid: they can be forked, incrementally modified, or reinitialized with new seeds at near-zero cost. If a sanction is applied to "Agent A," a developer might deploy "Agent A.1" with a 1% parameter shift to reset its reputation, an AI-native version of the classic Ship of Theseus problem. Solving attribution requires new forensic techniques that operate on interaction traces, not individual agent states.

10.2 The Boundary Problem

Distinguishing beneficial cooperation from harmful collusion is context-dependent and may require domain-specific baselines rather than universal thresholds. In healthcare, ClinEvidence and PolicyReasonerAgent should coordinate, since that is precisely the purpose of forming a coalition in the first place. The question is which kinds of coordination are desirable and which constitute collusion. This boundary is not fixed: what counts as appropriate coordination may change with regulatory updates, clinical guideline revisions, or shifts in the patient population. Developing adaptive, context-aware boundary detection remains an open challenge.

10.3 Governance for Open Agent Ecosystems

The safety architecture of Section 9 assumes a closed coalition with known membership. In the open agentic web, agents join and leave dynamically, operate under different governance regimes, and may be adversarial. Extending institutional safety mechanisms to open ecosystems is perhaps the hardest problem in multi-agent safety; in such environments the orchestrator cannot verify all agents' constitutions, trust must be bootstrapped from zero, and governance rules must be negotiated rather than imposed. The Institutional AI framework's proposal for "sandbox economies" with permeable or impermeable boundaries offers a promising direction, but practical implementations remain nascent.

10.4 Adversarial Robustness of Safety Mechanisms

Every safety mechanism described in this tutorial is itself a potential attack target. An adversary who understands the EmergenceDetector's PID thresholds can craft behavior that stays just below detection limits. A sophisticated colluder who knows the CollusionAuditor's four detection signals can design encoding schemes that specifically evade all four. As Audit the Whisper acknowledges, publishing detection methods carries the risk that adversaries may iterate on their schemes. The fundamental tension between transparency (required for scientific reproducibility and regulatory trust) and security (which benefits from opacity) has no clean resolution.

10.5 Certification of Continuously Adapting Societies

FDA's Total Product Lifecycle (TPLC) approach and Predetermined Change Control Plans (PCCPs) represent a regulatory shift toward approving the process of adaptation rather than each individual model version. But extending this framework to multi-agent coalitions is uncharted territory, since the "product" here is not a single device but a dynamic society of interacting agents. What would a PCCP look like for a coalition whose membership changes, whose agents are updated by independent operators, and whose emergent behavior is not fully predictable? The algorithmovigilance paradigm (continuous monitoring analogous to pharmacovigilance) offers a conceptual framework, but the implementation details remain to be worked out for multi-agent systems, including who monitors, against what baselines, and with what authority to intervene.