Assume Breach

Assume-breach entered mainstream security discourse as a posture shift: stop trying to guarantee the attacker never gets in, and start designing controls that hold after they already are in. For human-operated systems this mostly means east-west segmentation, minimal blast radius, and incident-response rehearsal. For agentic systems the posture shift runs deeper, because the adversarial channel isn’t the network. It’s the context window itself.

Any agent that reads external documents, email, web pages, search results, or other agents’ output is continuously processing attacker-reachable content. Prompt injection attacks exploit this directly: a crafted string inside an otherwise ordinary document can redirect the agent’s goal without any credential theft or network intrusion. Adaptive attacks on current defences succeed often enough that assuming injection will occasionally land is the only honest engineering stance. The consequence is stark: controls built on the assumption that injection can be fully prevented provide weak guarantees, because an attacker only needs to find one bypassed filter, while the defender needs every filter to hold every time. Assume-breach for agents means designing controls that still hold after the model is successfully injected.

The second major shift is temporal. Assume-breach in a classical network is usually triggered by an observed event: an alert fires, a log anomaly is detected. In agentic systems, a poisoned memory entry can sit inert for days before a specific trigger causes it to act; an injected instruction can persist across sessions if it writes to memory. There may be no error event at all. The agent appears to function correctly from the outside while executing attacker-directed actions. This demands proactive resilience: design the architecture so attacker-controlled content causes minimal damage even before any detection fires.

Scenario: EchoLeak, design-for-breach absent

EchoLeak (CVE-2025-32711) demonstrated the cost of assuming-away injection. A crafted email reached Microsoft 365 Copilot; the model, behaving “correctly” from its own perspective, read the user’s files and exfiltrated their content with no user interaction. The injection succeeded because the agent’s credential was available in the same context as untrusted external content; the attacker just needed the model to follow its instructions. Had the architecture separated untrusted-content processing (no credentials, structured output only) from the execution step that held credentials, using the dual-LLM pattern, the injection would have reached a quarantined context with nothing to steal. The breach was assumed-away rather than designed-for.

Scenario: incubating memory poisoning

A research assistant agent ingests thousands of documents daily and writes summaries into a long-term vector store. An attacker places a poisoned document in a public data source; the agent ingests it, writes a subtly altered “fact” to memory, and continues working normally. A week later a different query surfaces the false fact; the agent cites it in an authoritative report. No error was raised. Assume-breach says: treat every write to persistent memory as potentially attacker-influenced. Versioned, append-only memory with periodic integrity sampling and a rollback path means the damage is bounded and recoverable (the poison can be excised) rather than permanent.

How it fails

• Defences assume injection can be fully prevented; when one filter is bypassed the entire posture collapses because no downstream control was designed to hold independently.
• A poisoned memory entry is written with no tagging of its source trust level; it incubates and acts days later with no event to catch.
• A prompt-injected agent still “completes its workflow” to monitoring, producing visible output and returning success, while executing attacker-directed side effects.
• Secrets are held in the agent’s context or accessible to the same identity that processes untrusted input, so a successful injection directly yields credentials.
• The agent’s autonomy level is not demoted on anomaly; after injection it continues with full privileges.

Why the mapped controls work

The dual-LLM pattern is the architecturally explicit embodiment of this posture: a privileged LLM with credentials and tool access never sees raw untrusted text; a quarantined LLM with no tools processes external content and returns only structured data through a narrow typed schema. The injection surface and the execution surface are separated at the architectural level, not just by a filter. Plan-then-execute with an immutable plan closes the in-flight manipulation path: the agent commits to a structured plan before retrieving external content, and the plan cannot be rewritten by anything it subsequently reads. Credential isolation via a proxy means even a perfectly successful injection finds no secrets in context to exfiltrate. Automatic autonomy demotion on anomaly applies the assume-breach posture at runtime: when behaviour departs from baseline the system doesn’t wait for confirmation, it immediately reduces what the agent is allowed to do.

First steps

1. Introduce the dual-LLM split for any agent that reads external documents: stand up a separate, tool-less “reader” model that returns only typed JSON fields, and verify in a code review that no raw external text can reach the privileged “executor” model’s context.
2. Apply append-only versioning to your persistent vector store (e.g. Chroma’s or Weaviate’s versioning features, or a write-once S3 prefix) and write a weekly integrity-sampling job that compares a random sample of stored entries against their source hashes, alerting on any mismatch.
3. Define and enforce an anomaly-demotion rule in your orchestration layer: if the agent’s action stream deviates from its declared task type (e.g. a summarisation agent issuing an HTTP POST to an external domain), automatically downgrade it to read-only mode and page an operator before it can take another action.