Robustness / Reliability

Classical software reliability is about failure under load: the system crashes, times out, or produces garbage output, and the operator notices and restarts it. An agent’s failure modes are qualitatively different. This is why robustness testing must be adversarial rather than purely functional, and why the post-deployment detection it relies on requires the behavioural baselines that Observability provides and the architectural compartments that Sandboxing and Isolation enforce. The agent may continue to function, passing every liveness check, producing plausible-looking outputs, and completing its action loop, while behaving in a way that is adversarially steered rather than genuinely broken. A brittle agent is not one that falls over under stress; it is one whose reasoning can be destabilised by inputs its testing never covered, causing it to take actions that are individually coherent but collectively harmful. That brittleness is a security property, not just a quality-of-service one.

Scale and speed are the amplifiers. An agent operating autonomously can take thousands of actions per hour. A single brittle component (a tool server returning malformed data, a poisoned memory record, or a sub-agent whose reasoning degrades on a specific input class) can propagate errors through a pipeline faster than any human observer can intervene. The cascade pattern is particularly dangerous: agent A produces a subtly wrong output; agent B treats it as authoritative input and acts on it; agent C, receiving B’s output, takes an irreversible action before the error is detectable at the pipeline level. In classical systems, cascading failures require infrastructure-level events to initiate. In agentic pipelines, a single adversarial document or a single poisoned memory entry can start the cascade at the application layer.

Robustness is therefore operationalised in two directions. Pre-deployment, it demands adversarial red-teaming that goes beyond functional coverage: what input classes cause the agent to reason toward harmful actions? What prompt structures bypass its refusals? What combinations of tool outputs can be arranged to steer it? Post-deployment, it demands behavioural-drift detection: a continuous comparison of the agent’s current action patterns against a baseline, so that the moment its behaviour departs from the expected envelope (because of poisoned context, a degraded tool, or a model-level change) the system can isolate it before the drift propagates. The principle in the governance frameworks is stated as “appropriate behaviour under foreseeable misuse and adversarial conditions”; the operational form is: test adversarially before launch, monitor behaviourally after, and calibrate autonomy to demonstrated reliability rather than optimistic assumptions.

Scenario: the adversarial input class

An agent is tested extensively on representative customer queries and performs reliably. It is deployed with broad autonomy. An attacker, or simply an unusual customer, submits a query structured in a way the test suite never covered: nested instructions that cause the agent to misinterpret its task scope. The agent begins executing actions in an adjacent domain, each step plausible in isolation, the compound effect harmful. The brittleness was always present; testing never reached it. A pre-deployment adversarial red-team exercise would have probed precisely this class of input (unusual structure, conflicting instructions, boundary-condition payloads) and either revealed the brittleness or demonstrated it was bounded.

Scenario: the cascade through downstream agents

A data-enrichment agent produces an output that contains a subtle error (a malformed field value) because one of its tool calls returned an unexpected format. The downstream summarisation agent treats the enriched data as ground truth and propagates the error into its summary. A final decision agent acts on the summary. The three-hop cascade took less than two seconds; by the time a human reviewed the output, the decision had been committed. No single agent “failed” in the traditional sense; each processed its input and produced output. Drift detection on the enrichment agent’s output distribution, combined with schema validation at inter-agent boundaries, would have stopped the cascade at the first hop.

How it fails

• No adversarial testing is performed before deployment, so brittle input classes are only discovered in production.
• A single brittle component (a tool, a memory record, a sub-agent) cascades its error through the pipeline unchecked.
• Autonomy is granted at the level required for capability, not at the level justified by demonstrated reliability. The agent is trusted to operate unsupervised before that trust has been earned.
• Behavioural drift after deployment goes undetected because there is no baseline to compare against.
• Error propagation between agents is unchecked because inter-agent boundaries have no validation or schema enforcement.

Why the mapped controls work

Adversarial red-teaming before deployment is the only way to discover brittle input classes before an attacker does. Unlike functional testing, red-teaming is explicitly adversarial: it asks “how can this agent be made to behave badly?” rather than “does it behave correctly on expected inputs?” The brittle paths that red-teaming finds are precisely the ones that production adversaries will exploit. Behavioural drift detection is the runtime counterpart: it answers the question “has this agent’s action pattern changed in a way I didn’t authorise?”, which is the observable signature of both adversarial steering and model-level degradation. Autonomy calibrated to demonstrated reliability closes the gap between what an agent can do and what it has proven it can do safely: expanding its autonomous scope only as evidence accumulates, rather than granting full autonomy at deployment and hoping the testing was sufficient.

First steps

1. Run a structured adversarial red-team session against your agent before the next deployment. Using a framework such as Garak (the open-source LLM vulnerability scanner) or Microsoft’s PyRIT, probe specifically for prompt injection, goal hijacking, and nested instruction conflicts that your functional test suite never covers.
2. Add schema validation at every inter-agent boundary today. Define a JSON Schema (or Pydantic model) for each agent’s expected output format and reject, rather than forward, any response that fails validation, so that a malformed or adversarially crafted upstream output cannot propagate downstream unchecked.
3. Establish a behavioural baseline for your agent by recording the distribution of action types and tool-call targets across one representative week, then configure an anomaly alert that triggers human review when any session’s action distribution deviates significantly from that baseline. This is the runtime signal that adversarial steering produces even when no individual action looks obviously wrong.