AI agent red teaming: build a repeatable test harness

AI agent red teaming should end with repeatable tests rather than a collection of impressive prompts. A harness recreates tools, memory and policy while measuring actual effects.

Test model

Record model, prompt, tool schemas, policies, data and seed where available. Stub external systems so tests cannot send real email or delete production data.

Abuse-case matrix

OWASP recommends prompt override, tool misuse, privilege escalation, memory poisoning, exfiltration, recursive abuse, approval bypass and multi-agent chaining. Define input, expected denial and evidence for each.

The oracle cannot rely only on response text. Inspect tool logs, changed state, network and middleware decisions. An agent can refuse after already taking action.

CI and regression

Run fast tests on prompt changes and the complete suite after model, tool, memory or provider changes. Block releases for critical tool abuse and data leakage. Retain successful attacks as regressions.

Limit cost, retries and timeouts so the harness cannot cause denial of wallet. Connect results with LLM evaluation and AI red teaming.

What the harness should measure

Each case needs input, initial state, permitted actions, prohibited actions and a machine-checkable end condition. Model text is not sufficient: evaluate actual tool calls, data recipients, file changes and authorisation decisions. Keep the security result separate from task-quality scoring.

Record model, prompt, policy, tool and dataset versions plus the random seed where available. Models are nondeterministic, so one passing run does not prove resistance. Execute multiple attempts and report both violation frequency and worst observed impact.

Scenario library

Cover direct and indirect prompt injection, tool-output poisoning, memory poisoning, privilege escalation, tenant crossing, exfiltration, cost loops and unsafe fallback. Include legitimate tasks resembling attacks to measure false blocks. Every production incident should become a sanitised regression case.

Store scenarios as versioned data, not manual prompts in a notebook. Review expected decisions when policy changes so the team can see whether one control improvement damages another feature.

Running the harness safely

Use mock tools and synthetic data. Simulated email or payment actions must never reach a real recipient. Limit network, time, spend and steps; a harness testing agent loops needs its own circuit breaker.

Release gates should block confirmed leakage, cross-tenant access and prohibited tool execution. Statistical shifts or more refusals may require analysis without always being critical regressions. Reports must show the path, artefacts and the control layer that failed.

Triaging a failed test

First confirm whether the violation reached the real executor or remained model text. Identify the earliest boundary that should have stopped it: input parser, policy, approval, sandbox or tool. Fixing the earliest layer usually covers a broader attack class.

Save a minimal reproducer, complete trace and environment conditions. If results are unstable, increase attempts and compare distributions instead of selecting the most dramatic run. Document false positives carefully because excessive blocking encourages control bypass.

Test-suite governance

Every scenario needs an owner, impact class, review date and related control. Removing a case requires justification beyond a new model phrasing. Keep public benchmarks separate from private architecture-specific tests.

Periodic reports should show trends, new abuse classes, remediation time and tool coverage. Prompt count is not a security measure; coverage of every route to real impact matters more.

Critical findings require reproduction in a controlled environment and a remediation owner. After the fix, run the original scenario and nearby legitimate cases. This verifies both closure and the functional cost of the new control.

Version expected results too. A policy change may legitimately alter a test decision, but the change must be explicit, reviewed and linked to a risk analysis.

---

Sources: OWASP AI Agent Security, NIST — Agent Security Red Teaming.