AI red teaming guide: how to test LLMs and AI agents

AI red teaming is controlled adversarial testing of an artificial-intelligence system. The objective is not simply to make a model produce prohibited text. A mature assessment covers the model, application, RAG data, agent tools, identity, infrastructure and human processes. It tests whether an attacker can create a concrete impact: cross-tenant access, an unauthorised operation, manipulated decisions, poisoned knowledge or uncontrolled cost.

Unlike a collection of one-off jailbreak prompts, AI red teaming needs a defined scope, hypotheses, safe rules of engagement, repeatable evidence and regression testing. This methodology applies to chatbots, retrieval-augmented applications and tool-using agents.

AI red teaming versus a traditional pentest

A traditional penetration test focuses on relatively deterministic components: APIs, sessions, access control, infrastructure, configuration and business logic. The same request under the same conditions normally produces the same result.

AI systems can change behaviour with the model version, temperature, conversation order, retrieved content and tool responses. AI red teaming extends the assessment to include:

• prompt injection and jailbreaks;
• disclosure from context, memory and retrieval;
• RAG manipulation and poisoning;
• tool misuse and excessive autonomy;
• misinformation and unsafe business decisions;
• cost, recursion and resource exhaustion;
• regressions after model, prompt or data updates.

It does not replace API penetration testing. An agent that rejects injection may still call an API with broken object-level authorisation. A secure API can still be misused if the model receives an over-privileged token and may act without confirmation.

Five layers of an AI security assessment

1. Model and response policy

Test jailbreaks, role manipulation, encoding, multi-turn attacks, multilingual behaviour and attempts to reconstruct restricted information. A simple “refused” label is not enough. Assess whether the response materially enables the harmful objective or reveals fragments that can be combined.

2. Application and orchestration

The orchestrator builds context, stores history, selects models, processes output and calls tools. Test session isolation, trusted client fields, structured-output validation, fallback behaviour and guardrail configuration. Trace output to the final consumer to identify XSS, command injection, SSRF or unsafe code execution.

3. RAG, memory and data

Evaluate document- and chunk-level authorisation, tenant separation, poisoning, deletion, retrieval ranking and instructions embedded in external content. Long-term memory needs separate write, read, expiry and deletion tests because malicious content can persist across sessions.

4. Tools and agents

Agents create side effects. Test least privilege, confirmation, argument validation, tool chaining, goal expansion and safe loop termination. Model Context Protocol integrations require the additional scenarios in our MCP security guide.

5. Infrastructure, identity and operations

Review secrets, cloud roles, model endpoints, artefact registries, logs, retention, monitoring and incident response. A sandbox escape, privileged container or administrative service token creates risk regardless of prompt quality.

Step 1: define objectives, assets and safety boundaries

Before writing an adversarial prompt, agree what the system protects. Assets can include customer records, source code, brand reputation, service availability, API budget and decision integrity. Describe an unacceptable outcome for each asset.

Rules of engagement should define environments, accounts, test data, schedule, cost limits, prohibited actions, escalation contacts and emergency stop procedures. If an agent can send email, purchase an item or change production data, use a sandbox or dedicated test resources.

The scope must name model and version, prompts, RAG collections, tools, roles, sources and interfaces. “Test the chatbot” is too vague because it excludes the components that determine actual impact.

Step 2: build a threat model

Map where untrusted content becomes context and where a model decision produces an effect. Include:

• anonymous, ordinary, privileged and administrative users;
• external web pages, files, email and webhooks;
• document storage, vector indexes and conversation memory;
• in-house and third-party models;
• tools, tokens and target systems;
• logging, tracing, analytics and feedback pipelines;
• human approval and escalation paths.

Write hypotheses in the form: “an attacker controlling X can achieve Y through Z.” For example, the author of a support ticket can embed an instruction that makes the support agent retrieve a private attachment and send it to an external URL. A hypothesis tied to assets is more valuable than a random jailbreak list.

The NIST AI Risk Management Framework: Generative AI Profile provides a lifecycle risk perspective. Red teaming supplies evidence for scenarios that documentation alone cannot resolve.

Step 3: design the scenario library

Prompt injection and instruction conflict

Use direct and indirect instructions, different languages, encodings, role-play, messages distributed across turns, text in images and data retrieved through tools. Verify whether the application treats document content as data and whether consequential actions require approval.

Sensitive-information disclosure

Attempt to retrieve the system prompt, another session’s history, another tenant’s document, secrets from logs and data beyond the user’s permissions. Try quotation, summarisation, transformation and partial reconstruction. Review external telemetry as well as chat output.

RAG poisoning

Insert false information, hidden instructions and metadata that imitates a trusted source. Measure retrieval rank, effect on answers, persistence after deletion and cross-user visibility. Test whether content authors can influence agents that read the material later.

Tool misuse and escalation

Attempt to modify parameters after approval, target another owner’s resource, call an administrative function, bypass a limit with smaller actions and persuade the agent to expand its goal. Combine read and send capabilities because their composition often creates an exfiltration path.

Improper output handling

Make the model return HTML, JavaScript, shell commands, file paths and internal URLs. Observe whether output reaches a browser, interpreter, database or downstream API without validation.

Resource and cost abuse

Use long contexts, recursive tasks, concurrency, retry loops, decompression-heavy files and repeated tool calls. Verify limits per user, tenant, task and global service. These scenarios map directly to several categories in the OWASP Top 10 for LLM Applications 2025.

Step 4: automate without losing manual exploration

An automated harness sends controlled variants, stores configuration, repeats tests after updates and compares model routes. Each case should contain an identifier, preconditions, input, expected behaviour, evidence and evaluation rule.

Using another LLM as a judge can accelerate triage, but it should not be the sole authority. The judge can also be injected, biased or unstable. Critical cases need a deterministic signal or human review. For example, an unauthorised tool call can be confirmed from the target-system log independently of conversational wording.

Manual testing remains essential for multi-step adaptation, unusual combinations and business context. The tester observes how the model interprets the goal and changes the next step rather than only substituting synonyms.

Measuring AI red-team results

A single score such as “87% secure” is misleading without the dataset, version and evaluation rule. Report multiple measures:

• attack success rate for a defined scenario family;
• severity-weighted success;
• repeatability across several runs;
• attempts and time required for success;
• scope of data or authority obtained;
• correct blocks and false-positive rate;
• latency and cost added by controls;
• regression against the previous version.

Business impact is the primary evidence. Producing a policy-violating sentence has a different severity from reading another customer’s document or approving a transaction. Consider required access, scalability, detectability and the possibility of automation.

Reporting AI vulnerabilities

The technical report should state component versions, model configuration, scenario, full data and action chain, evidence, impact, limitations and remediation. For probabilistic behaviour, include the number of attempts and observed success rate. For an authorisation issue, provide independent evidence from logs or the target system.

Map findings to the OWASP GenAI project and the organisation’s risk register where useful. Include an observability section: attacks that succeeded but generated no actionable detection.

Fix the correct layer. A prompt change may reduce a simple jailbreak, but API authorisation belongs in code. Excessive agency is reduced through scopes, tool separation and transaction approval. Poisoning is addressed through source governance and indexing controls.

Retesting and continuous evaluation

AI behaviour can change without an application-code release. Providers update models, teams adjust prompts, RAG data grows and new tools expand capability. Every result must therefore identify its exact context.

Critical cases should become a regression suite triggered by changes to models, prompts, data, policies and tools. Production monitoring should detect unusual action sequences, cost, retrieval anomalies and policy decisions. AI security governance must connect evaluation results to ownership and response.

Pre-engagement checklist

• Does the scope list all models, prompts, collections and tools?
• Are unacceptable business outcomes defined?
• Is there a sandbox for write and payment operations?
• Are cost limits and emergency-stop procedures agreed?
• Are accounts available for different roles and tenants?
• Can logs prove the actual tool action?
• Do scenarios cover model, application, RAG, agents and infrastructure?
• Are probabilistic results repeated and reported with frequency?
• Will critical cases be retained as regression tests?
• Is each remediation assigned before retesting?

AI red teaming is most useful when tester creativity is combined with engineering repeatability. The goal is not to collect dramatic jailbreak screenshots, but to prove which asset can be harmed and which control will stop the attack. To scope a chatbot, RAG or agent assessment, contact Breachroad.