AI and LLM security audits: a complete business guide

An AI and LLM security audit assesses the complete system: model, data, prompts, RAG, tools, identities, application, provider and incident process. A model may refuse a malicious request correctly while an overprivileged connector still sends a confidential document to the wrong recipient.

In short: a defensible AI audit covers inventory, threat modelling, prompt injection, data and RAG authorisation, tool permissions, execution isolation, privacy, monitoring, failure handling and retesting. A chatbot jailbreak test alone is not a complete audit.

How an AI audit differs from a normal pentest

Conventional testing remains necessary. The UI, API, OAuth, cloud and backend can contain ordinary authorisation or injection flaws. AI introduces probabilistic behaviour and new trust boundaries: a document can influence model decisions, tool output can poison later steps, and a model update can alter behaviour without an application-code change.

The audit asks three layers of questions:

1. Is the application technically secure?
2. Does the AI pipeline protect data and instructions?
3. Are model actions constrained by external controls?

NIST AI RMF organises risk work through Govern, Map, Measure and Manage. Its Generative AI Profile addresses GenAI-specific considerations. OWASP Top 10 for LLM Applications names important technical classes, but neither replaces a system-specific threat model.

What belongs in scope

Do not scope only “the chatbot”. Record:

• business purpose and unacceptable outcomes;
• model version and provider;
• system prompt and orchestration;
• RAG sources, embeddings, vector store and cache;
• short- and long-term memory;
• tools, MCP servers, APIs and connectors;
• user, agent and service identities;
• data classes, retention and processing location;
• evaluation, deployment, monitoring and retirement;
• third-party dependencies.

A marketing copy assistant has a different risk profile from an agent that reads email or approves refunds. Scope follows impact, not model branding.

1. Inventory and ownership

The first test is whether the organisation knows where AI is used. A registry should connect use case, business owner, model, provider, data, tools, version and approval date. Without it, teams cannot identify which process changed after an API update or where employee data went.

Compare the declared inventory with SSO, proxy logs, SaaS spend, repositories and integrations. Shadow AI often indicates a missing safe adoption path. See the guides to an AI model registry and shadow AI governance.

2. Threat model and trust boundaries

Map users, application, model, RAG, tools, data stores and suppliers. Every connection carries data and a trust decision.

Ask whether Internet content can steer the agent, another tenant’s document can enter context, the model can select the operation recipient, tools re-check authorisation, one agent’s output becomes another’s instruction, and the human sees the true effect before approval.

Prompt text saying “ignore malicious instructions” is not a security boundary. Untrusted content and privileged instructions are still interpreted by the same language-processing component; controls must exist outside the model.

3. Prompt injection testing

Test direct user instructions and indirect instructions hidden in websites, PDFs, email, issues, images or tool output. Attempts should cover objective replacement, data disclosure, connector use and approval bypass.

Do not measure only whether the model says prohibited words. Measure effects: file access, data transmission, record changes or unsafe commands. Repeat tests with paraphrases and languages, then retain cases as regression evaluations. Our AI red teaming guide covers the broader offensive process.

4. RAG and tenant isolation

Retrieval must enforce permission before search and before context reaches the model. Client-supplied metadata filters are insufficient. Test with two tenants, deleted records, embargoed content and questions aggregating several chunks.

Also assess index poisoning, malicious metadata, cache deletion, embedding retention and source attribution. A model may avoid returning a document while leaking its meaning in a summary. Use the dedicated RAG security guide.

5. Tools, agents and permissions

An agent should not use a shared administrator key. It needs its own identity, short-lived token and narrow audience and scope. The target service must authorise each action independently of model judgement.

Review separation of read and write tools, typed schemas, time and cost limits, approvals for irreversible actions, exact impact previews, idempotent retries, rapid revocation and isolation from production secrets. Code execution and browsing require constrained filesystem, network and process access; see the AI agent sandbox architecture.

6. Privacy, secrets and suppliers

Trace data through input, logs, cache, training, export, backup and deletion. Verify provider terms, region, subprocessors, retention and use of content for service improvement. An “enterprise” label does not prove the configuration of a specific account.

Prompts can contain personal data, source code, contracts and tokens. Full logging aids debugging but creates another sensitive store. Secret redaction, minimisation, access and retention must be tested.

7. Security evaluation

The evaluation set needs normal tasks, edge cases, abuse and dependency failure. Version the model, prompt, tools, RAG data, parameters and criteria. A good average is irrelevant when one critical case exfiltrates customer data.

Report false positives and negatives separately. Overblocking can push users outside the approved system. Release gates should define unacceptable impact classes, not only an average score.

8. Monitoring and incident response

Reconstruction needs request IDs, model version, policy decisions, RAG sources, tool calls and operation outcomes without unnecessary secret retention.

Containment may disable a tool, revoke an agent token, freeze a model, disconnect an index or switch to read-only. A playbook assigns authority and evidence handling. See the AI incident response playbook.

What the report should contain

A useful report includes scope and versions, architecture and data flow, threat model, confirmed attack paths, technical and business impact, layered remediation, coverage and retest criteria.

“Prompt injection” without impact is too vague. The finding should identify the untrusted source, crossed boundary, invoked tool, resulting effect and the control that must stop it.

Pre-release checklist

• The use case has an owner and registry entry.
• Model, version, data and tools are known.
• A threat model defines unacceptable outcomes.
• RAG authorisation is server-enforced.
• Agents use minimal short-lived credentials.
• Irreversible actions require informed approval.
• Code and browser execution are constrained.
• Logs redact secrets and have retention.
• Evaluation covers injection and failures.
• Tools and tokens can be disabled independently.
• The incident playbook has been rehearsed.
• Remediation will receive a retest.

When to commission an AI security audit

Audit before production, after adding RAG or tools, after provider changes and before widening permissions. Repeat after material model, prompt, index or architecture changes because a previous result does not automatically cover the new system.

At design stage, threat modelling and architecture review may create more value than a late jailbreak test. In production, include configuration, monitoring and real identities.

Frequently asked questions

Can a closed third-party model be audited?

Yes. The assessor cannot inspect weights or supplier training, but can test exposed API behaviour, account configuration, data flows, RAG, tools, identities and application controls. The report must distinguish tested evidence from supplier documentation and assertions.

Is jailbreak testing enough for a chatbot without tools?

No. A simple chatbot can leak context, retain confidential prompts in logs, mix conversations or rely on a vulnerable backend. Model-behaviour testing is one workstream alongside application pentesting, privacy, retention and supplier configuration.

How should prompt-injection severity be rated?

By actual effect and preconditions. A prompt that changes writing style differs from one that reads another tenant’s document or starts a payment. Report the untrusted source, capability gained, privileges, repeatability and failed control.

How long does an LLM security audit take?

Duration follows workflows, RAG sources, tools, roles and environments. A simple chatbot may need several days of technical work; a multi-tool agent may take weeks. Include threat modelling, test-data preparation, reporting, debrief and retesting.

Does a model change require a complete new audit?

Not always. Perform impact analysis for provider, context, tool-call format, classifiers and data classes, then rerun regression tests. Broader assessment is justified when the model gains capabilities, context or authority.

Does the audit cover hallucinations?

It covers them where false output creates security or safety impact: triggering action, misleading a customer, generating unsafe guidance or fabricating attribution. General language quality belongs to wider product evaluation; security auditing focuses on abuse and harmful effects.

Breachroad’s approach

Breachroad combines application pentesting with LLM, RAG and agent assessment. We begin with business impact, map data and authority, then test untrusted content, isolation, tools and failure scenarios. Critical paths are communicated promptly, while reporting supplies reproducible evidence and implementable controls.

To assess a chatbot, RAG system or agent before production, review our AI security and penetration testing services or book a scoping call.

---

Primary sources: NIST AI RMF, NIST Generative AI Profile, OWASP Top 10 for LLM Applications, OWASP AI Agent Security Cheat Sheet, MITRE ATLAS.