AI agent observability with OpenTelemetry and safe logs

AI agent observability should show how a user objective became tool calls and effects. OpenTelemetry defines GenAI conventions for invoke_agent, invoke_workflow, execute_tool and retrieval.

Trace structure

Use the task as root span, with model invocation, retrieval, policy decision and tool call as children. Correlate session, user, agent, prompt version, model and environment.

Never place user identifiers or prompts in span names; this creates high cardinality and leakage.

Content is sensitive

Tool arguments and results contain secrets and PII; OpenTelemetry warns about these GenAI attributes. Collect tool type, status, duration, tokens and decision by default. Capture content only for controlled samples with redaction and retention.

Alerts

Detect new tools, domains, denied-action spikes, unusual scopes, retry loops, cost jumps and approval-rate changes. Correlate with model version and deployment.

Telemetry must resist agent tampering. Emit from middleware and executors, not model-generated text. Feed evidence into the AI incident playbook.

Trace the complete task

Create a parent span for the task, with child spans for model calls, context retrieval, policy decisions and tool execution. Join them with stable task and tenant identifiers rather than prompt content. Attributes can describe provider, model, version, tool, status, latency and usage where this does not expose sensitive data.

OpenTelemetry publishes semantic conventions for generative AI systems, but implementations must account for their current status and version. Use standard fields where available; put organisation-specific attributes in a documented namespace.

Log privacy and integrity

Do not record complete prompts, responses, RAG documents or secret-bearing arguments by default. Prefer classification, length, a digest, dataset identifier and an explicit flag for extended logging. Debug retention should not automatically inherit the audit retention period.

The agent does not self-report success. Infrastructure emits call status, allow or deny decisions, resolved destinations and response codes. Export logs outside the execution environment and restrict access so a compromised agent cannot rewrite them.

Sequence-based detection

A cost metric alone has limited context. A sequence is more valuable: untrusted document retrieval, denied new-domain access, repeated retries and a request for a higher-scope tool. Build alerts on span relationships and correlate behaviour with model, policy and deployment versions.

Test telemetry completeness with a controlled policy violation. The team should reconstruct who started the task, what data was retrieved, which decisions were made and whether any action reached an external system.

Agent SLO metrics

Beyond latency and errors, measure tasks completed without escalation, tool calls per task, denied actions, fallback share and cost. Segment by use case because a research agent and a transaction agent have different risk profiles.

Never optimise one metric alone. Fewer denials may reflect better precision or weaker policy. Higher task completion may come from excess scope. Compare every change with security evaluation results.

Response practice

Prepare saved queries for leakage, tool abuse, retry loops and cross-tenant access. Alerts should link to trace, deployment version and identity-revocation procedure. Restrict detailed-log access to incident responders.

In a tabletop exercise, simulate a poisoned document and locate all tasks that retrieved it and all dependent actions. If telemetry cannot answer quickly, the missing relationship is an architecture requirement, not merely a dashboard issue.

Set a cardinality budget for attributes. User identifiers, full URLs and arbitrary text in metric labels can increase cost and damage query performance. Keep high-cardinality values in traces or logs and build metrics from controlled dimensions.

Document fields required for investigation. A CI or staging test should catch the loss of task identifiers, policy decisions or executor outcomes before telemetry changes reach production.

Sample routine traffic carefully, while retaining complete denial and high-impact events. Responders must know the sampling rule so missing spans are not mistaken for missing activity.

---

Sources: OpenTelemetry GenAI Attributes, OpenTelemetry Semantic Conventions.