AI ROI Without Illusions: OpenAI's New Scorecard

OpenAI published A scorecard for the AI age today, 17 July 2026. Its central argument is straightforward: purchased seats, active users and token consumption do not show whether AI creates value. The company proposes “Useful Intelligence per Dollar”—useful work obtained from each unit of budget spent.

That is a valuable change in perspective, but it is not a ready-made accounting standard or independent technical standard. The article is a model vendor’s proposal. To support management decisions, the concept must be translated into measurable tasks, quality criteria, full cost and risk. This guide shows how to build a scorecard that does not reward AI consumption itself or hide the cost of corrections, oversight and security failures.

Four questions in today’s OpenAI scorecard

OpenAI suggests that business and finance leaders ask:

1. Is AI completing work that matters?
2. What does each successful task actually cost?
3. Can people depend on the result?
4. Does each AI dollar produce more value as usage grows?

“Successful” is the critical word. A cheaper token does not necessarily create a cheaper outcome. A model may need more attempts, a longer prompt, specialist input and manual review. A more expensive model may complete the task on the first pass. The economic unit should therefore be an outcome that meets a defined quality threshold, not an API call.

Why common adoption metrics mislead

AI deployment dashboards often emphasise users, conversations, tokens and minutes reportedly saved in a survey. Those figures describe activity, not value. One million tokens may produce a thousand useful answers or a thousand documents that need to be rewritten completely.

The problem has several causes:

• adoption is not an outcome — a user may open the tool but discard its answer;
• API spend is only one cost — integration, observability, evaluation, controls and corrections remain;
• an average hides critical tasks — strong summarisation does not compensate for a wrong financial decision;
• self-reported savings are inflated — users compare against intuition rather than a defensible baseline;
• volume may reward waste — more retries and tokens look like increased usage.

An AI programme should start with a catalogue of processes and units of work. An AI model and system registry helps record owner, purpose, data, vendor, version and risk tier.

The unit of measurement: a completed task

A task needs to be small enough for its outcome to be assessed, yet complete enough to represent business value. For customer support, it might be a case response accepted by both agent and customer. For a developer, it may be a pull request that passes tests and code review. For a SOC analyst, it could be alert enrichment containing correct entities, sources and a defensible recommendation.

Each task should define:

Element	Example definition
input	ticket, document, alert or requirement
expected outcome	structured answer or completed action
quality threshold	correctness, completeness, format and sources
risk limit	data that must not be disclosed and actions requiring approval
time limit	result required in five minutes or by end of day
assessment	automation, expert, recipient or a combination

Without this definition, “success” becomes subjective. The team selects examples on which the model performs best, while the cost of failed attempts disappears from the report.

Full cost per successful task

The most practical part of OpenAI’s proposal is moving beyond token price. A minimum cost model should include:

cost of success = models and infrastructure + user effort + review + rework + evaluation + security + allocated fixed costs

Direct costs include model calls, embeddings, retrieval, tools, data storage and orchestration. Human costs include data preparation, prompting, verification, escalation and repeated work. Platform costs include the gateway, logging, regression tests, secret management, DLP, IAM and integration maintenance.

If a system makes 10,000 attempts in a month but only 6,000 outcomes meet the quality bar, the denominator cannot be 10,000. Divide cost by the 6,000 successful tasks and report failure cost separately. Otherwise, a model producing many cheap but wrong answers will appear better than a system that attempts less work but delivers reliable outcomes.

Three outcome classes, not one score

OpenAI recommends tracking whether an outcome is ready to use, needs correction or requires escalation. That is a useful simple starting point:

• ready to use — the outcome meets criteria without a substantive correction;
• needs correction — a person can repair it quickly without repeating the process;
• needs escalation — the output is wrong, unsafe or beyond the automation boundary.

The classes need process-specific definitions. A minor style change should not count as a factual failure. Conversely, removing a fabricated citation, incorrect amount or disclosed personal record is not a “cosmetic correction”. Record the reason too: missing data, hallucination, incorrect tool use, policy breach, timeout, integration failure or ambiguous instruction.

Our guide to LLM evaluation metrics and deployment gates explains how to build test sets and technical gates. The management scorecard should consume the same evidence, not create a separate and more permissive version of reality.

AI ROI must account for risk

OpenAI says organisations should define which data AI can access, which systems it can use or change, and when human approval is required before scaling. This is not an economic afterthought. Risk has measurable cost.

An agent that makes an incorrect payment, configuration change or code publication can erase all earlier savings. A scorecard should therefore include at least:

• count and severity of data-policy violations;
• percentage of high-risk actions stopped by an approval gate;
• unauthorised tool calls and permission errors;
• prompt-injection and poisoned-data resilience results;
• time to detect, contain and recover from an incorrect action;
• log completeness sufficient to reconstruct an agent decision.

NIST’s voluntary AI Risk Management Framework organises risk management around Govern, Map, Measure and Manage. The NIST Generative AI Profile extends that approach with GenAI-specific risks and actions across the lifecycle. OpenAI’s economic proposal should be embedded in a broader risk-management system rather than treated as a standalone control.

Example scorecard for one process

Area	Metric	Why it matters
value	tasks meeting the quality threshold	shows actual delivery
quality	ready / correction / escalation	exposes dependence on people
cost	full cost per successful task	enables valid comparison
time	median and p95 completion time	reveals the long tail
risk	incidents and policy breaches per 1,000 tasks	connects scale and safety
reliability	timeout and integration-error rate	measures the system layer
control	test and log coverage	shows whether outcomes are defensible
scale economics	cost of success month by month	tests the improvement claim

Do not collapse every field into a single “AI score” without showing its components. Risk weight depends on the process, and one aggregate number can hide a critical regression behind a favourable cost result.

Comparing models and architectures fairly

Test the full system: prompt, retrieval, tools, safeguards, model and review. Comparing foundation models on a public benchmark does not answer a question about your workflow. Use a frozen task set, the same definition of success and the full cost.

A useful evaluation compares at least:

1. the process without AI as a baseline;
2. a small, inexpensive model without routing;
3. a stronger model for every task;
4. an AI model router that escalates difficult cases;
5. mandatory review for high-risk actions.

Report confidence intervals and sample sizes. A difference of several percentage points on a dozen examples does not justify migration. A test containing only easy tasks will not predict behaviour in rare, expensive cases.

A 30, 60 and 90-day implementation plan

First 30 days: definition and baseline

Choose one process with sufficient volume. Define the task, owner, quality bar and prohibited outcomes. Measure the no-AI variant: time, cost, errors and escalations. Complete the secure enterprise AI deployment controls and inventory Shadow AI using the discovery and governance guide.

Days 31–60: instrumentation and controlled test

Add task identifiers, prompt version, model, costs, latency, tools used, review decision and failure reason. Run a constrained pilot on traffic that can be reviewed safely. Do not let the system execute irreversible actions autonomously. Instrumentation can follow the patterns in our OpenTelemetry guide for AI agents.

Days 61–90: scale decision

Compare full cost of success with the baseline, inspect the quality distribution and investigate incidents. Increase traffic only after the gates pass. Define automatic rollback when cost, quality or security regress. If value rises only because employees perform invisible review after hours, the pilot has not demonstrated ROI.

Common AI scorecard traps

The first is choosing a convenient denominator. Cost per user looks attractive when many accounts exist, even if few tasks succeed. The second is omitting post-deployment labour: manual correction and escalation handling disappear into another department’s budget. The third is changing the test with the model, allowing every release to pass its own easier exam.

The fourth trap is optimising the metric rather than the objective. If ticket closures are rewarded, a system may write short replies and mark cases resolved prematurely. Outcome quality from the recipient’s perspective and reopen rates are needed. The fifth trap is omitting low-frequency, high-impact incidents.

What today’s publication changes for organisations

The most useful part of OpenAI’s proposal is not a new slogan. It moves attention from purchasing and using a tool to work delivered. CFOs, CISOs and process owners can finally use the same denominator: a correctly completed task with known cost and risk.

Treat “Useful Intelligence per Dollar” as a measurement hypothesis. The organisation must define “useful”, decide which errors are unacceptable and assign the full cost. Only then does the chart describe system economics instead of adoption marketing.

Want to establish whether an enterprise AI agent produces repeatable outcomes without hidden risk? Book a BreachRoad AI security assessment—we will build evaluation scenarios, test permission boundaries, logging and abuse resistance, then connect the findings to a measurable process scorecard.