OpenAI and Cerebras: 750 MW shows latency is AI's next battleground

On 14 January 2026, OpenAI announced a partnership with Cerebras to add 750 MW of low-latency AI compute to its platform. Capacity is expected to come online in stages through 2028. OpenAI framed it as part of an inference portfolio matching different systems to different workloads.

This is a capacity plan, not a claim that 750 MW operated on announcement day. Electrical capacity does not convert directly into users, tokens or cost; the outcome depends on model, utilisation, cooling, reliability and whole-data-centre efficiency.

Why wafer-scale architecture can lower latency

Cerebras places substantial compute, memory and bandwidth on one very large chip. In a conventional cluster, model layers span accelerators and sequential token generation waits for communication. Fewer chip boundaries can reduce transfer time and raise long-output throughput. That is an architectural argument; actual benefit must be measured end to end, not only as laboratory tokens/s.

Users experience time to first token, inter-token speed and total task time including tools. An agent may generate text quickly yet wait on a database, browser or API. Fast inference therefore matters most for interactive coding, voice, multi-step agents and long answers, not every batch workload.

A hardware portfolio, not one winner

OpenAI explicitly described a portfolio. Training, large-prompt prefill, decode and embeddings have different memory and communication profiles. Hardware excellent at sequential decode may be less economical for short, highly batched requests. An infrastructure router can select GPUs, custom silicon or wafer-scale systems by latency, price and availability.

API buyers should care more about SLOs than chip branding:

• p50 and p95 time to first token,
• p95 inter-token latency with real prompts,
• stability under load and limits,
• cost per completed task rather than per million tokens,
• processing region and failover path.

Faster responses can increase agent action count and risk. Tool limits, step budgets and tracing must scale with throughput. Our AI agent security guide covers the governance layer.

The OpenAI–Cerebras deal shows competition moving from “can the model do it?” to “can it do it fast and predictably?”. Evaluate latency, task cost and tool control on one realistic workflow.

Building a fair latency benchmark

Fix the model, temperature, prompt length and expected response length. Warm up, then measure at different times and production-like concurrency. Report percentiles rather than the fastest run. Separate network, queue, first-token and decode time.

Streaming improves perceived speed without shortening the whole task. Speculative decoding can raise throughput, but benefit depends on draft-model agreement. Compare equivalent output quality; a shorter or worse answer is not an equal speed-up.

Reliability and backend routing

An accelerator portfolio needs capability-aware routing. A model or quantised variant may behave differently across backends. Fallback must preserve limits, data policy and tool schema. If failure moves traffic to a slower system, uncontrolled retries can form a queue cascade.

Use task deadlines, bounded retries with jitter and load shedding. For agents, goal-completion time matters more than one call. Trace how many steps and tokens occurred before failure and ensure retry does not repeat a side effect.

Energy as part of the SLO

The 750 MW figure describes planned infrastructure capacity, not per-answer consumption. Evaluate completed tasks per kWh, facility PUE and accelerator utilisation. Idle reserve improves latency and resilience while reducing utilisation.

Do more tokens per second always improve UX? No. Short answers depend on first-token latency; agents depend on tool time and step count. Optimise the user’s metric.

Architecture decision

Do not couple an application to one backend promise. The access layer should switch region or inference class, preserve evaluations and compare cost. Define maximum task time, queue mode and degraded response for critical workflows. A fast model without controlled fallback can be less available than a slower predictable system.

---

Primary source: OpenAI — OpenAI partners with Cerebras.