400 MB per minute: how Mistral found a vLLM leak outside the heap
Mistral documented an RSS leak in vLLM disaggregated serving, NIXL and UCX. Heaptrack missed it; pmap, eBPF and targeted GDB found the cause.
- AUTHOR
- Karol Rapacz / CEO of Breachroad · OSCP · PNPT
- PUBLISHED
- 21 January 2026
- READING TIME
- 13 min read
- TOPIC
- AI Security
On 21 January 2026, Mistral published an investigation into a vLLM memory leak in pre-production. With Mistral Medium 3.1, graph compilation and separated prefill/decode phases, decode-process system memory climbed by roughly 400 MB per minute under production-like traffic. OOM followed after several hours.
This was not a universal vLLM defect. It appeared in a particular disaggregated-serving setup using NIXL, which relied on UCX to transfer KV cache across high-speed interconnects. That precise scoping is one of the report’s most useful features.
Why the profiler saw nothing
Memray and Guppy showed no issue. Heaptrack observed balanced malloc and free activity while RSS kept rising. The memory was not disappearing from the conventional heap. pmap exposed growing anonymous mappings, and BPFtrace found direct mmap raw syscalls bypassing glibc hooks.
Automated GDB stopped only at that specific mmap call and captured the complete stack. It led into UCX. For RDMA registration, UCX hooked memory mappings and moved regions into an invalidation queue after munmap rather than releasing them immediately. The queue’s pool grew, while the default UCX_RCACHE_MAX_UNRELEASED=inf did not force cleanup in the observed edge case.
Fix and workaround
The vLLM use case transferred one large contiguous KV-cache region, so disabling mmap hooks removed the leak without measured performance loss in Mistral’s setup. Alternatively, UCX_RCACHE_MAX_UNRELEASED=1024 bounded the queue and triggered cleanup. vLLM merged a fix, while UCX and NIXL teams planned behaviour changes.
Do not copy the environment variable blindly. A registration-cache limit can affect another RDMA workload. Reproduce on a canary, measure RSS and throughput, update dependencies, run a soak test and only then roll out.
Inference observability runbook
Monitor VRAM, RSS, anonymous mappings, file descriptors and cgroup memory separately. A Python-heap alert misses native libraries, drivers and RDMA. Use realistic prompt-length, concurrency and KV-transfer distributions in soak testing.
Record exact vLLM, NIXL, UCX, PyTorch, driver and firmware versions. Without them, “memory sometimes grows” becomes irreproducible. Our OpenTelemetry agent observability guide covers higher-level traces.
This case shows that a “memory leak” may live in an RDMA invalidation queue rather than Python. A production AI review should include long-running tests and native dependency inventory.
Distinguishing leaks, caches and fragmentation
Rising RSS does not always mean unreachable memory. A cache may intentionally retain regions, an allocator may not return them to the kernel, and fragmentation leaves unusable gaps. Stop traffic, hold an idle period and observe whether memory plateaus or falls after controlled cleanup.
Compare VmRSS, RssAnon, RssFile, /proc/<pid>/smaps_rollup and cgroup memory. For GPUs, separate allocated from framework-reserved memory. A stable-workload trend with no recovery after requests end supports a leak hypothesis.
A useful soak test
A short throughput benchmark may miss 400 MB/min growth. Run longer than the expected time to OOM at the real cgroup limit, with ramp-up, steady state, ramp-down and full idle phases. Record memory slope, not only peak.
Change one variable at a time: model, graph compilation, P/D disaggregation, KV-transfer backend and UCX version. The matrix identifies necessary conditions. Canary hardware should match production NUMA, RDMA and kernel.
OOM operational impact
OOMKill can remove a worker mid-response, while retries overload survivors. Readiness should drain a process whose memory slope becomes dangerous. The router must distinguish safe retries from tasks that already produced a side effect.
Why did ordinary APM miss it? APM agents typically observe application runtimes and conventional allocations, not raw syscalls and UCX internal pools. System-level observability was required.
Postmortem lesson
Separate symptom, condition, mechanism and fix. The symptom was linear RSS, the condition a specific NIXL P/D setup, the mechanism a UCX queue, and mitigation changed hooks or limits. This prevents the false conclusion that “vLLM always leaks”. Attach the slope graph, experiment matrix and every dependency version.
Primary source: Mistral AI — Heaps do lie: debugging a memory leak in vLLM.
