OpenSSL HollowByte: 11 Bytes Can Exhaust Server RAM

HollowByte is an unusual denial-of-service weakness in OpenSSL. Okta Red Team researchers demonstrated that an unauthenticated client can trigger a disproportionately large memory allocation with only 11 bytes of prepared TLS traffic. The client needs no valid certificate or account and does not have to complete the handshake. The important lesson is not the byte count alone but cost asymmetry: an attacker sends almost nothing while the server reserves memory and holds state for many unfinished connections.

Okta published the HollowByte research on 16 July 2026, and the issue reached wider industry alerts on 20 July. OpenSSL treated the change as resilience hardening rather than a conventional security vulnerability, so it did not receive a CVE or a standard advisory. The absence of a CVE does not remove operational risk. Administrators should assess the real exposure of their TLS terminators, the library build in use, connection limits, and allocator behavior.

What the researchers confirmed

The issue occurs at the beginning of TLS negotiation. A record carries a four-byte ClientHello header whose declared message length can be much greater than the data delivered in the initial packet. OpenSSL allocated a buffer corresponding to that declared length at an early stage, before receiving the full message. The connection stayed open waiting for the remaining data while the memory remained committed.

In the documented variant, an 11-byte record could lead one connection to allocate approximately 131 KB. That multiplier looks modest for a single session, but thousands of parallel slow connections change the calculation. The technique does not upload a large file or execute code. It exploits the gap between a cheap client request and more expensive state maintained by the service.

In a laboratory environment capped at 1 GB, the process was OOM-killed when roughly 547 MB remained in frozen, fragmented memory regions. In a 16 GB test constrained by the maximum connection count, the process held around one quarter of total system memory. These are research-environment results, not guaranteed outcomes for every deployment. The actual effect depends on the server, descriptor limits, timeouts, operating system, allocator, and how the application uses OpenSSL.

Why memory did not immediately return to the OS

Heap fragmentation is the second part of HollowByte. Even after some connections close, freed small regions do not necessarily form a contiguous block that glibc can return to the operating system. The process can regard memory as reusable while its resident set size remains high. From the perspective of a container, cgroup, or host limit, the resource is still occupied.

That behavior explains why shortening a timeout may not immediately restore the service to a safe state. A timeout reduces how long a new connection can remain pending, but it does not undo fragmentation or change the early allocation pattern. Restarting a process may temporarily release RAM, yet an unpatched library exposes the replacement instance to the same asymmetry.

HollowByte is therefore a useful example of why monitoring must cover more than memory percentage. Teams also need the number of open handshakes, new-connection rate, time spent waiting for the rest of a message, worker counts, allocation errors, and OOM-triggered restarts.

The fix makes the buffer grow with received data

OpenSSL changed the logic so the buffer is not immediately expanded to the full size declared by an untrusted client. Data is read and buffered incrementally. The server-side cost therefore follows the number of bytes actually received more closely.

The changes are visible in public pull requests #30792, #30793, and #30794. The mechanism shipped in OpenSSL 4.0.1, 3.6.3, 3.5.7, 3.4.6, and 3.0.21. A Linux distribution retaining an older base version may have backported the patch. In that case, the version string alone cannot settle the assessment; check the package changelog and vendor bulletin.

Teams should not mechanically search for a CVE either. The OpenSSL project classified this change as hardening. A scanner that reports only CVEs may not show it even when uncorrected code is deployed. This is a strong reason to connect vulnerability management with upstream monitoring and component inventory.

Which systems may be exposed

OpenSSL is a library, not a single server product. It can be present in a reverse proxy, API gateway, VPN, message broker, mail server, agent, embedded application, or custom service. However, finding the package on disk does not prove exposure. A managed load balancer may terminate public TLS while the application uses another library. A container may include OpenSSL only as an administrative tool.

An exposure review should answer four questions:

Which process accepts untrusted TLS connections?
Which library and build does that process actually load at runtime?
Has the vendor backported the correction without changing the main version number?
Which memory, connection, and timeout limits protect that process?

This is the same visibility problem addressed by an SBOM and deployed-component inventory. A container image, host, sidecar, and network appliance can carry different copies of the library. Updating one layer does not establish that production traffic now reaches corrected code.

An update plan based on evidence

Begin with a list of every public and internal TLS terminator. Include CDNs, WAFs, ingress gateways, reverse proxies, load balancers, application servers, email, VPNs, and appliances. Obtain written confirmation from vendors for managed services. For self-built images, inspect installed packages and the library mapped into the running process rather than relying solely on the command-line client’s openssl version output.

Assign a business and technical owner next. Internet-facing services without effective connection limits, small containers, and workloads sharing a node with critical systems deserve priority. Test the update in a canary first. The fix touches handshake handling, so check client compatibility, mTLS, session resumption, TLS inspection, and performance.

After deployment, prove the new artifact is actually active. Compare the image digest, process library map, package revision, and build identifier. Observation of canary traffic and stability should precede a wider rollout. Our TLS 1.3, mTLS, and PKI security audit guide describes the surrounding controls.

Temporary controls do not replace the correction

When an update requires a maintenance window, teams can shorten idle handshakes, lower per-source connection limits, impose a global connection budget, and isolate worker pools. A reverse proxy placed in front of the service may absorb part of the asymmetry, provided it does not use the same vulnerable allocation pattern. Resource isolation should also prevent one container from exhausting the memory of an entire node.

No single limit is a complete correction. Sources can be distributed, while aggressive rate limiting may block legitimate users behind carrier NAT. A WAF inspecting HTTP may never see the problem because traffic stalls before an HTTP request exists. Lessons from the NoName DDoS campaign help with volume and connection anomalies but do not remove the allocation behavior.

Detecting suspected exploitation

There is no single reliable indicator of compromise. Behavioral correlation is more useful:

a rapid increase in unfinished TLS handshakes;
high process RSS despite a small volume of received data;
many sessions waiting for the remaining ClientHello bytes;
growing fragmentation and no proportional RSS reduction after sessions close;
worker restarts, OOMKilled events, or allocation failures;
repeated tiny records that start a handshake but send nothing further;
simultaneous latency and availability degradation for workloads on the same host.

Network telemetry alone may show only thousands of small connections. Process metrics alone may resemble a memory leak. Joining both views in a SIEM, EDR, or XDR workflow helps distinguish a protocol anomaly from ordinary demand growth.

Do not reproduce the attack against production. Validate the correction in an isolated lab with a strict memory limit and allocator observation, using a test approved by the system owner. A public proof of concept is not necessary to confirm a package build and corrected behavior.

Responding to a suspected incident

If a process rapidly consumes RAM, protect availability first. Isolate the affected pool, direct traffic to healthy instances, and constrain new handshakes. Preserve traffic samples, connection metrics, kernel logs, cgroup events, the library build, and timeout configuration. A restart may be operationally necessary, but collect the minimum evidence first whenever service conditions permit.

Then determine whether the event was deliberate traffic, a defective client, a monitoring test, or an unintended connection storm. HollowByte does not by itself steal data. Its confirmed impact is resource consumption and potential unavailability. Do not expand an incident statement to code execution or server compromise without separate evidence.

The procedure belongs in the first 72 hours of incident response: who shifts traffic, who approves emergency limits, where a packet capture is stored, how the runtime build is established, and when customers are informed. Memory pressure at the edge can create a cascade, so exercising this scenario before an incident has practical value.

What HollowByte teaches about risk management

This case exposes the gap between conventional CVE scanning and practical resilience. A meaningful correction can have no CVE, CVSS score, or entry in a popular feed. A patch process limited to critical identifiers from an automated scanner can miss it entirely.

A mature process monitors upstream projects, distribution bulletins, and artifacts running in production. It combines that evidence with resilience tests, resource budgets, and secure design. During web application penetration testing, the TLS layer should be assessed not only for ciphers and certificates but also for the behavior of the connection terminator under controlled pressure.

HollowByte is not evidence that every application containing OpenSSL is vulnerable, and it is not a reason for panic. It is a reason for a specific verification: inspect the loaded library, confirm any backport, update exposed terminators, monitor incomplete handshakes, and test resource limits. That is a stronger defense than waiting for a CVE number.

Sources and recommendation

This article is based on the Okta Red Team technical publication and the public OpenSSL changes in PR 30792, PR 30793, and PR 30794. The memory figures above are the researchers’ laboratory results, not a universal forecast.

If you cannot identify which OpenSSL copy terminates traffic in your organization, begin with inventory and configuration testing. Contact Breach Road to verify public TLS endpoints, runtime dependencies, resource limits, and a safe update path.