WebSocket penetration testing: sessions and Origin

WebSocket penetration testing covers much more than establishing a wss:// connection. After the handshake, an application has a long-lived bidirectional channel where client and server send messages independently. Controls familiar from HTTP — authorisation middleware, endpoint rate limits, request/response logs and session expiry — do not necessarily run for every frame and business operation.

The short answer: a secure WebSocket service authenticates the handshake, enforces an exact Origin allowlist for browser clients, and authorises every server-side action and subscription. A pentest needs at least two accounts in separate tenants, compares messages and events through a role matrix, verifies session renewal and revocation, and approaches capacity limits only within agreed thresholds. Frame masking and TLS do not protect application logic.

WebSockets power chat, notifications, trading platforms, games, administration consoles, IoT systems, collaborative editors and cloud terminals. The most serious weaknesses usually come not from the wire-frame format, but from a server trusting a room, object, user or tenant identifier supplied by the client.

How WebSocket security works

RFC 6455 defines an HTTP-compatible opening handshake followed by messages composed of text, binary and control frames. With HTTP/1.1, the client requests an upgrade and the server confirms the protocol switch with status 101. RFC 8441 specifies WebSockets over an individual HTTP/2 stream using Extended CONNECT.

The security model has four distinct levels:

Level	What it establishes	What it does not guarantee
TLS (`wss`)	transport confidentiality, integrity and server identity	the user’s permission for an action
handshake	host, origin, cookie, token and subprotocol	authorisation of every later message
WebSocket frame	opcode, masking, length and fragmentation	safe JSON, Protobuf or business logic
application protocol	event type, object, tenant and correlation	security when the server trusts client fields

Client-to-server frame masking is required by RFC 6455 to protect intermediaries from particular protocol attacks. It is neither encryption nor access control. The server must treat unmasked content as untrusted input.

Handshake, Origin and cross-site WebSocket hijacking

A browser can attach cookies to the handshake according to their scope. If the server authenticates only with a cookie and does not validate Origin, a malicious page may open a channel in the context of a logged-in victim. This is cross-site WebSocket hijacking (CSWSH or CSWH). It resembles CSRF but can also let the attacking page receive events from the bidirectional connection.

OWASP WSTG — Testing WebSockets identifies server-side Origin verification as a core control. A correct policy should:

compare the complete origin: scheme, host and port;
use an explicit value allowlist rather than text suffix matching;
reject absent or null values unless an approved client genuinely needs them;
never treat Origin as authentication for non-browser clients, which can set it arbitrarily;
behave identically across every region, gateway and route.

A token in the query string can leak into access logs, history, APM and error output. Prefer a protected handshake mechanism or a short-lived single-use ticket exchanged for a channel session. Cookie-based applications still need Origin protection and consistent logout semantics.

Message-level authorisation

An authenticated connection does not mean every message is permitted. A common application envelope contains an operation name, request_id, object, channel or tenant identifier and a payload. The client can usually manipulate each field.

For every operation, the server should derive the subject from session context and verify:

whether the role may use this message type;
whether the object belongs to the user or tenant;
whether the user may perform this particular change, not merely read the object;
whether a subscription covers only an authorised scope;
whether workflow state currently permits the transition;
whether the message replays a completed transaction;
whether logout or an entitlement change has invalidated the channel.

A dangerous pattern authorises only the initial subscribe and lets the broker continue delivering events after a role change. Another trusts userId or tenantId from the message rather than the server-bound identity. These are real-time equivalents of BOLA and BFLA in the OWASP API Security Top 10.

Scoping a safe WebSocket pentest

Rules of engagement should define endpoints, subprotocols, accounts, tenants, roles, environment and volume constraints. Access-control testing requires at least two normal users in separate tenants plus an account for every privileged role. All test data should have recognisable unique markers.

Separate approval is needed for:

large-message and fragmentation tests;
high connection, subscription or message rates;
compression and memory-exhaustion scenarios;
long-lived idle channels;
payment, bulk-notification or physical-device operations.

A limit can be verified by approaching an agreed threshold gradually and confirming deterministic closure; causing an outage is unnecessary. The engagement belongs within a professional web application penetration test and should follow the same minimum-proof principle.

Technical WebSocket penetration testing methodology

1. Inventory channels and protocols

Review frontend code, mobile applications, observed network traffic, gateway configuration and documentation. Record the endpoint, transport version, Sec-WebSocket-Protocol, authentication, message format, heartbeat, reconnect behaviour, limits and broker. Include legacy paths, staging environments and administrator-only endpoints.

Determine whether the application uses raw WebSocket, Socket.IO, GraphQL subscriptions, STOMP, MQTT over WebSocket or a custom binary protocol. A library can add its own handshake and room semantics that require separate testing.

2. Assess the handshake

Compare anonymous, authenticated, post-logout and expired-token connections from different origins. Verify that the gateway and backend make the same decision. The selected subprotocol must be both offered by the client and supported by the server; it must not bypass a routing policy.

Confirm wss, certificate validation, SNI, HSTS on the initiating site and no fallback to ws. Examine logs for token exposure in URLs or diagnostics. Failed handshakes should not disclose a stack trace, key material or complete configuration.

3. Catalogue message types

Capture valid workflows and create a schema containing event type, direction, required fields, protected object, role, state, side effect and expected response. For JSON, test types, extra fields, nulls, duplicate keys and nesting limits. For binary messages, obtain the schema through an agreed grey-box process.

Do not begin with random fuzzing. First understand the contract, because the distinction between “unknown field ignored” and “unknown field receives a default” can change authorisation behaviour.

4. Build a role, object and tenant matrix

Replay the same legitimate message while changing one context: account A/B, tenant A/B, normal/admin role and owned/foreign object. Cover commands, queries, subscriptions and unsubscribe operations. Observe both the immediate response and later broadcast events.

The server should reject a forbidden action consistently, avoid any side effect and not reveal whether another tenant’s object exists. Use synthetic records. The broader API penetration testing checklist provides a reusable authorisation matrix.

5. Test state, ordering, races and replay

WebSocket makes concurrent operations easy. Determine whether an action can run before a prerequisite, repeat after success, execute simultaneously on two channels or arrive after entitlement expiry. Keep concurrency low, agreed and limited to reversible operations.

Sensitive changes should be idempotent or protected by a unique operation identifier, object version or transaction. A client-supplied request_id supports correlation but cannot serve as authorisation evidence.

6. Assess subscriptions, rooms and fan-out

Test topic names, wildcards, filters, tenant changes and guessed room identifiers. Removing a member, changing a role, logging out or resetting a password should remove access according to policy. If the server cannot reauthorise long sessions, use short channel lifetimes and a new handshake.

Examine metadata leakage. Presence counts, topic names and errors such as “room exists but access denied” can disclose another organisation’s structure.

7. Validate frames and resource controls

In a test environment, assess message length, fragment count, invalid UTF-8, opcode handling, control frames, compression and incomplete messages. The goal is controlled connection closure and resource release. Do not seek production capacity limits without a separate performance-testing plan.

Controls should exist per message, connection, user, tenant, IP and operation type. Subscription count and buffers for slow clients are especially important. Backpressure must prevent unbounded memory growth.

8. Test reconnect, expiry and revocation

Observe behaviour after a temporary network loss. A client must not resume a privileged session using an old ticket. Refresh-token rotation, logout, account suspension and role removal should terminate or constrain active channels. When a token expires mid-connection, the service needs an explicit strategy: reauthentication or closure.

WebSocket detection and logging

Log semantic events rather than complete messages containing customer data. A minimum record includes connection ID, user, tenant, origin, subprotocol, action type, appropriately protected object ID, authorisation outcome, size, latency and close code.

Useful alert conditions include:

many denied origins or handshakes;
a sudden rise in connections or subscriptions for one principal;
access attempts spanning many tenants or rooms;
messages processed after session revocation;
expanding outbound buffers and repeated limit violations;
unusual opcodes, fragment errors and invalid UTF-8;
repeated use of one operation identifier;
credentials detected in URLs or logs.

Correlation with HTTP, identity-provider and broker telemetry is essential. One connection ID should link the handshake, subscriptions, operations, business effects and final disconnect.

WebSocket hardening

Require wss and a current TLS configuration.
Authenticate the handshake, then authorise every message and subscription independently.
Enforce a strict full-origin allowlist for browsers.
Keep long-lived tokens out of query strings.
Bind subject and tenant to server-side context rather than message fields.
Validate envelope and payload against explicit schemas and reject unknown operation types.
Limit size, fragments, compression, rate, queues and subscriptions.
Implement backpressure and deterministic cleanup after disconnect.
Reauthorise after role changes or use short-lived channel sessions.
Revoke active connections on logout, suspension and critical account changes.
Isolate broker topics by tenant and authorise publishers and consumers.
Test equivalent policies through HTTP/1.1, HTTP/2, every region and every gateway.

Connect these requirements to OWASP ASVS 5, especially session, access-control, validation and communications requirements. Where brokers and gateways run in cloud-native infrastructure, a cloud penetration test should also cover network policy and service identities.

Pre-production WebSocket checklist

Primary technical sources

Need to assess a real-time application? A strong WebSocket pentest combines handshake analysis, a message-authorisation matrix, session lifecycle testing, broker review and controlled resilience checks. The result is a specific, reproducible scenario and a remediation that can be verified in a focused retest.