RFC 6455, published by the IETF in December 2011, defines a persistent bidirectional TCP channel between browser and server, opened by an ordinary HTTP handshake. The detail to look at is not the handshake but a seemingly arbitrary rule in section 5: every frame the client sends to the server must be masked with a 32-bit XOR key. That rule is the residue of a concrete attack against the network infrastructure of the time.

Context

Before RFC 6455, low-latency browser-server communication rested on workarounds bolted on top of HTTP: short polling, long polling, the techniques that went under the name Comet, and Server-Sent Events for the one-way server-to-client stream. All of them reuse HTTP’s request/response semantics to simulate a channel HTTP does not give. For low-latency bidirectional traffic the price is steep: one request per message, repeated headers, connections held open by force.

WebSocket exists to give an explicit TCP channel. The client opens with a GET request carrying Upgrade: websocket and Connection: Upgrade; a server that accepts replies 101 Switching Protocols. From there on the TCP connection no longer speaks HTTP: it carries WebSocket frames. URIs use the ws:// scheme (port 80) and wss:// (port 443, over TLS).

The handshake includes a check that is not authentication but protocol confirmation. The client sends Sec-WebSocket-Key with 16 random bytes in base64. The server concatenates the fixed string 258EAFA5-E914-47DA-95CA-C5AB0DC85B11, computes its SHA-1, base64-encodes the result and returns it in Sec-WebSocket-Accept. It serves to guarantee that the peer is genuinely speaking WebSocket and not bouncing back some unrelated HTTP request. About the content it guarantees nothing.

The transparent proxy problem

The first widely deployed draft, known as Hixie-76, did not mask client data. In November 2010 Lin-Shung Huang, Eric Chen, Adam Barth, Eric Rescorla and Collin Jackson brought to the working group an analysis — later published as Talking to Yourself for Fun and Profit — of an attack against transparent network proxies.

The mechanism is this. Many intermediary proxies inspect traffic on port 80 and treat it as HTTP, even when the handshake has already negotiated an upgrade the proxy does not understand. An attacker controlling JavaScript inside a page opens a WebSocket connection and writes bytes onto the channel that the proxy, reading them as if they were an HTTP request, interprets as a GET toward a third-party resource. The proxy executes the forged request and caches the attacker’s response under the legitimate URL. From that moment every client served by that proxy gets the poisoned content. The authors estimated a cost under one dollar per execution, and could not identify the faulty proxies precisely because they were transparent.

The reaction was immediate. Firefox 4, from beta 8 (December 2010), disabled WebSocket; Opera did the same. Mozilla and Opera asked the hybi group for a protocol change before re-enabling it.

The countermeasure’s architecture

The solution the working group chose takes away from the attacker the control over the bytes that end up on the wire. Every frame sent by the client carries a set mask bit and a 32-bit masking key, drawn from a strong-entropy source and different for each frame. The payload goes into XOR with the key repeated byte by byte; the server, which receives the key inside the frame itself, applies the same XOR and reconstructs the data.

 0               1               2               3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-------+-+-------------+-------------------------------+
|F|R|R|R| op    |M| Payload len |    Extended payload length    |
|I|S|S|S| code  |A|     (7)     |             (16/64)           |
|N|V|V|V|       |S|             |                               |
+-+-+-+-+-------+-+-------------+- - - - - - - - - - - - - - - -+
|     Masking-key (present if MASK=1)           | Payload Data  |
+-----------------------------------------------+- - - - - - - -+

Since the key is random and changes with each frame, the attacker can no longer predetermine the byte sequence the proxy would read: they cannot compose a valid GET nor choose the URL to poison. The framing defines opcodes for data (0x1 text, 0x2 binary) and for channel control (0x8 close, 0x9 ping, 0xA pong), with the FIN bit governing message fragmentation.

The critical point

Masking is asymmetric: mandatory from client to server, absent in the opposite direction. The reason lies in the threat model. The danger comes from hostile code running in a victim’s browser behind a shared proxy; it is the client that can be induced to write attacker-chosen bytes. The server is not under third-party JavaScript, so in that direction there is no payload to neutralise.

From this it follows that masking is not encryption and adds no confidentiality: the key travels in the clear inside the frame itself. Anyone wanting confidentiality and integrity against an active intermediary must use wss:// over TLS, not rely on the mask. The mask solves one problem — stopping the sender from choosing the raw bytes on the wire — and solves it at the cost of one XOR pass per frame on both ends.

Implications and limits

For anyone writing a WebSocket server the rule is binding: a conforming server closes the connection if it receives an unmasked frame from the client, and a conforming client closes if it receives a masked frame from the server. The nascent 2011 libraries — among them ws for Node.js and the Socket.IO fallbacks for browsers without native support — already embed the rule.

The cost can be measured. The XOR on every frame leaving the client and on every frame arriving at the server consumes cycles proportional to payload size; on high-frequency streams it tells. It remains a tolerated trade-off to keep WebSocket usable across the HTTP infrastructure already installed, which does not update at the pace of the standard.

The underlying limit is that masking defends the infrastructure, not the application. It does not authenticate the parties, does not protect against cross-origin messages toward a server that does not check the Origin header, does not replace TLS. It is a countermeasure aimed at a class of faulty proxies common in 2011, written into the standard because those proxies exist and will not vanish any time soon.

Cover image: Sequence diagram between a client (laptop) and a server: HTTP Upgrade handshake, bidirectional messages over a persistent connection,… — diagram by Brivadeneira, CC BY-SA 4.0 — https://commons.wikimedia.org/wiki/File:Websocket_connection.png