CDP over CBOR, msgpack and Snappy: three formats, one bug

A Chrome DevTools Protocol message sent from one of our scrapers to a browser tab is serialized, on the wire, three different times. CBOR between the browser and our agent. Msgpack between the agent and the scraper. Snappy on top of msgpack when the payload is large enough.

The browser end of that link is Scrapium, our in-house stealth-patched Chromium build. It is not stock Chrome: it is Chromium with our anti-detection patches, our fingerprint hooks, and our pipe transport selected by default. When we say “the browser” in the rest of this post, we mean Scrapium. The CDP it speaks is upstream-compatible at the message level but the transport underneath is our binary stack, which is what the rest of this post is about.

Up front, the thing to know: we do not speak a vanilla CDP wire protocol. The public CDP transport you get with puppeteer, playwright, or chromedp is text WebSocket frames carrying JSON. We replace JSON with msgpack and wrap large payloads in Snappy. That choice rules out drop-in compatibility with off-the-shelf CDP clients (we keep a JSON fallback for them), and in exchange it buys us materially fewer bytes on the wire, native-code parsing on the hot path, binary payloads that stay binary end-to-end, and the ability to fix wire-format bugs at the producer.

Outline

1. The shape of the link
2. A 30 second primer on CBOR
3. Step 1: Scrapium speaks CBOR over a pipe
4. Step 2: msgpack on the WebSocket link (incl. vanilla-CDP trade-off, binary end-to-end)
5. Step 3: Snappy at 64KB
6. Primer: UTF-8 vs UTF-16 inside CBOR
7. “What is a string?” the bug we caught in monitoring (incl. the fix)
8. What we got wrong
9. What we would change next

The shape of the link

Three protocols, one logical call. Every translation step is a place where two implementations can drift.

A 30 second primer on CBOR

CBOR (RFC 8949) is a binary serialization format with a JSON-shaped data model: maps, arrays, strings, numbers, booleans, null. The wire encoding is a 1-byte type tag plus a payload, where the top three bits of the tag pick a “major type” (0 = uint, 1 = neg int, 2 = byte string, 3 = text string, 4 = array, 5 = map, 6 = tagged item, 7 = float/simple).

Two CBOR features matter for the rest of this post.

Indefinite-length containers. A map can be opened with 0xBF and closed with 0xFF, with key/value pairs streamed in between. Useful when the encoder does not know the size up front, which is exactly what Chromium’s crdtp does as it serializes DevTools events.

Tags. Major type 6 wraps a value with a semantic tag. The Chromium-derived pipe uses two:

• tag(24) (Encoded CBOR data item): a byte string whose contents are themselves a CBOR-encoded value. The encoder uses it to wrap sub-objects so they can be passed around as opaque blobs.
• tag(22) (Base64 binary): semantic marker for byte arrays that should round-trip as base64 in JSON-shaped views.

Anyone who has parsed CDP by hand knows the format is almost JSON-clean. CBOR is where the “almost” lives.

Step 1: Scrapium speaks CBOR over a pipe

CBOR is what Chromium gives you when you opt into the binary CDP transport (the --remote-debugging-pipe flag plus the cbor channel). Scrapium inherits that transport. The point of the format here is throughput and latency: a 1-byte type tag instead of a JSON character, no string escaping, no whitespace, no number-to-text round-trip. Parsing is roughly a state machine over byte tags rather than a tokenizer over UTF-8.

Our agent talks to its Scrapium subprocess via a pair of OS pipes (not a WebSocket). The framing is Chromium’s own:

0xD8 0x18 0x5A [4-byte uint32 big-endian content_size] [CBOR payload]

That prefix is tag(24) (0xD8 0x18) plus a byte-string-of-length-uint32 marker (0x5A), then the length, then the bytes. Each message is a self-describing CBOR envelope wrapping a single map.

The reader is small and hot. We pool the 7-byte header buffer per goroutine to avoid an allocation per read:

var cborEnvelopePrefix = []byte{0xD8, 0x18, 0x5A}
const cborEnvelopeHeaderSize = 7

type CBORPipeReader struct {
    r      io.Reader
    header [cborEnvelopeHeaderSize]byte
}

func (cr *CBORPipeReader) ReadMessage() (*Message, error) {
    if _, err := io.ReadFull(cr.r, cr.header[:]); err != nil {
        return nil, err
    }
    // validate prefix, parse 4-byte size, then ReadFull(payload)
}

The decoded Message keeps its Params, Result and ID fields as opaque CBOR byte slices. We do not re-decode them unless we need to inspect them, which keeps pass-through messages cheap.

Step 2: msgpack on the WebSocket link

Between the agent and the scraper we use msgpack, not CBOR. Msgpack is in the same family as CBOR (binary, schema-less, JSON-shaped) but the wire is simpler: no tag major type, no indefinite-length containers, no float-format negotiation. For our link that simplicity is the feature, not a missing one.

Three concrete reasons:

1. Drop-in replacement for JSON, only faster. This is what made the migration cheap. Msgpack’s type system is a strict superset of JSON’s: maps, arrays, strings, integers, floats, booleans, null. Swap json.loads(payload) for msgpack.unpackb(payload) and the resulting Python dict / Go map[string]interface{} is structurally identical. No schema file, no codegen, no struct re-definition. Existing CDP handlers, dict-walking telemetry, pytest fixtures all keep working. On a Network.responseReceived event the size win over JSON is typically 30-40%, and the decode-time win is larger than that. The two additions msgpack brings over JSON are exactly the two things JSON was missing: bin for raw bytes (see Binary stays binary) and ext for custom tagged types (we do not use it). This is the property protobuf, Cap’n Proto, and FlatBuffers do not give you: they all force a schema definition up front, which is a multi-week refactor on a CDP codebase that has been growing dict-walking handlers for years.
2. Native-code parsers in every language we care about, which matters most for Python. Production msgpack implementations are thin bindings over a C or Rust core. Python: ormsgpack (Rust), msgpack-python (C). Node: @msgpack/msgpack. Go: vmihailenco/msgpack. The decoder almost never touches the slow path of the host runtime, which is the difference between “shipping CDP throughput in Python” and “burning your event loop on JSON parsing.” On the hot path for our Python scraper, msgpack unpackb is 5-10x faster than json.loads on the same payload shape; for a scrape that generates thousands of CDP events that gap is the difference between a healthy worker and one that falls behind.
3. Simple wire, binary framing. No tag system (we want this: every CDP payload becomes a flat map of JSON-shaped values by the time it leaves the agent, with no tag(22)/tag(24) ambiguity to negotiate). WebSocket binary frames are the natural fit. Text frames force a UTF-8 validation we do not need.

The serializer is picked by the client at connect time via query parameters on the WebSocket URL:

wss://agent/run/<id>/cdp?serializer=msgpack&compression=snappy&compression_threshold=65536

serializer is msgpack or json. compression is snappy or absent. compression_threshold is bytes. JSON stays as a fallback for debugging in a browser dev tools panel.

On the Go side the codec is a small decorator stack:

func NewCodec(serializer, compression string, compressionThreshold int) Codec {
    var c Codec
    if serializer == "msgpack" {
        c = msgpackCodec{}
    } else {
        c = jsonCodec{}
    }
    if compression == "snappy" {
        if compressionThreshold <= 0 {
            compressionThreshold = defaultCompressionThreshold
        }
        c = &compressedCodec{inner: c, threshold: compressionThreshold}
    }
    return c
}

This is not vanilla CDP anymore

Once you put msgpack on the wire instead of JSON, you are no longer speaking the CDP transport that puppeteer, playwright, chromedp, or any other off-the-shelf client knows how to talk to. Those libraries assume a text WebSocket carrying JSON. Connect them to our agent with the default settings and they get binary frames they cannot parse.

We keep the JSON codec in the decorator stack for exactly that reason. A customer who points puppeteer at our endpoint gets a working JSON link by appending ?serializer=json (or by omitting serializer entirely, since JSON is the fallback). They give up the bytes-on-the-wire and decode-time wins, but the existing tooling works. Our SDK and our internal scraper opt into msgpack + Snappy by default.

Binary stays binary, end to end

This is the win that does not show up in any benchmark micro-comparison and matters more than all the others combined.

JSON is a text format. It has no way to carry raw bytes. Every binary payload that crosses a JSON boundary has to be base64-encoded first, then base64-decoded on the other side. Base64 is a 3-bytes-in-4-chars expansion, so it adds ~33% bloat to every byte of binary you ship, plus an encode pass on the sender and a decode pass on the receiver. Both passes touch every byte.

In CDP that tax is everywhere binary lives:

• Screenshots (Page.captureScreenshot, Page.startScreencast frames). A 2MB PNG becomes ~2.7MB of base64 text.
• Response bodies (Network.getResponseBody with base64Encoded: true). A 5MB image response: ~6.7MB on the wire.
• PDFs (Page.printToPDF). Same story, often larger.
• File uploads / downloads (Browser.setDownloadBehavior interception).

CBOR (between Scrapium and the agent) has a native byte string type. Msgpack (between the agent and the scraper) has a native bin type. Snappy compresses whichever bytes it sees. There is no base64 anywhere in our stack. A 2MB PNG screenshot leaves Scrapium as 2MB of CBOR byte string, becomes 2MB of msgpack bin, gets ~600KB-700KB after Snappy on typical screenshot entropy, and lands in the scraper at its original 2MB after decompression. The bytes ride at native size the whole way.

The same property is what makes the msgpack bin fix in the second half of this post even possible. Vanilla CDP has no native binary type to retreat to; if you want to ship raw bytes you base64 them inside a JSON string. We have bin, so we can route non-UTF-8 bytes (like the Norwegian Location header we discuss below) through a type that does not pretend to be text.

Step 3: Snappy, only when it pays

Snappy is the third layer, and the only one that is conditional. The reason we use it at all is simple: Snappy costs almost no CPU and still gives a real compression ratio. At our scale (millions of CDP events per minute across the fleet) the bytes-on-the-wire savings turn into a measurable reduction in WebSocket buffer pressure, fewer TCP segments per event, and lower egress on the agent pods. The CPU we pay to encode is small enough that it does not show up on the latency tail.

The threshold is the lever for not regressing latency on small payloads. We turn Snappy on at 64KB: below that, the per-frame compress/decompress overhead is a larger fraction of the message lifetime than the bytes-saved win, and we leave the payload alone.

Two design choices for the framing:

No negotiation, just a magic prefix. A compressed payload is prepended with two bytes, 0x00 0x73. 0x00 is not a legal first byte for JSON (must be {, [, ", digit, etc.) or for a msgpack map/array marker, so detection on the receive side is unambiguous. 0x73 is ASCII s. The receiver does not need to know the sender’s threshold; it inspects the first two bytes:

// Marshal path
if len(payload) > c.threshold {
    compressed := s2.EncodeSnappy(nil, payload)
    if len(compressed) < len(payload) {
        result := make([]byte, len(snappyMagic)+len(compressed))
        copy(result, snappyMagic)
        copy(result[len(snappyMagic):], compressed)
        return result, nil
    }
}
return payload, nil

// Parse path
if len(data) > 2 && data[0] == 0x00 && data[1] == 0x73 {
    decompressed, _ := s2.Decode(nil, data[2:])
    data = decompressed
}
return c.inner.Parse(data)

Asymmetric thresholds per direction. Each direction picks its own threshold; Snappy is one-way per leg because the sender alone decides what to compress and the receiver only checks the magic prefix. The downstream link (events from Scrapium) is where the win lives: a 1000-frame screencast burst sees most frames in the compressible-and-large bucket, and Snappy on that traffic is the difference between a comfortable agent pod and one queuing WebSocket writes. The upstream link (commands from the client) is almost always under 1KB and rarely benefits.

Only if the compressed payload is actually smaller. The len(compressed) < len(payload) guard catches the case where a small or already-entropic payload gets bigger after framing. We send the original bytes in that case and skip the magic prefix entirely.

We picked Snappy over zstd or gzip on one criterion: decompression CPU under contention. The agents run hot, often 80%+ CPU on the WebSocket reader goroutine when a tab is streaming Network.responseReceived events for a heavy page. Snappy decompression at ~2 GB/s/core via klauspost/compress/s2 was the only option that did not start showing up in flame graphs.

A primer on UTF-8 vs UTF-16 inside CBOR

This is the part of CBOR that bites every new CDP client author. If you take one thing from this post, take this section.

CBOR has two string types with two different major types:

• Major type 2 (byte string). Opaque bytes. CBOR makes no claim about the contents.
• Major type 3 (text string). Per RFC 8949 §3.1, contents MUST be a “valid sequence of UTF-8 bytes.”

You would expect text to land in major type 3 and binary in major type 2. That is not what happens. Chromium’s crdtp encoder has its own conventions for what goes where, and they do not match the CBOR spec’s intent:

This table is the entire problem. The byte string type is overloaded for three different things (UTF-16LE text, base64-binary, nested CBOR), and the text string type is not actually guaranteed to be UTF-8 the way the spec promises. As a client, you have to do four things:

1. For byte strings without a tag, decode them as UTF-16LE. Chromium’s internal string type is std::u16string (a survivor of the early-2000s “UCS-2 is enough” era), and crdtp emits those as raw little-endian byte pairs.
2. For byte strings with tag(22), treat them as opaque binary. Do not try to decode.
3. For byte strings with tag(24), recurse: the contents are themselves a CBOR-encoded value.
4. For text strings, validate the UTF-8. If it is not valid, you have a Chromium-shaped HTTP header that needs transcoding (see the next section).

The two-byte heuristic that actually works on the wire for UTF-16LE detection: if the byte string is at least 2 bytes long, even-length, and a significant fraction of the bytes at odd offsets are 0x00, it is almost certainly UTF-16LE ASCII text. If it is even-length but no null high bytes appear, it might be UTF-16LE CJK text or it might be base64-shaped binary. If it is odd-length, it is not UTF-16LE and you should treat it as raw bytes.

There is one trap inside the trap: never run a fallback “replace invalid UTF-8 with U+FFFD” pass before you have ruled out UTF-16LE. Japanese character ん (U+3093) in UTF-16LE is bytes [0x93, 0x30]. The 0x93 byte is invalid as a UTF-8 start byte. If you replace it with U+FFFD first, you get the bytes for the replacement character (0xEF 0xBF 0xBD) followed by 0x30, and any subsequent UTF-16LE decode produces garbage CJK. Order your checks UTF-16LE first, surrogate-recovery second, raw-bytes-transcode last.

The drift we caught: “what is a string?”

The stack above ran in production for months without trouble. Then enough traffic accumulated that the long tail started landing in our dashboards: our decode-error counter on the agent-to-scraper link began ticking up in lockstep with one specific shape of origin response. We pulled the captures and traced the failures back to a single mode: a non-ASCII byte arriving inside an HTTP response header.

The trigger was a redirect from a Norwegian origin where the Location: value contained VÅR (the Norwegian word for “spring”). The Å was the smoking gun.

The root cause is one sentence long: HTTP/1.1 header values are nominally ASCII, and we received a non-ASCII byte. That sounds trivial, but every layer downstream had a different opinion about what to do with it, and the disagreement only became visible when a real origin sent us bytes that none of the layers were happy with.

Here is what was actually happening, by layer:

1. The origin server emitted Location: ...VÅR... with the Å byte as raw latin-1 0xC5, not UTF-8 0xC3 0x85. RFC 7230 §3.2.4 lets header values carry latin-1, so the Norwegian origin was within spec. In practice, HTTP/1.1 callers assume header values are ASCII, and most code paths downstream of “header parsing” never expect a high byte.
2. Chromium’s network stack received those bytes via HandleString8 (Chromium third_party/inspector_protocol/crdtp/cbor.cc, around lines 471-475) and emitted them into a CBOR text string (major type 3) without re-encoding to UTF-8.
3. CBOR major type 3 is supposed to be UTF-8. Chromium’s crdtp encoder does not re-validate HandleString8 input. So we got a CBOR text string whose contents were not valid UTF-8. A spec violation we, as the consumer, had to tolerate.
4. Our Go agent decoded the CBOR text string into a Go string. Go strings are byte slices; they do not have to be valid UTF-8. The raw byte survived intact.
5. We msgpack-encoded that Go string as a msgpack str type. The msgpack spec also requires str contents to be valid UTF-8 (since msgpack 2.0). vmihailenco/msgpack on the Go side does not validate.
6. On the client side, the strict Rust msgpack decoder rejected the frame with ValueError: invalid UTF-8 string. We do not run a lenient surrogateescape fallback on purpose: silently rewriting bad bytes into surrogates would hide the wire-format mismatch behind a downstream BSON crash that points nowhere useful.
7. The decode failure surfaced as a dropped frame in the CDP event stream. The scraper kept running with one fewer event. The damage showed up later: the response object it eventually returned was missing the redirect chain that the dropped frame would have carried, and downstream code that expected the chain crashed at the persistence layer. The decode rejection was the right immediate behavior; the lesson was that dropped frames need loud telemetry, which is why our decode-error counter (the thing that caught this incident in the first place) now alerts on any non-zero count.

The bug was that two implementations on either side of a wire boundary had different ideas about whether “text string” meant “bytes that happen to be UTF-8 most of the time” or “validated UTF-8.” Chromium’s CBOR encoder takes the loose view. The strict msgpack decoder on the client takes the strict view (and refuses to paper over the difference). Our agent in the middle was passing the loose bytes through unchanged.

The fix: sanitize at the agent

The right place to fix this is the agent, not the client. Two reasons. One, the agent already has to walk every CBOR value to re-encode as msgpack, so the cost is marginal. Two, the agent has more information than the client: it knows whether a string came from a CBOR text string, a CBOR byte string, or a tag-wrapped sub-object. The client only sees the msgpack output.

The sanitizer handles four cases that show up in real Scrapium traffic, in this strict order:

func sanitizeString(s string) string {
    // 1. UTF-16LE detection MUST come first. A surrogate-recovery pass would
    //    destroy CJK content (Japanese 'ん' U+3093 is [0x93, 0x30] in UTF-16LE;
    //    0x93 is invalid UTF-8 and would be replaced with U+FFFD).
    if IsNullByteUTF16LE(s) {
        s = utf16LEToUTF8(s)
    } else if !utf8.ValidString(s) {
        // 2. CESU-8 lone surrogates (Chromium escape encoding for stray bytes).
        if hasCESU8Surrogates(s) {
            s = decodeSurrogateBytes(s)
        } else {
            // 3. All-CJK UTF-16LE that the null-byte heuristic missed.
            s = DecodeString16(s)
        }
        // 4. Raw non-UTF-8 bytes (the Norwegian Location case).
        //    Chardet picks a decoder, default Windows-1252.
        if !utf8.ValidString(s) {
            s = decodeRawInvalidUTF8Bytes(s)
        }
    }
    // Null bytes become newlines (Netlog uses NUL as a multi-value separator).
    if strings.ContainsRune(s, 0) {
        s = strings.ReplaceAll(s, "\x00", "\n")
    }
    return s
}

The ordering is load-bearing. Skip step 1 and you destroy CJK. Skip step 2 and you misidentify CESU-8 as UTF-16LE. Skip step 4 and the Norwegian byte leaks through.

CBOR byte strings (major type 2) get a deterministic version of the same pipeline: per the table above, byte strings without a tag are always UTF-16LE, so the heuristics disappear and we transcode unconditionally.

What we got wrong

We trusted “CBOR text string = UTF-8” as a guarantee instead of an aspiration. RFC 8949 §3.1 is unambiguous. Chromium’s encoder ignores it for forwarded HTTP bytes. We had no defensive check at the agent, and the strict client decoder has nothing to fall back to on purpose: a lenient surrogateescape decode would silently rewrite bad bytes into lone surrogates, and that hides the real wire-format mismatch behind a downstream BSON crash that points at the wrong service. The right answer is to fix the producer, not paper over it at the consumer.