Moving a Live Connection Between Kernels Without the Client Noticing

A client opens a WebSocket, streams over it for twenty minutes, and somewhere in the middle the infrastructure under it deploys and the socket dies. The client sees a close it did not ask for, and whatever the two ends had agreed on over those twenty minutes has to be negotiated again from nothing. A proxy tunnel has the same shape: an association that has been carrying traffic for an hour is one pod eviction away from an ECONNRESET.

The common answer is to push the problem onto the client. Reconnect logic, resume tokens, replay buffers, sequence-number negotiation, idempotent message handling. That works when the client is yours. When it belongs to a customer, runs code you do not control, and is holding state that is expensive to rebuild, the interesting question is whether the connection has to break at all.

It does not: a connection is state, all of it enumerable, and state can be moved. This post is the theory of how, the practice on Linux, and the measurements that corrected three things we had originally written down as fact.

What a connection is made of

“Move the connection” hides how much a connection is. It decomposes cleanly, and the decomposition is what makes the problem tractable, because each layer has a different owner and a different extraction mechanism.

Read it as a rule about ownership. The transport layer’s state lives in the kernel and can only be reached through socket options. The record layer’s state lives in the kernel too, but only if you put it there. Everything above is yours, in your own process, and is only as hard to serialize as you made it. The record-layer row is present only for connections that are encrypted, so a plain SOCKS5 or HTTP CONNECT tunnel skips it and has that much less to move.

“As hard as you made it” is a live decision at the top of that diagram, and WebSocket compression is where it bites. Negotiate permessage-deflate¹ with client_no_context_takeover and server_no_context_takeover, and there is zero cross-message deflate state to serialize. The alternative is shipping two 32 KiB LZ77 sliding windows between hosts and proving zlib bit-exactness across versions. The compression ratio lost is cheaper than that correctness argument.

The unacked bytes in the send queue are the entry people miss. They sit in the origin kernel waiting for an ACK that may never arrive, and the destination kernel has to own them so that it, and not a dead origin, performs the retransmission. Extract the whole send queue, restore the whole send queue, and the new kernel retransmits naturally.

Some of it is worth leaving behind deliberately. Congestion window, RTT estimators and pacing rate can stay, and the destination restarts in slow start, costing a few round trips of throughput. And in a proxy the upstream connection is yours, so the destination can dial a fresh one rather than migrating it. Only the client-facing socket has to physically move.

The theory: the only byte that matters is the one the client can see

A migration is a disconnect the moment the client observes anything that says the connection ended. That is one FIN, or one RST, on the wire. Everything else is a stall, and a stall is survivable.

So the requirement is narrower than it first appears. It reduces to two capabilities: destroying a socket without the kernel emitting the teardown packet it would normally emit, and getting the packets that were heading for the old location to arrive at the new one.

Linux supplies both halves, and the first one has been there since 3.5. TCP_REPAIR² was added for CRIU³, the checkpoint/restore project. It puts a socket into a mode where its queues and sequence numbers can be read and written directly, and where close() emits no FIN and no RST. It needs CAP_NET_ADMIN, and specifically not CAP_SYS_ADMIN, which is worth knowing when you are writing the pod security context.

The silent close is what makes it usable here. Queue extraction is reproducible at the application level by buffering, whereas every approach that lacks the silent close eventually has the old kernel emit something observable, and that observable packet is what turns a migration into a disconnect.

The second half is steering: the client’s 4-tuple names the old location, so something in the network path has to deliver that 5-tuple to the new one. On a single host, an established socket passed between processes carries its own binding and nothing is needed. Across pods or nodes, the dataplane has to be told.

What a frozen socket actually does

We measured this on kernels 6.8.0 and 7.0.13 with packet captures, and three of the results contradicted what we had written down.

A frozen socket is silent on close, and on nothing else. Two consequences follow, and both change the shape of the protocol you build on top.

Fence the flow to DROP before you freeze. Because the frozen origin keeps acknowledging, any byte arriving between your snapshot and your flip is acknowledged to the peer while being absent from the state you shipped. The peer will never retransmit it, because as far as the peer knows it was delivered. That is silent data loss, the worst failure available to this design.

With no drop and a freeze longer than the roughly 40 ms delayed-ACK⁵ timer, the connection lost data and then hung forever. With a dataplane drop in place the same freeze survived, because the client’s segments are dropped rather than absorbed, and TCP retransmits them after the flip.

The drop is what makes a freeze of non-trivial length safe at all.

Close the frozen fd only after the flip. The close is silent, and the hole it leaves is not. Once the socket is gone the 4-tuple no longer exists in that kernel, and the next packet the client sends draws a stateless RST from it. Flip the dataplane, confirm the destination is live, then close.

The protocol

Two-phase commit, with the origin holding a veto until the last possible moment. The whole cycle, with the client’s packets in flight throughout:

The client emits nothing new in that whole sequence. It retransmits what the fence dropped, and its retransmit lands on the destination.

Every failure before COMMIT ends with the origin thawed and serving. A NAK on PREPARE means nothing was ever frozen. A transport failure after the freeze means the origin sets TCP_REPAIR = 0 and resumes, and the client saw a stall. A failed restore means the destination closes its half-built socket in repair mode, which is silent, and NAKs. The flip is the only irreversible step and it happens after the destination has confirmed.

Split brain is the remaining hazard: two endpoints both unfrozen, both answering, sequence spaces diverging. Prevent it with an epoch written into the dataplane entry as a fencing token, so a stale origin can never reclaim a flow. The invariant to hold is that exactly one endpoint is unfrozen at any instant.

The practice: restore ordering, and the error codes that do not explain themselves

Rebuilding a socket in repair mode has an order the kernel enforces, and it enforces it with EPERM and EADDRINUSE rather than with anything descriptive. connect() is the boundary, and the two halves are mirror images of each other:

int fd = socket(family, SOCK_STREAM | SOCK_CLOEXEC, IPPROTO_TCP);

// 1. Repair mode first. It sets sk->sk_reuse = SK_FORCE_REUSE, which is what
//    lets the bind below succeed. Never set SO_REUSEADDR on a repair socket:
//    it overwrites SK_FORCE_REUSE with the weaker SK_CAN_REUSE and bind()
//    then fails with EADDRINUSE.
const int on = 1;
setsockopt(fd, IPPROTO_TCP, TCP_REPAIR, &on, sizeof(on));

// 2. A cross-node restore binds an address this host does not own.
//    IP_FREEBIND alone is not enough: bind() succeeds and connect() then
//    fails ENETUNREACH. IP_TRANSPARENT is the one that works.
setsockopt(fd, IPPROTO_IP, IP_FREEBIND,    &on, sizeof(on));
setsockopt(fd, IPPROTO_IP, IP_TRANSPARENT, &on, sizeof(on));
bind(fd, local_addr, local_len);

// 3. Sequence numbers, both queues, while sk_state is still TCP_CLOSE.
//    After connect() these return EPERM. TCP_REPAIR_QUEUE selects which
//    queue the next TCP_QUEUE_SEQ, send() or recv() applies to.
setsockopt(fd, IPPROTO_TCP, TCP_REPAIR_QUEUE, &recv_queue,    sizeof(int));
setsockopt(fd, IPPROTO_TCP, TCP_QUEUE_SEQ,    &rcvq_head_seq, sizeof(int));
setsockopt(fd, IPPROTO_TCP, TCP_REPAIR_QUEUE, &send_queue,    sizeof(int));
setsockopt(fd, IPPROTO_TCP, TCP_QUEUE_SEQ,    &sndq_head_seq, sizeof(int));

// 4. No handshake. In repair mode connect() only installs the 4-tuple.
connect(fd, peer_addr, peer_len);

Everything above happens while the socket is still TCP_CLOSE. Everything below requires it to be ESTABLISHED, which is what makes the two halves mirror images:

// 5. TCP_REPAIR_OPTIONS is the mirror image of TCP_QUEUE_SEQ: it requires
//    ESTABLISHED with no bytes sent, so strictly AFTER connect() and before
//    the queue bytes. It is write-only; getsockopt returns ENOPROTOOPT.
//    MSS, window scale, timestamps and SACK-permitted travel here.
setsockopt(fd, IPPROTO_TCP, TCP_REPAIR_OPTIONS, opts, n * sizeof(opts[0]));

// 6. Queue bytes last, because tcp_connect_init() calls
//    tcp_write_queue_purge() and resets snd_una/snd_nxt, destroying anything
//    pushed earlier. Select the queue, then send() into it.
setsockopt(fd, IPPROTO_TCP, TCP_REPAIR_QUEUE, &send_queue, sizeof(int));
send(fd, sndq_bytes, sndq_len, MSG_NOSIGNAL);
setsockopt(fd, IPPROTO_TCP, TCP_REPAIR_QUEUE, &recv_queue, sizeof(int));
send(fd, rcvq_bytes, rcvq_len, MSG_NOSIGNAL);

// 7. Window state after the bytes, since pushing them moves the sequences.
setsockopt(fd, IPPROTO_TCP, TCP_REPAIR_WINDOW, &win, sizeof(win));

// Repair mode stays ON here. Thaw only once the dataplane points at this
// socket, so that exactly one endpoint is ever unfrozen.

The MSS carried in those options needs one guard. Under repair, getsockopt(TCP_MAXSEG) reports the negotiated mss_clamp, which on loopback reads 65495, while setsockopt rejects anything outside [88, 32767] with EINVAL. Clamping to that range keeps a migration alive that a blind round-trip would kill.

TLS has to be in the kernel, and that sets a hard version floor

OpenSSL exposes no supported way to export its record-layer state, so a userspace TLS session cannot be serialized without forking the library or writing your own record layer. kTLS⁶ solves it by moving the record layer into a kernel ULP attached to the socket. getsockopt(SOL_TLS, TLS_TX) then hands back a crypto_info including the current record sequence number, and restore is the mirror: attach the ULP on the new socket, push the crypto_info back.

The floor is measured rather than read off a changelog. TLS 1.3 receive offload landed in OpenSSL 3.2 and was never backported:

3.0.16 and 3.1.8 were the newest releases of their series when we measured them, and 3.1 has stayed there since. The 3.0 line has moved on to 3.0.21, which we have not measured, and the reason to expect the same answer is the backport rather than the version number: receive offload is a 3.2 feature that was never carried back, so the version to move to is 3.2 or newer.

The failure is silent: TX engages, RX does not, and no error is raised anywhere. Assert BIO_get_ktls_send and BIO_get_ktls_recv after every handshake, and treat a session that fails the assertion as non-migratable from that moment, rather than discovering it at freeze time.

kTLS shapes the design in two further ways.

A TLS 1.3 KeyUpdate is kernel-dependent. On kernel 6.8 a second setsockopt(SOL_TLS, TLS_TX/TLS_RX) returns EBUSY, the kernel’s own comment reading “Currently we don’t support set crypto info more than one time”. OpenSSL then aborts with “no suitable record layer” and sends a fatal internal_error alert, while kernel 7.0.13 rekeys in place. On a kernel without rekey support the honest policy is to refuse migratability for that session, since there is no absorbing a KeyUpdate there. Where rekey works, a KeyUpdate arriving mid-migration aborts the migration, with the origin still frozen and intact, and it retries once the re-key settles.

The receive queue stops being bytes on the wire. TCP_REPAIR_QUEUE = TCP_RECV_QUEUE followed by recv(MSG_PEEK) returns decrypted plaintext once the TLS ULP is attached, because tls_sw_recvmsg intercepts the call, and that peek is destructive of the underlying state. A short read strands decrypted plaintext in the kernel’s rx_list, where FIONREAD cannot see it and repair extraction cannot reach it. Any design assuming the receive queue holds ciphertext is wrong the moment kTLS is on.

Where a connection is allowed to stop

A proxy that migrates only the client-facing socket has an asymmetry to respect: the destination’s upstream is brand new and has seen nothing.

A byte boundary is not a safe stopping point, because the tail of a partially forwarded message arriving at a fresh upstream is a message with no beginning. A message boundary is not sufficient either, and this one only showed up under load. A request already forwarded upstream may still be unanswered. That answer arrives at the origin’s upstream socket, which is about to close, and the destination’s fresh upstream will never produce it. The client then waits for a reply that no longer exists anywhere.

The signature is unambiguous in the numbers. At 20,000 messages, two runs in five stalled after the flip, and every stalled run’s serialized state was exactly one message smaller than a healthy run’s: 1072 bytes against 2116.

The drain also has to be ordered rather than symmetric. Against a client that streams continuously neither direction is ever idle, so waiting for both to fall silent independently just burns the grace period and drops the session. The client side stops first, at a message boundary. Only then can the upstream side drain, because only then is nothing new being handed to it.

With that ordering, six consecutive 20,000-message runs produced byte-identical serialized state, a freeze of 322 to 465 microseconds, no loss, no reordering, and no FIN or RST anywhere on the client-side capture.

One more ordering trap lives on the destination. A restored socket sits in TCP_REPAIR until it is thawed, and a repair-mode socket is silent on close only, so it still ACKs. Once the dataplane points at the destination, anything arriving before the thaw is acknowledged by a socket that will never deliver it, and never retransmitted. That is the origin-side hazard reproduced on the other end.

Dialling the upstream after receiving the state put a TCP connect and an HTTP round trip inside exactly that window, and stalled roughly one run in three. The dial belongs at channel-accept time, while the origin is still serving and nothing is frozen.

What the client actually feels

An internal freeze budget of 100 ms against Linux’s 200 ms minimum RTO⁷ measures your own work, and reading it as the client’s experience hid a real problem for a while.

An early run measured a freeze of 16 to 41 ms and a client-observed stall of about 205 ms. The gap is one segment that was in flight during the freeze: the fence drops it, and its retry waits for an RTO. The freeze was comfortably inside budget and the client still stalled longer than the RTO minimum.

Quiescing with nothing in flight is what keeps the stall small. With the ordered drain in place, a 20,000-message run measures a 322 to 465 microsecond freeze and a 26 to 38 ms client-observed stall, that stall being the drain window plus the freeze. All of it sits under the 200 ms minimum RTO, so the client emits no retransmits at all.

Process-to-process migration on one host is cheaper still. Six concurrent WebSocket sessions carrying 48,000 CRC-tagged messages moved to a different process mid-stream with a worst client-observed stall of 4.7 ms, and 136 microseconds on bare metal.

Three orders of magnitude separate those numbers, so they are worth seeing on one logarithmic axis, against the two lines that decide whether any of it matters:

The number to instrument is the client-observed stall, read from the client side, plus tcp_info retransmit counters and a packet capture asserting zero FIN and zero RST. Freeze duration is a measure of your own work.

The unit of migration is a set of sockets

SOCKS5 UDP ASSOCIATE is the case that settles the design. The TCP control connection and the UDP relay socket are one association, and moving either without the other breaks the client. So the unit is a set of sockets with a shared fate:

• the serialized state carries 1..N socket sections, each with its own transport mechanism
• freeze is all-or-nothing across the set, and so is thaw
• the dataplane flip is one atomic update covering every 5-tuple in the set
• a partial failure rolls the whole set back

The whole set should cross in a single sendmsg, because a partial handover is a session split across two processes with no way back. A SOCKS5 UDP association is three sockets and migrates as one, which we measured at 400 datagrams with zero lost.

Building for the single-socket case and generalizing later means rewriting the freeze ordering, the wire format, and the fencing protocol. That held a second time without being planned: a UDP relay needs one socket per client as a NAT binding, and those bindings are migration units of exactly this shape, so the machinery took them unchanged.

Two bugs came out of the generalization, both found by running it rather than by reading it.

The first built its fd array from an assumed shape, so a unit with no upstream socket passed an empty fd and sendmsg failed with EBADF, which read as a migration failure and was bookkeeping. The second restarted a fixed pair of coroutines regardless of what the unit was, so an adopted association got an upstream pointed at a placeholder, read EOF, and tore down a session whose sockets had all arrived intact. Which work to restart is a property of the unit.

Getting the packets to the new kernel

The fd handoff is the easy half, and it comes with a trap. A socket passed over SCM_RIGHTS⁸ keeps its creator’s netns forever, so its packets still leave through the origin pod’s veth even after another process owns the fd. Pod-to-pod migration on one node therefore still needs a dataplane override, and fd passing alone will not do it.

An eBPF⁹ hash map keyed by 5-tuple, read from a TC ingress program and redirecting to the destination veth, flips in 4 to 10 microseconds as a single bpf_map_update_elem. An nftables netdev rule is a workable fallback. For cross-node moves the flow table has to live at the ingress node, in the XDP L4LB shape, with the entry changing from a local veth to a GENEVE or IPIP encapsulation toward the destination.

A smaller detail that costs an hour if you meet it cold: abstract AF_UNIX sockets are netns-scoped and return ECONNREFUSED across namespaces, so a node-agent broker socket has to be a pathname socket.

We have moved a live connection from kernel 6.8.0 to kernel 7.0.13, so the restore path is genuinely kernel-independent rather than tuned to one build.

Datagrams, and the protocol that was designed for this

The datagram side of the first diagram has no queues and no sequence space, so there is nothing to freeze. The NAT binding is the entire flow, which makes it simultaneously the easiest state to move and the only state there is: lose it and the flow is gone with nothing to rebuild it from.

What QUIC’s own connection migration covers

QUIC has a feature called connection migration already, and it is worth being precise about its scope, because it describes the client’s move rather than the server’s.

RFC 9000 §9¹⁰ describes an endpoint changing its address and revalidating the path with PATH_CHALLENGE and PATH_RESPONSE, then scopes that narrowly: “This document limits migration of connections to new client addresses, except as described in Section 9.6. Clients are responsible for initiating all migrations.”

Section 9.6¹¹ is the server’s preferred address, offered once at handshake time, and it closes the door explicitly on the case we care about: “Migrating a connection to a new server address mid-connection is not supported by the version of QUIC specified in this document.” So the supported case is a phone moving from wifi to cellular, with the server sitting still throughout.

Routing by connection ID instead

What the connection ID¹² gives you on top of that is a routing handle. The address the client sends to stays exactly where it was, and the CID decides which backend behind that address serves the connection, so the server address the RFC pins down never moves.

Relocating the connection to a new backend then means issuing a NEW_CONNECTION_ID frame whose value routes to the new node, with a Retire Prior To that pulls the old one out of use. No freeze, no fencing epoch, no privileges, no kernel support.

Retire Prior To is what makes it a migration rather than an offer, and §5.1.2¹³ is worth quoting because it is a MUST rather than an invitation:

Upon receipt of an increased Retire Prior To field, the peer MUST stop using the corresponding connection IDs and retire them with RETIRE_CONNECTION_ID frames before adding the newly provided connection ID to the set of active connection IDs.

Issue a new CID without it and the client may keep using the old one indefinitely, leaving the connection served from two places at once, which is the exact state the TCP path spends an epoch-fenced dataplane preventing.

The CID layout

Encode the backend into the CID following the QUIC-LB draft¹⁴ rather than inventing a layout: three bits of config rotation and five of self-encoded length in the first octet, then the server ID, then a nonce of at least four octets.

The point of QUIC-LB is that a load balancer nobody wrote can route to a server nobody told it about, and the self-encoded length is what lets that balancer read a short header’s destination CID length off the wire instead of being told out of band. The draft is blunt about the plaintext algorithm’s cost: “without a key for the encoding, QUIC-LB makes no attempt to obscure the server mapping”, so an observer learns which backend serves a connection.

Two catches in the stack

OpenSSL manages connection IDs internally and exports no way to choose one, to issue a NEW_CONNECTION_ID with a value you picked, or to set a Retire Prior To, and that is still true of the 4.0 manual, so it is not a packaging lag. Stream IDs are exposed and are a different thing: a stream ID names a stream within a connection, a connection ID names the connection, and routing needs the second. Either patch the stack or pick one whose API exposes CID issuance.

The second catch is that RFC 9000 §5.1¹⁵ lets a client use a zero-length connection ID, meaning identify me by address, and OpenSSL’s client does. Every reply then carries dcid=0, so the reverse direction has no routing information at all, by the client’s own choice.

The forward direction still needs no state. The return path has to be a NAT, one socket per client, where the socket a reply arrives on is what identifies it. Those sockets are migration units, so they cross with the listener and the connection never notices.

Measured against a real OpenSSL 4.0.1 client and server with the relay replaced mid-connection: 900 exchanges, every echo correct, 21.3 ms worst stall against a 21.2 ms baseline taken straight at the origin. The migration itself costs about 0.1 ms.

What each layer of the stack buys

Six technologies stand behind the TCP path described above, each supplying exactly one capability the others cannot, and the migration is what falls out when all six are present at once.

Take any one away and the property collapses in its own specific way: without the silent close the client sees a teardown, without kTLS the keys stay locked inside OpenSSL, without the dataplane the packets keep arriving at a pod that no longer exists, and without the fencing token two kernels answer for the same connection.

Proving it, because an unverified migration is a rumor

The harness worth building is one whose only job is to catch the proxy dropping something. Ours drives a deterministic CRC-tagged bidirectional stream, detects any gap, duplicate or reorder, verifies a CRC over every payload, distinguishes how a broken stream broke (RST, FIN, protocol close, protocol error, timeout), and records per-message RTT percentiles so the stall shows up as the max.

Three assertions are worth building in from the start, because each one catches a class of vacuous pass:

• Fail on a skipped privileged test. TCP_REPAIR tests skip without CAP_NET_ADMIN, and an all-skipped run looks exactly like an all-passed one in any summary line.
• Assert that a session actually migrated, rather than that the client survived. A test that never triggers the migration passes beautifully.
• Watch the wire, not the socket. Inferring a disconnect from your own ECONNRESET tells you the connection died without telling you which packet killed it.

The measurement that communicates the result fastest is the client’s reconnect count across several chained migrations, per protocol. A topology diagram and a throughput graph both fail to distinguish a connection that survived from one that was rebuilt quickly. On the datagram side, where there is no connection to break, the equivalent is whether the NAT binding changed, which is the same failure in the only form a flow without a connection can have it.

What this solves, beyond deployments

The deployment case is the obvious one, and the same machinery answers two more that have nothing to do with shipping code.

A relay or an exit going down. In the cross-kernel move the client-facing socket is the only thing that travels, and the destination dials its own upstream. That asymmetry is what lets a session leave a node whose egress is degrading: the client keeps one unbroken connection while the path behind it is rebuilt somewhere healthier.

The honest scope is that the connection survives and in-flight upstream work might not. A request already handed to the failing upstream and not yet answered dies with it, which is why the drain is ordered and why the flip waits for the drain rather than racing it. If the exit is already dead the drain cannot complete at all, so this is a mechanism for leaving a degrading path rather than for recovering a lost one.

Quality rerouting. The flow map decides where a 5-tuple lands, and nothing requires that decision to be about failure. An exit with a better path to the target, a node closer to the client, a pool with different egress characteristics: any of them is a reason to move a live session, and the client stays connected across the change.

The cost is the state deliberately left behind. Congestion window, RTT estimators and pacing rate do not travel, so the moved connection restarts in slow start and spends a few round trips getting its throughput back. A move made for quality has to be worth more than that, which makes it a decision for a controller with real path measurements rather than a thing to do speculatively.

Draining a node on purpose. Eviction, autoscaling, kernel patching and hardware retirement are all the deployment case wearing different clothes, with one difference worth stating: they are scheduled. A migration you can schedule is one you can rate-limit, verify per session, and abort on the first rollback, which is a much easier operational position than a migration triggered by something already going wrong.

Scrapium is Scrapfly’s scraping browser, and the infrastructure under it is built so that our deployments are not our customers’ problem. Our engineering blog goes deep on the rest of the stack.