Writes to a Dead Socket Succeed

The Circuit Breaker agent is a small Go process that sits on a machine, collects host facts and probe results, and streams them to a server over a WebSocket. It is about as simple as a network client gets: connect, then write samples forever.

It spent an entire outage writing samples into nothing and reporting that everything was fine.

The failure that produces no event#

Every reconnect strategy people write assumes the connection ends. The peer closes and you get a FIN. The peer crashes and its kernel sends a RST. Either way there is an event, your read returns an error, and your reconnect loop runs.

Three extremely ordinary things sever a connection without producing either one:

• A firewall rule changes to DROP. Not REJECT – DROP is silent by definition, and packets in flight simply stop arriving.
• A container is detached from a network, or a virtual interface goes away underneath an established socket.
• A NAT or conntrack entry expires on a middlebox between the two ends. The translation that made the connection work is gone; both endpoints still believe they have it.

None of these tell anybody anything. What is left is a black hole: a socket that is open on both ends, connected to a path that no longer carries packets.

Why the writes keep succeeding#

A successful write() on a TCP socket does not mean the data arrived. It means the data was accepted into the kernel’s send buffer. Delivery, retransmission and acknowledgement all happen afterwards, out of the calling code’s sight.

So the agent writes a sample, gets no error, and writes the next one. The kernel queues them and retransmits, patiently, for a long time. On Linux the ceiling is tcp_retries2, which defaults to 15 and corresponds to somewhere between thirteen and thirty minutes of retrying depending on the round-trip timeout, before the socket finally errors.

Two consequences follow, and the second is worse than the first.

The first is that a write only fails long after the outage started. Thirteen minutes of samples are already gone by the time anything is reported.

The second is that if the agent goes quiet – no samples due, nothing to send – then nothing is ever retransmitted, nothing ever exhausts, and the socket stays cheerfully open indefinitely. There is no idle probing in TCP unless you ask for it. A connection that is never written to is never discovered to be dead.

TCP keepalive is not the answer you want#

The protocol does have an answer, and it is not usable for this. SO_KEEPALIVE defaults to two hours of idle before the first probe. You can tune that per socket on Linux, and then discover that the value you need is not portable, that nothing in the path is obliged to preserve it, and that a keepalive probe answered by a proxy in the middle tells you the proxy is alive rather than the peer.

The layer that knows what a healthy connection means for your application is your application. So the heartbeat belongs there.

A heartbeat is not a detector#

This is the part worth being precise about, because sending pings is the obvious half and it accomplishes nothing on its own.

If you send a ping every twenty seconds and never require anything back, you have added traffic and learned nothing – the pings vanish into the black hole exactly like the samples did. The detector is not the ping. The detector is a read deadline: a bound on how long you are willing to go without hearing anything at all, after which you declare the peer gone.

The heartbeat exists to make that deadline meaningful. Without a ping, silence is ambiguous: an idle peer and a dead peer look identical. With a ping on a known interval, silence is a fact. A peer that owes you a message every twenty seconds and has said nothing for sixty is not idle.

That relationship is the whole design, and it fixes both numbers at once. The deadline has to be a multiple of the interval – Circuit Breaker uses twenty seconds and sixty, so two heartbeats can be lost to ordinary jitter before anything is declared dead. Set the deadline too close to the interval and a slow network becomes an outage. Set it too far and you are back to waiting out tcp_retries2 by hand.

Symmetric, because either end can go deaf#

Both ends run the same pair of timers. This matters more than it looks like it should: the failures above are frequently one-directional. A firewall rule that drops traffic in one direction leaves the other direction working, so one end still receives while the other has been talking to itself for ten minutes.

If only the client enforces a deadline, the server holds a dead session open, keeps its resources, and keeps believing the host it represents is reporting. The rule is that a silent peer is a disconnected peer, and it has to be true on whichever side goes silent.

Detection is only worth what catches the data#

Noticing an outage faster is not the goal. Not losing the outage is the goal, and that needs somewhere for the samples to go.

The agent spools to a 64 MiB ring buffer on disk, which survives a reboot as well as a disconnect, and a tripped read deadline is the signal to start writing there instead of to the socket. The reconnect loop backs off behind it. The interesting problem is not the outage; it is the end of the outage.

An agent that has been offline for a day is holding thousands of queued samples, and the naive catch-up is to send them as fast as the socket accepts them. Do that across a fleet of agents that all reconnect when the server comes back and you have converted one outage into a second one, this time self-inflicted, arriving as a thundering herd on the ingest path.

So catch-up is paced – forty frames per second and 2.5 MiB per second from the spool head. A day of one-sample-per-thirty-seconds telemetry is about 2,880 frames, which clears in roughly seventy-two seconds. Nobody notices seventy-two seconds. Everybody notices an ingest worker falling over.

The shape of the fix#

The change that closed this was three lines: a write deadline on the ping, a read deadline on the connection, and the same pair on the server side. Around them sits a comment several times longer than the code, explaining why those particular numbers relate to each other and what breaks if somebody "simplifies" the deadline away.

That ratio is the honest summary of the bug. The hard part was never the fix. It was believing that a socket reporting success could be lying, when every mental model of a network failure says that failures announce themselves.

Sources#

• Linux man7, tcp(7) – tcp_retries2, tcp_keepalive_time and their defaults
• The Circuit Breaker architecture reference, which documents the agent link, its spool and its pacing budget