Jane Street measured TCP connections rising from 429 to 474,033 in about 30 seconds after network connectivity returned. Adam Yi's October 2 engineering account traces recurring maintenance failures through six interacting bugs.
The maintenance partitioned a cabinet for roughly 15–25 minutes. Kafka retries could time out while their underlying connection attempts stayed alive. A fourteen-year-old Async regression also left DNS resolution outside the connection timeout. Attempts accumulated during the partition, then progressed together when connectivity returned.
The numerical reconstruction is unusually persuasive. An outage of 1,435 seconds, about eighteen retry cycles and roughly 1,500 consumer groups predicted around 27,000 attempts per relay. Logs on two surviving relays showed 27,898 and 27,786, with similar open-file-descriptor counts. The retry schedule accounts for the scale of the work released during recovery.
A separate branch ran through UDP ephemeral-port pressure. A failed bind leaked its newly created socket. File-descriptor pressure could then cause resolver reinitialization to fail partway; an eleven-year-old glibc defect left a nonzero address count with null pointers, allowing a later lookup to crash. Jane Street says the affected glibc versions ran from 2.22 through 2.43, with the upstream fix in 2.44. The explanation for UDP demand is less certain than the quantified TCP reconstruction.
Meanwhile, a move to systemd user services stopped applying earlier descriptor-limit overrides. The soft limit fell eightfold, from 262,144 to 32,768, changing which older defect reached a hard resource boundary first.
The fixes addressed distinct links: resolver cleanup, closing sockets after failed binds, DNS inside the connection timeout, cancellation of abandoned Kafka attempts, deduplicated bootstrap lookups and restored limits. Some were upstreamed; others were slated for the next public Async release. This is Jane Street's own infrastructure account, with no trading-performance result attached. Read the complete investigation.