Google’s November 4 account of its October London cloud outage describes two controllers reaching the same wrong conclusion. During repairs, a breaker stood open. Voltage was still present on the supply side, but no power could cross the breaker. Both the active and standby controllers used the same logic; neither combined that voltage reading with breaker position. Neither started the generators before the batteries ran down.

Adjacent zones received traffic they did not have capacity to absorb. Some otherwise healthy requests still waited for metadata services in the offline zone. Engineers isolated those endpoints and prioritized active reservations. Restoring electricity was not enough to reopen immediately: storage devices that had lost power needed integrity checks. At the report’s six-hour-eight-minute impact endpoint, 99.5% of Compute Engine VMs were healthy. Its subsequent repair dates were targets, not completion notices.

Sources: status.cloud.google.com