ClickHouse describes an EC2 RAID-member failure whose ext4 errors continued while Kubernetes marked the node Ready. Temporary problem-detector events expired after an hour; service degraded without reported data loss. Its replacement system converts patterns and repeated checks into node conditions, then lets a separate controller evaluate, drain or terminate.

Two NVMe timeouts within a minute trigger a host-level condition. Network-wide probe failures do not automatically prompt migration, since moving pods cannot fix a regional fault. Disruption budgets, exclusions, dry-run, a kill switch and a cluster-wide cooldown constrain action. Diagnostics are best-effort rather than a prerequisite that can hang mitigation. Completely unresponsive nodes and external VM-health signals remain gaps.

clickhouse.com