ClickHouse incident

Elevated error rate in AWS

Resolved after 3 days, affecting us-east-1 and eu-west-2 and 1 more, on January 31, 2026.

resolved Feb 03, 2026, 10:09 AM UTC

We have confirmed that all systems are stable and operating normally. The networking configuration has been fully restored across all regions, and we have implemented additional monitoring and safeguards to prevent recurrence. We will continue to monitor systems closely over the coming days as part of our standard post-incident procedures, and share an RCA as soon as possible.

Started
Jan 31, 2026, 10:03 AM UTC
Resolved
Feb 03, 2026, 10:09 AM UTC
Duration
3d
Detected by Pingoru
Jan 31, 2026, 10:03 AM UTC

Affected components

us-east-1eu-west-2ap-southeast-1eu-central-1AWS us-east-1

Update timeline

  1. investigating Jan 31, 2026, 10:03 AM UTC

    We are investigating a partial outage affecting services in the AWS region us-east-1.

  2. identified Jan 31, 2026, 10:55 AM UTC

    Our engineering team has identified the root cause of the service disruption affecting ClickHouse some instances in US-EAST-1. We are currently deploying a fix across the region. ClickHouse instances should begin resuming normal operations within the next 10-30 minutes. We will continue to monitor closely and provide updates as services are restored.

  3. resolved Jan 31, 2026, 11:51 AM UTC

    This issue is now fixed.

  4. investigating Feb 01, 2026, 09:27 AM UTC

    We are investigating an elevated error rate in AWS us-east-1 and eu-west-2.

  5. resolved Feb 01, 2026, 09:45 AM UTC

    All components are operational after the restart of an internal component (Cilium) in the affected regions. The team is working on an RCA.

  6. investigating Feb 02, 2026, 09:35 AM UTC

    We are investigating an elevated error rate in AWS eu-central-1 and ap-southeast-1.

  7. monitoring Feb 02, 2026, 10:23 AM UTC

    The team mitigated the issue in affected regions and continues to investigate the root cause.

  8. monitoring Feb 02, 2026, 06:13 PM UTC

    Additional configuration changes have been applied to prevent the recurrence of this issue. The team continues to monitor the systems.

  9. monitoring Feb 02, 2026, 10:47 PM UTC

    We believe this is related to a known bug in the version of our networking stack currently in use and are mitigating the issue by resetting this layer across all AWS regions. The team continues to monitor all systems closely.

  10. monitoring Feb 03, 2026, 03:06 AM UTC

    We have completed the reset of the networking stack across all regions and continue to monitor all systems closely.

  11. resolved Feb 03, 2026, 10:09 AM UTC

    We have confirmed that all systems are stable and operating normally. The networking configuration has been fully restored across all regions, and we have implemented additional monitoring and safeguards to prevent recurrence. We will continue to monitor systems closely over the coming days as part of our standard post-incident procedures, and share an RCA as soon as possible.

Your product runs on more than ClickHouse

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →