Omni Analytics incident

Omni pages not accessible

Resolved after 1 hour 18 minutes, affecting EastUsa, on September 8, 2025.

resolved Sep 08, 2025, 07:15 PM UTC

We are now considering this incident resolved. A recent code change generated unexpectedly large database queries that saturated CPU resources, leading to query timeouts and subsequent connection failures. We initiated a rapid rollback once the cause was identified. Since mitigating we have reverted the change and are working to prevent similar issues in the future.

Started
Sep 08, 2025, 05:57 PM UTC
Resolved
Sep 08, 2025, 07:15 PM UTC
Duration
1h 18m
Detected by Pingoru
Sep 08, 2025, 05:57 PM UTC

Affected components

EastUsa

Update timeline

  1. investigating Sep 08, 2025, 05:57 PM UTC

    Our monitoring picked up a spike in errors and we have begun getting reports from users. The team is investigating on highest priority

  2. identified Sep 08, 2025, 06:02 PM UTC

    We have identified the source of the issue and are in the process of rapidly reverting a change to resolve.

  3. identified Sep 08, 2025, 06:29 PM UTC

    We have completed rolling out the mitigation for this incident and are now in the monitoring phase.

  4. resolved Sep 08, 2025, 07:15 PM UTC

    We are now considering this incident resolved. A recent code change generated unexpectedly large database queries that saturated CPU resources, leading to query timeouts and subsequent connection failures. We initiated a rapid rollback once the cause was identified. Since mitigating we have reverted the change and are working to prevent similar issues in the future.

Your product runs on more than Omni Analytics

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →