Cribl incident

High CPU utilization on Lakehouses

Resolved after 3 days, affecting Cribl Lake, on June 23, 2025.

monitoring Jun 26, 2025, 06:29 PM UTC

A fix has been implemented and we are monitoring the results.

Started
Jun 23, 2025, 06:24 PM UTC
Resolved
Jun 26, 2025, 06:29 PM UTC
Duration
3d
Detected by Pingoru
Jun 23, 2025, 06:24 PM UTC

Affected components

Cribl Lake

Update timeline

  1. investigating Jun 23, 2025, 06:24 PM UTC

    We are currently investigating this issue.

  2. investigating Jun 23, 2025, 06:26 PM UTC

    We're currently investigating an issue with high CPU utilization on small Lakehouses. No customer action required at this time. If you are impacted, you will see delayed or failing searches to Lakehouse.

  3. investigating Jun 23, 2025, 07:35 PM UTC

    Update: We have restarted the impacted Lakehouses, which is allowing them to slowly return to a normal state. There is still no Customer action required at this time. Investigating: We are still in the process of investigating a long-term resolution, and the incident is still active.

  4. identified Jun 23, 2025, 08:46 PM UTC

    The issue has been identified and a fix is being implemented.

  5. monitoring Jun 23, 2025, 08:48 PM UTC

    A fix has been implemented and we are monitoring the results.

  6. monitoring Jun 24, 2025, 04:01 PM UTC

    Update: The impacted tenants have been restarted, and are being actively monitored for the impact-thresholds. We will proactively intervene if necessary, but tenants should be performing as expected. Investigating: Engineering has identified the trigger and is finalizing long-term remediation plans.

  7. identified Jun 26, 2025, 12:56 PM UTC

    The issue has been identified and a fix is being implemented.

  8. identified Jun 26, 2025, 06:08 PM UTC

    Cribl will be performing an Emergency Maintenance at 2pm ET and 4pm ET in order to remediate this Incident. During this time, Customers should expect small windows of slow searches from Cribl Search to Lakehouse datasets during this time. If your Lakehouse does not perform as expected after this time, please contact Cribl Support for assistance.

  9. monitoring Jun 26, 2025, 06:29 PM UTC

    A fix has been implemented and we are monitoring the results.

  10. resolved Jun 26, 2025, 06:29 PM UTC

    This incident has been resolved.

Your product runs on more than Cribl

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →