EU Platform Service Disruption - Insights Reports Failing To Load Intermittently

Update timeline

1. investigating Sep 09, 2026, 02:06 PM UTC

   We are currently investigating reports where Insights Reports are intermittently failing to load for EU customers only. We will provide an update within 1 hour.

2. identified Sep 09, 2026, 03:00 PM UTC

   We have identified a potential root cause for this issue and are working towards mitigation. Another update will be provided within the next 1 hour.

3. monitoring Sep 09, 2026, 04:00 PM UTC

   We have fixed the underlying issue with loading Insights Reports and completed a data backfill task to restore any data that may have been missed in the reports. We will be placing the affected components under monitoring for now.

4. resolved Sep 09, 2026, 09:10 PM UTC

   Insights Reports have remained stable and fully available after the fix was released in the production environment. This incident is now resolved. We will publish a Root Cause Analysis here once a full incident postmortem has been completed.

5. postmortem Oct 28, 2026, 04:22 PM UTC

   **Summary:** On September 9th, 2026, at 12:20 AM PT, a system monitor alerted us of a backend system process exceeded its allotted runtime allocation and was therefore halted. At 3:51 AM PT, we received a customer report stating that Insights reports were not loading as expected. After initial triaging and troubleshooting, a platform incident was declared, and Insights Reports functionality was restored at 7:55 AM PT, with a backfill of missing reports data being completed at 8:25 AM PT. **Severity:** Sev-2 **Scope:** Only customers on the Firstup EU platform were in scope of this incident. Customers on the Firstup US1 and US2 platform remained unimpacted. **Impact:** Studio users observed intermittent issues with loading data in Insights Reports. **Root Cause:** The root cause for this incident was determined to be process locks that were introduced by the addition of a new node in our data warehouse during the Software Release Maintenance window on September 8th, 2026. The new node cancelled currently running queries and left process locks on the primary data tables. Each subsequent process and transformation job that tried to run against the EU data warehouse instance was unable to complete due to the existing locks on those tables, which resulted in delays and incomplete data being presented in Insights Reports. **Mitigation:** To mitigate this incident, the process locks were manually cleared at 9:55 AM PT, stabilizing the tables and allowed new data to flow through, which restored the Insights Reports functionality. A data backfill task was also run to restore data that was missed during the incident, and Insights Reports were up to date as of 8:25 AM PT. **Recurrence Prevention:** To prevent such an incident from recurring, Firstup has committed to the following actions: * Check for process locks following each Software Release Maintenance window to identify and mitigate potential issues faster. * Add monitoring to check for stuck processes that could become problematic. * Updating our core backfill scripts to be parameterized, allowing for faster running of backfill scripts whenever backfills tasks are needed.