Resolved after 1 day 5 hours, affecting AdBook+ System Performance, on September 30, 2026.
### **Summary of Impact** On September 30, 2026, at 8:17 AM PT, an AdBook deployment of v4.10.01 intended for the sandbox \(SBX\) environment was unintentionally run against Production. The deployment shut down all environments, and AdBook was unavailable to all customers for 25 minutes. Access was restored at 8:42 AM PT. Some reporting and inventory functions stayed degraded until 2:24 PM PT, when full functionality was restored. No customer data was lost, and no customer action is required. Summary of events: * 8.17 am PT - v4.10.01 deployment begins on Production; all environments shut down * 8.22 am PT - Incident communication protocols begin * 8.24 am PT - v4.09.03 redeployed; application access restored * 8.33 am PT - Customers notified of incident via Status page * 9.30 am PT - DQT \(Data Query Tool\) reporting restored * 1.19 pm PT - Inventory availability validation restored * 2.24 pm PT - Inventory jobs that failed during the incident reprocessed; full functionality restored ### **Root Cause** While deploying v4.10.01 to SBX, the Engineering team hit deployment failures. During troubleshooting, the deployment pipeline's target environment was unintentionally changed from SBX to Production. Two gaps in tooling and process allowed this: * The environment selector's clickable area extends well beyond its visible controls, so a click elsewhere on the page can change the target without any visible indication. * The pipeline has no confirmation step specific to Production, so nothing prompted a final check of the target before services stopped. The extended recovery after the application was rolled back was a result of.v4.10.01 including database changes that aren't backward compatible with v4.09.03. ### **Resolution** The Engineering team redeployed v4.09.03 - the current production version - as the fastest and lowest-risk way to restore access. Application access was restored at 8:42 AM PT. Redeploying the application did not reverse the database changes made by v4.10.01.DQT reporting and inventory availability validation remained affected because database artifacts required by v4.09.03 had been removed or changed during the v4.10.01 deployment. The team restored these objects as they were identified and reprocessed the affected inventory jobs, completing recovery at 2:24 PM PT. ### **Mitigation and Next Steps** To reduce risk of a similar incident we are adding a mandatory production-intent confirmation gate. The gate and subsequent confirmation will prominently identify key deployment parameters \(such as the target environment, release version\), and require explicit acknowledgment before any services are stopped or database changes are applied. Second-person approval will be added as an additional safeguard, taking staff availability and emergency recovery requirements into account.
Affected components
Update timeline
- investigating Sep 30, 2026, 03:33 PM UTC
We are aware of and urgently investigating an issue with AdBook system performance. Users may experience errors when trying to access the system. Further updates will be provided here as soon as they are available. Please contact us at support.fattail.com with questions.
- investigating Sep 30, 2026, 03:39 PM UTC
We are continuing to investigate this issue.
- identified Sep 30, 2026, 04:01 PM UTC
We have identified the issue and AdBook system access has been restored. Users may still experience issues with accessing DQT. Our engineering team is working to resolve that issue. Further updates will be provided here as soon as they are available. Please contact us at support.fattail.com with questions.
- identified Sep 30, 2026, 04:12 PM UTC
We have identified the issue and AdBook system access has been restored. Users may still experience issues with accessing DQT and other reporting. Our engineering team is working to resolve that issue. Further updates will be provided here as soon as they are available. Please contact us at support.fattail.com with questions.
- monitoring Sep 30, 2026, 05:03 PM UTC
The issue with DQT and other reporting has been fixed. We are continuing to monitor for any further issues.
- resolved Oct 01, 2026, 09:19 PM UTC
This issue has been resolved. A postmortem of this incident will be posted once available. Please contact us at support.fattail.com with questions.
- postmortem Oct 14, 2026, 10:15 PM UTC
### **Summary of Impact** On September 30, 2026, at 8:17 AM PT, an AdBook deployment of v4.10.01 intended for the sandbox \(SBX\) environment was unintentionally run against Production. The deployment shut down all environments, and AdBook was unavailable to all customers for 25 minutes. Access was restored at 8:42 AM PT. Some reporting and inventory functions stayed degraded until 2:24 PM PT, when full functionality was restored. No customer data was lost, and no customer action is required. Summary of events: * 8.17 am PT - v4.10.01 deployment begins on Production; all environments shut down * 8.22 am PT - Incident communication protocols begin * 8.24 am PT - v4.09.03 redeployed; application access restored * 8.33 am PT - Customers notified of incident via Status page * 9.30 am PT - DQT \(Data Query Tool\) reporting restored * 1.19 pm PT - Inventory availability validation restored * 2.24 pm PT - Inventory jobs that failed during the incident reprocessed; full functionality restored ### **Root Cause** While deploying v4.10.01 to SBX, the Engineering team hit deployment failures. During troubleshooting, the deployment pipeline's target environment was unintentionally changed from SBX to Production. Two gaps in tooling and process allowed this: * The environment selector's clickable area extends well beyond its visible controls, so a click elsewhere on the page can change the target without any visible indication. * The pipeline has no confirmation step specific to Production, so nothing prompted a final check of the target before services stopped. The extended recovery after the application was rolled back was a result of.v4.10.01 including database changes that aren't backward compatible with v4.09.03. ### **Resolution** The Engineering team redeployed v4.09.03 - the current production version - as the fastest and lowest-risk way to restore access. Application access was restored at 8:42 AM PT. Redeploying the application did not reverse the database changes made by v4.10.01.DQT reporting and inventory availability validation remained affected because database artifacts required by v4.09.03 had been removed or changed during the v4.10.01 deployment. The team restored these objects as they were identified and reprocessed the affected inventory jobs, completing recovery at 2:24 PM PT. ### **Mitigation and Next Steps** To reduce risk of a similar incident we are adding a mandatory production-intent confirmation gate. The gate and subsequent confirmation will prominently identify key deployment parameters \(such as the target environment, release version\), and require explicit acknowledgment before any services are stopped or database changes are applied. Second-person approval will be added as an additional safeguard, taking staff availability and emergency recovery requirements into account.
Your product runs on more than FatTail
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services