Resolved after 43 minutes, affecting API and Chrome and 1 more, on June 23, 2025.
The root cause of the downtime and slowness from the API was a new database query that we deployed a few days ago. Once it had some traffic we started noticing slow queries coming from the new query. It took us a about an hour to track everything down, and we quickly reverted the code that added the new query. We will continue to monitor things but as of right now the system is stable.
Affected components
Update timeline
- resolved Jun 23, 2025, 03:59 PM UTC
We are continuing to monitor an issue which caused most Happo jobs and API calls to fail. We are seeing things recover right now and will keep an eye on this to make sure we're not regressing again. Issue started at 15:00 UTC and was ongoing until 15:43 UTC.
- postmortem Jun 23, 2025, 04:27 PM UTC
The root cause of the downtime and slowness from the API was a new database query that we deployed a few days ago. Once it had some traffic we started noticing slow queries coming from the new query. It took us a about an hour to track everything down, and we quickly reverted the code that added the new query. We will continue to monitor things but as of right now the system is stable.
Your product runs on more than Happo
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services