Is Honeycomb.io down?

Canvas is not loading for customers in the US and EU regions and we have identified what we believe to be the cause and are rolling out a fix.

We have deployed the fix and are seeing recovery of normal Canvas functionality.

This incident has been resolved.

From 11:36UTC to 12:43UTC, queries in US1 that read older data returned errors or timed out. Queries over recent data were un-affected, and no telemetry is being lost. The issue has been identified, and a fix is in place. We are currently monitoring the situation to confirm resolution.

The fix is in place and we've confirmed resolution. Queries and SLOs are processing normally.

Between 10:30 and 11:00 AM EDT, we experienced elevated querying failures and slowness. The impact has subsided, and we are currently investigating and monitoring the situation.

Starting at 9:50 AM Central Time on July 31st we received alerts around querying health and latency. The cause was eventually tracked down to a clash of two different factors. First, we have been introducing Lambda Managed Instances \(LMI\) into our query architecture, which behave similar to classic lambdas in many ways and had been introduced as part of work around querying performance improvements with the goal of a better and faster user experience. Unfortunately, a difference between classic lambdas and LMIs we discovered is the 15 minute hard stop of classic lambdas does not exist on LMIs, which means that certain assumptions built into our querying system no longer held. At the time of the incident, our newer Canvas feature was running batches of queries for a customer which should have a self imposed one minute time out. This timeout was not properly communicated back to the lambda, and while with classic lambdas the task would automatically cut off at 15 minutes, the new LMIs were not enforcing this sort of safety mechanism. As a result, some large queries were running for up to 30 minutes, driving up query latency across the board and resulting in errors for users trying to run fresh queries. The system recovered by 10:10 AM CT, 20 minutes later, when the large queries cleared out, but we continued to investigate the cause of the incident. Once we identified this weakness in our querying architecture we began work to fix it and prevent the same incident from occurring again. We have shipped all associated incident follow ups.

We are seeing triggers intermittently failing in the EU region. We are actively investigating the cause.

We are continuing to see issues with triggers and are also noticing issues with querying. We are investigating for the cause of both.

We have identified the source of the issue, applied a temporary fix, and are working on a permanent fix. We continue to monitor the situation.

We are seeing a regression in performance with partial degradation of both triggers and querying. We are working to remediate.

The partial degradation has been resolved for all customers except for those specifically contacted. We are working on mitigation measures so that we can fully restore trigger functionality for everyone.

We are continuing to monitor for any further issues.

The degradation in querying and triggers has been resolved and we're back to full functionality.

We are currently investigating this issue.

Clearing the CDN cache appears to have solved the issue.

This incident has been resolved.