Honeycomb.io Outage History

Started Apr 15, 2026, 07:04 PM UTC· resolved Apr 15, 2026, 07:25 PM UTC

1. identified Apr 15, 2026, 07:04 PM UTC

   Activity Log is experiencing a delay in processing events. We are working on remediating the issue.

2. resolved Apr 15, 2026, 07:25 PM UTC

   Activity Log is ingesting events again. We had no loss in data.

Read the full incident report →

Started Apr 14, 2026, 09:36 PM UTC· resolved Apr 14, 2026, 10:37 PM UTC

1. identified Apr 14, 2026, 09:36 PM UTC

   Activity Log events may be delayed, starting from around 19:55UTC. We're actively working on remediation.

2. monitoring Apr 14, 2026, 10:08 PM UTC

   A fix has been implemented and we are monitoring recovery.

3. resolved Apr 14, 2026, 10:37 PM UTC

   All activity log data is now live and up to date.

Read the full incident report →

Started Apr 10, 2026, 01:41 PM UTC· resolved Apr 10, 2026, 05:24 PM UTC

1. identified Apr 10, 2026, 01:41 PM UTC

   We have confirmed reports of some teams getting issues ingesting their data. We have identified a probable source behind this behavior and are currently trying to correct it.

2. identified Apr 10, 2026, 01:46 PM UTC

   We have further narrowed down that only teams on Classic environments are currently seeing issues. We are figuring out mitigation mechanisms at the moment.

3. identified Apr 10, 2026, 01:55 PM UTC

   We've identified a manual fix and are deploying that by hand to each affected environment, while also investigating a systemic solution.

4. monitoring Apr 10, 2026, 02:19 PM UTC

   We've applied the manual fix to nearly all affected datasets (less than 1% remaining). The systemic solution is still in progress.

5. monitoring Apr 10, 2026, 02:32 PM UTC

   The manual fix has been applied to all environments.

6. monitoring Apr 10, 2026, 03:21 PM UTC

   We are continuing to work on the systemic solution and monitor. We'll leave the incident open until we're able to confirm the issue won't recur.

7. resolved Apr 10, 2026, 05:24 PM UTC

   We can confirm that the fix was applied uniformly to all impacted and potentially impacted environments; no teams should see ingest errors related to this issue anymore.

Read the full incident report →

Started Apr 09, 2026, 05:43 PM UTC· resolved Apr 09, 2026, 07:00 PM UTC

1. resolved Apr 09, 2026, 05:43 PM UTC

   We have detected and remediated an issue around our SLO processing pipeline. From 17:00 to 17:15 UTC, events were being delayed, and alerts may likewise have been a few minutes late. Processing has caught up and is currently within normal operational ranges.

Read the full incident report →

Started Apr 09, 2026, 11:52 AM UTC· resolved Apr 10, 2026, 07:10 PM UTC

1. identified Apr 09, 2026, 11:52 AM UTC

   We have identified the cause and are working to resolve the issue.

2. identified Apr 09, 2026, 08:24 PM UTC

   We have identified an issue with triggers and SLOs and are actively investigating. Customers may experience delays when creating or updating triggers and SLOs. A fix for the activity log delay has been implemented and is in the process of recovering. Customers may continue to see activity log unavailability while this completes.

3. identified Apr 09, 2026, 09:03 PM UTC

   We have identified an issue with triggers and SLOs and have implemented a fix. Customers should no longer see degraded performance for triggers or SLOs. A fix for the activity log delay has been implemented and is in the process of recovering. Customers may continue to see activity log unavailability while this completes.

4. identified Apr 09, 2026, 11:43 PM UTC

   SLOs experienced a period of additional processing delays. A fix has been deployed and recovery is ongoing.

5. identified Apr 10, 2026, 12:05 AM UTC

   SLO processing has fully recovered.

6. resolved Apr 10, 2026, 07:10 PM UTC

   All activity log data is now live and up to date.

Read the full incident report →

Started Apr 09, 2026, 08:58 AM UTC· resolved Apr 09, 2026, 09:18 AM UTC

1. resolved Apr 09, 2026, 10:12 AM UTC

   We experienced a partial querying outage from approximately 8:58 - 9:18 UTC. This outage may have resulted in some Trigger and SLO alert failures for the duration of the outage. The cause of the outage has been identified and mitigated.

Read the full incident report →

Started Mar 18, 2026, 06:07 PM UTC· resolved Mar 18, 2026, 06:18 PM UTC

1. investigating Mar 18, 2026, 06:07 PM UTC

   We are currently investigating this issue.

2. identified Mar 18, 2026, 06:10 PM UTC

   We have identified the issue and are working on a fix.

3. resolved Mar 18, 2026, 06:18 PM UTC

   The issue has been identified and we have shipped a fix.

Read the full incident report →

Started Mar 12, 2026, 11:55 AM UTC· resolved Mar 12, 2026, 01:39 PM UTC

1. identified Mar 12, 2026, 11:55 AM UTC

   We are investigating elevated error rates affecting queries in our US1 region. Our team is actively working on a resolution. We will provide updates as we learn more.

2. identified Mar 12, 2026, 01:17 PM UTC

   We have rolled back a code change to our query engine, but our triggers service has not recovered. We are investigating.

3. resolved Mar 12, 2026, 01:39 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 10, 2026, 01:50 PM UTC· resolved Mar 10, 2026, 02:14 PM UTC

1. investigating Mar 10, 2026, 01:50 PM UTC

   We are currently seeing elevated errors in connecting to some of our databases, which results in low amounts of Ingest traffic being rejected. We are looking into correcting the situation.

2. monitoring Mar 10, 2026, 01:56 PM UTC

   We have stabilized the cluster and are monitoring the state while investigating to understand what triggered the degradation. Overall, 0.04% traffic was rejected with an error due to this degradation during the 20 minute window we were impaired.

3. resolved Mar 10, 2026, 02:14 PM UTC

   This incident is now resolved. We are also noting that some other parts of honeycomb (the query UI) may have seen spurious failures as well during the impacted period (13:22—13:44 UTC).

Read the full incident report →

Started Feb 18, 2026, 10:10 PM UTC· resolved Feb 18, 2026, 10:40 PM UTC

1. resolved Feb 18, 2026, 11:07 PM UTC

   Between 22:10 and 22:40 UTC on Feb18, 2026 Honeycomb experienced a partial query outage in our EU environment. The incident has been resolved, and we are continuing to investigate the contributing factors.

Read the full incident report →

Started Feb 10, 2026, 08:18 PM UTC· resolved Feb 10, 2026, 08:30 PM UTC

1. investigating Feb 10, 2026, 08:18 PM UTC

   We are aware of elevated errors in trigger evaluations and queries, and are working to resolve it

2. resolved Feb 10, 2026, 08:30 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jan 22, 2026, 11:27 PM UTC· resolved Jan 22, 2026, 11:42 PM UTC

1. investigating Jan 22, 2026, 11:27 PM UTC

   We are currently investigating what appears to be a transient issue related to querying and triggers

2. resolved Jan 22, 2026, 11:42 PM UTC

   This appears to have been a minor transient network issue; we do not expect this to recur

Read the full incident report →

Started Jan 15, 2026, 03:46 PM UTC· resolved Jan 15, 2026, 03:46 PM UTC

1. resolved Jan 15, 2026, 03:46 PM UTC

   We observed a period of increased latency accompanied by a higher-than-normal error rate in some event ingest requests in the US. The service has since stabilized and is operating normally. We are continuing to monitor the system and are evaluating improvements to reduce the impact of similar conditions in the future. We apologize for the disruption and appreciate your patience.

Read the full incident report →

Started Jan 06, 2026, 10:14 PM UTC· resolved Jan 07, 2026, 07:54 PM UTC

1. identified Jan 06, 2026, 10:14 PM UTC

   We are seeing a spike in querying errors and the Activity Log has delayed ingest of new activity. We have identified the issue and are working to resolve

2. identified Jan 06, 2026, 10:37 PM UTC

   We have resolved the issue which impacted querying but are still working to get Activity Log up to date

3. identified Jan 06, 2026, 11:05 PM UTC

   We are currently expecting to lose the last three hours of data for the beta activity log feature as part of remediation of this issue.

4. identified Jan 07, 2026, 01:17 AM UTC

   We have restored basic activity log functionality, though some delays remain, and we will analyze further tomorrow during business hours.

5. identified Jan 07, 2026, 07:53 PM UTC

   We have stabilized activity log. We have confirmed activity log entries lost from 4:54pm-7:04pm ET in the US and 9:43pm-11:55pm GMT in the EU. Event ingest was unimpacted. All systems are currently operating normally

6. resolved Jan 07, 2026, 07:54 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Dec 05, 2025, 08:49 PM UTC· resolved Dec 17, 2025, 03:47 PM UTC

1. investigating Dec 05, 2025, 08:49 PM UTC

   We are currently investigating this issue.

2. investigating Dec 05, 2025, 09:19 PM UTC

   A subset of customer environments may see higher than usual error rates when sending events to api.eu1.honeycomb.io, and notifications for SLOs and Triggers may be delayed for that subset. We are continuing to investigate the issue.

3. investigating Dec 05, 2025, 10:33 PM UTC

   We have identified that 0.23% of datasets are fully affected, and a larger percentage are seeing intermittent ingestion and query failures (500s at the API level). We are also investigating a replication error in our ingestion pipeline.

4. investigating Dec 05, 2025, 11:46 PM UTC

   We are continuing the investigation after business hours to stabilize ingest and determine what work will be needed to fully recover. Known impact will be updated here as the situation changes. We will post at least one more update this evening.

5. investigating Dec 06, 2025, 12:04 AM UTC

   Ingest, querying, SLOs, and Trigger alerting is back to normal. Activity Log is still impacted, we have identified the cause and are working to resolve. We are still evaluating the scope of the outage for ingest and expect to have a full answer for that posted during US business hours on Monday. We will post at least one more update this evening.

6. investigating Dec 06, 2025, 07:30 AM UTC

   Automated systems are still working to catch up and Activity Log will remain offline until that that completes. We will post another update Saturday whether or not the Activity Log outage has been remediated.

7. investigating Dec 06, 2025, 03:19 PM UTC

   We have temporarily disabled event ingestion for the EU Region, in service of restoring full functionality to our EU Kafka fleet. External API access will also be disabled. Additionally, during the ongoing outage, our Service Level Objectives feature has been impacted and down since 12:30 PM Pacific time on Friday, December 5th. SLO data will not be correct until our systems catch up and we rebuild the cache

8. investigating Dec 06, 2025, 04:15 PM UTC

   External API access is restored, event ingestion is still impacted

9. investigating Dec 06, 2025, 07:20 PM UTC

   We are continuing to investigate issues with the Kafka cluster.

10. investigating Dec 06, 2025, 09:23 PM UTC

    We are continuing to investigate issues with the Kafka cluster. We will continue providing updates every 2 hours unless there are significant changes.

11. identified Dec 06, 2025, 09:51 PM UTC

    We have resumed ingest to api.eu1.honeycomb.io as we were able to restore partial service to the Kafka cluster.

12. identified Dec 06, 2025, 10:08 PM UTC

    Ingest has been fully re-enabled, and SLOs and Triggers are now up-to-date.

13. identified Dec 07, 2025, 12:10 AM UTC

    We have gotten the Kafka cluster to a stable-enough state, however it will still require remediation before the Activity Log is fully functional and it is back to a level of service sustainable until business hours. We will provide another update in 2 hours or as the situation changes.

14. identified Dec 07, 2025, 02:18 AM UTC

    The Kafka cluster is still operational, but is not back to full capacity and resiliency. Work will continue tomorrow to ensure sufficient capacity, as well as during business hours to resume full Activity Log functionality. We will provide more updates tomorrow and as-needed if there are any changes to availability.

15. identified Dec 07, 2025, 08:23 PM UTC

    Our Kafka cluster is still operational and we are continuing to work towards restoring full reliability. We will resume work during business hours to restore full Activity Log functionality. We will continue to update as work progresses.

16. identified Dec 08, 2025, 09:52 PM UTC

    The Activity Log is operational but is experiencing ingestion delays, which means recent activity may not be reflected immediately. We are working to restore full functionality. Currently, a small subset of data for certain customers may not be queryable due to a partition needing repair. We are working to return the data to queryability. After investigation of the impact starting on Friday to SLO and Trigger functionality, we're providing the update below with the details. Overview SLOs and Triggers were affected by Kafka and ingest outages beginning Friday, December 5 at 8:40 PM UTC. Trigger evaluations are now behaving normally except for some baseline triggers. SLOs with evaluation periods that include missing ingest or are on the affected partition will reflect that currently unqueryable data in their ongoing calculations. SLOs - No burn alerts were fired between December 5, 2025 at 8:40 PM UTC and December 6, 2025 at 7:15 PM UTC due to a processing delay. - Now that ingest is restored, burn alerts should behave normally again minus a subset that are also impacted by the small amount of currently unqueryable data. Triggers - When ingest was down on December 6, no events were received, so all EU Trigger evaluations were impacted. - Baseline Triggers with lookback windows that query data in the affected partition returning will also be affected by the small amount of currently unqueryable data. - Non-baseline triggers (which can only look back 24 hours) should behave normally going forward. We will provide an update at the end of the workday (5PM PT) or as the situation changes.

17. identified Dec 09, 2025, 01:02 AM UTC

    We continue to work on restoring full functionality to Activity Log and repair the partition causing certain customers to have subsets of unqueryable data. We will next provide an update during US business hours Tuesday.

18. identified Dec 09, 2025, 02:01 AM UTC

    We fixed the issue that was causing subsets of temporarily missing data in queries against certain partitions. We continue to work on restoring full functionality to Activity Log. We'll provide another update during US business hours Tuesday.

19. identified Dec 09, 2025, 02:05 AM UTC

    We are continuing to work on a fix for this issue.

20. identified Dec 09, 2025, 10:40 PM UTC

    We continue to work toward fully restoring the Activity Log. While functionality has not yet improved, we have a clear path to recovery, and remediation work is underway. Our next steps involve coordinated infrastructure changes that will allow us to resume Activity Log processing. We’ll provide another update during US business hours Wednesday.

21. identified Dec 10, 2025, 05:34 PM UTC

    We continue to work toward fully restoring the Activity Log. While functionality has not yet improved, we’ve made progress on the underlying issue and have begun the next phase of remediation. Our engineering team is actively working through the remaining steps required to resume Activity Log processing.

22. identified Dec 11, 2025, 01:08 AM UTC

    We continue to work toward fully restoring the Activity Log. While functionality has not yet improved, we’ve made progress on the underlying issue and have begun the next phase of remediation. Our engineering team is pausing for the evening and will resume working through the remaining steps required to resume Activity Log processing during US business hours tomorrow. We will provide another update tomorrow morning.

23. identified Dec 11, 2025, 06:02 PM UTC

    We continue to work toward fully restoring the Activity Log. We’ve made meaningful progress on the underlying issues and continue to work through our remediation plan. Our engineering team is actively working through the next steps required to restore Activity Log processing.

24. identified Dec 12, 2025, 12:09 AM UTC

    We continue to work toward restoring the Activity Log. We’ve made meaningful progress on the underlying issues, but have had to pivot on our remediation plan due to previously unknown hard retention limits. Our engineering team is actively working through the next steps required to restore Activity Log processing.

25. identified Dec 12, 2025, 11:20 PM UTC

    We continue to work toward restoring the Activity Log. We’ve made meaningful progress on the underlying issues, but have had to pivot on our remediation plan due to previously unknown hard retention limits. Our engineering team is working through the next steps required to restore Activity Log processing. Next updates to come during Monday US business hours

26. identified Dec 16, 2025, 01:52 AM UTC

    As of today, our old EU cluster is still functional, yet in a bad state. We do not consider this outage over. However, after running a full evacuation in a pre-production cluster, we are now ready to run a production version of it. This will only be in the impacted EU cluster. We are planning to do it on 6pm UTC December 16th. We expect no ingestion downtime, although we will expect delays on processing events and alerts, will suffer minor data loss in our Service Maps feature, and lose 11 days worth of Activity log events starting the night of December 5th. We will provide updates throughout the migration.

27. monitoring Dec 16, 2025, 08:17 PM UTC

    We successfully created and migrated over to a new kafka cluster in the EU. We have resumed consuming events for our SLO and Service Maps features. No further data loss is expected for service maps. SLOs are caught up and are fully synchronized. Activity logs are still delayed and expect to be caught up in about 5 hours. We consider this incident to be mitigated, but will continue to monitor until Activity logs are caught up.

28. resolved Dec 17, 2025, 03:47 PM UTC

    As of 23:23UTC on December 16, the Activity Log has fully caught up and has remained up to date. All systems are now operational.

29. postmortem Dec 18, 2025, 07:09 PM UTC

    This incident started on December 5th, and is one of the longest in Honeycomb history, having been actively worked on and closed only on December 17th. Due to its impact and duration, we wanted to offer a partial and preliminary report to explain, at a high level, what happened. On December 5th, at 20:23 UTC, our Kafka cluster suffered a critical loss of redundancy. Our Kafka cluster contains multiple topics, including all telemetry events submitted by Honeycomb users, the rematerialization of state changes into the activity log, and multiple metadata topics used by Kafka to manage its own workloads. This led multiple partitions leaderless and by 20:35 UTC, we were getting alerts that roughly a quarter of our usual event topic partitions were unable to accept writes. For most Honeycomb customers, this does not result in ingest outages, because traffic gets redirected to other partitions environments can be assigned to. However, we noted some teams for whom all their assigned partitions fit within the impacted set, and for whom all ingest was down \(0.23% of all datasets were impacted\). By 21:30 UTC, we had identified impacted teams, were working on a traffic reassignment script, and thought other features \(such as SLO evaluation\) were unimpacted. However, at 22:08 UTC, we noticed, through the noise of all the alerts that had been going on, that our ingestion fleet had been in overload protection since 20:38 UTC, impacted everyone more broadly by returning errors on `api.eu1.honeycomb.io` that would have led to dropped events. Stabilization work was done and 6 minutes later \(22:14 UTC\), ingestion was stabilized, yielding roughly 1h25 of increased ingest error rates on `api.eu1.honeycomb.io` for everyone. At 01:30 UTC on Saturday, December 6th, our responders had managed to force new leader elections on all impacted ingestion partitions, which re-established working traffic internally on all but one of them, which remained shuttered in read-only mode. By then, our brokers were severely imbalanced, we attempted to tweak retention settings, and monitored disk usage to come back during the day to repair the remaining broken partitions. By 11:00 UTC, our disk usage on Kafka brokers reached a threshold where it became necessary to devise and perform emergency operations. We noticed that our metadata partitions, some of which handle the tracking of Kafka offloading its storage to S3 and its autobalancing features, were still leaderless, and we thought this could be the problem. In fixing them, we also repaired consumer group metadata topics, which revealed, at 14:27 UTC, that our SLO product had been partially stuck since the start of the outage. By having broken consumer groups topic partitions, our event consumers for SLOs had been working fine on some partitions but were fully stalled on others, despite reporting healthy—they were idling as “online” but not seeing that they were late. They caught up and by 19:15 UTC, they were backfilled and brought back to normal. However, before then, the issue with Kafka disk storage got worse. We feared that with full disks, the entirety of our Kafka cluster would reach an irreparable state \(where the only way to free space is to do dangerous untested operations to free storage data\), and at 15:09 UTC, we instead chose to turn off ingest entirely with less than 5% of disk space left. Ingest would turn itself off if the disk were to be filled, so we elected to keep recovery simpler by cutting traffic off a few minutes ahead of time. We quickly shuffled to try and add new Kafka brokers and partitions, which wouldn’t be full, to shift traffic onto them – but before we were done, we thought of disabling our tiered storage. Our Kafka brokers store only a few hours of data locally \(“hotset” data\), and tier out a longer retention period \(2-3 days\) to S3. All data written to Kafka is quickly replicated to our storage engine and longer retention is only kept for disaster recovery. We had tried repairing topics, and reducing the size of our hotset for many hours before, but nothing had the desired effect. Our theory was that tiering data was broken anyway, and we were stuck waiting for the offload of data that wouldn’t happen. At 21:32 UTC, we disabled tiered storage altogether and most of the disk space was recovered. We abandoned the cluster expansion, which was minutes from completion, and at 21:49 UTC, Ingestion of customer data was turned back on. In total, we were unable to accept traffic for 6 hours and 23 minutes. Our responders were online trying to clean up issues further until roughly 5 am UTC on December 7, and they disbanded to come back during the day. Sunday was mostly spent stabilizing the emergency configuration changes that were made, and the response team, who had worked around the clock since Friday night, was able to take some resting time while our EU cluster was mostly healthy again, aside from the activity log and one of our storage partitions that was unavailable still for querying. On Monday December 8, employees who weren’t on call over the weekend took over stabilization work. Efforts went to successfully restore the partial querying outage of roughly 1/40th of our storage data. No significant progress was made on improving our activity log feature’s storage, and instead the team increased the storage retention of its changesets to 7 days \(the max allowed in our storage engine\). Our thinking was that once the Kafka cluster would be stable and these partitions fixed, we could simply replay events and insert everything back. Tuesday December 9 was spent trying to further stabilize our Kafka cluster and to turn on some features, but we were able to make little progress in salvaging partitions, and started doing corrective work while our Kafka experts tried to see what they could do with the many still damaged metadata and internal topics that were less critical to staying up, but still very important. On December 10, knowing we only had a few days to salvage Activity Log data, we decided to stop trying to save its Kafka topics, and to instead recreate them. At 19:25 UTC, we found that deletion operations fail, and that our Kafka cluster’s control plane no longer lets us do any manipulation whatsoever aside from listing topics. We can’t delete, create, describe, or mutate any of them. They are fine accepting and transmitting events, but we are essentially unable to administer the cluster anymore. What we realized at that point in time is that our initial cluster outage severely damaged multiple internal partitions used to manage the cluster itself; as we cut off tiered storage \(and as it aged out as well\), the damage became more or less irreversible. We then considered our chances of salvaging the cluster to be rather low, and feared that most small changes \(such as assigning a new controller\) could collapse the cluster and result in a large outage with data loss. Internal details of some of our applications are relevant here: our storage engine is tightly coupled to Kafka’s own event offsets. To prevent data consistency issues, it will refuse to “roll back” to an older offset, which could indicate a misconfiguration or some other problem that could lead to re-reading, duplicating, or losing data. As such, just “resetting” the cluster was not doable at this point in time, without adding a lot of infrastructure, or writing emergency fixes that significantly change our storage engine’s boot sequences and safety checks. Later, still on December 10, we decided to split efforts into a\) keep trying to save our cluster to the extent it doesn’t risk its stability, and b\) start an emergency migration project that requires figuring out how to modify our storage engine, infrastructure, and multiple services to tolerate moving from the damaged Kafka cluster to a new one. We also prepared for multiple contingencies in case the current cluster were to die before the migration was ready to run. December 11 and December 12 were spent working on this migration at a high priority. Likewise, during that time, our low-risk efforts to salvage the Kafka cluster yielded no great results. As we neared the end of Friday December 12, our retention of activity for the Activity Log also came to an end. In deciding between causing a major outage to rush an evacuation of our Kafka cluster before we were ready or losing days of Activity Log data to wait for a safer migration path, we chose the latter. On Monday December 15, we managed to boot a new Kafka cluster with new infrastructure, and had all the fixes required for all services to do a migration, along with runbooks from every team involved. We decided to do a “dress rehearsal” in a pre-production environment, where we tried a full evacuation from a functional Kafka cluster to a brand new one, finding what the tricky parts were and making sure that if we were to damage or lose data, it would be internal telemetry, and not customer data. Fortunately, everything went well, and on Tuesday December 16, we ran the full emergency evacuation. Running it required doing a switchover of all ingest from the old damaged Kafka cluster to the new Kafka cluster. We started by switching components that write data into the cluster first \(producers\), letting the consumers catch up on all topics and partitions. We then started migrating our consuming services, first with the query engine, and then the other consumers. This order of operation ensured that we would not corrupt, damage, or miss any data, but forced us to have delays on alerting and the freshness of query data. This happened without major issues. During the migration, we believe our trigger data was stale and runs using incomplete data for roughly 30 minutes, if we average out most partitions. SLOs were delayed for a bit more than one hour, and service maps will have flat out skipped that hour as well. Activity Log data was lost between December 5 at 20:23 UTC and December 9 at 23:45 UTC. These are added to the 10 days during which Activity Log data was unavailable, the ingestion issues of December 5, the 6h23 minutes of full ingest outage on December 6, and the 18h of delays on SLO processing between December 5 and 6. At this point in time, we have fully mitigated this incident and developed new processes that promise better readiness to respond if similar outages were to happen again. A more in-depth review will be published in a few weeks, in January. This has been a significant outage, and we will need some time to do a proper analysis of it. FEBRUARY 2025 EDIT: The public incident report has been published at [https://www.honeycomb.io/blog/incident-report-exercises-cleanups-and-evacuations](https://www.honeycomb.io/blog/incident-report-exercises-cleanups-and-evacuations)

Read the full incident report →

Started Nov 25, 2025, 10:51 PM UTC· resolved Nov 27, 2025, 12:14 AM UTC

1. identified Nov 25, 2025, 10:51 PM UTC

   We have identified the cause and are working to resolve the issue.

2. identified Nov 26, 2025, 05:02 AM UTC

   We are continuing to work on a fix for this issue.

3. monitoring Nov 26, 2025, 05:07 AM UTC

   Due to issues with replication streams, the activity log will have a gap in its data. We estimate that a little over 6 hours of content will be missing, from 20:27 UTC until 2:43 UTC. We do not expect missing data to be backfilled. No other Honeycomb data or feature has been impacted. We are currently working on catching up on late data after this window.

4. monitoring Nov 26, 2025, 05:24 PM UTC

   We are continuing to work on catching up late data from yesterday night's replication issues and expect to complete this process in approximately 6 hours.

5. resolved Nov 27, 2025, 12:14 AM UTC

   As of 23:00 UTC, all systems are back to synchronized.

Read the full incident report →

Started Oct 20, 2025, 02:16 PM UTC· resolved Oct 21, 2025, 03:41 PM UTC

1. monitoring Oct 20, 2025, 02:16 PM UTC

   Due to constrained EC2 instance capacity in the AWS us-east-1 region, we are choosing to allocate the capacity we have to incoming events and telemetry. As such, customers may see over 5 minute delays in our processing of - SLOs - Service Maps We do not expect a degradation of our core ingest -> query flow, and we do not expect triggers to be impacted

2. monitoring Oct 20, 2025, 03:14 PM UTC

   SLO processing has recovered. Service Maps continues to be degraded but historical data can be queried.

3. monitoring Oct 20, 2025, 03:30 PM UTC

   The networking issues in us-east-1 is affecting our query engine - customers may see errors or delays when running queries as a result.

4. monitoring Oct 20, 2025, 05:39 PM UTC

   We’re currently observing that querying is seeing signs of recovery. We are continuing to monitor the situation

5. monitoring Oct 20, 2025, 08:06 PM UTC

   AWS is starting to show signs of recovery. Querying remains partially impacted and may take longer to return results. We are continuing to monitor the situation.

6. monitoring Oct 20, 2025, 11:08 PM UTC

   Service maps are up and running and errors have resolved. Querying, triggers and SLOs are continuing to improve and customers may see delays as functionality continues to improve. We are continuing to monitor the situation.

7. monitoring Oct 21, 2025, 12:11 AM UTC

   Honeycomb core functionality is operational. Service maps are up and running and errors have resolved. Querying, triggers and SLOs are continuing to improve and customers may see delays as functionality continues to improve. This is our final update for the night unless the situation degrades. We will continue to monitor the situation.

8. resolved Oct 21, 2025, 03:41 PM UTC

   This incident has been resolved. At no point did we lose any customer data that hit our load balancers. Querying has been stable for hours, and all features that were degraded are functional.

Read the full incident report →

Started Oct 20, 2025, 07:24 AM UTC· resolved Oct 20, 2025, 10:04 AM UTC

1. identified Oct 20, 2025, 07:24 AM UTC

   We are currently experiencing degraded performance for queries that access older data stored in cold storage. Due to an ongoing issue with AWS, some queries over historical data may fail to complete successfully or take longer than usual. This includes queries for SLOs and Triggers. Queries over recent or “hot” data are not impacted. Our team is closely monitoring the situation. We’ll provide updates here as more information becomes available.

2. identified Oct 20, 2025, 08:14 AM UTC

   We are currently experiencing degraded performance for queries. Due to an ongoing issue with AWS, some queries over historical data may fail to complete successfully or take longer than usual. This includes queries for SLOs, Triggers, and Traces. Our team is closely monitoring the situation. We’ll provide updates here as more information becomes available.

3. identified Oct 20, 2025, 08:46 AM UTC

   We are currently experiencing degraded performance. Due to an ongoing issue with AWS, some queries over historical data may fail to complete successfully or take longer than usual. This includes trace queries. SLO and Trigger alerts may be missed or delayed. Queries may return results, but those results may not be saved, meaning that loading queries that other people have run may fail. After the problem resolves, queries may need to be rerun. Our team is closely monitoring the situation. We’ll provide updates here as more information becomes available.

4. monitoring Oct 20, 2025, 09:49 AM UTC

   We are beginning to see recovery across our systems. We will continue to monitor the situation in case of any regressions.

5. resolved Oct 20, 2025, 10:04 AM UTC

   All affected systems have fully recovered.

6. postmortem Oct 22, 2025, 10:52 PM UTC

   At 07:05 UTC on October 22, 2025, Honeycomb Platform On-Call received a page indicating that our [End to End tests](https://www.honeycomb.io/blog/how-honeycomb-uses-honeycomb-part-3-end-to-end-failures) that run every 5 minutes had failed. The responding engineer logged in to find multiple systemic failures, involving querying, SLO evaluation, and noted the lack of ability to log in to the AWS console. 6 minutes later at 07:11 UTC, AWS declared an outage affecting `us-east-1`. This outage would not be declared resolved until 22:53 UTC, nearly sixteen hours later. Even at the time of AWS-indicated resolution, services continued to be impacted by throttling imposed to permit system recovery. During the duration of the incident, we remained able to ingest events; **no events that were received by our ingest hosts were dropped due to system failures**. Querying was affected between approximately 13:30 and 14:06 UTC, 15:00 and 16:00 UTC, 16:10 UTC and 16:45 UTC, 17:12 UTC and 17:21 UTC, and from 18:40 to 19:06 UTC, which also created delays in evaluations of Triggers and SLOs during those periods. AWS described the trigger of the outage as being DNS resolution for internal DynamoDB endpoints. Once this DNS resolution issue was handled, issues affecting dependent systems within AWS became visible. The most notable symptoms for Honeycomb were errors while launching instances and failed Lambda invocations. The Lambda failures increased the difficulty of seeing exactly which systems were impacted since under normal operations Honeycomb relies on Lambda to power the query functionality, and EC2 scheduling issues reduced our capacity to keep up with SLO evaluation; however, we were able to maintain the EC2 capacity we already had in order to ensure ingest continued to function. We also had degraded performance on ancillary systems like Service Maps due to reduced overall capacity. Multiple providers we rely on also experienced significant hardship with the AWS outage, limiting our ability to make on-the-fly changes to the running system. Ordinarily, if particular features are inaccessible or otherwise in a problematic state, we have levers to de-activate the features and ensure the stability of the key parts of Honeycomb. We were able to mitigate some aspects of the failure without this, but several of these were unavailable to us in this case - for example, to allow querying to continue in a limited capacity while Lambda was inaccessible. Because of the duration and severity of the incident, we plan on conducting a full incident review with further discussions where we will re-evaluate our Disaster Recovery story, reliance on external vendors, and regional dependence.

Read the full incident report →

Started Oct 08, 2025, 02:26 PM UTC· resolved Oct 08, 2025, 03:24 PM UTC

1. identified Oct 08, 2025, 02:26 PM UTC

   We are currently experiencing issues with our configuration with one of our AI providers. Features such as Query Assistant and our Calculated Field assistant are not functional, and some other parts of Honeycomb Intelligence may be partially degraded at this moment.

2. monitoring Oct 08, 2025, 03:10 PM UTC

   We have made changes to our configuration and errors rates seem to have gone back to normal. We are keeping an eye on things to know the features remain stable.

3. resolved Oct 08, 2025, 03:24 PM UTC

   The system has remained stable, we do believe everything to now be in order.

Read the full incident report →

Started Sep 25, 2025, 12:07 AM UTC· resolved Sep 25, 2025, 01:22 AM UTC

1. investigating Sep 25, 2025, 12:07 AM UTC

   An outage with an upstream service provider is currently affecting Honeycomb's infrastructure. Functionality of the ServiceMaps feature is degraded at this time.

2. resolved Sep 25, 2025, 01:22 AM UTC

   Service Maps functionality has been restored.

Read the full incident report →

Started Aug 28, 2025, 04:30 AM UTC· resolved Aug 28, 2025, 04:30 AM UTC

1. resolved Aug 28, 2025, 04:40 PM UTC

   A PagerDuty outage prevented some Trigger and SLO notifications from reaching your workspaces. The issue has since been resolved, and notification systems are now operating normally. Impact Window: 21:54 PT – 23:27 PT

Read the full incident report →

Started Aug 27, 2025, 03:19 PM UTC· resolved Aug 27, 2025, 03:36 PM UTC

1. investigating Aug 27, 2025, 03:19 PM UTC

   We're see a partial degradation of data liveness, which may impact triggers and SLOs in a small percentage of users

2. resolved Aug 27, 2025, 03:36 PM UTC

   We have determined the cause and implemented a fix. Customers should be seeing normal operations again.

Read the full incident report →

Started Aug 13, 2025, 09:30 AM UTC· resolved Aug 13, 2025, 11:08 AM UTC

1. resolved Aug 13, 2025, 05:10 PM UTC

   For a period of approximately 1h30, from 9:34 UTC to 11:08 UTC, our ingestion of events in the EU region was delayed. All events were eventually processed, but for that duration, triggers, SLOs, and regular queries could have seen missing data for a brief period of time. As we re-balanced traffic loads and scaled the system, ingestion caught up and data has since then been fully available. We are still actively working on improving the scaling margins and behaviors of the EU instance of Honeycomb to better sustain load variations we are observing at this time.

Read the full incident report →

Started Aug 12, 2025, 03:17 PM UTC· resolved Aug 12, 2025, 03:37 PM UTC

1. identified Aug 12, 2025, 03:17 PM UTC

   Data ingestion is currently being delayed in the EU region. All traffic is being accepted, but it may take some time to show up in queries, and may have an impact on trigger-based alerting as well.

2. monitoring Aug 12, 2025, 03:30 PM UTC

   We have load-balanced ingestion traffic to bring back performance under control, and are currently monitoring to make sure it stays that way.

3. resolved Aug 12, 2025, 03:37 PM UTC

   Our load-balancing of ingestion traffic is doing its job and everything is back to normal.

Read the full incident report →

Started Jul 21, 2025, 06:01 PM UTC· resolved Jul 21, 2025, 08:53 PM UTC

1. identified Jul 21, 2025, 06:01 PM UTC

   We are seeing a querying slowdown in the US. We have identified the cause and are working to mitigate

2. monitoring Jul 21, 2025, 07:52 PM UTC

   A fix has been implemented and customers should see query speed returning to normal

3. resolved Jul 21, 2025, 08:53 PM UTC

   This incident has been resolved.

Read the full incident report →