Resolved after 31 minutes, affecting Publish/Subscribe Service and North America Points of Presence, on June 10, 2026.
## Problem Description, Impact, and Resolution At 19:50 UTC on June 10, 2026, we observed a small fraction of publishes originating from US-EAST-1 failing to replicate to subscribers globally. We removed the degraded publisher pod from service and the issue was resolved at 21:21 UTC on June 10, 2026. The root cause of the incident was triggered by a single process that fell into a degraded state where it continued receiving inbound traffic and passing health checks, but traffic sent outbound from the process was failing at an abnormally high rate. Our automated health check/recovery system did not auto-detect and replace the degraded process because its health check API reported itself as healthy. ## Mitigation Steps and Recommended Future Preventative Measures To prevent a similar issue from occurring in the future, we are improving the data within our health check APIs to return more complete performance metrics over a rolling time window. We are enhancing the issue detection logic to detect more patterns that infer process failure, even if the process itself is reporting as healthy.
Affected components
Update timeline
- investigating Jun 10, 2026, 08:14 PM UTC
Starting at 17:00 UTC on June 10, a subset of publishes originating from the North America POP failed to replicate to subscribers globally. PubNub Technical Staff is investigating, and more information will be posted as it becomes available.
- monitoring Jun 10, 2026, 08:27 PM UTC
The PubNub Technical Staff identified that failures are limited to publishes from US-East and has identified, have applied a fix, and are now monitoring the results. If you are experiencing issues that you believe to be related to this incident, please report the details to PubNub Support ([email protected]).
- resolved Jun 10, 2026, 08:45 PM UTC
Services have returned to normal. A root cause analysis will be published in the coming days. If you believe you were impacted and would like to speak with us, please report impact to [email protected].
- postmortem Jun 15, 2026, 11:58 PM UTC
## Problem Description, Impact, and Resolution At 19:50 UTC on June 10, 2026, we observed a small fraction of publishes originating from US-EAST-1 failing to replicate to subscribers globally. We removed the degraded publisher pod from service and the issue was resolved at 21:21 UTC on June 10, 2026. The root cause of the incident was triggered by a single process that fell into a degraded state where it continued receiving inbound traffic and passing health checks, but traffic sent outbound from the process was failing at an abnormally high rate. Our automated health check/recovery system did not auto-detect and replace the degraded process because its health check API reported itself as healthy. ## Mitigation Steps and Recommended Future Preventative Measures To prevent a similar issue from occurring in the future, we are improving the data within our health check APIs to return more complete performance metrics over a rolling time window. We are enhancing the issue detection logic to detect more patterns that infer process failure, even if the process itself is reporting as healthy.
Your product runs on more than PubNub
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services