Resolved after 24 minutes, affecting North America Points of Presence and Asia Pacific Points of Presence and 1 more, on May 16, 2025.
### **Problem Description, Impact, and Resolution** On May 16, 2025 at 09:05 AM UTC, some customers in US and APAC regions experienced elevated error rates and increased latency when using the Presence service. This disruption was caused by an interruption of underlying infrastructure supported by a third-party provider service. This failure also led to delays in resolving the issue, which prolonged the impact. We were able to adjust resource capacity to overcome this and bring the service back to normal. Services were fully restored on May 16, 2025 at 10:23 AM UTC. ### **Mitigation Steps and Recommended Future Preventative Measures** We increased memory on the Presence pods and are working with the third-party provider to address the issues encountered during the incident. To prevent such issues from recurring in the future, we will also work to improve the Presence service to be more resilient to fluctuations in traffic.
Affected components
Update timeline
- monitoring May 16, 2025, 09:59 AM UTC
At 09:05 UTC we detected increased errors and latency for the Presence service in US and AP-North regions. The service has stablized as of 9:30 UTC. Our engineers are monitoring the system.
- resolved May 16, 2025, 10:23 AM UTC
This incident has been resolved.
- postmortem Jun 05, 2025, 09:25 PM UTC
### **Problem Description, Impact, and Resolution** On May 16, 2025 at 09:05 AM UTC, some customers in US and APAC regions experienced elevated error rates and increased latency when using the Presence service. This disruption was caused by an interruption of underlying infrastructure supported by a third-party provider service. This failure also led to delays in resolving the issue, which prolonged the impact. We were able to adjust resource capacity to overcome this and bring the service back to normal. Services were fully restored on May 16, 2025 at 10:23 AM UTC. ### **Mitigation Steps and Recommended Future Preventative Measures** We increased memory on the Presence pods and are working with the third-party provider to address the issues encountered during the incident. To prevent such issues from recurring in the future, we will also work to improve the Presence service to be more resilient to fluctuations in traffic.
Your product runs on more than PubNub
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services