Apideck incident
Delays in webhook event processing with connectors that utilize Virtual Webhooks
Resolved after 1 day, affecting Webhooks, on March 4, 2025.
Last week, we identified an issue in our system that led to delays in webhook event processing and manual sync triggers. A backlog of messages accumulated in our processing queue, causing high resource utilization and degraded performance. **Root Cause** The issue was traced to a bug in our pagination logic when processing data. This caused an unexpected loop, leading to repeated processing attempts and message buildup in our queue. **Resolution** We implemented a series of fixes, including: * Correcting the pagination logic to prevent looping * Optimizing message processing and error handling * Enhancing system monitoring to detect similar issues earlier **Impact** During the incident, some customers experienced delays in receiving webhook events and manually triggering syncs. We have since resolved the issue and taken steps to prevent recurrence. **Next Steps** We are implementing additional monitoring and safeguards to improve system resilience and ensure seamless webhook processing. We appreciate your patience and remain committed to maintaining a reliable experience.
Affected components
Update timeline
- investigating Mar 04, 2025, 07:44 AM UTC
We are currently investigating this issue.
- resolved Mar 05, 2025, 08:23 AM UTC
This incident has been resolved.
- postmortem Mar 06, 2025, 08:26 PM UTC
Last week, we identified an issue in our system that led to delays in webhook event processing and manual sync triggers. A backlog of messages accumulated in our processing queue, causing high resource utilization and degraded performance. **Root Cause** The issue was traced to a bug in our pagination logic when processing data. This caused an unexpected loop, leading to repeated processing attempts and message buildup in our queue. **Resolution** We implemented a series of fixes, including: * Correcting the pagination logic to prevent looping * Optimizing message processing and error handling * Enhancing system monitoring to detect similar issues earlier **Impact** During the incident, some customers experienced delays in receiving webhook events and manually triggering syncs. We have since resolved the issue and taken steps to prevent recurrence. **Next Steps** We are implementing additional monitoring and safeguards to improve system resilience and ensure seamless webhook processing. We appreciate your patience and remain committed to maintaining a reliable experience.
Your product runs on more than Apideck
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services