PDF-Generator API Outage History
PDF-Generator API is up right nowPingoru has recorded 49 incidents for PDF-Generator API: 2 in the last 12 months and 0 in the last 90 days.
Show as a table
| Month | Incidents | Hours with an open incident |
|---|---|---|
| April 2026 | 0 | 0 of 720 (0%) |
| May 2026 | 0 | 0 of 744 (0%) |
| June 2026 | 0 | 0 of 720 (0%) |
| July 2026 | 0 | 0 of 744 (0%) |
| August 2026 | 0 | 0 of 744 (0%) |
| September 2026 | 0 | 0 of 720 (0%) |
| October 2026 | 0 | 0 of 744 (0%) |
| November 2026 | 0 | 0 of 19 (0%) |
Incidents, newest first
Major January 6, 2026 Elevated API Errors 29m Document Generation Service, Customer Portal, Editor Service
Started Jan 06, 2026, 08:03 PM UTC· resolved Jan 06, 2026, 08:32 PM UTC
- investigating Jan 06, 2026, 08:03 PM UTC
We're experiencing an elevated level of API errors and are currently looking into the issue.
- identified Jan 06, 2026, 08:27 PM UTC
We have identified the root cause of the issue and are applying a fix.
- resolved Jan 06, 2026, 08:32 PM UTC
We have resoled the issue.
Critical December 22, 2025 Enterprise US and EU Service Interruption 15m Document Generation Service, Document Generation Service, Customer Portal, Customer Portal and 2 more
Started Dec 22, 2025, 09:26 AM UTC· resolved Dec 22, 2025, 09:42 AM UTC
- investigating Dec 22, 2025, 09:26 AM UTC
We're experiencing an elevated level of API errors and are currently looking into the issue.
- investigating Dec 22, 2025, 09:41 AM UTC
We are continuing to investigate this issue.
- resolved Dec 22, 2025, 09:42 AM UTC
This incident has been resolved.
Critical October 20, 2025 Service Disruption Due to Ongoing AWS Outage –
Started Oct 20, 2025, 07:00 AM UTC· resolved Oct 20, 2025, 07:00 AM UTC
- resolved Oct 20, 2025, 11:03 AM UTC
We are currently experiencing service disruptions due to a major AWS (Amazon Web Services) outage which is affecting our systems and functionalities. We sincerely apologize for the inconvenience this may cause. Our team is actively monitoring the situation and working on mitigation steps where possible. Thank you for your patience and understanding. We will keep you informed as we learn more and as the situation evolves.
Major October 2, 2025 Asynchronous Document Generation service failure –
Started Oct 02, 2025, 10:00 PM UTC· resolved Oct 02, 2025, 10:00 PM UTC
- resolved Oct 03, 2025, 05:13 AM UTC
We have resolved the issue affecting our asynchronous document generation service, and all systems are now operating normally. The service disruption began at approximately 1:00 AM EEST and was fully resolved by 7:50 AM EEST. The incident was caused by a failure in our queue database system, which prevented a replica node from being promoted to handle new processing requests.
Critical August 13, 2025 Service Degradation: Intermittent Failures in Document Generation 38m Document Generation Service
Started Aug 13, 2025, 05:15 AM UTC· resolved Aug 13, 2025, 05:53 AM UTC
- identified Aug 13, 2025, 05:15 AM UTC
Our team is currently addressing a service degradation impacting our document generation feature. We understand the importance of this service to your workflow, and our teams are working diligently to restore full functionality. What is happening? Starting at approximately 12:00 AM UTC, we identified an issue causing intermittent failures for document generation requests. Our monitoring indicates that approximately 50% of these requests are currently failing. Our investigation has traced the root cause of performance issues within our Redis cache layer in our primary AWS Region. What are we doing? Our DevOps and Engineering teams are actively working on a resolution. We are currently implementing remediation steps to restore cache stability and bring the service back to full health. We are confident in our team's ability to resolve this matter promptly. We sincerely apologise for the disruption this has caused and appreciate your patience.
- monitoring Aug 13, 2025, 05:50 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Aug 13, 2025, 05:53 AM UTC
This incident has been resolved.
- postmortem Aug 13, 2025, 08:25 AM UTC
### Summary On August 13, 2025, between approximately 02:30 and 08:44 \(UTC\+3\), the PDF Generator API service experienced a severe degradation. During this period, a significant percentage of API requests failed. The issue was resolved by failing over a component of our caching infrastructure, and the service has been stable since. ### Impact * **Service:** PDF Generator API * **Impact:** A high rate of failed API requests for all users. * **Duration:** Approximately 6 hours and 15 minutes of severe degradation. ### Timeline of Events \(UTC\+3\) * **August 12, 22:00:** A low volume of connection errors to our caching layer was first detected by our monitoring systems. The overall service health remained stable with no noticeable impact on users. * **August 13, 02:30:** A sharp increase in connection failures occurred, leading to a severe degradation of the PDF Generator API service. The on-call engineering team was engaged. * **August 13, 02:30 - 08:44:** The engineering team actively investigated the issue. During this time, the service continued to experience a high failure rate. * **August 13, 08:44:** A successful manual failover of a key component in our caching infrastructure was completed. Service functionality was immediately restored. ### Root Cause Analysis The direct cause of the incident was a widespread failure of our application servers to establish connections with our primary caching instance, which runs on Redis in an AWS cluster. Our investigation confirmed that the Redis server itself was healthy, responsive to direct connections, and processing data normally. All internal metrics for the server appeared normal throughout the incident. The issue was isolated to the network path between the application and the cache. Evidence, including a concurrent spike in reported issues for Amazon Web Services \(AWS\) on third-party monitoring sites like Downdetector, suggests the incident may have been related to a transient issue within the underlying cloud infrastructure. ### Resolution The immediate fix was to redirect all application traffic from the primary cache instance to a secondary hot-standby replica. This action successfully bypassed the connectivity problem, and full service functionality was restored upon completion of the failover. ### Action Items & Lessons Learned To prevent recurrence and improve our response, we are taking the following actions: * **Enhance Monitoring:** We are implementing more granular, application-level alerts to track connection success rates for our caching tier. This will allow for faster detection of this specific failure scenario. * **Automate Failover:** We will develop and test an automated failover procedure for the caching tier. This will significantly reduce recovery time \(RTO\) if a similar incident occurs in the future. * **Review Incident Protocols:** We will review and update our incident response protocols to formalise the process for investigating and escalating potential issues with our cloud provider. We apologise for the disruption this incident caused our users. We are committed to learning from this event and improving the reliability of our services.
Major August 12, 2025 Elevated API Errors –
Started Aug 12, 2025, 09:00 PM UTC· resolved Aug 12, 2025, 09:00 PM UTC
- resolved Aug 13, 2025, 05:10 AM UTC
We're experiencing an elevated level of API errors and are currently looking into the issue.
Notice July 16, 2025 General Cloud Service: Release Window 2.211.0 –
Started Jul 16, 2025, 06:00 AM UTC· resolved Jul 16, 2025, 06:00 AM UTC
- resolved Jul 16, 2025, 06:57 AM UTC
A weekly release window to deploy updates and fixes. The update can cause temporary downtime up to 5 seconds. If the deploy includes customer facing updates, we will also publish release notes here: https://support.pdfgeneratorapi.com/en/category/release-notes-uouwx9/
Major June 18, 2025 Elevated API Errors 25m Document Generation Service
Started Jun 18, 2025, 10:30 AM UTC· resolved Jun 18, 2025, 10:56 AM UTC
- identified Jun 18, 2025, 10:30 AM UTC
We're experiencing an elevated level of API errors due to cache issues. The team is working on solving the issue.
- identified Jun 18, 2025, 10:42 AM UTC
We are continuing to work on a fix for this issue.
- monitoring Jun 18, 2025, 10:51 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jun 18, 2025, 10:56 AM UTC
Root cause: Redis ran out of memory when a huge base64-encoded PDF got stuck in the document-delivery queue, pushing usage over 90%. The team temporarily upgraded the Redis node to a bigger instance, which cleared the alert, and then deleted the oversized queue key to free up about 1.2 GB.
Major March 2, 2025 Document Generation Delays 28m Document Generation Service
Started Mar 02, 2025, 09:17 AM UTC· resolved Mar 02, 2025, 09:46 AM UTC
- investigating Mar 02, 2025, 09:17 AM UTC
Our document generation processes are taking longer time than usual. We have identified the root cause and working on solving the issue.
- monitoring Mar 02, 2025, 09:39 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Mar 02, 2025, 09:46 AM UTC
This incident has been resolved.
Major December 18, 2024 Document Generation Downtime in Enterprise Deployment (EU) 42m Document Generation Service, Customer Portal, Editor Service
Started Dec 18, 2024, 08:34 AM UTC· resolved Dec 18, 2024, 09:16 AM UTC
- identified Dec 18, 2024, 08:34 AM UTC
A configuration issues caused downtime in Enterprise Deployment EU. We have found the root cause and are updating the deployment configuration.
- monitoring Dec 18, 2024, 08:40 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Dec 18, 2024, 09:16 AM UTC
The issue affected Enterprise EU customers using API v4 for document generation. The problem was caused by the misconfiguration of the Generator API endpoints required by the latest release updates to generation logic. We are looking into how we can improve the configuration management to mitigate risks in the future.
Major November 13, 2024 Document Generation Delays 34m Document Generation Service
Started Nov 13, 2024, 09:16 AM UTC· resolved Nov 13, 2024, 09:50 AM UTC
- investigating Nov 13, 2024, 09:16 AM UTC
Our document generation processes are taking longer time than usual. We have identified the root cause and working on solving the issue.
- resolved Nov 13, 2024, 09:50 AM UTC
This incident has been resolved.