Resolved on April 30, 2024, affecting API.
Date: 30/04/2024 Authors: André Chaves, Eduardo Milhomen, Douglas Ferreira Status: Solved Impact: Partial latency on document creation ## Summary: On April 30, 2024, at 08:43 \(UTC-3\), our team detected significant latency issues in document creation endpoint. This problem caused some requests to continue processing beyond the expected timeframe ### Root Causes: The primary cause of the latency was that requests were held for an extended duration without timely processing. ### Trigger: The issue was initially identified through an alert from DataDog, which reported excessive latency affecting Itau Endpoints. ### Resolution: To resolve the issue, we scaled up the number of pods handling the traffic and implemented a strict time limit on the endpoint. Now, no request is held for more than 60 seconds. This adjustment has stabilized the document creation process and resolved the latency issues. ### Lessons Learned #### What went well The monitoring application alerted us of the latency promptly. #### What went wrong We took more time than expected to identify the root cause of the problem. #### Where we got lucky Despite the significant latency, the impact was confined to partial delays on document creation rather than a complete system outage. ### Timeline * 30/04/2024 08:43 \(UTC-3\): We detected a high latency on some endpoints of the application * 30/04/2024 09:00 \(UTC-3\): Changes implemented on scalability of the application * 30/04/2024 09:40 \(UTC-3\): Another occurrence of high latency * 30/04/2024 09:44 \(UTC-3\): Change implemented on endpoint
Affected components
Update timeline
- resolved Apr 30, 2024, 04:01 PM UTC
On April 30, 2024, at 08:43 (UTC-3), our team detected significant latency issues in document creation endpoint. This problem caused some requests to continue processing beyond the expected timeframe
- postmortem Apr 30, 2024, 04:02 PM UTC
Date: 30/04/2024 Authors: André Chaves, Eduardo Milhomen, Douglas Ferreira Status: Solved Impact: Partial latency on document creation ## Summary: On April 30, 2024, at 08:43 \(UTC-3\), our team detected significant latency issues in document creation endpoint. This problem caused some requests to continue processing beyond the expected timeframe ### Root Causes: The primary cause of the latency was that requests were held for an extended duration without timely processing. ### Trigger: The issue was initially identified through an alert from DataDog, which reported excessive latency affecting Itau Endpoints. ### Resolution: To resolve the issue, we scaled up the number of pods handling the traffic and implemented a strict time limit on the endpoint. Now, no request is held for more than 60 seconds. This adjustment has stabilized the document creation process and resolved the latency issues. ### Lessons Learned #### What went well The monitoring application alerted us of the latency promptly. #### What went wrong We took more time than expected to identify the root cause of the problem. #### Where we got lucky Despite the significant latency, the impact was confined to partial delays on document creation rather than a complete system outage. ### Timeline * 30/04/2024 08:43 \(UTC-3\): We detected a high latency on some endpoints of the application * 30/04/2024 09:00 \(UTC-3\): Changes implemented on scalability of the application * 30/04/2024 09:40 \(UTC-3\): Another occurrence of high latency * 30/04/2024 09:44 \(UTC-3\): Change implemented on endpoint
Your product runs on more than ZapSign
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services