Jenji incident

Elevated API Latency on iOS Signups

Resolved on May 25, 2022.

postmortem Jun 13, 2022, 04:12 PM UTC

May 25th, 9:54am - our monitoring showed elevated latency on the iOS signup API calls, in p95 and p99 latencies. Users clicking the Signup \(not Login\) button on iOS have a longer wait time than normal. May 25th, 10:02am - we identified the root cause, hitting the AWS service limit on concurrent lambda execution in this specific AWS account, making some of those calls timeout or take up to 10-15 seconds. May 25th, 10:18am - a fix has been pushed to production, the iOS signups are back to normal.

Started
May 25, 2022, 07:30 AM UTC
Resolved
May 25, 2022, 07:30 AM UTC
Duration
–
Detected by Pingoru
May 25, 2022, 07:30 AM UTC

Update timeline

  1. resolved Jun 13, 2022, 04:12 PM UTC

    We're experiencing an elevated level of API latency affecting iOS signups and are currently looking into the issue.

  2. postmortem Jun 13, 2022, 04:12 PM UTC

    May 25th, 9:54am - our monitoring showed elevated latency on the iOS signup API calls, in p95 and p99 latencies. Users clicking the Signup \(not Login\) button on iOS have a longer wait time than normal. May 25th, 10:02am - we identified the root cause, hitting the AWS service limit on concurrent lambda execution in this specific AWS account, making some of those calls timeout or take up to 10-15 seconds. May 25th, 10:18am - a fix has been pushed to production, the iOS signups are back to normal.

Your product runs on more than Jenji

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →