Production API Outage Due to Database Connection Exhaustion

Update timeline

1. resolved Sep 14, 2026, 11:20 AM UTC

   We experienced a production API outage after a database schema migration on a large MySQL table caused increased database load and connection buildup. The RDS instance reached its maximum connection limit, resulting in Too many connections errors and 500 responses across the API. Application and background worker services were temporarily stopped to prevent new database connections. Existing connections cleared, services were restarted, and the API returned to normal operation. We are reviewing our production migration process and database safeguards to reduce the risk of similar incidents.

2. investigating Sep 14, 2026, 11:20 AM UTC

   We experienced a production API outage after a database schema migration on a large MySQL table caused increased database load and connection buildup. The RDS instance reached its maximum connection limit, resulting in `Too many connections` errors and 500 responses across the API. Application and background worker services were temporarily stopped to prevent new database connections. Existing connections cleared, services were restarted, and the API returned to normal operation. We are reviewing our production migration process and database safeguards to reduce the risk of similar incidents.