Telerivet incident

Database connectivity issue

Resolved after 14 hours 5 minutes, affecting Web Dashboard and REST API, on May 22, 2021.

resolved May 23, 2021, 03:47 AM UTC

Telerivet has not detected any further issues with database connectivity or data integrity, and has verified that no scheduled messages or other data were lost due to this issue.

Started
May 22, 2021, 01:41 PM UTC
Resolved
May 23, 2021, 03:47 AM UTC
Duration
14h 5m
Detected by Pingoru
May 22, 2021, 01:41 PM UTC

Affected components

Web DashboardREST API

Update timeline

  1. investigating May 22, 2021, 01:41 PM UTC

    Telerivet is currently investigating an issue with database connectivity.

  2. investigating May 22, 2021, 01:41 PM UTC

    We are continuing to investigate this issue.

  3. identified May 22, 2021, 01:56 PM UTC

    The issue has been identified as a particular query causing Telerivet's database server to crash. Telerivet has disabled scheduled message functionality to avoid this issue and is continuing to investigate the issue.

  4. identified May 22, 2021, 02:32 PM UTC

    We are continuing to work on a fix for this issue.

  5. monitoring May 22, 2021, 02:39 PM UTC

    The issue was caused by a corrupted database table storing scheduled messages and other scheduled events, which caused the database server to crash when triggering scheduled events. The corrupted database table has been repaired, and scheduled messages are being sent now. All Telerivet functionality should be working normally.

  6. resolved May 23, 2021, 03:47 AM UTC

    Telerivet has not detected any further issues with database connectivity or data integrity, and has verified that no scheduled messages or other data were lost due to this issue.

Your product runs on more than Telerivet

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →