TRMNL incident

DB Replica lag affecting site responsiveness

Resolved after 5 hours 36 minutes, affecting Display API and Firmware API and 1 more, on March 26, 2026.

resolved Mar 26, 2026, 07:58 PM UTC

All screen generation workers and APIs have been restored. Internal infrastructure analysis and adjustments will continue with no impact on customers or operations.

Started
Mar 26, 2026, 02:22 PM UTC
Resolved
Mar 26, 2026, 07:58 PM UTC
Duration
5h 36m
Detected by Pingoru
Mar 26, 2026, 02:22 PM UTC

Affected components

Display APIFirmware APIScreen Generation Workers

Update timeline

  1. investigating Mar 26, 2026, 02:58 PM UTC

    We are currently investigating the issue as the main database has 32GB of replica lag.

  2. monitoring Mar 26, 2026, 03:33 PM UTC

    We've identified the issue and restored site functionality. We are still monitoring the system as we identify the underlying cause. Screen generation will still see lag due to the backlog and one worker being taken offline.

  3. monitoring Mar 26, 2026, 05:45 PM UTC

    Screen generation workers continue to work through the backlog. This can result in: - Existing plugins not showing up-to-date information (old screeens) - New devices showing "You're all caught up!" due to no screens yet generated

  4. monitoring Mar 26, 2026, 06:22 PM UTC

    All workers are caught up, but one worker is currently still offline. While this is technically degraded, the existing workers can handle the ongoing load.

  5. resolved Mar 26, 2026, 07:58 PM UTC

    All screen generation workers and APIs have been restored. Internal infrastructure analysis and adjustments will continue with no impact on customers or operations.

Your product runs on more than TRMNL

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →