Parade Outage History

Started Oct 14, 2026, 10:00 PM UTC· resolved Oct 14, 2026, 10:00 PM UTC

1. resolved Oct 14, 2026, 10:00 PM UTC

   An upstream outage with our inference infrastructure provider has taken down our third party dialogue LLM cluster, resulting in agent silence. They have blocked further outbound dispatch and are queueing calls. They are moving workloads over to restore functionality.

Read the full incident report →

Started Feb 06, 2026, 01:29 AM UTC· resolved Feb 06, 2026, 01:29 AM UTC

1. resolved Feb 06, 2026, 01:31 AM UTC

   ### Incident Summary Between **February 4 and February 6, 2026 (UTC)**, some customers experienced **intermittent service disruptions** affecting load posting and related workflows that rely on Truckstop syndication. During these periods, scheduled postings failed and downstream systems (including load visibility and carrier engagement) were temporarily unavailable. The issue occurred on an **hourly basis**, lasting approximately **5–10 minutes per occurrence**, and primarily impacted customers actively using Truckstop integrations. ### Customer Impact * Scheduled load postings to Truckstop and other loadboards failed or were delayed * Some loads were not visible to carriers during the affected windows * Downstream applications experienced temporary unavailability * A subset of customers may need to **re-authenticate their Truckstop accounts** due to token invalidation during the incident No data loss occurred. --- ## Timeline (UTC) **February 4, 2026** * **10:46 PM** – Truckstop begins returning unexpected errors for authentication refresh requests * **10:46–11:16 PM** – First reports of recurring, hourly service disruptions detected * **11:26 PM** – Preventative rollback performed on recent customer-facing changes to reduce system load * **11:30 PM** – Monitoring continued; impact assessed as limited overnight **February 5, 2026** * **10:00 AM** – Full investigation resumes * **10:40 AM** – Root cause identified as a failure pattern tied to Truckstop authentication refresh behavior * **11:00 AM** – Initial mitigation attempted by disabling Truckstop for affected accounts * **11:15 AM** – Issue persists * **11:28 AM** – Hourly scheduled posting job disabled to stop recurring outages * **12:15 PM** – System stability confirmed after disabling scheduled postings * **Afternoon–Evening** – Engineering work focused on a permanent fix * **10:05 PM** – Initial fixes deployed to staging * **11:04 PM** – Fixes validated in staging **February 6, 2026** * **12:18 AM** – Fixes deployed to production * **1:08 AM** – Scheduled postings re-enabled; no further outages observed --- ## Root Cause Two issues combined to cause this incident: 1. **External dependency failure** Truckstop experienced an issue that caused authentication refresh requests to fail unexpectedly. 2. **System behavior under failure conditions** When these authentication requests failed, our system retried them in a way that unintentionally created contention during peak scheduled activity. Under normal conditions this behavior is safe, but the external failure caused retries to stack up and temporarily block other operations. This combination turned a recoverable external error into a broader, time-boxed service disruption. --- ## Resolution ### Immediate Actions Taken * Disabled hourly scheduled posting jobs to immediately stop recurring disruptions * Reduced system load while isolating and validating the root cause * Coordinated internally and with Truckstop to confirm external failure conditions ### Permanent Fix * Updated our authentication handling to prevent system-wide contention during external API failures * Ensured external calls no longer block other operations during retries * Added additional safeguards so similar external issues cannot cascade into broader outages These changes were fully deployed and validated in production by **February 6, 2026, 01:08 AM UTC**. --- ## Customer Follow-Up Actions * **Some customers may need to re-authenticate their Truckstop accounts** due to token invalidation during the Truckstop outage * Our team will proactively reach out to affected customers with instructions and support * No action is required for customers who are not contacted --- ## Prevention Going Forward * Improved isolation between external partner failures and core system operations * Additional monitoring and alerting around authentication and scheduled job behavior * Stronger guardrails to ensure retries cannot impact overall system availability

Read the full incident report →

Started Feb 05, 2026, 12:39 PM UTC· resolved Feb 25, 2026, 03:05 PM UTC

1. investigating Feb 05, 2026, 12:39 PM UTC

   We are currently experiencing a period of instability, and some services may be temporarily disrupted. Our teams are actively working on a fix and monitoring the situation closely. Updates will be shared as progress is made.

2. investigating Feb 05, 2026, 02:20 PM UTC

   We have pinpointed the root cause of this issue to be an authentication issue with one of our integration partners, that is causing some locking throughout our system. We have temporarily stopped our scheduled postings feature, to contain the impact, and to prevent this issue from causing any further degradation. Scheduled postings will remain non-functional until we can work with the partner to fix authentication. Other features should not be affected.

3. monitoring Feb 06, 2026, 12:51 AM UTC

   We have deployed a fix to scheduled postings and are currently monitoring performance.

4. monitoring Feb 06, 2026, 01:21 AM UTC

   We have confirmed that our deploy has fixed the scheduled posting instability issues from this morning. The scheduled posting feature is now fully enabled for all customers. However, due to Truckstop instability yesterday (see their incident here: https://truckstop.statuspage.io/incidents/s6x1x7bwb8dp) - Some of our Truckstop auth tokens were invalidated with no way of recovering without assistance from Truckstop. This is causing the Truckstop integration to remain broken for those affected customers. Our support team will be reaching out if your Truckstop posting integration is affected, so that we can work together with Truckstop to get your integration back into a healthy state. We will be closing this incident now, as it is no longer global, and scheduled postings are running again.

5. resolved Feb 25, 2026, 03:05 PM UTC

   This incident has been resolved.

Read the full incident report →