Kollective incident

Degraded Live Event Playback

Resolved on June 26, 2025.

postmortem Jun 27, 2025, 03:23 AM UTC

On June 25th 2025 at 1:59 PM Kollective released KSDK 3.6.3. This release contained refactored code that caused the KSDK to hang when forced to fallback to CDN, preventing playback of a live event. Kollective's development team was alerted at 8:41 PM \(PDT\) of a potential outage. The outage was verified and service was restored at 9:07 PM \(PDT\). The changes to the KSDK were reverted from the testing branch at 2:00PM on June 26th. QA has identified the testing failure as an invalid configuration in the testing environment that allowed for playback despite the configuration settings. Current manual testing steps have been fixed and work is proceeding on automated testing that will identify these issues in the future.

Started
Jun 26, 2025, 03:41 AM UTC
Resolved
Jun 26, 2025, 03:41 AM UTC
Duration
–
Detected by Pingoru
Jun 26, 2025, 03:41 AM UTC

Update timeline

  1. resolved Jun 27, 2025, 03:15 AM UTC

    Customers using Teams Live events are experiences degraded performance

  2. postmortem Jun 27, 2025, 03:23 AM UTC

    On June 25th 2025 at 1:59 PM Kollective released KSDK 3.6.3. This release contained refactored code that caused the KSDK to hang when forced to fallback to CDN, preventing playback of a live event. Kollective's development team was alerted at 8:41 PM \(PDT\) of a potential outage. The outage was verified and service was restored at 9:07 PM \(PDT\). The changes to the KSDK were reverted from the testing branch at 2:00PM on June 26th. QA has identified the testing failure as an invalid configuration in the testing environment that allowed for playback despite the configuration settings. Current manual testing steps have been fixed and work is proceeding on automated testing that will identify these issues in the future.

Your product runs on more than Kollective

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →