TestingBot incident

Elevated wait time

Resolved after 3 hours 30 minutes, on September 5, 2025.

postmortem Sep 05, 2025, 05:44 AM UTC

Today we experienced long waiting times for Windows \+ macOS machines. The issue was caused by a server issue, where memory issues were causing restarts of our hub \(h[ub.testingbot.com\)](http://hub.testingbot.com) This in turn caused the machines running the tests to fail to communicate with the hub, causing tests from clients to wait up to 5 minutes before a test could start. If your test runner is configured with a http connect timeout < 5 minutes, this may have caused issues where tests would fail to start. The memory issue was resolved and we are looking into a solution to prevent this from happening again.

Started
Sep 05, 2025, 01:00 AM UTC
Resolved
Sep 05, 2025, 04:30 AM UTC
Duration
3h 30m
Detected by Pingoru
Sep 05, 2025, 01:00 AM UTC

Update timeline

  1. resolved Sep 05, 2025, 05:43 AM UTC

    Since 1:00 UTC time we have been experiencing elevated wait times, causing some tests to wait for up to 5 minutes before they could start. We deployed a fix and are monitoring the waiting times. They are back to < 10 seconds since 04:30 UTC.

  2. postmortem Sep 05, 2025, 05:44 AM UTC

    Today we experienced long waiting times for Windows \+ macOS machines. The issue was caused by a server issue, where memory issues were causing restarts of our hub \(h[ub.testingbot.com\)](http://hub.testingbot.com) This in turn caused the machines running the tests to fail to communicate with the hub, causing tests from clients to wait up to 5 minutes before a test could start. If your test runner is configured with a http connect timeout < 5 minutes, this may have caused issues where tests would fail to start. The memory issue was resolved and we are looking into a solution to prevent this from happening again.

Your product runs on more than TestingBot

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →