Weights & Biases Outage History
Weights & Biases is up right nowPingoru has recorded 68 incidents for Weights & Biases: 68 in the last 12 months and 1 in the last 90 days.
Show as a table
| Month | Incidents | Hours with an open incident |
|---|---|---|
| April 2026 | 5 | 192 of 720 (27%) |
| May 2026 | 4 | 43 of 744 (6%) |
| June 2026 | 4 | 104 of 720 (14%) |
| July 2026 | 6 | 30 of 744 (4%) |
| August 2026 | 0 | 0 of 744 (0%) |
| September 2026 | 1 | 0 of 720 (0%) |
| October 2026 | 0 | 0 of 571 (0%) |
Incidents, newest first
Notice September 1, 2026 API Degraded –
Started Sep 01, 2026, 02:40 PM UTC· resolved Sep 01, 2026, 02:40 PM UTC
- resolved Sep 02, 2026, 08:02 PM UTC
An incident with Google Cloud caused our API and site to become degraded or unavailable. We apologize for the disruption. https://status.cloud.google.com/incidents/J5ia5t9p3g9Q5Wi7r8Ev
Notice July 15, 2026 Errors fetching media files 1h 31m Frontend
Started Jul 15, 2026, 10:35 PM UTC· resolved Jul 16, 2026, 12:06 AM UTC
- investigating Jul 15, 2026, 10:35 PM UTC
We are currently experiencing elevated errors fetching media files in Weights & Biases.
- resolved Jul 16, 2026, 12:06 AM UTC
This incident has been resolved.
Minor July 15, 2026 Metric Ingestion Backup 1m Metrics Ingestion Pipeline
Started Jul 15, 2026, 06:46 AM UTC· resolved Jul 15, 2026, 06:48 AM UTC
- identified Jul 15, 2026, 06:46 AM UTC
We are currently experiencing elevated ingestion delay due to high volume.
- identified Jul 15, 2026, 06:48 AM UTC
We are continuing to work on a fix for this issue.
- resolved Jul 15, 2026, 06:48 AM UTC
This incident has been resolved.
Minor July 13, 2026 Metrics Ingestion Backup 13h 30m Metrics Ingestion Pipeline
Started Jul 13, 2026, 06:04 AM UTC· resolved Jul 13, 2026, 07:35 PM UTC
- identified Jul 13, 2026, 06:04 AM UTC
We are experiencing metric ingestion delay of ~1 hour
- resolved Jul 13, 2026, 07:35 PM UTC
This incident has been resolved.
Minor July 12, 2026 Metric Ingestion Backlog 13h 13m Metrics Ingestion Pipeline
Started Jul 12, 2026, 04:38 AM UTC· resolved Jul 12, 2026, 05:52 PM UTC
- identified Jul 12, 2026, 04:38 AM UTC
We're currently experiencing elevated ingestion time of about 30 minutes for some run metrics.
- resolved Jul 12, 2026, 05:52 PM UTC
This incident has been resolved.
Major July 11, 2026 Timeouts and 503s for serverless inference 34m Inference Endpoint
Started Jul 11, 2026, 03:11 AM UTC· resolved Jul 11, 2026, 03:45 AM UTC
- identified Jul 11, 2026, 03:11 AM UTC
Serverless Inference users could experience timeouts and 503 response codes while attempting to leverage models on the product. We are actively engaged in troubleshooting and will post an update here as soon as we know more.
- monitoring Jul 11, 2026, 03:36 AM UTC
Impact has been mitigated.
- resolved Jul 11, 2026, 03:45 AM UTC
This incident has been resolved.
Minor July 11, 2026 Metric Ingestion Delay 56m Metrics Ingestion Pipeline
Started Jul 11, 2026, 02:11 AM UTC· resolved Jul 11, 2026, 03:07 AM UTC
- identified Jul 11, 2026, 02:11 AM UTC
We are currently seeing a backup in metric ingestion of up to 20 minutes
- resolved Jul 11, 2026, 03:07 AM UTC
This incident has been resolved.
Notice June 26, 2026 Logins failing 3d 11h Frontend, Auth0 User Authentication
Started Jun 26, 2026, 07:13 AM UTC· resolved Jun 29, 2026, 06:15 PM UTC
- investigating Jun 26, 2026, 07:13 AM UTC
We are investigating an issue where logging in to wandb.ai may fail for some users.
- identified Jun 26, 2026, 07:18 AM UTC
We have identified an issue with Auth0 that may cause some users to be unable to log in.
- monitoring Jun 26, 2026, 07:28 AM UTC
An Auth0 incident that had earlier prevented logins seems to have recovered, allowing restored access to the site and we are continuing the monitor the status. Auth0 Incident: https://status.auth0.com/incidents/z8sc9m1gzqb7
- resolved Jun 29, 2026, 06:15 PM UTC
This incident has been resolved.
Critical June 14, 2026 Web App loading issue 13h 22m Frontend
Started Jun 14, 2026, 07:37 AM UTC· resolved Jun 14, 2026, 08:59 PM UTC
- investigating Jun 14, 2026, 07:37 AM UTC
We're investigating reports of the W&B web app (app.wandb.ai) showing a blank page on load for some users. The API and SDK data logging remain operational. A fix is in progress and we'll share an update soon.
- investigating Jun 14, 2026, 07:37 AM UTC
We are continuing to investigate this issue.
- monitoring Jun 14, 2026, 09:30 AM UTC
A fix has been deployed and we are monitoring.
- resolved Jun 14, 2026, 08:59 PM UTC
Web app has been restored.
Minor June 9, 2026 Metric ingestion delayed 7h 20m Metrics Ingestion Pipeline
Started Jun 09, 2026, 11:47 PM UTC· resolved Jun 10, 2026, 07:08 AM UTC
- identified Jun 09, 2026, 11:47 PM UTC
We are currently responding to delay of up to 5 hours for metrics written into wandb. We have identified the issue and are working to process metrics as quickly as possible. There is no data loss and all data will be complete once the backlog has drained. We're very sorry for the disruption.
- monitoring Jun 10, 2026, 05:44 AM UTC
We identified an infrastructure issue as the root cause and have resolved it. We are rapidly working through the backlog of ingested metrics and should fully catch up soon.
- resolved Jun 10, 2026, 07:08 AM UTC
This incident has been resolved.
Minor June 9, 2026 Metrics ingestion delay 22m Metrics Ingestion Pipeline
Started Jun 09, 2026, 09:32 AM UTC· resolved Jun 09, 2026, 09:55 AM UTC
- investigating Jun 09, 2026, 09:32 AM UTC
We are currently investigating an issue where metric ingestion is delayed for some runs up to 1 hour.
- resolved Jun 09, 2026, 09:55 AM UTC
This incident has been resolved
Notice May 26, 2026 Slowness and missing metrics 6h 15m Backend API, Frontend
Started May 26, 2026, 07:47 PM UTC· resolved May 27, 2026, 02:02 AM UTC
- monitoring May 26, 2026, 07:47 PM UTC
Starting at 10:48AM PST we experienced an incident causing missing metrics and slowness across the application. As of 12:10 PM metrics should now be restored (there was no data loss) and we are continuing to monitor performance.
- resolved May 27, 2026, 02:02 AM UTC
This incident has been resolved.
Minor May 15, 2026 Delayed Metrics Processing – Workspace & API Updates Impacted 2h 47m Backend API, Metrics Ingestion Pipeline
Started May 15, 2026, 01:51 PM UTC· resolved May 15, 2026, 04:39 PM UTC
- identified May 15, 2026, 01:51 PM UTC
We are currently experiencing delays in processing newly logged metrics. As a result, recent metrics from active runs may take longer than usual to appear in charts, dashboards, reports, and API queries. No data is lost. All metrics are being processed and will become visible once the backlog clears. We are actively working to restore normal processing times and will provide updates here.
- resolved May 15, 2026, 04:39 PM UTC
This incident has been resolved.
Notice May 10, 2026 Run Update Ingest Delays 4h 50m Backend API
Started May 10, 2026, 01:19 PM UTC· resolved May 10, 2026, 06:09 PM UTC
- identified May 10, 2026, 01:19 PM UTC
We identified an issue where updates to W&B Models runs may be delayed for up to 30 minutes. There is no data loss.
- monitoring May 10, 2026, 02:56 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved May 10, 2026, 06:09 PM UTC
This incident has been resolved.
Notice May 4, 2026 API Errors 12h 13m Backend API, Frontend
Started May 04, 2026, 03:50 PM UTC· resolved May 05, 2026, 04:03 AM UTC
- investigating May 04, 2026, 03:50 PM UTC
We are currently investigating this issue.
- monitoring May 04, 2026, 05:57 PM UTC
The frontend and backend APIs have fully recovered after we mitigated excess load on the system. We will continue to actively monitor the situation to ensure stability.
- investigating May 04, 2026, 10:42 PM UTC
We are currently investigating this issue.
- monitoring May 05, 2026, 12:31 AM UTC
A fix has been implemented and we are monitoring the results.
- monitoring May 05, 2026, 12:36 AM UTC
We are continuing to monitor for any further issues.
- resolved May 05, 2026, 04:03 AM UTC
This incident has been resolved.
Minor April 28, 2026 Backend API Errors 3h 26m Backend API, Frontend
Started Apr 28, 2026, 10:41 PM UTC· resolved Apr 29, 2026, 02:07 AM UTC
- investigating Apr 28, 2026, 10:41 PM UTC
We are currently investigating an issue where API requests may sporadically fail.
- resolved Apr 29, 2026, 02:07 AM UTC
This incident has been resolved.
Notice April 22, 2026 Elevated API Errors 5d 22h Backend API, Frontend
Started Apr 22, 2026, 06:00 AM UTC· resolved Apr 28, 2026, 04:30 AM UTC
- investigating Apr 22, 2026, 06:51 AM UTC
We are currently investigating an issue where requests are failing.
- monitoring Apr 22, 2026, 07:43 AM UTC
The issue has been identified and a fix has been rolled out. We are monitoring for any further issues.
- monitoring Apr 22, 2026, 03:30 PM UTC
We are continuing to monitor for any further issues.
- resolved Apr 28, 2026, 04:30 AM UTC
This incident has been resolved.
Notice April 6, 2026 API and UI outage 12h 56m Backend API, Frontend
Started Apr 06, 2026, 03:26 PM UTC· resolved Apr 07, 2026, 04:22 AM UTC
- identified Apr 06, 2026, 03:26 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Apr 06, 2026, 04:51 PM UTC
A fix has been implemented and we are monitoring the results.
- monitoring Apr 06, 2026, 05:12 PM UTC
We are seeing the site recover and be fully operational - we are continuing to actively monitor the situation to ensure stability.
- resolved Apr 07, 2026, 04:22 AM UTC
This incident has been resolved.
Major April 2, 2026 Delay in files and media getting uploaded 44m Backend API
Started Apr 02, 2026, 02:15 AM UTC· resolved Apr 02, 2026, 03:00 AM UTC
- identified Apr 02, 2026, 02:15 AM UTC
We have identified a delay in files and media getting uploaded.
- identified Apr 02, 2026, 02:15 AM UTC
We are continuing to work on a fix for this issue.
- resolved Apr 02, 2026, 03:00 AM UTC
This incident has been resolved.
Major March 31, 2026 W&B Inference Maintenance 1h 19m Inference Endpoint
Started Mar 31, 2026, 04:52 PM UTC· resolved Mar 31, 2026, 06:12 PM UTC
- investigating Mar 31, 2026, 04:52 PM UTC
The W&B Inference endpoint is currently unavailable. This is due to some maintenance on the gateway that was scheduled but that we missed the announcement step for. We expect intermittent outages between 9am and 11am PDT. We apologize for any inconvenience.
- resolved Mar 31, 2026, 06:12 PM UTC
Maintenance has completed.
Minor March 11, 2026 Elevated API request latencies 23h 34m Backend API
Started Mar 11, 2026, 11:58 PM UTC· resolved Mar 12, 2026, 11:33 PM UTC
- investigating Mar 11, 2026, 05:50 PM UTC
We are aware of an issue causing elevated API request latencies, resulting in degraded performance in the UI and SDK. Recent changes in traffic patterns are resulting in increased load, which we urgently are working to address.
- monitoring Mar 11, 2026, 11:58 PM UTC
We've deployed mitigations and are actively monitoring. Request latencies have decreased but are not yet back to baseline. We appreciate your patience as we work to resolve this issue.
- resolved Mar 12, 2026, 11:33 PM UTC
We've been monitoring request latencies over the past 24 hours and they are back within our normal ranges.
Minor March 5, 2026 High latency - Workspaces & API Impacted 5h 15m Backend API, Frontend
Started Mar 05, 2026, 07:03 PM UTC· resolved Mar 06, 2026, 12:19 AM UTC
- investigating Mar 05, 2026, 07:03 PM UTC
We're experiencing an elevated level of latency across the site and API. We are actively investigating the issue.
- investigating Mar 05, 2026, 07:07 PM UTC
We are continuing to investigate this issue.
- investigating Mar 05, 2026, 08:02 PM UTC
We have implemented a set of mitigations - site performance is significantly improved and we are investigating further fixes.
- monitoring Mar 05, 2026, 09:36 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Mar 06, 2026, 12:19 AM UTC
UI & API latency has returned to normal ranges
Notice February 9, 2026 Error loading some projects with active runs 2h 26m Backend API
Started Feb 09, 2026, 09:41 PM UTC· resolved Feb 10, 2026, 12:07 AM UTC
- identified Feb 09, 2026, 09:41 PM UTC
A recent deployment caused an issue affecting read access to some projects with active runs. We're actively working on fixing the issue.
- resolved Feb 10, 2026, 12:07 AM UTC
This incident has been resolved.
Notice January 21, 2026 Elevated latency affecting app and SDK 6h 16m Backend API
Started Jan 21, 2026, 04:09 PM UTC· resolved Jan 21, 2026, 10:25 PM UTC
- investigating Jan 21, 2026, 04:09 PM UTC
We are currently investigating elevated latency across the application since 7:08am PST.
- monitoring Jan 21, 2026, 04:45 PM UTC
Issue appears resolved as of 8:18am. We are continuing to monitor.
- resolved Jan 21, 2026, 10:25 PM UTC
This incident has been resolved.
Minor January 15, 2026 Ingestion delay for some projects 13h 38m Backend API, Frontend
Started Jan 15, 2026, 04:48 AM UTC· resolved Jan 15, 2026, 06:26 PM UTC
- identified Jan 15, 2026, 04:48 AM UTC
We are aware of ingestion delay for some projects and working to resolve it.
- identified Jan 15, 2026, 07:13 AM UTC
We have implemented a mitigation and are making progress against the backlog.
- resolved Jan 15, 2026, 06:26 PM UTC
Our systems have fully caught up with the backlog. The API should correctly return up-to-date metrics for experiments in all projects. We sincerely appreciate your patience as we worked to resolve the issue.