Windmill Outage History

Started Oct 12, 2026, 04:07 PM UTC· resolved Oct 12, 2026, 05:28 PM UTC

1. investigating Oct 12, 2026, 04:07 PM UTC

   Seeing degraded performance on Web App. Some users may experience high load times.

2. resolved Oct 12, 2026, 05:28 PM UTC

   Web app degraded due to increased load. Additional resources have come online.

Read the full incident report →

Started Oct 12, 2026, 02:10 PM UTC· resolved Oct 13, 2026, 01:22 AM UTC

1. investigating Oct 12, 2026, 02:10 PM UTC

   Windmill is experiencing high-load and elevated error rates. We have determined the root cause and are actively investigating a fix

2. resolved Oct 13, 2026, 01:22 AM UTC

   Web App, Windy and MCP Unavailable Background Windmill generates Windy reports in the background to prepare content for our users. On October 12, 2026, a large number of these reports started at the same time. Our services connect to the database through a connection proxy. The proxy usually shares a small number of database connections between many requests. But a problem in how our services use their database connections stopped the proxy from sharing them. Each request held its own database connection, so the large report load opened thousands of connections, and each connection used memory on the database server. At 10:00 AM ET, the database server that handles reads ran out of memory and restarted two times. We added more read servers and the service recovered at 10:40 AM. At 11:44 AM, when the reports started to finish, the load moved to our primary database server, which handles writes. That server ran out of memory, and our database provider automatically moved the primary role to a standby server three times. Each move interrupted database connections for 1–2 minutes. During these periods, the Windmill web app, Windy and MCP were unavailable or slow, and many requests failed. Some review chats were sent later than planned. Timeline 10/12 @ 10:00 AM ET - Report generation starts. The read database server runs out of memory. 10/12 @ 10:09 AM ET - Web app becomes unavailable. Status page updated at 10:10 AM. 10/12 @ 10:10 AM ET - Cause linked to report generation. 10/12 @ 10:16 AM ET - Read database server restarts and recovers, but some requests continue to fail. 10/12 @ 10:25 AM ET - Read database server runs out of memory a second time. 10/12 @ 10:33 AM ET - Additional read servers added. 10/12 @ 10:40 AM ET - Error rate drops to 0%. Incident resolved and monitoring continues. 10/12 @ 11:25 AM ET - Read servers slow down again under load. More read servers added. 10/12 @ 11:44 AM ET - Primary database server runs out of memory. The primary role moves to a standby server. 10/12 @ 12:00 PM ET - Reports finish, load moves to write database. 10/12 @ 12:04 PM ET - Primary database server runs out of memory a second time. Status page updated at 12:07 PM. 10/12 @ 12:42 PM ET - Primary database server runs out of memory a third time. 10/12 @ 1:05 PM ET - Query improvements deployed. Database memory recovers and errors stop. 10/12 @ 1:28 PM ET - Incident resolved and monitoring continues. Remediation We understand the impact this downtime had; our customers rely on Windmill for critical work and we apologize greatly for interrupting it. Internally, we are implementing changes to prevent similar incidents from occurring in the future. This includes: Lowering the maximum number of connections each database server can hold Continuing to optimize our database queries for agent-driven workloads Upgrading our performance review systems to spread report generation over a longer period of time

3. investigating Oct 13, 2026, 01:22 AM UTC

   # Web App, Windy and MCP Unavailable ## Background Windmill generates Windy reports in the background to prepare content for our users. On October 12, 2026, a large number of these reports started at the same time. Our services connect to the database through a connection proxy. The proxy usually shares a small number of database connections between many requests. But a problem in how our services use their database connections stopped the proxy from sharing them. Each request held its own database connection, so the large report load opened thousands of connections, and each connection used memory on the database server. At 10:00 AM ET, the database server that handles reads ran out of memory and restarted two times. We added more read servers and the service recovered at 10:40 AM. At 11:44 AM, when the reports started to finish, the load moved to our primary database server, which handles writes. That server ran out of memory, and our database provider automatically moved the primary role to a standby server three times. Each move interrupted database connections for 1–2 minutes. During these periods, the Windmill web app, Windy and MCP were unavailable or slow, and many requests failed. Some review chats were sent later than planned. ## Timeline - 10/12 @ 10:00 AM ET - Report generation starts. The read database server runs out of memory. - 10/12 @ 10:09 AM ET - Web app becomes unavailable. Status page updated at 10:10 AM. - 10/12 @ 10:10 AM ET - Cause linked to report generation. - 10/12 @ 10:16 AM ET - Read database server restarts and recovers, but some requests continue to fail. - 10/12 @ 10:25 AM ET - Read database server runs out of memory a second time. - 10/12 @ 10:33 AM ET - Additional read servers added. - 10/12 @ 10:40 AM ET - Error rate drops to 0%. Incident resolved and monitoring continues. - 10/12 @ 11:25 AM ET - Read servers slow down again under load. More read servers added. - 10/12 @ 11:44 AM ET - Primary database server runs out of memory. The primary role moves to a standby server. - 10/12 @ 12:00 PM ET - Reports finish, load moves to write database. - 10/12 @ 12:04 PM ET - Primary database server runs out of memory a second time. Status page updated at 12:07 PM. - 10/12 @ 12:42 PM ET - Primary database server runs out of memory a third time. - 10/12 @ 1:05 PM ET - Query improvements deployed. Database memory recovers and errors stop. - 10/12 @ 1:28 PM ET - Incident resolved and monitoring continues. ## Remediation We understand the impact this downtime had; our customers rely on Windmill for critical work and we apologize greatly for interrupting it. Internally, we are implementing changes to prevent similar incidents from occurring in the future. This includes: - Lowering the maximum number of connections each database server can hold - Continuing to optimize our database queries for agent-driven workloads - Upgrading our performance review systems to spread report generation over a longer period of time

Read the full incident report →

Started Oct 01, 2026, 12:30 PM UTC· resolved Oct 01, 2026, 08:30 PM UTC

1. investigating Oct 01, 2026, 12:30 PM UTC

   We are experiencing intermittent issues connecting to our MCP server. We are investigating the situation.

2. resolved Oct 01, 2026, 08:30 PM UTC

   Background Early this week, Windmill rolled out enhanced monitoring for the core infrastructure which processes all requests coming into all our services (Windy, web app, MCP, and integrations). We noticed a small number (>0.01%) of requests were failing due to a networking misconfiguration in how our services talk to each other. Although there was no customer impact since these failing requests get retried, we proactively rolled out a fix the evening of 9/30/2026. Windmill runs its services in a high-availability configuration, to ensure our services are always available. To maintain high-availability during upgrades, we use rolling deploys where old versions of services stay online until the new version is ready. Once the new version is ready, the old version enters a draining state where it will finish servicing all pending requests before it goes offline. The configuration change we rolled out on 9/30/2026 caused a latent bug in how we handle draining to block the old versions from shutting down. We balance load evenly across all instances, meaning the 50% of requests going to the stuck old version would fail. Timeline 9/30 @ 11:52 PM EST - Fix deployed for networking misconfiguration, exposing latent bug in service shutdown 10/1 @ 7:00 AM EST - A small number of MCP servers get shut down during load balancing, but fail to drain, causing minor degradation in MCP service. 10/1 @ 8:30 AM EST - Error rates increase, reported internally and incident declared. 10/1 @ 9:00 AM EST - Root cause identified. 10/1 @ 9:46 AM EST - Fix is identified, applied, and roll out begins. 10/1 @ 9:52 AM EST - Roll out reaches our API servers, error rates begin to rise as the old version fails to drain properly. 10/1 @ 10:01 AM EST - First user-facing errors accessing app.gowindmill.com reported and incident escalated. 10/1 @ 10:09 AM EST - Roll out of new API servers completes, but old servers continue to fail to drain. 50% error rate observed. 10/1 @ 10:21 AM EST - Infrastructure vendor involved to forcefully shutdown the erroring API servers. 10/1 @ 10:25 AM EST - Force shutdown triggered by vendor, error rates remain elevated 10/1 @ 10:30 AM EST - Identified another service was blocking the force shutdown, restart on the other service triggered 10/1 @ 10:36 AM EST - Restart completes, all the erroring servers terminate, error rate drops to 0%. 10/1 @ 10:43 AM EST - Continued monitoring reveals no further errors, incident resolved. Remediation We understand the impact this downtime had - our customers rely on Windmill for critical work and we apologize greatly for interrupting it. Internally, we are implementing changes to prevent similar incidents from occurring in the future. This includes: Enhanced testing in staging prior to promotion to production Comprehensive audits for similar latent risks and remediation Continuing to improve monitoring and shorten incident response time More clear error pages For continuous updates on Windmill’s real-time status, we encourage customers to subscribe to status page here: https://status.gowindmill.com/

3. investigating Oct 01, 2026, 08:30 PM UTC

   # Background Early this week, Windmill rolled out enhanced monitoring for the core infrastructure which processes all requests coming into all our services (Windy, web app, MCP, and integrations). We noticed a small number (>0.01%) of requests were failing due to a networking misconfiguration in how our services talk to each other. Although there was no customer impact since these failing requests get retried, we proactively rolled out a fix the evening of 9/30/2026. Windmill runs its services in a high-availability configuration, to ensure our services are always available. To maintain high-availability during upgrades, we use rolling deploys where old versions of services stay online until the new version is ready. Once the new version is ready, the old version enters a draining state where it will finish servicing all pending requests before it goes offline. The configuration change we rolled out on 9/30/2026 caused a latent bug in how we handle draining to block the old versions from shutting down. We balance load evenly across all instances, meaning the 50% of requests going to the stuck old version would fail. # Timeline - 9/30 @ 11:52 PM EST - Fix deployed for networking misconfiguration, exposing latent bug in service shutdown - 10/1 @ 7:00 AM EST - A small number of MCP servers get shut down during load balancing, but fail to drain, causing minor degradation in MCP service. - 10/1 @ 8:30 AM EST - Error rates increase, reported internally and incident declared. - 10/1 @ 9:00 AM EST - Root cause identified. - 10/1 @ 9:46 AM EST - Fix is identified, applied, and roll out begins. - 10/1 @ 9:52 AM EST - Roll out reaches our API servers, error rates begin to rise as the old version fails to drain properly. - 10/1 @ 10:01 AM EST - First user-facing errors accessing app.gowindmill.com reported and incident escalated. - 10/1 @ 10:09 AM EST - Roll out of new API servers completes, but old servers continue to fail to drain. 50% error rate observed. - 10/1 @ 10:21 AM EST - Infrastructure vendor involved to forcefully shutdown the erroring API servers. - 10/1 @ 10:25 AM EST - Force shutdown triggered by vendor, error rates remain elevated - 10/1 @ 10:30 AM EST - Identified another service was blocking the force shutdown, restart on the other service triggered - 10/1 @ 10:36 AM EST - Restart completes, all the erroring servers terminate, error rate drops to 0%. - 10/1 @ 10:43 AM EST - Continued monitoring reveals no further errors, incident resolved. # Remediation We understand the impact this downtime had - our customers rely on Windmill for critical work and we apologize greatly for interrupting it. Internally, we are implementing changes to prevent similar incidents from occurring in the future. This includes: Enhanced testing in staging prior to promotion to production Comprehensive audits for similar latent risks and remediation Continuing to improve monitoring and shorten incident response time More clear error pages For continuous updates on Windmill’s real-time status, we encourage customers to subscribe to status page here: https://status.gowindmill.com/

Read the full incident report →

Started Sep 18, 2026, 05:06 PM UTC· resolved Sep 18, 2026, 05:27 PM UTC

1. investigating Sep 18, 2026, 05:06 PM UTC

   We are investigating delays from our queuing provider affecting some Windmill background workflows. Some jobs may start or finish later than expected. We are working to restore normal operation and will share another update when we have more information

2. resolved Sep 18, 2026, 05:27 PM UTC

   We are continuing to monitor this but the syncing system is healthy now and is working as normal

Read the full incident report →

Started Sep 14, 2026, 08:27 PM UTC· resolved Sep 14, 2026, 11:14 PM UTC

1. investigating Sep 14, 2026, 08:27 PM UTC

   Some customers with previously completed review cycles are receiving notifications from Windy. We are investigating the root cause and will reach out to anyone impacted directly

2. resolved Sep 14, 2026, 11:14 PM UTC

   The issue has been resolved. Approximately 2% of users were impacted and we have emailed everyone affected

Read the full incident report →

Started Sep 08, 2026, 12:42 AM UTC· resolved Sep 08, 2026, 12:57 AM UTC

1. investigating Sep 08, 2026, 12:42 AM UTC

   Web App is unavailable due to a faulty autoscaling event. Additional resources coming online.

2. resolved Sep 08, 2026, 12:57 AM UTC

   Additional resources have come online. This is now resolved.

Read the full incident report →

Started Aug 19, 2026, 03:02 PM UTC· resolved Aug 24, 2026, 01:04 AM UTC

1. investigating Aug 19, 2026, 03:02 PM UTC

   We're experiencing delays syncing user calendars and meetings. Users may experience delays seeing these update in the platform. We're investigating this incident.

2. resolved Aug 24, 2026, 01:04 AM UTC

   A fix has been released. Meeting syncing will gradually resolve over the next day.

Read the full incident report →

Started Jul 28, 2026, 04:28 PM UTC· resolved Jul 28, 2026, 07:10 PM UTC

1. investigating Jul 28, 2026, 04:28 PM UTC

   We have implemented a fix and are working on restoring missing peer feedback and nominations. We'll send an update when it is complete.

2. resolved Jul 28, 2026, 07:10 PM UTC

   We've restored all nominations and feedback. This is now resolved. Thank you for waiting.

Read the full incident report →

Started Jun 29, 2026, 02:47 PM UTC· resolved Jun 29, 2026, 03:45 PM UTC

1. investigating Jun 29, 2026, 02:47 PM UTC

   Seeing high load, users may experience slowness and downtime on web app.

2. resolved Jun 29, 2026, 03:45 PM UTC

   Experienced excessive load, and additional resources came online to resolve the slow performance.

Read the full incident report →

Started Jun 22, 2026, 02:40 PM UTC· resolved Jun 22, 2026, 02:45 PM UTC

1. investigating Jun 22, 2026, 02:40 PM UTC

   We have identified an issue with the web app not loading

2. resolved Jun 22, 2026, 02:45 PM UTC

   The issues loading the web app should be resolved.

Read the full incident report →

Started May 28, 2026, 01:41 AM UTC· resolved May 28, 2026, 01:41 AM UTC

1. resolved May 28, 2026, 01:41 AM UTC

   Slack issues are resolved: https://slack-status.com/2026-05/46d9d5d41fbedd9f

Read the full incident report →

Started May 14, 2026, 02:45 PM UTC· resolved May 14, 2026, 03:34 PM UTC

1. investigating May 14, 2026, 02:45 PM UTC

   Slack is having issues sending and responding to messages. As a result, Windy in Slack is having issues. We recommend using Chat In Platform in the meantime.

2. resolved May 14, 2026, 03:34 PM UTC

   Slack has resolved the issue.

Read the full incident report →

Started Apr 15, 2026, 02:10 PM UTC· resolved Apr 15, 2026, 10:37 PM UTC

1. investigating Apr 15, 2026, 02:10 PM UTC

   We are currently experiencing delays in our queues resulting in slow Windy response times and slower than expected integration syncing.

2. resolved Apr 15, 2026, 10:37 PM UTC

   The underlying root cause has been resolved and Windy should be responding normally.

Read the full incident report →

Started Mar 19, 2026, 03:03 AM UTC· resolved Mar 19, 2026, 03:27 AM UTC

1. investigating Mar 19, 2026, 03:03 AM UTC

   We are experiencing issues with db connectivity due to a configuration update.

2. resolved Mar 19, 2026, 03:27 AM UTC

   Due to a incorrect configuration update on our DB, we experienced ~17 minutes of downtime from 11:03 to 11:20pm. During this time all services were affected. We apologize for any inconvenience.

Read the full incident report →

Started Feb 09, 2026, 06:46 PM UTC· resolved Feb 09, 2026, 08:00 PM UTC

1. investigating Feb 09, 2026, 06:46 PM UTC

   We’re currently experiencing an outage affecting the web app. We’re investigating and working on a fix.

2. resolved Feb 09, 2026, 08:00 PM UTC

   We have resolved the issue

Read the full incident report →