Gigalixir Outage History

Started Sep 01, 2026, 03:25 PM UTC· resolved Sep 01, 2026, 04:11 PM UTC

1. investigating Sep 01, 2026, 03:25 PM UTC

   The us-central1-b availability zone appears to be offline for GCP. Some of our standard tier database in us-central1 region are unreachable at the moment. We are currently investigating this incident and awaiting GCP action.

2. monitoring Sep 01, 2026, 03:56 PM UTC

   We are seeing the zone recovery and most databases should be reachable again. we are still monitoring and will post when all is healthy.

3. monitoring Sep 01, 2026, 03:56 PM UTC

   Note from GCP: _Google Cloud multi-product customers are experiencing elevated packet loss and errors to/from us-central1-b._ _Most recent update – 17 minutes ago September 1, 2026 at 10:38:32 AM UTC-5_ _Google Cloud multi-product customers are experiencing elevated packet loss and errors to/from us-central1-b. We are experiencing an issue with Hybrid Connectivity Tuesday, 2026-09-01 07:44 US/Pacific. Our engineering team continues to investigate the issue. We will provide more information by Tuesday, 2026-09-01 09:15 US/Pacific with current details._

4. monitoring Sep 01, 2026, 04:06 PM UTC

   The AZ is recovering, we are seeing most back online now. If you still are experiencing issues, write us immediately at [[email protected]](mailto:[email protected]) so we can investigate.

5. resolved Sep 01, 2026, 04:11 PM UTC

   This incident has been resolved, noted both on our end and by GCP below. All databases appears to be functioning. Any continuing issues, please email us immediately at [email protected]. GCP Note regarding the outage: September 1, 2026 at 11:09:16 AM UTC-5 Multiple products in us-central1-b are experiencing network service degradation. We are experiencing a networking issue starting Tuesday, 2026-09-01 07:41 US/Pacific. The issue is affecting a portion of the us-central1-b zone. The remaining zones in the region are unaffected. Our network engineering team has put in mitigations and we are seeing services starting to recover. We will provide more information by Tuesday, 2026-09-01 09:30 US/Pacific with current details.

Read the full incident report →

Started Mar 03, 2026, 11:14 PM UTC· resolved Mar 04, 2026, 12:34 AM UTC

1. investigating Mar 03, 2026, 11:14 PM UTC

   We are currently investigating this incident..

2. identified Mar 03, 2026, 11:29 PM UTC

   We are continuing to work on a fix for this incident. .

3. investigating Mar 03, 2026, 11:54 PM UTC

   Investigating: Some applications in the US-Central-1 region are experiencing intermittent failures. This includes elevated 502 responses, occasional TLS/certificate errors, and the Gigalixir console () currently being unavailable. Our engineering team is actively investigating the issue and working on mitigation. Updates will be posted here as we learn more..

4. monitoring Mar 04, 2026, 12:16 AM UTC

   The earlier issues affecting applications in the US-Central-1 region, including intermittent 502 errors, TLS/certificate errors, and temporary unavailability of the Gigalixir console (), appear to be resolved. Services have recovered and applications should be operating normally. We are continuing to monitor the system to ensure stability..

5. resolved Mar 04, 2026, 12:34 AM UTC

   Applications in the US-Central-1 region experienced intermittent 502 responses, TLS/certificate errors, and temporary unavailability of the Gigalixir console. The root cause was a configuration distribution issue within our ingress system that caused certificate and routing disruptions across the region. Our engineering team identified and resolved the issue, and all services have been restored. We are implementing changes to further isolate ingress configurations and prevent similar issues in the future. We will continue monitoring to ensure stability..

6. monitoring Mar 05, 2026, 05:29 PM UTC

   Updated March 4, 2026 # Description of Issue A configuration distribution issue within our ingress system caused certificate and routing disruptions across the us-central1 region. This resulted in intermittent 502 responses, TLS/certificate errors, and temporary unavailability of the Gigalixir console. # Scope of the Issue The issue affected applications running in the us-central1 GCP region. During the remediation process, each ingress system required a rolling restart, which caused a brief but broader period of disruption. For the affected applications, this downtime lasted 9 minutes on average. # Prevention Measures We are implementing changes to further isolate ingress configurations across our infrastructure. These measures include, but are not limited to: * improved isolation of ingress configuration distribution * additional validation of configuration changes before application * enhanced monitoring and alerting for ingress health # Customer Recommendations There are no customer recommendations for this incident. The incident was entirely internal to the infrastructure of Gigalixir. # Incident Timeline ## 3 March - 22:15 UTC / 16:15 CST We began to see intermittent disruptions affecting a small number of applications on one of our shared ingress systems. The impact was limited and sporadic at this stage. ## 3 March - 22:35 UTC / 16:35 CST Our monitoring systems raised alerts indicating elevated error rates. Our engineering team began investigating the issue as a possible concern for more than just the one application we had detected at 16:15 CST. Our first customer report came in at 22:40 UTC. ## 3 March - 23:13 UTC / 17:13 CST We determined the issue was not isolated to individual applications but was a system-level ingress configuration problem. We declared an incident and began working on determining the root cause. ## 3 March - 23:32 UTC / 17:32 CST The root cause was identified. Invalid configurations were being applied to the ingress systems in an intermittently destructive manner. We began to investigate the best solution to resolve the problem with the least amount of additional disruption. ## 3 March - 23:46 UTC / 17:46 CST It was determined the best fix would require each of our ingress systems to be forcibly restarted. Although this would result in guaranteed downtime for applications, it was deemed the safest way to recover fully. We began applying the fix to the ingress system. This required a rolling restart of the ingress infrastructure, which temporarily caused a broader disruption. During this window, most applications experienced intermittent SSL errors and 503 responses. The average downtime per application during this phase was approximately 9 minutes. ## 4 March - 00:04 UTC / 18:04 CST The fix was fully applied and all services were restored. We continued to monitor the system to ensure stability. ## After resolution We have also added additional verifications in our configuration distribution to prevent application of errant configurations. We are implementing additional isolation measures for our ingress configuration distribution to prevent similar issues in the future..

Read the full incident report →

Started Feb 09, 2026, 07:12 PM UTC· resolved Feb 09, 2026, 09:03 PM UTC

1. investigating Feb 09, 2026, 07:12 PM UTC

   We are currently investigating the repercussions of GitHub's issues for git operations. This will possibly affect some Gigalixir operations if you're pushing to production new builds. .

2. identified Feb 09, 2026, 07:36 PM UTC

   We are continuing to monitor GitHub's status for their expected timeline to fix their incident..

3. monitoring Feb 09, 2026, 08:26 PM UTC

   We continue to monitor, please reach out to [[email protected]](mailto:[email protected]) if you are affected and need assistance. .

4. resolved Feb 09, 2026, 08:49 PM UTC

   This incident has been resolved. We'll continue to monitor, please reach out if you have questions. .

5. resolved Feb 09, 2026, 09:03 PM UTC

   This incident has been resolved..

Read the full incident report →

Started Sep 19, 2025, 06:10 PM UTC· resolved Sep 19, 2025, 09:35 PM UTC

1. identified Sep 19, 2025, 06:10 PM UTC

   GCP us-central region. Increase in traffic started approx 1:10 PM central. Requests are intermittent, but we have put in some fixes to relieve the system. --- We are monitoring and continuing to investigate the performance. .

2. monitoring Sep 19, 2025, 07:14 PM UTC

   We have applied mitigations. Our systems have recovered. We are actively monitoring the situation..

3. resolved Sep 19, 2025, 09:35 PM UTC

   This incident has been resolved and we will continue to monitor. .

4. monitoring Sep 22, 2025, 02:48 PM UTC

   2025-09-19 us-central1 Ingress Incident Report Updated September 22, 2025 # Description of Issue We received a DDoS attack on one of our shared ingress systems. The attack originated from many distinct sources. A distributed denial-of-service (DDoS) attack is a malicious attempt to disrupt the normal traffic of a targeted server, service or network by overwhelming the target or its surrounding infrastructure with a flood of Internet traffic. DDoS attacks achieve effectiveness by utilizing multiple compromised computer systems as sources of attack traffic. # Scope of the Issue The attack was able to degrade our ability to process incoming requests on one of our shared ingress systems in the us-central1 GCP region. The attack prevented or degraded incoming traffic to applications that are on the same shared ingress setup. Applications on our dedicated ingress infrastructure were not affected. Applications on adjacent shared ingress systems were also not affected. We were able to identify and mitigate the effects relatively quickly, restoring traffic to affected applications. We were able to block the attacker(s) after applying additional measures. # Prevention Measures We have applied several layers of protections to make our system more resilient to this type of attack in the future. These measures include, but are not limited to: * dynamic rate limiting * dynamic firewall rules * additional monitors and alerts Additionally we are working to split our shared ingress systems into smaller pieces to limit the scope of any similar attacks in the future. # Customer Recommendations Gigalixir is always working to improve our infrastructure to prevent and lessen the impact of However, for general application protection we would recommend the following: 1\. **Run more than one Replica** When you run more than one replica, we run them on multiple servers and across multiple zones. This helps prevent an issue on a single server or zone from completely taking your application offline. 2\. **Consider Dedicated Ingress** Dedicated Ingress gives your applications their own load balancer and ingress resources. Applications on dedicated ingress also run in a separate runtime server pool from our common runtime server pool, which provides further isolation. 3\. **Consider DDoS Protections and/or WAF** There are a handful of good products that offer protections for traffic coming through your domain and/or hostnames. _(We can provide individual recommendations upon request.)_ 4\. **Source Filtering** We commonly work with customers to add source filtering in our system to ensure traffic only comes from trusted sources. The details of this vary depending on the customer's domain setup. One of our preferred solutions is to utilize Cloudflare for your DNS with a rule that applies a signature to all requests. If you couple this with our Dedicated Ingress, we can limit all traffic through the ingress system to only traffic coming from your Cloudflare setup. Cloudflare offers DDoS protections for all plan levels, including their free plan. # Incident Timeline ## 19 September - 16:40 UTC 11:40 CDT We started to receive elevated levels of ingress traffic on the affected shared ingress system. Our system was handling this scenario; scaling to account for the additional traffic. This traffic remained elevated, but managed until 12:50pm CDT. At 12:50pm, the traffic levels settled back to a nominal level. ## 19 September - 18:10 UTC / 13:10 CDT A new flood of traffic came into the affected ingress system. This time our system began to drop packets in the affected ingress system. Our team increased the resources we were dedicating to ingress and worked to identify the patterns and sources of the traffic. After these changes, our system was delivering traffic to customer applications again, but the threat was still active. ## 19 September - 18:40 UTC / 13:40 CDT With the additional resources we added, customer traffic was healthy until 1:40pm CDT. At this time a few of our ingress servers restarted, but the large majority remained online and healthy. We increased our resources limits further at this time help with the situation. ## 19 September - 18:50 UTC / 13:50 CDT At this time we were able to uniquely identify the traffic pattern used in the attack. We were able to isolate this traffic away from the other applications on the affected ingress system. We added some additional identification systems to help us block/reject the traffic further upstream. We began to block the traffic at our firewall. Our team continued to monitor the traffic and our system’s response. ## 19 September - 21:30 UTC / 16:30 CDT We considered the incident to be resolved. We were able to block the attack at our firewall and had implemented several measures to identify and block similar attacks in the future. ## After resolution We continued to implement measures, alerts, and strategies to prevent the problem as well as to improve our response and recovery times. .

Read the full incident report →

Started Jun 12, 2025, 06:10 PM UTC· resolved Jun 12, 2025, 10:06 PM UTC

1. identified Jun 12, 2025, 06:10 PM UTC

   AWS, GCP, and Cloudflare are showing large outages. These are all downstream providers for us. At the moment, new builds/deploys are not going out. Applications largely continue to run, but degraded performance can be expected. (Updated status with proper start date).

2. monitoring Jun 12, 2025, 07:06 PM UTC

   Per Google: **Multiple GCP products are experiencing impact due to Identity and Access Management Service Issue** .

3. monitoring Jun 12, 2025, 08:07 PM UTC

   Cloudflare and GCP continue to try to get things back to normal, but no new updates providing a clear path to resolution yet. .

4. monitoring Jun 12, 2025, 08:23 PM UTC

   It appears we're seeing progress towards recovery, except for GCP US-Central1 Per Google: _Our infrastructure has recovered in all regions except us-central1._.

5. resolved Jun 12, 2025, 10:06 PM UTC

   It appears the multiple downstream incidents have been resolved. All services and components should be fully functional. As always, we'll keep monitoring and let us know if you see anything not behaving as expected so we can investigate. .

6. monitoring Jun 13, 2025, 08:36 PM UTC

   Postmortem from Google: Multiple GCP products are experiencing Service issues - .

Read the full incident report →

Started Mar 30, 2025, 04:15 PM UTC· resolved Mar 30, 2025, 04:49 PM UTC

1. monitoring Mar 30, 2025, 04:15 PM UTC

   We noticed an issue for roughly 25 minutes due to processing an unusual flow of traffic through this region. We have taken steps to mitigate the impacts on the system and are addressing the root cause. We will continue to monitor the situation, but everything is operating as expected..

2. resolved Mar 30, 2025, 04:49 PM UTC

   We will continue to monitor the situation, but everything is operating as expected, so we will mark this resolved..

Read the full incident report →

Started Nov 07, 2024, 04:31 PM UTC· resolved Nov 07, 2024, 05:10 PM UTC

1. investigating Nov 07, 2024, 04:31 PM UTC

   We've seen a few customers with issues related to NPM builds not deploying successfully. It appears that NPM has changed their packaging system, causing the build failures. We should have a solution deployed shortly..

2. resolved Nov 07, 2024, 05:10 PM UTC

   As we believed, there was a change in the packaging system, causing build failures. We have applied a fix for the nodejs resolution issues during the build process. Please contact support if you are still experiencing issues with deploying..

Read the full incident report →