Neo4j Aura Outage History

Started Oct 20, 2025, 09:15 AM UTC· resolved Oct 20, 2025, 11:13 AM UTC

1. identified Oct 20, 2025, 09:15 AM UTC

   An active issue on AWS us-east-1 is affecting Neo4j Aura operations in the regions where we have some dependency on an impacted AWS service. See more details https://health.aws.amazon.com/health/status Other regions not currently impacted.

2. identified Oct 20, 2025, 09:39 AM UTC

   Operations such as Create, Pause, Resume, Clone are affected globally due to a dependency. Backups in the affected region of AWS us-est-1 will be impacted The affected AWS service is now starting to recover and we are monitoring the situation

3. monitoring Oct 20, 2025, 09:58 AM UTC

   We see clear signs of recovery and our impacted service are progressively getting back to normal. We will monitor and update when we can confirm full recovery and normal operations.

4. monitoring Oct 20, 2025, 10:32 AM UTC

   We are continuing to see good recovery across the service. We will update when this is completed.

5. resolved Oct 20, 2025, 11:13 AM UTC

   The underlying AWS incident has bene resolved and our services have now resumed and fully recovered

6. postmortem Jan 15, 2026, 05:34 PM UTC

   ### **What Happened** Between 08:41 UTC on October 20th and 09:00 UTC on October 21st, Neo4j Aura experienced service disruptions affecting the us-east-1 \(N. Virginia\) region in AWS. The incident was triggered by a broad AWS regional outage impacting Identity and Access Management \(IAM\) and the EC2 control plane. This resulted in delayed backups, temporary loss of database fault tolerance for a subset of users, and internal delays in administrative actions due to toolchain failures. A detailed summary of the AWS regional incident can be found here: [https://aws.amazon.com/message/101925/](https://aws.amazon.com/message/101925/) ### **How the service was affected** The primary cause was several AWS regional service disruptions in us-east-1. We will cover how each of these affected Neo4j Aura and its users. Neo4j Aura is designed to isolate regional failures. This is achieved through deploying customer instances in Orchestras that have instances in three availability zones and no cross region dependencies. AWS IAM and Identity Center became unresponsive, preventing Neo4j Aura’s automated systems from authenticating with AWS resources in us-east-1. This affected Neo4j Aura backups to be written to AWS S3 buckets. Customers were also not able to resume paused instances during this period as the resume process was not able to authenticate with AWS S3 buckets to retrieve the paused data set. Neo4j Aura’s inability to authenticate with Route53 to create new DNS entries affected Neo4j Aura DB creation, as new databases were created. AWS Network Load Balancer health check system failures and AWS EC2 “request limit exceeded” or “insufficient capacity” errors in us-east-1 were false negatives with the NLB heath checks which resulted in some Neo4j Aura DB clusters in us-east-1 losing fault tolerance \(1 out of 3 cluster members unavailable\). When this happened instances were removed by kubernetes and we were not able to provision new ones due to the EC2 failures noted earlier, leaving the clusters without fault tolerance for a prolonged period of time. All of these clusters still had full availability of the other two cluster members. ### **What are we doing now** Largely Neo4j Aura responded as designed to these events, isolating failures to us-east-1, with no to minimal cross system failure propagation. The cross cloud impact of creating DNS records for new instances was the main outlier. To mitigate the scope of impact of future regional outages, Neo4j is implementing the following measures: * Full Neo4j Aura reviews to ensure Neo4j Aura design is implemented as intended across all sub-systems to surface any cross region dependencies. * Remove identified cross-region dependencies.

Read the full incident report →

Started Aug 25, 2025, 01:34 PM UTC· resolved Aug 27, 2025, 04:46 PM UTC

1. investigating Aug 25, 2025, 01:34 PM UTC

   We are currently investigating this issue.

2. investigating Aug 25, 2025, 01:36 PM UTC

   We are aware of an issue that could result in a degradation in write performance for all Aura instance. We have identified the cause and are preparing a fix.

3. investigating Aug 25, 2025, 01:47 PM UTC

   We are continuing to investigate this issue.

4. monitoring Aug 25, 2025, 03:34 PM UTC

   The issue impact has been refined and currently could impact AuraDB Virtual Dedicated Cloud and Business Critical instances only. We have identified the cause and the faulty component has been reverted for the majority of instances. We will continue to monitor the situation, while the full fix is prepared.

5. monitoring Aug 25, 2025, 05:40 PM UTC

   We continue to work towards the issue resolution.

6. monitoring Aug 25, 2025, 09:44 PM UTC

   We continue to work towards the issue resolution.

7. monitoring Aug 26, 2025, 04:06 AM UTC

   We are continuing to investigate this issue.

8. monitoring Aug 26, 2025, 10:53 AM UTC

   We are continuing to monitor the situation whilst a full fix for the solution is prepared.

9. monitoring Aug 26, 2025, 02:48 PM UTC

   We are continuing to monitor the situation. The full fix for the solution is currently undergoing final testing and will start deployment once testing is completed.

10. monitoring Aug 26, 2025, 05:17 PM UTC

    The full fix for the solution is starting to be deployed across the estate and are continuing to monitor the situation.

11. monitoring Aug 26, 2025, 08:13 PM UTC

    The full fix for the solution is starting to be deployed across the estate and are continuing to monitor the situation.

12. monitoring Aug 26, 2025, 11:12 PM UTC

    The full fix for the solution is starting to be deployed across the estate and are continuing to monitor the situation.

13. monitoring Aug 27, 2025, 06:36 AM UTC

    We are continuing to monitor for any further issues.

14. monitoring Aug 27, 2025, 11:16 AM UTC

    The full fix for the solution is continuing to be deployed across the estate and are continuing to monitor the situation.

15. resolved Aug 27, 2025, 04:46 PM UTC

    The full fix for the solution has now been deployed across the estate and this issue is now resolved.

16. postmortem Oct 06, 2025, 01:32 PM UTC

    ### **What happened** Between August 22 and August 28, 2025, a performance issue affected some of our database services following a recent update. Our team quickly responded and discovered that the problem was due to a bug in the latest update, which caused certain memory settings to be incorrectly configured. We promptly applied the previous version of those settings to stabilize the affected databases. By August 28, all databases were successfully transferred to a new, stable version, and the incident was fully resolved. The issue was caused by a misconfiguration in the system's data processing module. Specifically, an incorrect parameter setting in the data pipeline led to a bottleneck, which slowed down the processing speed. This misconfiguration affected the way data was being queued and processed, resulting in delays. Our technical team has identified the root cause and implemented a fix to ensure that the data pipeline operates efficiently, preventing future occurrences of this issue. ### **How the service was affected** Some customers reported a performance change on their instance. ### **What we are doing now** Neo4j remains committed to providing reliable service and is implementing additional safeguards to prevent similar incidents in the future. New mitigations being deployed: * Enhancing our monitoring systems to detect similar issues faster * Implementing additional automated checks to prevent service disruptions * Improving our service resilience to handle similar scenarios * Introducing dynamic configuration management to ensure seamless updates * Enabling multiple configuration defaults to better manage database priorities * Improving notification systems for faster response to potential issues * Enhancing our tools to streamline database management and transfers

Read the full incident report →

Started Aug 22, 2025, 11:02 AM UTC· resolved Aug 22, 2025, 01:31 PM UTC

1. investigating Aug 22, 2025, 11:02 AM UTC

   Engineers are investigating an issue whereby instance actions are not working. You may be unable to import data, pause, resume, create, delete or run manual backups for your instances. Regular Neo4j scheduled backups are continuing to run as normal. You may be unable to see your instances in the Aura Console. CMI is also experiencing an outage, you may be unable to ingest metrics from your Aura instances.

2. monitoring Aug 22, 2025, 11:42 AM UTC

   A fix has been identified and implemented. Instances should be showing on the console and instance operations should now be available. You should now also be able to ingest metrics using CMI. We are continuing to monitor the status of these services.

3. resolved Aug 22, 2025, 01:31 PM UTC

   This incident has been resolved.

4. postmortem Sep 12, 2025, 03:45 PM UTC

   ### What happened On August 22, 2025, from 10:10 to 11:27 UTC, users experienced an issue with the Aura Console where instance operations were unavailable. This was due to a timing issue with refreshing security keys that affected the visibility and operations of instances in the Aura Console and also caused an outage in the Customer Metrics Integration \(CMI\) and Aura API. The issue was identified within 5 minutes and efforts to resolve it began promptly. The problem was traced back to a recent update and issues with authentication were resolved by refreshing the system cache, with the service fully restored by 11:27 UTC. The issue arose because of a timing mismatch between the creation of new security keys and their recognition by our system. When new keys were generated, they were quickly used by the Console API. However, another part of our system, the database manager, was still using an older set of keys to verify requests. This mismatch caused requests to fail because the database manager could not recognize the new keys. No unauthorized operations were allowed and this did not affect the security of the service in any way. After approximately an hour, the system automatically updated its keys, and everything started working correctly again. ### How customers were affected Aura Console features were impacted, including the visibility of instances and therefore ability to connect to instances through the console. Driver connections to instances were not impacted. The issue also impacted the use of parts of the service including Customer Metrics Integration \(CMI\) and the Aura API. ### What we are doing now Neo4j remains committed to providing reliable service and is implementing additional safeguards to prevent similar incidents in the future. We are taking steps to prevent similar issues in the future by improving our processes and monitoring systems. Changes we are making: * Enhancing our caching mechanisms to ensure seamless updates and prevent similar issues in the future. * Designing a robust process for updating service accounts and keys to guarantee uninterrupted service reliability.

Read the full incident report →

Started Aug 19, 2025, 09:19 AM UTC· resolved Aug 19, 2025, 12:04 PM UTC

1. investigating Aug 19, 2025, 09:19 AM UTC

   We are currently investigating an issue with backups. We will update shortly as we get more details

2. investigating Aug 19, 2025, 10:05 AM UTC

   We are continuing to investigate the issue with backups. New backups after 0:00 last night are failing to complete. Existing backups are unaffected and remain available.

3. investigating Aug 19, 2025, 10:34 AM UTC

   We are continuing to investigate this issue.

4. investigating Aug 19, 2025, 10:51 AM UTC

   We continue to investigate and have identified the issue more clearly. Beyond backups, other operations are also affected: pause and clone. Operations may fail or take a long time to complete.

5. monitoring Aug 19, 2025, 11:30 AM UTC

   The fix has been deployed and we will continue to monitor this incident.

6. resolved Aug 19, 2025, 12:04 PM UTC

   This incident has been resolved.

7. postmortem Sep 12, 2025, 07:41 PM UTC

   ### What happened On August 18, 2025, at 19:18 UTC, the release of a faulty component in Neo4j Aura impacted the ability to perform backups of customer instances. The problem was identified when a critical component was updated without aligning with another dependent component, leading to failed backup attempts. This resulted in Neo4j not being able to take backups of customer instances for a period of approximately 17 hours. Our team quickly identified the root cause and began mitigation efforts early on August 19, 2025. By 11:54 UTC, the issue was fully resolved, and normal backup operations resumed. We apologize for any inconvenience this may have caused and are taking steps to prevent similar issues in the future. The issue arose because the deployment of the neo4j-backup system was not properly synchronized with the deployment of the neo4j-operator. This misalignment led to critical changes being introduced that impacted the ability to create new backups effectively. Essentially, the two components were not updated in harmony, which caused disruptions in the backup process. ### How customers were affected Customers were unable to get successful backups of their instances for 17 hours. We promptly addressed the issue by reverting the recent backup changes that caused disruptions and ensured the stability of our service by cleaning up old, failing backups. ### What we are doing now Neo4j remains committed to providing reliable service and is implementing additional safeguards to prevent similar incidents in the future. New mitigations being deployed: * Enhancing our monitoring systems to detect similar issues faster * Implementing additional automated checks to prevent service disruptions * Improving our service resilience to handle similar scenarios

Read the full incident report →

Started Jun 30, 2025, 08:40 PM UTC· resolved Jul 03, 2025, 06:32 PM UTC

1. identified Jul 01, 2025, 04:38 PM UTC

   We have identified an issue where the PUBLIC role cannot be modified on certain Business Critical instances. The public role is available to all database users, and contains a limited set of privileges outlined in our documentation: https://neo4j.com/docs/operations-manual/current/authentication-authorization/built-in-roles/#access-control-built-in-roles-public

2. identified Jul 02, 2025, 05:17 AM UTC

   We have identified the issue and a fix is being rolled out

3. resolved Jul 03, 2025, 06:32 PM UTC

   We have applied a fix and verified this is fully resolved.

4. postmortem Jul 15, 2025, 02:51 PM UTC

   ## **What happened** On June 27th, the new predefined roles feature\* for AuraDB customers inadvertently removed database Admin access from some Business Critical \(BC\) instances. As a temporary fix, the team disabled the feature for Business Critical instances. While the fix took place, it also replaced the database public RBAC role privileges which are available on Business Critical with enhanced public role privileges available on Free/Pro instances. System logic prevented customers from modifying the public role during that time.\*[https://neo4j.com/docs/aura/user-management/#\_predefined\_roles](https://neo4j.com/docs/aura/user-management/#_predefined_roles) Neo4j restored the public role to its original limited permissions for all Virtual Dedicated Cloud instances on June 30, and for all Business Critical instances on July 1. ## **How customers were affected** Customers were affected by this issue from June 27th to July 1st. The issue impacted some Business-Critical \(BC\) instances and a small number of Virtual Dedicated Cloud \(VDC\) instances. All project admin users with affected instances received an email confirming the issue was resolved on July 1st. ## **What we are doing now** Neo4j remains committed to providing reliable service and is implementing additional safeguards to prevent similar incidents in the future: * We have implemented safeguards in our system to prevent misconfigured role changes and better isolate feature toggles from production permission models * We are reviewing our internal release processes to improve testing and validation of role-based access control \(RBAC\) changes * We are implementing additional detection methods for misconfigured RBAC which will alert our operational teams

Read the full incident report →

Started Jun 27, 2025, 09:48 AM UTC· resolved Jun 27, 2025, 10:36 AM UTC

1. investigating Jun 27, 2025, 11:48 AM UTC

   AuraDB Business Critical customers may face issues with running queries. We are investigating Error returned: ACCESS on database 'neo4j' is not allowed for user 'neo4j' with roles [PUBLIC]

2. resolved Jun 27, 2025, 12:02 PM UTC

   We have addressed the issue and rolled back a feature involved with this problem.

3. postmortem Jul 15, 2025, 02:50 PM UTC

   ## **What happened** On June 27th, the new predefined roles feature\* for AuraDB customers inadvertently removed database Admin access from some Business Critical \(BC\) instances. As a temporary fix, the team disabled the feature for Business Critical instances. While the fix took place, it also replaced the database public RBAC role privileges which are available on Business Critical with enhanced public role privileges available on Free/Pro instances. System logic prevented customers from modifying the public role during that time.\*[https://neo4j.com/docs/aura/user-management/#\_predefined\_roles](https://neo4j.com/docs/aura/user-management/#_predefined_roles) Neo4j restored the public role to its original limited permissions for all Virtual Dedicated Cloud instances on June 30, and for all Business Critical instances on July 1. ## **How customers were affected** Customers were affected by this issue from June 27th to July 1st. The issue impacted some Business-Critical \(BC\) instances and a small number of Virtual Dedicated Cloud \(VDC\) instances. All project admin users with affected instances received an email confirming the issue was resolved on July 1st. ## **What we are doing now** Neo4j remains committed to providing reliable service and is implementing additional safeguards to prevent similar incidents in the future: * We have implemented safeguards in our system to prevent misconfigured role changes and better isolate feature toggles from production permission models * We are reviewing our internal release processes to improve testing and validation of role-based access control \(RBAC\) changes * We are implementing additional detection methods for misconfigured RBAC which will alert our operational teams

Read the full incident report →

Started Jun 12, 2025, 06:42 PM UTC· resolved Jun 13, 2025, 05:18 AM UTC

1. investigating Jun 12, 2025, 06:42 PM UTC

   We've detected that Aura Console is currently unreachable. We are actively investigating this incident. Individual instances should not be effected.

2. investigating Jun 12, 2025, 06:57 PM UTC

   Our cloud provider is currently experiencing an outage that is impacting Aura Console reachability. At this point it is reachable but may be intermittent until the the outage is fully resolved.

3. identified Jun 12, 2025, 07:13 PM UTC

   The identified issue is reported on our cloud provider's status page: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW We are continuing to work to restore Console availability.

4. identified Jun 12, 2025, 07:33 PM UTC

   We are updating this to partial outage as Console is reachable more consistently in recent tests. The underlying cause of the outage for our cloud providers is still under investigation. According to that, services are restoring and we expect full operations once that is fully restored.

5. monitoring Jun 13, 2025, 12:23 AM UTC

   Aura Console should now be fully operational and available now that the underlying cloud provider's services have all recovered.

6. resolved Jun 13, 2025, 05:18 AM UTC

   This incident has been resolved Aura Console is operational and is available to use without any issues

7. postmortem Jul 23, 2025, 01:45 PM UTC

   ### What happened On June 12, 2025, at 17:49 UTC, a service disruption occurred affecting the Aura Console, Aura API, CMI, and Aura Graph Analytics, making them fully unavailable for approximately 1.5 hours. The issue was caused by a change in the Google Cloud Platform \(GCP\) that led to errors in communication with the Aura Console API. The problem was identified quickly, and services began to recover around 19:04 UTC. By 01:27 UTC on June 13, GCP confirmed that all services were fully restored. We apologize for any inconvenience this may have caused and appreciate your understanding. ### How customers were affected The outage was caused by a recent update in Google Cloud Platform \(GCP\) affecting the Data store service. This update inadvertently affected the way our systems interact with GCP services, leading to a temporary disruption. Our team actively worked with GCP to monitor the issue and ensure that our systems became fully operational as quickly as possible. Details here: [https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW](https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW) The issue was promptly resolved by GCP reversing the change that caused the disruption, ensuring that our services were restored to normal operation. ### What we are doing now Neo4j remains committed to providing reliable service and is implementing additional safeguards to prevent similar incidents in the future. New mitigations being deployed: * Enhancing our monitoring systems to detect similar issues faster * Implementing additional automated checks to prevent service disruptions

Read the full incident report →

Started Jun 10, 2025, 04:41 AM UTC· resolved Jun 10, 2025, 09:46 AM UTC

1. investigating Jun 10, 2025, 04:41 AM UTC

   Some Free-tier users are experiencing issues connecting to their instances through the default console experience. As a temporary workaround, we recommend switching to the classic console experience by going to your profile → Experience → Classic experience. Our team is actively investigating the issue and working toward a resolution

2. investigating Jun 10, 2025, 06:13 AM UTC

   Some Free-tier users are experiencing issues connecting to their instances through the default console experience. As a temporary workaround, we recommend switching to the classic console experience by going to your profile → Experience → Classic experience. Our team is actively investigating the issue and working toward a resolution

3. identified Jun 10, 2025, 08:12 AM UTC

   We’ve identified the potential cause of the issue and are currently working on mitigating it. We’ll keep you updated as soon as we have more information to share.

4. resolved Jun 10, 2025, 09:46 AM UTC

   This issue is resolved after rolling out a fix in production. Users can now connect to the free tier instances from the new console without any issues.

Read the full incident report →

Started May 21, 2025, 01:40 PM UTC· resolved May 22, 2025, 09:09 AM UTC

1. investigating May 21, 2025, 01:40 PM UTC

   Engineering have identified an increase in latency with our Customer Metric Integration endpoints, they are currently investigating the cause. If you are seeing timeouts when attempting to fetch metrics we recommend increasing your timeout value to 20 seconds temporarily until this incident is resolved.

2. investigating May 21, 2025, 02:44 PM UTC

   Engineering are continuing to investigate the root cause of the issue. If you are seeing timeouts when attempting to fetch metrics we recommend increasing your timeout value to 20 seconds temporarily until this incident is resolved.

3. investigating May 21, 2025, 04:13 PM UTC

   Engineering are continuing to investigate the root cause of the issue. If you are seeing timeouts when attempting to fetch metrics we recommend increasing your timeout value to 20 seconds temporarily until this incident is resolved.

4. investigating May 21, 2025, 05:35 PM UTC

   Engineering are continuing to investigate the root cause of the issue. If you are seeing timeouts when attempting to fetch metrics we recommend increasing your timeout value to 20 seconds temporarily until this incident is resolved.

5. investigating May 21, 2025, 06:51 PM UTC

   Engineering are continuing to investigate the root cause of the issue. If you are seeing timeouts when attempting to fetch metrics we recommend increasing your timeout value to 20 seconds temporarily until this incident is resolved.

6. investigating May 21, 2025, 07:57 PM UTC

   Engineering are continuing to investigate the root cause of the issue. If you are seeing timeouts when attempting to fetch metrics we recommend increasing your timeout value to 20 seconds temporarily until this incident is resolved.

7. investigating May 21, 2025, 08:50 PM UTC

   Engineering are continuing to investigate the root cause of the issue. If you are seeing timeouts when attempting to fetch metrics we recommend increasing your timeout value to 20 seconds temporarily until this incident is resolved.

8. investigating May 21, 2025, 09:48 PM UTC

   Engineering are continuing to investigate the root cause of the issue. If you are seeing timeouts when attempting to fetch metrics we recommend increasing your timeout value to 20 seconds temporarily until this incident is resolved.

9. investigating May 21, 2025, 10:47 PM UTC

   Engineering are continuing to investigate the root cause of the issue. If you are seeing timeouts when attempting to fetch metrics we recommend increasing your timeout value to 20 seconds temporarily until this incident is resolved.

10. investigating May 21, 2025, 11:46 PM UTC

    Engineering are continuing to investigate the root cause of the issue. If you are seeing timeouts when attempting to fetch metrics we recommend increasing your timeout value to 20 seconds temporarily until this incident is resolved.

11. investigating May 22, 2025, 12:45 AM UTC

    Engineering are continuing to investigate the root cause of the issue. If you are seeing timeouts when attempting to fetch metrics we recommend increasing your timeout value to 20 seconds temporarily until this incident is resolved.

12. investigating May 22, 2025, 04:30 AM UTC

    Our Engineering team is continuing to investigate the root cause of the issue. If you are seeing timeouts when attempting to fetch metrics, we recommend temporarily increasing your timeout value to 20 seconds until this incident is resolved.

13. monitoring May 22, 2025, 07:37 AM UTC

    The issue has been resolved, and we are currently monitoring the system to ensure continued stability. If you were experiencing timeouts while fetching metrics, those should now be resolved. We will continue to observe the system and provide further updates if necessary.

14. resolved May 22, 2025, 09:09 AM UTC

    The latency issue affecting our Customer Metric Integration endpoints has been resolved. All services operate normally now, and timeout errors should no longer occur.

15. postmortem Jun 13, 2025, 10:11 AM UTC

    ### **What happened** At approximately 17:00 UTC on 2025-05-20 our cloud provider released and rolled out a change on the managed version of Prometheus we use to provide the Customer Metrics Integration \(CMI\) endpoint. This change affected our production PromQL query performance because “_the change to the PromQL query path now evaluates queries that previously had empty results_”. This was a change we had no warning and no control over and effected multiple customers. We quickly raised the issue to our cloud provider and they rolled back the change. While we were checking the root cause of the issue we immediately recommended increasing the timeout value to 20 seconds as a remediation. ### **How the service was affected** Customers with low timeout settings on their PromQL queries to fetch metrics from the Neo4j Aura CMI endpoint would see an increase in query timeouts \(HTTP error 499\). ### **What we are doing now** This incident was not caused by anything Neo4j directly controls but we have been looking at improving our handling of this situation and have devised the following actions: * Provided feedback to our cloud provider on the impact this has had on our service. * Added CMI endpoint on the statuspage: Aura Customer Metrics \([customer-metrics-api.neo4j.io](http://customer-metrics-api.neo4j.io)\) to better represent the status of the service. * Updated our documentation and recommended a larger timeout be set. * Reviewed improvements to our detection and alerting in timeouts and errors to queries \(499 and 5xx errors\) * Review our own queries to make them more efficient and more resilient to a performance degradation from our suppliers’ service

Read the full incident report →

Started May 07, 2025, 08:18 AM UTC· resolved May 07, 2025, 09:24 PM UTC

1. investigating May 07, 2025, 08:18 AM UTC

   We have identified an issue impacting the availability of metrics on instances across all tiers. Our team is currently investigating the root cause. In the meantime, you may experience difficulties monitoring your instances or accessing metric data.

2. investigating May 07, 2025, 08:20 AM UTC

   We have identified an issue impacting the availability of metrics on instances across all tiers. Our team is currently investigating the root cause. In the meantime, you may experience difficulties monitoring your instances or accessing metric data.

3. investigating May 07, 2025, 10:09 AM UTC

   We are still investigating the issue affecting the availability of metrics. As a result, you may experience difficulties monitoring your instances or accessing metric data. We appreciate your patience as we work to resolve it.

4. identified May 07, 2025, 01:43 PM UTC

   We have identified the issue and have a change that we will rollout shortly.

5. identified May 07, 2025, 03:40 PM UTC

   We have identified the issue and have a change that we will rollout shortly.

6. monitoring May 07, 2025, 04:47 PM UTC

   We have rolled out the fix and the metrics are available again. We will monitor for some time.

7. resolved May 07, 2025, 09:24 PM UTC

   The fix implemented by our engineers has resolved the issue. Users can once again use Aura Metrics

8. postmortem Jun 04, 2025, 03:51 PM UTC

   ### **What happened** As we rolled out our regular monthly release \(2025.04\), we introduced the possibility for the metric `log.appended_bytes` in the DBMS to return a negative value, which became apparent because with this version we released a new datastore version. The process caused the metrics HTTP endpoint of the Neo4j DBMS to fail. This issue was undetected because it affected only a subset of instances and it got corrected for the instances that had rolled subsequently to another component change we delivered as part of the release. The issue only occurred on instances that were not part of that restart due to the roll. This issue also prevented us from fully collecting metrics from those DBMS instances, which impacted monitoring of the instances, troubleshooting by engineers. We filtered out the metric causing the issue and rolled the affected instances to overcome the issue. ### **How the service was affected** Affected customers \(a random subset of instances across tiers\) could not retrieve any instance metrics via the endpoint [customer-metrics-api.neo4j.io](http://customer-metrics-api.neo4j.io) and this also affected the built-in metrics included in the monitoring section of the Aura console \([console.neo4j.io](http://console.neo4j.io) \) ### **What we are doing now** Following a review of the sequence of events and their impact, we have identified a number of actions to implement so that we improve the Neo4j Aura and prevent, detect, mitigate as well as better handle any similar issue. * Prevention * Improve metrics endpoint robustness: requests should not fail if 1 metric is invalid * Fix and prevent negative counters for metrics and log occurrences. * Detection * Implement an alert on metrics not being collected successfully * Mitigation, handling and troubleshooting * Improve access to raw metrics from any instances * Provide a configuration option to exclude a metric * Communication * Represent the metric endpoint on the status page and work towards automating the report of its status

Read the full incident report →

Started Apr 03, 2025, 06:54 PM UTC· resolved Apr 04, 2025, 08:27 AM UTC

1. investigating Apr 03, 2025, 06:54 PM UTC

   We're experiencing an issue with the Support Portal and some users may face difficulties accessing the portal.

2. resolved Apr 04, 2025, 08:27 AM UTC

   We have now addressed the issue and ensured the certificate chain is complete.

Read the full incident report →

Started Mar 31, 2025, 01:15 PM UTC· resolved Mar 31, 2025, 03:36 PM UTC

1. investigating Mar 31, 2025, 01:15 PM UTC

   We are currently investigating an issue which impact the CMI feature, where requests return HTTP 401.

2. monitoring Mar 31, 2025, 02:08 PM UTC

   A fix has been implemented and the CMI feature is responding normally. We will monitor for a short time before considering this resolved.

3. resolved Mar 31, 2025, 03:36 PM UTC

   CMI functionality has been fully restored and this issue is considered resolved.

4. postmortem Apr 10, 2025, 09:38 PM UTC

   ### What happened On March 31, 2025, we experienced an incident that temporarily affected the availability of the Customer Metrics Integration service for Aura instances. This issue arose due to a change in our internal systems that inadvertently caused errors in processing customer requests. As a result, customers received error messages instead of the expected metrics. Our team quickly identified and resolved the problem. By rolling back the recent changes, we restored service functionality within 2.5 hours. We are committed to improving our processes to prevent similar issues in the future and ensure a seamless experience for our customers. ### Root Cause We encountered an issue due to the way our services interact with each other. One of the Aura Console API response fields structure changed and resulted in some unexpected dependency problems. We're actively working to improve the reliability of our services to ensure a smoother experience for you in the future. ### Customer Impact * Customers experienced service disruptions as they were unable to access metrics for their Aura instances. * All customer requests returned an HTTP 401 error, preventing users from retrieving necessary information. ### Resolution We rolled back the recent changes to restore service functionality quickly. Following that, implemented some changes in the Console API to reduce dependencies, and redeployed changes successfully. ‌ **What we are doing now** Neo4j remains committed to providing reliable service and is implementing additional safeguards to prevent similar incidents in the future. New mitigations being deployed: * Enhancing our incident response playbook to improve handling of similar situations. * Establishing a standardized internal Console API with clear versioning and documentation to prevent compatibility issues. * Implementing comprehensive testing measures to identify potential issues before they affect our production environment.

Read the full incident report →

Started Mar 28, 2025, 04:39 PM UTC· resolved Mar 28, 2025, 06:20 PM UTC

1. investigating Mar 28, 2025, 04:39 PM UTC

   We are currently investigating unavailability of the Aura Console. If you are logged into the Classic view, you are likely unaffected.

2. investigating Mar 28, 2025, 04:57 PM UTC

   As we are continuing to investigate and remediate this issue with Aura Console, please use https://console.neo4j.io, which is still working normally.

3. identified Mar 28, 2025, 05:42 PM UTC

   We have identified the issue being related to web browser cached files. Clearing all files linked to the domain neo4j.io in your browser should allow you to login. Alternatively please use https://console.neo4j.io We continue to investigate a way to fix and will update shortly.

4. monitoring Mar 28, 2025, 05:58 PM UTC

   We reverted a change that was rolled out earlier today and we can confirm this has addressed the issue but will keep on monitoring for some time.

5. resolved Mar 28, 2025, 06:20 PM UTC

   We can confirm that the change has resolved the issue and that Aura Console is fully operational at this time.

6. postmortem Apr 16, 2025, 04:20 PM UTC

   ### What happened On March 28, 2025, some users experienced difficulties accessing the Aura Console due to a temporary issue with our system. The problem was identified quickly, and a solution was implemented to restore full functionality. We apologize for any inconvenience this may have caused and appreciate your patience as we worked to resolve the issue and improve the service. ### Root Cause We encountered an issue where under certain circumstances an older version of the @neo4j/graphql library was loaded which was not compatible with the browser environment. As a result, you might have experienced some disruptions or errors while using our service. ### Customer Impact * Some users were unable to access the Aura console for a short period. * Users needed to refresh their browser’s cache or wait for a fix to regain access to the Aura Console. ### Resolution We have updated our service, removing the feature from the console which loaded the library in the browser, which avoids this situation altogether. ### What we are doing now Neo4j remains committed to providing reliable service and is implementing additional safeguards to prevent similar incidents in the future. New mitigations being deployed: * Enhancing our testing environments to better reflect real-world conditions, ensuring more reliable service updates. * Implementing stricter release protocols to ensure only fully verified updates are deployed. * Improving our monitoring systems to detect potential issues before they impact our customers. * Increasing the reliability of our automated testing to ensure consistent service quality. * Developing a more robust notification system to keep our teams informed of any service issues promptly. * Enhancing our documentation and training to ensure all team members are well-prepared to handle service updates and incidents.

Read the full incident report →

Started Mar 18, 2025, 06:24 AM UTC· resolved Mar 18, 2025, 01:16 PM UTC

1. investigating Mar 18, 2025, 06:24 AM UTC

   Neo4j Support Portal (https://support.neo4j.com/) is experiencing a service disruption - We are assessing the impact at this stage - Users may encounter difficulty in signing-in and accessing the portal - Users may have difficulty in opening New support cases, managing existing cases and in accessing the knowledgebase articles Note: Users can update existing cases via email without any challenges In the meantime, please report any issues via email at [email protected].

2. investigating Mar 18, 2025, 07:30 AM UTC

   We are still actively investigating the ongoing service disruption affecting the Neo4j Customer Support Portal (https://support.neo4j.com/). Users may continue to experience difficulties signing in, opening new support cases, managing existing cases, and accessing knowledge base articles. In the meantime, please continue reporting any issues via email at [email protected]. We appreciate your patience and will provide further updates as we learn more.

3. investigating Mar 18, 2025, 08:44 AM UTC

   We are still actively working to resolve the ongoing disruption impacting the Neo4j Support Portal (https://support.neo4j.com/). Users may experience challenges when signing in, creating new support cases, managing existing cases, or accessing knowledge base articles. For now, please continue submitting any issues via email at [email protected], as mentioned in our previous update. We appreciate your patience and will provide further updates as soon as possible.

4. investigating Mar 18, 2025, 10:35 AM UTC

   We are still investigating the issue affecting the Neo4j Support Portal (https://support.neo4j.com/). Users may continue to experience access difficulties, but updating existing cases via email remains unaffected. Please report any issues via [email protected] appreciate your patience and will share updates as soon as we have more information

5. investigating Mar 18, 2025, 01:08 PM UTC

   We are still investigating the issue affecting the Neo4j Support Portal (https://support.neo4j.com/). Users may continue to experience access difficulties, but updating existing cases via email remains unaffected. Please report any issues via [email protected] appreciate your patience and will share updates as soon as we have more information

6. resolved Mar 18, 2025, 01:16 PM UTC

   We have fixed the issue and the support portal is now fully functional.

7. postmortem Apr 15, 2025, 01:12 AM UTC

   ### What happened On Monday March 18th 2025 at 05:35 UTC, a Neo4j Customer Portal outage was reported which made it impossible for customers to create or manage their support tickets through the portal \([https://support.neo4j.com](https://support.neo4j.com)\). This outage was caused by a CA-signed certificate expiring which customers used to securely gain access to Neo4j’s support portal via Salesforce. Access was restored for customers around 13:00 UTC on March 18th as a result of creating a new CA-signed certificate and applying it to production. While the portal was inaccessible to customers, Neo4j Support Engineers could still work cases internally and accept email submitted cases. We apologize for the inconvenience this caused and appreciate your understanding and patience. ### Root Cause The Customer Support certificate for [support.neo4j.com](http://support.neo4j.com) was created as part of Neo4j’s Service Cloud migration in March 2024. As a best practice, Salesforce limits CA-signed certificates to a one-year validity period primarily for security and compliance reasons. The teams that manage this system at Neo4j misunderstood how this certificate worked and did not renew it on time as a result. ### Customer Impact * Neo4j Customer Portal was unavailable for users for 7.5 hours. * Customers were able to receive case updates via email, and submit new cases via email. ### Resolution To prevent future missed certificate renewals, our team has dedicated resources who will be responsible for proactively monitoring, renewing, and managing all expiring certificates. We now have alerts configured that message the whole team 60 days in advance to make sure we don’t miss deadlines in the future. The certificate process has also been documented thoroughly in our internal systems so that there is visibility on renewal steps across the teams responsible for managing the system.

Read the full incident report →

Started Mar 13, 2025, 09:27 PM UTC· resolved Mar 14, 2025, 10:41 AM UTC

1. identified Mar 13, 2025, 09:27 PM UTC

   We have identified an issue in GCP causing some new AuraDB Pro, AuraDS Pro and Aura BC instances to get stuck in creation for an extended period of time. Updating instances in this cohort may also experience delay. We are already preparing a fix.

2. identified Mar 14, 2025, 12:59 AM UTC

   we are continuing work on a fix for this issue.

3. identified Mar 14, 2025, 12:59 AM UTC

   We are in the process of releasing a fix for this issue and would update with a status when it has been rolled out across all impacted tiers

4. monitoring Mar 14, 2025, 06:48 AM UTC

   The fix for this issue has been successfully rolled out across all impacted tiers. We are currently monitoring the situation to ensure stability.

5. resolved Mar 14, 2025, 10:41 AM UTC

   We have confirmed that the fix has addressed issues and all operations fully recovered.

6. postmortem Apr 14, 2025, 03:48 PM UTC

   ### What happened On March 13, 2025, some of our customers experienced delays in the creation of AuraDB Pro, AuraDS Pro, and Aura BC databases on our platform. This was due to an unexpected issue with our database management system, which caused some databases to become stuck during the creation process. Our team quickly identified the problem and worked diligently to resolve it. By March 14, all affected databases were successfully created and operational. We are committed to ensuring the reliability of our services and have taken steps to prevent similar issues in the future. We apologize for any inconvenience this may have caused and appreciate your understanding and patience. ### Root Cause Recently, we implemented a new security feature to enhance the way we manage access to our database systems. This involved setting up additional security permissions in the IAM Policy for each of our paid database services. While we were careful to stay within known limits for these permissions, we encountered an unexpected issue due to an undocumented restriction on the size of these permissions. This caused a problem where our system couldn't create new permissions for databases in certain areas, leading to delays in processing and updates. ### Customer Impact * AuraDB Pro, AuraDS Pro, and Aura BC Databases experienced issues where databases would get stuck during creation. * A total of 8 AuraDB Pro databases were directly affected by this issue. ### Resolution We promptly addressed the issue by reversing recent IAM changes to ensure the stability of our services. ### What we are doing now Neo4j remains committed to providing reliable service and is implementing additional safeguards to prevent similar incidents in the future. New mitigations being deployed: * Modifying the access management method to eliminate the possibility of hitting IAM limits * Implementing additional automated checks to prevent service disruptions * Improving our service resilience to handle similar scenarios

Read the full incident report →

Started Jan 17, 2025, 11:45 AM UTC· resolved Jan 20, 2025, 08:25 AM UTC

1. identified Jan 20, 2025, 09:23 AM UTC

   New users registering for Aura faced an error: "Error project not found" or "An error occurred" with some Event ID This prevented access for new users to progress further. All existing users were unaffected and no production workload was affected.

2. resolved Jan 20, 2025, 09:25 AM UTC

   A fix was released and rolled-out. All new users registering to Aura can access the Neo4j Aura Console

3. postmortem Jan 31, 2025, 01:39 PM UTC

   ### **What happened** We deployed a fix to a bug to address a race condition in registering new users on Aura. The fix aimed to address specifically a situation whereby multiple records could be created accidentally because they were bundled in the same transaction for the backend database and we therefore decided to split these operations. Unfortunately, in fixing it this way we blocked new user registration as the sequence of operations in registering users would be incorrect. ### **How the service was affected** New users registering for Aura faced an error: "Error project not found" or "An error occurred" with some Event ID. This prevented access for new users to progress further. However, all existing users were unaffected and no production workload was affected. We were notified by the customers of issues and needed to deploy the fix as no workaround was identified at the time. As registering to the support portal is bundled with registering to Aura this error hindered new customers from notifying us of the problem. As a fallback channel, emails sent to [[email protected]](mailto:[email protected]) from un-registered Support Portal users create a ticket but these tickets do not carry severity or SLA for response. ### **What we are doing now** We have identified several actions that can prevent similar problems from occurring in the future, including but not limited to those concerning new user registrations in Aura. * Add some automated tests for operations on our backend database managing the registration details so that we validate against the good usage with regards to the specific needs of transactions. Tests would verify that READ and WRITES aren’t bundled in a conflicting way for each of our API operations. * Implement a User Registration release check: to avoid issues creeping into production * To improve our troubleshooting capabilities we will be adding further information into the logs to establish a connection between Event Error ID and its corresponding log in the backend database.

Read the full incident report →

Started Jan 16, 2025, 03:56 PM UTC· resolved Jan 16, 2025, 08:16 PM UTC

1. identified Jan 16, 2025, 03:56 PM UTC

   Endpoint ingress degradation in AWS us-east-1, us-west-2 and ap-northeast-2 was introduced earlier on January 16, 2025. We have identified a fix and are currently deploying it to all impacted regions.

2. monitoring Jan 16, 2025, 06:48 PM UTC

   A fix has been deployed across all impacted regions. This issue has been resolved and we will monitor to ensure the service remains healthy before considering this Resolved.

3. resolved Jan 16, 2025, 08:16 PM UTC

   The incident is resolved. A postmortem will be available once it is complete.

4. postmortem Feb 21, 2025, 01:40 PM UTC

   ## **What happened** A configuration change in Aura’s DB Ingress service caused intermittent connectivity issues for customer databases across multiple AWS regions. The issue was due to a misconfiguration of AWS Network Load Balancers \(NLBs\), which resulted in dropped inbound traffic. A change focused on making the service more efficient rolled on 2025-01-15 and reduced db-ingress replicas to three per region, affecting AWS regions with more than three availability zones \(AZs\). The AWS Network Load Balancer \(NLB\) couldn't route traffic correctly when requests landed in AZs without a db-ingress pod. Cross-zone load balancing was not enabled, preventing the NLB from distributing traffic across zones. This led to intermittent connection failures in us-east-1, us-west-2, and ap-northeast-2. On 2025-01-16 the fix was deployed. ## **How the service was affected** Intermittent connectivity failures impacted Aura Professional, Business Critical, and DS Enterprise orchestras. The issue was caused by traffic being dropped by the AWS Network Load Balancer \(NLB\) due to improper routing. As a result, failure rates \(for queries using the Bolt protocol\) reached approximately 40% in us-east-1, 25% in us-west-2, and 25% in ap-northeast-2. We reverted a change that reduced the number of db-ingress replicas, ensuring that instances were running in all availability zones and this restored normal database operations. ## **What we are doing now** Neo4j remains committed to providing reliable service and is implementing additional safeguards to prevent similar incidents in the future. To prevent similar incidents in the future, we are implementing the following improvements: * **Enable cross-zone Load Balancing**: Ensuring traffic is correctly distributed across all availability zones. * **Adding automated monitoring** to detect ingress failures before deployment. * **Adding improved Alarms** to detect and respond to connectivity issues quickly.

Read the full incident report →

Started Jan 15, 2025, 06:20 PM UTC· resolved Jan 15, 2025, 10:17 PM UTC

1. identified Jan 15, 2025, 06:20 PM UTC

   The Neo4j Aura Customer Metrics Integration (CMI) is currently unavailable. We have identified a fix and are preparing to roll it out.

2. monitoring Jan 15, 2025, 07:39 PM UTC

   A fix has been rolled out and we are monitoring to ensure CMI is fully operational for all instances.

3. resolved Jan 15, 2025, 10:17 PM UTC

   A fix has been in place for some time and this incident is considered resolved. A portmortem will be forthcoming.

4. postmortem Feb 07, 2025, 11:45 AM UTC

   ### **What happened** On 2025-01-15 at 15:42:02 UTC our secure endpoint to provide Customer Metrics Integration became unavailable and returned the error \{"message":"Failed to validate JWT."\} due to the expiry of the SSL certificate. Whilst we had renewed the certificate our framework for deploying and rolling out components had updated service accounts but not updated the associated service key secret in the correct sequence. ### **How the service was affected** Customers collecting and ingesting metrics from their Neo4j Aura instances, were no longer able to do so as a result of connectivity issues with the provided endpoints. The issue was due to the requirement for valid encryption \(and an up to date certificate\) to connect to [customer-metrics-api.neo4j.io](http://customer-metrics-api.neo4j.io/). We detected it internally when rolling out an update and soon after received reports of issues from customers. We worked to create a new service account and associated secret key to be rolled out immediately. ### **What we are doing now** We recognise that this issue caused serious issues in monitoring and operating Neo4j Aura instances and we have committed to the following actions: * Monitoring: We built additional monitoring metrics and dashboards and derived alarms to our cloud operations team to detect issues with failed connection * Mitigation: We are improving how we roll these changes related to service accounts secrets key. * Prevention: For any service account secret key deletion and updating with a new one we will not bundle this work anymore but split the tasks associated with the changes.

Read the full incident report →

Started Jan 10, 2025, 07:49 PM UTC· resolved Jan 13, 2025, 01:41 PM UTC

1. investigating Jan 10, 2025, 07:49 PM UTC

   We are investigating an issue where GDS functions are missing from AuraDS Enterprise and AuraDS instances.

2. identified Jan 10, 2025, 10:02 PM UTC

   We have identified the issue causing the GDS plugin to be missing from the impacted instances and are working towards restoring the plugin to all missing instances.

3. identified Jan 11, 2025, 12:45 AM UTC

   We have identified the issue causing the GDS plugin to be missing from the impacted instances and we are working on releasing the fix.

4. monitoring Jan 11, 2025, 03:54 AM UTC

   We have identified the issue causing the GDS plugin to be missing from the impacted instances and have rolled out a fix. If your GDS plugins are missing from your instance, please pause and resume your instance as a workaround until all remaining impacted instances are remediated.

5. resolved Jan 13, 2025, 01:41 PM UTC

   We have monitored the service and this incident is now resolved.

6. postmortem Jan 31, 2025, 02:37 PM UTC

   ## **What happened** On January 10, 2025, following a system update, some AuraDS instances experienced an issue where Graph Data Science \(GDS\) functionality became temporarily unavailable resulting in users receiving “unknown function” error messages while attempting to use GDS functions. Users with AuraDS Self-serve and AuraDS Enterprise instances were affected. ### **Timeline:** 2025-01-10 15:39:00 - Code change merged and rolled out to production 2025-01-10 17:26:59 - Incident started - Customers begin reporting GDS function unavailability 2025-01-10 19:49:00 - Status Page updated with incident once initial scope determined 2025-01-11 00:09:00 - Fix merged and deployed to production. Begin process of restoring plugins to missing instances 2025-01-14 10:45:15 - Incident resolved - Fix fully deployed and verified ## **Root Cause** Neo4j rolled out an update which affects how GDS plugin assignments are managed for AuraDS instances. Incident-1966 unexpectedly resulted in the removal of the GDS library from AuraDS Self-Serve and AuraDS Enterprise instances. ## **Customer** **Impact** * AuraDS instances lost access to GDS functionalities * A number of projections were lost due to the remediation activities to restore GDS functionality, affecting a number of user instances * A number of users needed to restart their GDS workflows, once a fix was put in place ## **Resolution** Neo4j worked to identify the root cause, affected instances, and deploy a fix to restore the GDS functionality as the first step. After step one, Neo4j updated affected instances to load the GDS plugins and verify the fix across all impacted customers’ instances. ## **What we are doing now** Neo4j remains committed to providing reliable service and is implementing additional safeguards to prevent similar incidents in the future. New mitigations being deployed: * Improve on our Console reconciler testing methodologies * Increased observability metrics intended to detect and fix plugin-related issues more quickly * Improve deployment procedures by adding more effective guardrails against unintended plugin changes

Read the full incident report →

Started Nov 18, 2024, 03:15 PM UTC· resolved Nov 18, 2024, 08:05 PM UTC

1. identified Nov 18, 2024, 03:15 PM UTC

   Our cloud provider has confirmed some capacity issues in the northeurope region. As a result, some operations that require compute are delayed or not completing. We continue to assess the full impact and we are working with our cloud provider.

2. resolved Nov 18, 2024, 08:05 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 28, 2024, 06:31 PM UTC· resolved Oct 29, 2024, 04:06 PM UTC

1. investigating Oct 28, 2024, 06:31 PM UTC

   We have received some reports of queries failing after the roll-out of version 5.25. The affected queries have a CASE within a WITH clause. We are investigating. Customers who had marked their instances as production are not affected, as the roll-out is delayed for those instances.

2. investigating Oct 28, 2024, 07:36 PM UTC

   We are continuing to investigate this issue.

3. identified Oct 28, 2024, 07:39 PM UTC

   We have identified the issue and are working on a fix. In the meantime, work around this issue by replacing Simple Case statements (https://neo4j.com/docs/cypher-manual/current/queries/case/#case-simple) with Generic Case (https://neo4j.com/docs/cypher-manual/current/queries/case/#case-generic) statements. We will continue to update this incident as we progress with the fix.

4. identified Oct 28, 2024, 10:42 PM UTC

   We have identified a fix and are in the process of building and releasing it. Please utilize the workaround if you are impacted until the fix is released.

5. identified Oct 29, 2024, 01:47 AM UTC

   We have identified a fix and are in the process of building and releasing it. In the meantime, work around this issue by replacing Simple Case statements (https://neo4j.com/docs/cypher-manual/current/queries/case/#case-simple) with Generic Case (https://neo4j.com/docs/cypher-manual/current/queries/case/#case-generic) statements.

6. identified Oct 29, 2024, 05:35 AM UTC

   We have identified a fix and are in the process of building and releasing it. In the meantime, work around this issue by replacing Simple Case statements (https://neo4j.com/docs/cypher-manual/current/queries/case/#case-simple) with Generic Case (https://neo4j.com/docs/cypher-manual/current/queries/case/#case-generic) statements.

7. identified Oct 29, 2024, 08:34 AM UTC

   We have identified a fix and are in the process of building and releasing it. Please utilize the workaround if you are impacted until the fix is released.

8. identified Oct 29, 2024, 10:01 AM UTC

   We will be proceeding to the roll-out of the fix as soon as checks are completed. ETA 3 hours for this phase to be initiated

9. identified Oct 29, 2024, 01:35 PM UTC

   We have started the roll-out of the fix to the affected customers.

10. resolved Oct 29, 2024, 04:06 PM UTC

    We have verified the fix and confirmed it has resolved this issue as reported by customers.

11. postmortem Nov 08, 2024, 12:00 PM UTC

    ### **What happened** We rolled out a new release \(version 5.25\) of the Neo4j database on 2024-10-25 starting with AuraDB Free tier. On 2024-10-28 at 10:30 UTC we progressed the roll-out to AuraDB Professional, and then next to AuraDB Business Critical and AuraDB Virtual Dedicated Cloud tiers. As we maintain a 48 hours delay before we roll-out to all instances marked as production, those instances marked as production were not affected and were still running the previous version \(5.24.2\) of the database. The new release version 5.25 contained a change aimed at handling the newer GQL syntax for the CASE expression to support the “[simple CASE](https://neo4j.com/docs/cypher-manual/current/queries/case/#case-simple)” mode. The implementation affected the optimization phase of such queries. The recently modified handling of the WITH clause also leveraged an optimization, and the two clashed when both involved in the same query, causing the Cypher statements to fail. The existing test suite for Cypher covered the individual keywords but not the combination of these two exact cypher keywords in this particular sequence that would trigger both optimization phases. ### **How the service was affected** Customers running Cypher statements that comprise a CASE \(simple\) inside a WITH clause saw their queries fail and produce the error “`Neo.ClientError.Statement.SyntaxError”.` Customers reported issues on some specific queries without any changes on their side. We took the decision to halt further progress of the roll-out of the release until we had a confirmed fix for this issue. As we identified both the pattern triggering the error and its root cause, we communicated a workaround of adjusting the queries that use a “[simple CASE](https://neo4j.com/docs/cypher-manual/current/queries/case/#case-simple)” inside a WITH clause and modify them to use a “[generic CASE](https://neo4j.com/docs/cypher-manual/current/queries/case/#case-generic)”. ‌ A fix was prepared, but to prevent other issues, we ensured it completed the full Cypher test suite before being rolled out \(2024-10-29 at 13:35 UTC\) to all our customers. ### **What we are doing now** We don’t believe that the issues our customers experienced are acceptable for a DBaaS. We have carried out a thorough analysis of what went wrong in this situation. The actions that we are carrying out to ensure that nothing like this can happen again fall into multiple areas: * Test: We are enhancing the capability and coverage of our Cypher test suite. The syntax from any Cypher queries identified as bugs will automatically be included in the test suite. * Detection: We are building a set of metrics and alarms to detect sudden spikes of errors on Cypher queries run in Aura whilst we roll any new release and get alerted of possible issues linked to a new release. * Release roll-out: We are improving the agility and speed of rolling-out the releases to better handle problematic releases. * Bug fixing: We are building a fast-track release pipeline to enable swifter delivery of hotfixes in safe conditions. Communication: We will be reminding customers to have development instances and [designate their production instance](https://neo4j.com/docs/aura/preview/managing-instances/instance-actions/#_mark_an_instance_as_production) to better isolate them from possible issues during roll-out.

Read the full incident report →