DFDS IT Outage History

DFDS IT is up right now

Pingoru has recorded 50 incidents for DFDS IT: 8 in the last 12 months and 3 in the last 90 days.

Source: status.dfds.cloud

50incidents recorded
8last 12 months
3last 90 days
5 daystypical duration

Incidents, newest first

Major September 15, 2026 Log ingestion issue, some logs are dropped 1d 10h Grafana Cloud

Started Sep 15, 2026, 10:07 AM UTC· resolved Sep 16, 2026, 09:07 PM UTC

  1. investigating Sep 15, 2026, 10:07 AM UTC

    We are currently seeing log ingestion issues, where some logs are dropped. We are investigating.

  2. identified Sep 15, 2026, 11:57 AM UTC

    We have identified the issue. We haven't fixed the cause yet, but things are certainly better now.

  3. monitoring Sep 15, 2026, 07:32 PM UTC

    Backfilling of the logs that could be rescued has been completed. Looking into the possibility of discovering what pods have missing logs from the time period 2 PM CEST till 15:00 PM CEST ish. Capabilities that hasn't had all of their logs backfilled will be notified if possible.

  4. monitoring Sep 16, 2026, 12:43 PM UTC

    Affected Capabilities have been notified via email. There's still 3 nodes that are occasionally struggling, a fix for those are being rolled out.

  5. resolved Sep 16, 2026, 09:07 PM UTC

    This incident has been resolved.

Read the full incident report →

Notice August 20, 2026 Kubernetes node NotReady. 51 pods affected 2h 51m

Started Aug 20, 2026, 10:42 AM UTC· resolved Aug 20, 2026, 01:33 PM UTC

  1. investigating Aug 20, 2026, 10:42 AM UTC

    We are currently investigating this issue.

  2. monitoring Aug 20, 2026, 11:11 AM UTC

    The node has been marked unhealthy. A new node was spun up and took over the workloads.

  3. resolved Aug 20, 2026, 01:33 PM UTC

    This incident has been resolved.

Read the full incident report →

Minor June 12, 2026 Capabilities created since the 19th of May not receiving emails from third-party vendors 32d 23h Capability provisioning, AAD-AWS-Sync

Started Jun 12, 2026, 12:21 PM UTC· resolved Jul 15, 2026, 11:28 AM UTC

  1. identified Jun 12, 2026, 12:21 PM UTC

    Due to removal of permissions for Exchange Online, we are currently not able to create distribution groups and email aliases. The effects of this is that Capabilities created after that point are unable to receive emails from AWS and other third-party providers. We're awaiting details from the team managing Exchange Online as to why this is now broken..

  2. resolved Jul 15, 2026, 11:28 AM UTC

    Resolved for now.

Read the full incident report →

Critical May 18, 2026 Selfservice portal down (build.dfds.cloud) 6m Capability self-service portal (Blaster)

Started May 18, 2026, 09:32 AM UTC· resolved May 18, 2026, 09:38 AM UTC

  1. investigating May 18, 2026, 09:32 AM UTC

    Selfservice portal is fully down. We are investigating.

  2. investigating May 18, 2026, 09:32 AM UTC

    We are continuing to investigate this issue.

  3. resolved May 18, 2026, 09:38 AM UTC

    Problem resolved

Read the full incident report →

Minor February 13, 2026 Problematic node on hellman Kubernetes cluster must be killed 28m Kubernetes [Hellman] - Capacity/Scheduling

Started Feb 13, 2026, 12:35 PM UTC· resolved Feb 13, 2026, 01:04 PM UTC

  1. investigating Feb 13, 2026, 12:35 PM UTC

    We are currently investigating this issue.

  2. identified Feb 13, 2026, 12:37 PM UTC

    The node is cordoned off and now marked as unhealthy. Once a new node has been started up, we will start draining all the workloads on this node. The root cause is the kubelet having problems with managing Pod lifecycle, in particular termination, so if draining adds to the problem, all workloads on that node will be forcefully terminated. Downtime can be expected.

  3. monitoring Feb 13, 2026, 12:46 PM UTC

    All workloads on the faulty node have now started up on a new node.

  4. resolved Feb 13, 2026, 01:04 PM UTC

    This incident has been resolved.

Read the full incident report →

Major December 1, 2025 Grafana Cloud - Missing metrics 3h 25m

Started Dec 01, 2025, 10:15 AM UTC· resolved Dec 01, 2025, 01:40 PM UTC

  1. resolved Dec 01, 2025, 01:50 PM UTC

    In an attempt to optimise costs for metrics we accidentally ended up not gathering metrics for quite a significant amount of services for a few hours (11:15 CET till 14:40 CET) due to a bug that was triggered by our change. Apologies.

Read the full incident report →

Minor November 28, 2025 Selfservice platform permissions issues 3h 35m Capability self-service portal (Blaster), Capability provisioning, Topic management

Started Nov 28, 2025, 09:52 AM UTC· resolved Nov 28, 2025, 01:27 PM UTC

  1. investigating Nov 28, 2025, 09:52 AM UTC

    Several offerings on the Selfservice Platform are currently unavailable. This includes but is not necessarily limited to - approving memberships - creating new capabilities successfully - requesting Azure Resource groups We are working to resolve this as quickly as possible.

  2. investigating Nov 28, 2025, 01:24 PM UTC

    We are continuing to investigate this issue.

  3. resolved Nov 28, 2025, 01:27 PM UTC

    A problem with our Kafka setup prevented the consumption of events leading to the deteriorated state. This has now been addressed

Read the full incident report →

Notice September 30, 2025 Issues with the Kubernetes nodes after node rollover 2h 39m Kubernetes [Hellman] - Capacity/Scheduling, Kubernetes [Hellman] - Ingress/Networking

Started Sep 30, 2025, 09:40 AM UTC· resolved Sep 30, 2025, 12:20 PM UTC

  1. investigating Sep 30, 2025, 09:40 AM UTC

    We are troubleshooting the issues ASAP.

  2. identified Sep 30, 2025, 10:01 AM UTC

    We're in the process of restoring the nodes that were faulty.

  3. monitoring Sep 30, 2025, 10:14 AM UTC

    New nodes are using correct launch template now, monitoring the situation.

  4. resolved Sep 30, 2025, 12:20 PM UTC

    The issues should be now resolved.

Read the full incident report →

Major September 8, 2025 AWS eu-central-1 inaccessible 1h 40m AWS ec2-eu-central-1, AWS elb-eu-central-1, AWS s3-eu-central-1

Started Sep 08, 2025, 07:32 AM UTC· resolved Sep 08, 2025, 09:13 AM UTC

  1. investigating Sep 08, 2025, 07:32 AM UTC

    We are currently investigating this issue.

  2. monitoring Sep 08, 2025, 07:41 AM UTC

    A fix has been implemented and we are monitoring the results.

  3. resolved Sep 08, 2025, 09:13 AM UTC

    Fixed by AWS - https://health.console.aws.amazon.com/health/home#/account/event-log?End%20time=2025-09-07T11%3A12%3A52%20to%202025-09-08T11%3A12%3A52&Event%20category=issue&Status=closed&eventID=arn:aws:health:global::event/MANAGEMENTCONSOLE/AWS_MANAGEMENTCONSOLE_OPERATIONAL_ISSUE/AWS_MANAGEMENTCONSOLE_OPERATIONAL_ISSUE_4EEBD_2A76BAF307B&eventTab=details

Read the full incident report →

Critical August 27, 2025 selfservice platform unreachable 58m Capability self-service portal (Blaster), Capability provisioning, Topic management, AAD-AWS-Sync

Started Aug 27, 2025, 06:03 AM UTC· resolved Aug 27, 2025, 07:01 AM UTC

  1. identified Aug 27, 2025, 06:03 AM UTC

    The issue has been identified and a fix is being implemented.

  2. resolved Aug 27, 2025, 07:01 AM UTC

    This incident has been resolved.

Read the full incident report →

Major May 29, 2025 Prometheus data source missing gaps of data 1h 56m

Started May 29, 2025, 10:38 AM UTC· resolved May 29, 2025, 12:35 PM UTC

  1. identified May 29, 2025, 10:38 AM UTC

    Due to metrics collector going OOM sporadically since yesterday, metrics shipped from Hellman has been partly missing in Grafana Cloud. Working on implementing a fix now.

  2. monitoring May 29, 2025, 11:32 AM UTC

    Fix implemented. Seems stable for now, but without having resolved the root cause I suspect it'll probably happen again. Will keep monitoring it.

  3. resolved May 29, 2025, 12:35 PM UTC

    Incident itself is resolved, but the actual origin of the issue not quite yet. If you experience further issues with no data don’t hesitate to reach out directly via Slack to emcla.

Read the full incident report →

Minor May 21, 2025 Metrics inaccessible in Grafana tribe stacks 3h 8m Grafana Cloud

Started May 21, 2025, 12:02 PM UTC· resolved May 21, 2025, 03:11 PM UTC

  1. investigating May 21, 2025, 12:02 PM UTC

    Currently look into an issue where queries to the data source "grafanacloud-all-metrics" gives the error "authentication error: invalid authentication credentials"

  2. monitoring May 21, 2025, 12:23 PM UTC

    A fix has been implemented and we are monitoring the results.

  3. resolved May 21, 2025, 03:11 PM UTC

    This incident has been resolved.

Read the full incident report →

Minor May 19, 2025 build.dfds.cloud was partially unavailable 1m

Started May 19, 2025, 12:28 PM UTC· resolved May 19, 2025, 12:30 PM UTC

  1. resolved May 19, 2025, 12:28 PM UTC

    A new feature was enabled for build.dfds.cloud mid Friday. Unfortunately something was missing in the production deployment and it caused certain events to not be triggered. In total 6 events has been pending from that time until today 2:15 PM CEST. They are no longer pending, so previously created resources like Kafka topics or Capability membership changes should now have been applied. Apologies for the interruption.

Read the full incident report →

Critical April 14, 2025 DFDS Developer Portal shows blank pages and in some cases gives 404 error message 1d Capability self-service portal (Blaster), Capability provisioning

Started Apr 14, 2025, 01:12 PM UTC· resolved Apr 15, 2025, 01:17 PM UTC

  1. investigating Apr 14, 2025, 01:12 PM UTC

    We are currently investigating this issue.

  2. identified Apr 14, 2025, 01:14 PM UTC

    The issue has been identified and a fix is being implemented.

  3. monitoring Apr 14, 2025, 02:49 PM UTC

    Hotfixed applied, monitoring.

  4. resolved Apr 15, 2025, 01:17 PM UTC

    This incident has been resolved.

Read the full incident report →

Minor April 8, 2025 metrics: kafka-exporter missing data frequently 1d 23h Kafka metrics exporter

Started Apr 08, 2025, 07:22 AM UTC· resolved Apr 10, 2025, 07:06 AM UTC

  1. investigating Apr 08, 2025, 07:22 AM UTC

    Same as the incident a few weeks ago: https://dfdsit.statuspage.io/incidents/vllndmlym8xd Looking into it. Hopefully a more permanent solution this time. If you have alerts that are currently triggering because of the lack of data from this exporter, a temporary measure could be to set "Alert state if no data or all values are null" to "Keep Last State" for those alerts. It's not ideal, but it's much better than muting/silencing the alert itself. See screenshot: https://blob.juna.seq.tf/pub/vrsWnTTVXk37Srs.jpg

  2. monitoring Apr 09, 2025, 04:04 PM UTC

    A fix has been implemented and we are monitoring the results.

  3. resolved Apr 10, 2025, 07:06 AM UTC

    Since fix was implemented not a single scrape has been missed. Will continue to monitor for the next few days If it becomes problematic again, a different exporter is currently being evaluated to replace the current one.

Read the full incident report →

Major March 17, 2025 Some public API's returning 404 –

Started Mar 17, 2025, 06:30 PM UTC· resolved Mar 17, 2025, 06:30 PM UTC

  1. resolved Mar 18, 2025, 08:44 AM UTC

    A configuration change to Traefik which routes http traffic into the Kubernetes cluster caused endpoints that have /api in the path to return 404 not found to requests. The incident was resolved at ~00:30

Read the full incident report →

Minor March 14, 2025 metrics: kafka-exporter missing data often 2h 47m Kafka metrics exporter

Started Mar 14, 2025, 07:53 AM UTC· resolved Mar 14, 2025, 10:40 AM UTC

  1. investigating Mar 14, 2025, 07:53 AM UTC

    Currently looking into metrics from kafka-metrics having no data frequently. Currently the most probable cause is a big increase in consumer groups per topic for a few topics leading to much longer scrape times, resulting in a timeout.

  2. monitoring Mar 14, 2025, 09:49 AM UTC

    Issue seem potentially related to slower response times from Confluent Cloud. This seems to be improving now and the exporter can seemingly reliably export metrics within the set scrape threshold. Will continue to monitor the situation.

  3. resolved Mar 14, 2025, 10:40 AM UTC

    Seems stable... for now.

Read the full incident report →

Minor March 3, 2025 Some Kubernetes Pods failed scheduling –

Started Mar 03, 2025, 04:30 PM UTC· resolved Mar 03, 2025, 04:30 PM UTC

  1. resolved Mar 03, 2025, 08:53 PM UTC

    Some Kubernetes pods failed to schedule due to available Node CPU and Memory capacity

  2. postmortem Mar 18, 2025, 08:55 AM UTC

    [https://raw.githubusercontent.com/dfds/postmortems/refs/heads/master/PM2025-001 - Node system resource starvation and scheduling capacity](https://raw.githubusercontent.com/dfds/postmortems/refs/heads/master/PM2025-001%20-%20Node%20system%20resource%20starvation%20and%20scheduling%20capacity)

Read the full incident report →

Minor March 3, 2025 Unstable Kubernetes Worker Nodes 2d 19h Kubernetes [Hellman] - Capacity/Scheduling

Started Mar 03, 2025, 01:32 PM UTC· resolved Mar 06, 2025, 08:37 AM UTC

  1. investigating Mar 03, 2025, 01:32 PM UTC

    We are currently investigating an issue that is causing the odd Kubernetes worker node to become unresponsive and evict its workloads to other nodes

  2. monitoring Mar 03, 2025, 06:49 PM UTC

    We have implemented a potential fix and are monitirong the results

  3. resolved Mar 06, 2025, 08:37 AM UTC

    Nodes have been stable since the fix was applied so we consider the issue resolved

  4. postmortem Mar 18, 2025, 08:54 AM UTC

    [https://raw.githubusercontent.com/dfds/postmortems/refs/heads/master/PM2025-001 - Node system resource starvation and scheduling capacity](https://raw.githubusercontent.com/dfds/postmortems/refs/heads/master/PM2025-001%20-%20Node%20system%20resource%20starvation%20and%20scheduling%20capacity)

Read the full incident report →

Critical January 13, 2025 Capability self-service portal is not loading 15m Capability self-service portal (Blaster), Capability provisioning, Topic management

Started Jan 13, 2025, 02:47 PM UTC· resolved Jan 13, 2025, 03:03 PM UTC

  1. investigating Jan 13, 2025, 02:47 PM UTC

    We are investigating why the selfservice API is crash looping and causing the selfservice portal to not load properly.

  2. monitoring Jan 13, 2025, 02:51 PM UTC

    An invalid kafka message has been removed.

  3. resolved Jan 13, 2025, 03:03 PM UTC

    This incident has been resolved.

Read the full incident report →

Major December 4, 2024 Some Grafana stacks are experiening outage 3h 1m Grafana Cloud

Started Dec 04, 2024, 10:09 AM UTC· resolved Dec 04, 2024, 01:10 PM UTC

  1. investigating Dec 04, 2024, 10:09 AM UTC

    Data and CET stacks are confirmed down. We will contact Grafana Support.

  2. identified Dec 04, 2024, 10:14 AM UTC

    The issue is confirmed by Grafana Support. Follow https://status.grafana.com/incidents/dr2xbss3cxns for updates.

  3. monitoring Dec 04, 2024, 11:49 AM UTC

    A fix has been implemented and we are monitoring the results.

  4. resolved Dec 04, 2024, 01:10 PM UTC

    This incident has been resolved.

Read the full incident report →

Minor November 14, 2024 K8s access via SSO issues 18m Kubernetes [Hellman] - Developer access, AAD-AWS-Sync

Started Nov 14, 2024, 02:19 PM UTC· resolved Nov 14, 2024, 02:37 PM UTC

  1. investigating Nov 14, 2024, 02:19 PM UTC

    Currently looking into an issue where some Capabilities can't be accessed in Kubernetes via SSO. Pipeline access should still be operational.

  2. resolved Nov 14, 2024, 02:37 PM UTC

    False positive.

Read the full incident report →