DFDS IT incident

metrics: kafka-exporter missing data frequently

Resolved after 1 day 23 hours, affecting Kafka metrics exporter, on April 8, 2025.

resolved Apr 10, 2025, 07:06 AM UTC

Since fix was implemented not a single scrape has been missed. Will continue to monitor for the next few days If it becomes problematic again, a different exporter is currently being evaluated to replace the current one.

Started
Apr 08, 2025, 07:22 AM UTC
Resolved
Apr 10, 2025, 07:06 AM UTC
Duration
1d 23h
Detected by Pingoru
Apr 08, 2025, 07:22 AM UTC

Affected components

Kafka metrics exporter

Update timeline

  1. investigating Apr 08, 2025, 07:22 AM UTC

    Same as the incident a few weeks ago: https://dfdsit.statuspage.io/incidents/vllndmlym8xd Looking into it. Hopefully a more permanent solution this time. If you have alerts that are currently triggering because of the lack of data from this exporter, a temporary measure could be to set "Alert state if no data or all values are null" to "Keep Last State" for those alerts. It's not ideal, but it's much better than muting/silencing the alert itself. See screenshot: https://blob.juna.seq.tf/pub/vrsWnTTVXk37Srs.jpg

  2. monitoring Apr 09, 2025, 04:04 PM UTC

    A fix has been implemented and we are monitoring the results.

  3. resolved Apr 10, 2025, 07:06 AM UTC

    Since fix was implemented not a single scrape has been missed. Will continue to monitor for the next few days If it becomes problematic again, a different exporter is currently being evaluated to replace the current one.

Your product runs on more than DFDS IT

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →