Is Hosted Mender down?

We noticed an issue with the email sending system from both hosted Mender US and hosted Mender EU, we found the root cause and fixed it. No email has been sent starting from 12:30AM UTC on Tuesday 27th, to about 10AM UTC on Wednesday 28th. We are really sorry for this incident, we're gathering all the data and we'll publish a post mortem soon.

We are investigating an issue affecting outgoing emails from Hosted Mender. Emails such as password resets, user invitations and notifications may be delayed or not delivered. Device connectivity, deployments and all other platform functionality are not affected.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved

The PHP project recently announced two critical security vulnerabilities. You can read more about these on their websites: https://www.php.net/archive/2026.php#2026-10-21-5 As IT teams are handling these vulnerabilities, figuring out where they are affected and what systems need to be upgraded, we've gotten requests to clarify if Mender is affected or not from customers. Thus, to make this information easily available to everyone we've decided to put this clarification here; Mender Client, Mender Server / Backend, and hosted Mender do not use PHP, and are not affected by these vulnerabilities. The mender.io website is hosted by Hubspot, and does not use PHP, so is also not affected. This is from HubSpot's website / incident status page: HubSpot is aware of the recently disclosed PHP security vulnerability. PHP isn't part of HubSpot's infrastructure, and we are not aware of any impact to our platform or customers. We will continue to watch the situation and provide an update if our assessment changes. The Mender documentation, at docs.mender.io, does use PHP, and was assumed to be affected, so it was upgraded from PHP 8.3.35 to 8.3.36 this morning (Oct 22nd), at around 6AM UTC+2. The documentation site is hosted on infrastructure entirely separate from the hosted Mender production systems, and we have not seen any indication that the documentation site was being attacked or exploited prior to the upgrade. So, after upgrading the Mender documentation site, no part of the Mender product or its websites should be affected by these PHP vulnerabilities. We wish IT and security teams the best of luck in handling these vulnerabilities. This announcement has been published to our Blog Post as well: https://mender.io/blog/clarifications-regarding-recent-php-vulnerabilities-cve-2026-109618-cve-2026-109619

Some users get an "An error occured." message when they open Deployments > Finished. It happens on smaller screens and in browser windows that aren't maximized. We have found the cause, a display issue in the web interface, and are working on a fix. Devices, deployments and the API are not affected: deployments keep running and finishing as normal. Workaround: Make the browser window wider, or zoom out (Ctrl/Cmd and -), until the page loads.

We rolled back the release to quickly solve the issue. We're monitoring the results.

This incident has been resolved.

We are currently investigating this issue which has been reported by customers.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we're monitoring the results.

This incident has been resolved.

## Executive Summary We renew the signing key for our apt repositories every two years. The renewal on 2026-09-15 produced a key that Debian 13 \(trixie\) rejects. For about half a day, anyone installing Mender packages on Debian 13 got "`the repository is not signed`" and could not install anything. Existing devices were unaffected. Debian 12 and older were unaffected. No package was altered and nothing was exposed; this was a verification failure, not a security breach. ## What happened A signing key carries a signature saying "`this key is valid`". Renewing the key is supposed to extend that statement. The tool we use replaces it instead, so the renewed key effectively said "`valid as of today`" with no record of having been valid before. Debian 13 shipped a stricter signature checker. It asks whether the key was valid _at the moment the repository was signed_. Our repositories were last signed before the renewal, so the answer was no and it refused them. Debian 12 and older do not ask that question, which is why everything looked fine to us. ## Impact * Debian 13 users could not install or update Mender packages. ## Resolution We rebuilt the key so it carries both its old and its new validity statements. It is the same key with the same fingerprint, so nothing changed for existing devices, and it now satisfies both the old and the new checkers. Verified on Debian 12 and Debian 13, against every repository. **If you applied a workaround, re-fetch the key.** Some users unblocked themselves by fetching the key from `https://downloads.mender.io/repos/device-components/gpg`. During the incident that URL served the old key, which expires on **2026-10-01**, so those installs will start failing again in October. That URL now serves the corrected key, so fetch it once more: ``` curl -fsSL https://downloads.mender.io/repos/device-components/gpg \ | sudo gpg --dearmor -o /usr/share/keyrings/mender.gpg` ``` The expiry should now read 2028-09-14: ``` gpg --show-keys /usr/share/keyrings/mender.gpg ``` Adjust the output path if you installed the key somewhere else; the official docs place it at `/etc/apt/trusted.gpg.d/mender.asc`. No change to your `sources.list` is needed, and devices already running Mender need no action. ## Why we did not catch it 1. We have an internal renewal and verification runbook which doesn't include Debian 13, so we didn't catch it. 2. The repositories kept serving. Only clients on a newer distribution saw a problem, and we had no monitoring for that. ## Actions to avoid this issue the next time * We updated the internal runbook to also include Debian 13 and a note to check for future distributions. * We added a verification pipeline to the renewal process. ## Apology We are sorry. This was our mistake, and it blocked people from installing our software on a current, supported version of Debian. We have updated our internal renewal process to test against current Debian releases, and added a verification pipeline so this is checked automatically the next time. We did not find this ourselves. A customer did, and reported it. We are grateful for that report, which was precise enough that we could reproduce the problem within minutes, but catching this should not have been their job.