Formant Outage History
Formant is up right nowPingoru has recorded 48 incidents for Formant: 7 in the last 12 months and 0 in the last 90 days.
Incidents, newest first
Major May 11, 2026 Some queries against the Analytics SQL API return incomplete results 19h 4m Formant Web Application, Admin API, Query API
Started May 11, 2026, 09:32 PM UTC· resolved May 12, 2026, 04:37 PM UTC
- investigating May 11, 2026, 09:32 PM UTC
We are currently investigating this issue.
- identified May 12, 2026, 01:27 AM UTC
The issue has been identified. Working to identify the full fix.
- monitoring May 12, 2026, 02:05 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved May 12, 2026, 04:37 PM UTC
This incident has been resolved.
Minor April 21, 2026 Degraded performance in V1 view modules like bitset 3h 13m Formant Web Application
Started Apr 21, 2026, 03:12 PM UTC· resolved Apr 21, 2026, 06:26 PM UTC
- investigating Apr 21, 2026, 03:12 PM UTC
We are currently seeing an issue where data is not loading in some module types
- resolved Apr 21, 2026, 06:26 PM UTC
This incident has been resolved.
Minor March 26, 2026 We're currently experiencing some slowness in the platform 3h 29m Formant Web Application, Admin API
Started Mar 26, 2026, 02:24 PM UTC· resolved Mar 26, 2026, 05:54 PM UTC
- investigating Mar 26, 2026, 02:24 PM UTC
We are currently investigating this issue.
- monitoring Mar 26, 2026, 03:05 PM UTC
Performance is returning to normal levels
- resolved Mar 26, 2026, 05:54 PM UTC
Incident has been resolved, but we'll continue to monitor the situation.
Major March 25, 2026 Widespread issues reported by various customers 1h 22m Formant Web Application, Ingestion Pipeline, Admin API, Query API and 1 more
Started Mar 25, 2026, 01:17 PM UTC· resolved Mar 25, 2026, 02:39 PM UTC
- investigating Mar 25, 2026, 01:17 PM UTC
Various customers are reporting issues with the platform.
- investigating Mar 25, 2026, 01:21 PM UTC
We are continuing to investigate this issue.
- investigating Mar 25, 2026, 01:33 PM UTC
We are continuing to investigate this issue.
- resolved Mar 25, 2026, 02:39 PM UTC
This incident has been resolved. Thank you for your patience.
Minor March 17, 2026 Data Processing Delays - Alerting Worker falling behind - Reporting Tools Affected 1h 40m Formant Web Application, Ingestion Pipeline
Started Mar 17, 2026, 04:33 PM UTC· resolved Mar 17, 2026, 06:14 PM UTC
- investigating Mar 17, 2026, 04:33 PM UTC
Our data processing infrastructure is running behind which is causing inaccuracies in the reporting tools. No data has been lost. We are currently investigating the issue.
- monitoring Mar 17, 2026, 05:54 PM UTC
We've reverted to the previous version, upgraded the number of kubernetes pods, and are monitoring as the system is recovering
- resolved Mar 17, 2026, 06:14 PM UTC
Everything is back to normal
Minor February 5, 2026 Some users are getting logged out after 5 minutes of inactivity 2h 35m Formant Web Application
Started Feb 05, 2026, 03:14 PM UTC· resolved Feb 05, 2026, 05:50 PM UTC
- investigating Feb 05, 2026, 03:14 PM UTC
We are currently investigating an issue where some users are getting logged out after 5 minutes of inactivity.
- identified Feb 05, 2026, 03:33 PM UTC
The issue has been identified and a fix is incoming.
- monitoring Feb 05, 2026, 05:25 PM UTC
A fix has been deployed. Testing and monitoring the fix.
- resolved Feb 05, 2026, 05:50 PM UTC
The issue has been fixed and tested.
Notice December 3, 2025 Service degradation – Backend API errors 31m
Started Dec 03, 2025, 10:45 PM UTC· resolved Dec 03, 2025, 11:16 PM UTC
- resolved Dec 03, 2025, 09:00 PM UTC
We experienced a period of severely degraded backend performance between approximately 2:45 pm and 3:16 pm US Central Time on December 3, 2025 due to a database query change in a new release. We promptly reverted the change and confirmed that services returned to normal. No data loss occurred.
Critical October 20, 2025 AWS outage caused disruptions to Coherence, Teleop, and other Formant.io services –
Started Oct 20, 2025, 07:00 AM UTC· resolved Oct 20, 2025, 07:00 AM UTC
- resolved Oct 20, 2025, 07:00 AM UTC
We experienced disruptions to our Coherence, Telelop and other Formant.io services due to the widespread Amazon Web Services (AWS) outage that began around 2:11 AM Central Time on October 20th, 2025. Although most services had returned to normal by 1 PM Central Time, AWS did not provide the all clear until 5 PM Central Time. All Formant services have returned to normal operation.
Minor May 8, 2025 Ingestion lag 1h Ingestion Pipeline
Started May 08, 2025, 08:04 PM UTC· resolved May 08, 2025, 09:05 PM UTC
- investigating May 08, 2025, 08:04 PM UTC
We are currently investigating delayed data ingestion for some customers. Our team is actively working to identify the root cause and restore normal performance. We will provide updates as more information becomes available. Thank you for your patience.
- resolved May 08, 2025, 09:05 PM UTC
This incident has been resolved.
Minor March 26, 2025 Partial Outage Causing Ingestion Lag 1h 14m Ingestion Pipeline
Started Mar 26, 2025, 10:29 PM UTC· resolved Mar 26, 2025, 11:43 PM UTC
- investigating Mar 26, 2025, 10:29 PM UTC
We are currently investigating a partial outage that is resulting in delayed data ingestion for some customers. Our team is actively working to identify the root cause and restore normal performance. We will provide updates as more information becomes available. Thank you for your patience.
- identified Mar 26, 2025, 11:23 PM UTC
The issue has been identified.
- monitoring Mar 26, 2025, 11:42 PM UTC
A fix has been implemented and we're monitoring the performance of ingestion.
- resolved Mar 26, 2025, 11:43 PM UTC
The ingestion lag issue has been resolved.
Major February 27, 2025 Asset ingestion disruption –
Started Feb 27, 2025, 10:30 PM UTC· resolved Feb 27, 2025, 10:30 PM UTC
- resolved Feb 27, 2025, 10:30 PM UTC
Formant agent version 1.232.0 contained an issue which caused disruption to ingestion of asset data. This was fixed in version 1.232.6. To resolve this issue, update your Formant agent. For more information on asset data, see How telemetry streams work in our documentation center.
Major February 27, 2025 'Failed to import rosgraph' when installing Formant agent 13h 38m Formant Agent
Started Feb 27, 2025, 08:52 PM UTC· resolved Feb 28, 2025, 10:30 AM UTC
- identified Feb 27, 2025, 08:52 PM UTC
We've discovered an issue causing an import failure when installing the Formant agent on a ROS 1 machine in agent version 1.232. To resolve this issue, add `export FORMANT_ROS_VERSION=1` to `/var/lib/formant/.bashrc`.
- resolved Feb 28, 2025, 10:30 AM UTC
This incident has been resolved. To fix this issue, update to the latest version of the agent.
Minor January 16, 2025 Formant agent intermittently fails to connect to ROS 19d 20h Formant Agent
Started Jan 16, 2025, 10:39 PM UTC· resolved Feb 05, 2025, 07:38 PM UTC
- monitoring Jan 16, 2025, 10:39 PM UTC
We've identified an issue that sometimes causes the Formant agent to fail to reconnect to the ROS Master when the ROS Master is restarted. We have identified a fix and are testing it. In the meantime, if you are affected by this issue, you can roll back to agent version 1.212.0 See Install the Formant agent for instructions: https://docs.formant.io/docs/getting-started-install-the-formant-agent#install-specific-agent-version
- resolved Feb 05, 2025, 07:38 PM UTC
This incident has been resolved in agent version 1.226.
Critical January 16, 2025 Formant disruption 12m
Started Jan 16, 2025, 12:00 AM UTC· resolved Jan 16, 2025, 12:12 AM UTC
- resolved Jan 16, 2025, 12:00 AM UTC
The Formant application was unavailable for 12 minutes due to a backend deployment issue. There was no data loss during this time.
Minor January 3, 2025 Data ingestion lag 23m
Started Jan 03, 2025, 10:15 PM UTC· resolved Jan 03, 2025, 10:38 PM UTC
- resolved Jan 03, 2025, 10:00 PM UTC
On January 3, 2025, from 5:15 pm to 5:38pm EST, data ingestion to Formant experienced degraded service with high lag times. Data ingested during that period did not appear until after 5:38 pm EST. No data was lost during this incident.
Minor December 3, 2024 Intermittent scheduled command issue –
Started Dec 03, 2024, 11:00 AM UTC· resolved Dec 03, 2024, 11:00 AM UTC
- resolved Dec 03, 2024, 11:00 AM UTC
From December 3, 2024 to December 9, 2024, we saw intermittent issues with scheduled commands. Occasionally, scheduled commands failed to fire. This was fixed on December 9, 2024.
Minor November 13, 2024 Some users experiencing degraded performance in Formant web app and Admin API 1h 7m Formant Web Application, Admin API
Started Nov 13, 2024, 06:55 PM UTC· resolved Nov 13, 2024, 08:02 PM UTC
- investigating Nov 13, 2024, 06:55 PM UTC
We are currently investigating this issue.
- monitoring Nov 13, 2024, 07:10 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Nov 13, 2024, 08:02 PM UTC
This incident has been resolved.
Major November 5, 2024 Some users cannot log in 10h 12m Formant Web Application
Started Nov 05, 2024, 06:00 PM UTC· resolved Nov 06, 2024, 04:12 AM UTC
- investigating Nov 05, 2024, 06:00 PM UTC
We’re currently experiencing intermittent login issues. Our team is actively working to restore access as soon as possible. Thank you for your patience and understanding.
- investigating Nov 05, 2024, 08:22 PM UTC
We are continuing to investigate this issue.
- investigating Nov 05, 2024, 08:28 PM UTC
We are continuing to investigate this issue.
- identified Nov 05, 2024, 11:01 PM UTC
The issue has been identified and a fix is being implemented. Most users are unaffected.
- identified Nov 05, 2024, 11:04 PM UTC
We are continuing to work on a fix for this issue.
- resolved Nov 06, 2024, 04:12 AM UTC
All affected users have been reinstated and can now reset their password to log in.
- postmortem Nov 07, 2024, 11:33 PM UTC
We found an issue that prevented a limited number of users from logging in. Access for affected users has been restored. Affected users will have to reset their passwords. No user data was ever exposed as a result of this issue.