Is SignalFx US1 down?

We are investigating delays in processing Splunk Observability Synthetic Monitoring test results. Synthetic Monitoring tests are operational but metrics may be missing until the issue is resolved.

We are continuing to investigate this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

Some customers may be experiencing delays in some charts and detectors. Datapoint ingest is not affected. We are investigating and will provide an update shortly.

We continue investigating the delays experienced in some charts and detectors. Our engineering teams continue troubleshooting and will share an update shortly.

A fix has been implemented and we are monitoring the results.

Service recovery has been confirmed. We will continue to monitor system performance closely and share updates accordingly.

Our engineering team has confirmed the fix deployed has stabilized impacted services across US1 region. This incident is now considered resolved.

From 5:45pm to 6:20pm PT, degradation in the performance of the Splunk APM trace processing pipeline was causing Troubleshooting MetricSets to be delayed by more than five minutes. As a result, the APM Troubleshooting experience, service maps and Tag Spotlight do not have access to the most recent data. The processing of metrics for Business Workflows, which also depends on this pipeline, are equally delayed. Trace data ingest is not impacted at this time; service-level and endpoint-level Monitoring MetricSets and the detectors built from them are also not impacted.

Some of the customer are seeing login failures due to certificate errors. Customers that are already logged are not impacted. The issue has been resolved.

Beginning at 5:50 AM PDT, datapoint ingestion for the archival tier was impacted, resulting in a small number of datapoints not being accepted. The impact was limited to the archival tier. Detectors and charts were not affected. We have implemented remediation and are actively monitoring the system to ensure continued stability. We will provide updates every 15 minutes.

As of 7:48 AM PDT, we are no longer observing any data being dropped. The system is continuing to recover, and we are actively monitoring it to ensure continued stability.

The system is continuing to recover, and we are actively monitoring it to ensure continued stability.

The system is showing steady progress in its recovery. Our teams remain actively engaged, closely observing system metrics to ensure continued stability

We have successfully restored datapoint ingestion and are seeing steady progress as the data backlog is processed. We are actively monitoring this process to ensure full service functionality returns as quickly as possible.

We continue monitoring to ensure full service functionality.

We are continuing to monitor the system to ensure ongoing stability. Our teams remain actively engaged in observing performance metrics to confirm that all operations are functioning as expected.

To accelerate the recovery process, we are currently adding additional capacity to our clusters. Our teams remain actively engaged and are continuing to monitor system performance as it stabilizes.

The system has recovered and we ingesting archived data with no delay.