Alert ingestion via assemblers delayed

Update timeline

1. investigating Jun 23, 2025, 09:51 AM UTC

   At approximately 4am EDT, we began experiencing a problem with our assembler VPN system that is preventing alerts from being ingested through assemblers. Alert ingestion is delayed for some devices behind an assembler. We will provide an update by 6:30am EDT.

2. identified Jun 23, 2025, 10:30 AM UTC

   We have identified the root cause and are working on a remediation. Alert ingestion via some assemblers continues to be delayed. SOC operations are not affected. We will provide another update by 7:00 EDT.

3. identified Jun 23, 2025, 11:00 AM UTC

   We have identified the root cause and are working on a remediation. Alert ingestion via some assemblers continues to be delayed. SOC operations are not affected. We will provide another update by 7:30 EDT.

4. identified Jun 23, 2025, 11:47 AM UTC

   We have identified the root cause and are working on a remediation. Alert ingestion via some assemblers continues to be delayed. SOC operations are not affected. We will provide another update by 8:00 EDT.

5. identified Jun 23, 2025, 12:07 PM UTC

   We are continuing to work on a fix for this issue.

6. identified Jun 23, 2025, 12:08 PM UTC

   We have identified a further cause and are working on a remediation. Alert ingestion via some assemblers continues to be delayed. SOC operations are not affected. We will provide another update by 8:30 EDT.

7. identified Jun 23, 2025, 12:42 PM UTC

   We have identified a further cause and are working on a remediation. Alert ingestion via some assemblers continues to be delayed. SOC operations are not affected. We will provide another update by 9:30 EDT.

8. identified Jun 23, 2025, 01:30 PM UTC

   We have identified a further cause and are working on a remediation. Alert ingestion via some assemblers continues to be delayed. SOC operations are not affected. We will provide another update by 10:30 EDT.

9. identified Jun 23, 2025, 02:31 PM UTC

   We have identified a further cause and are working on a remediation. Individual assemblers are coming back online. Alert ingestion via some assemblers continues to be delayed. SOC operations are not affected. We will provide another update by 11:30 EDT.

10. identified Jun 23, 2025, 03:32 PM UTC

    We have identified a further cause and are working on a remediation. Individual assemblers are coming back online. Alert ingestion via some assemblers continues to be delayed. SOC operations are not affected. We will provide another update by 12:30 EDT.

11. monitoring Jun 23, 2025, 07:50 PM UTC

    We have implemented a fix and believe the incident to be resolved. We are monitoring the health of the infrastructure as individual assemblers come back online, and will resolve this if there are no outstanding issues. Please let us know if you are experiencing any problems. We will provide another update by 4:30pm EDT.

12. resolved Jun 23, 2025, 08:04 PM UTC

    The incident has been resolved, and all previously-available assemblers are once again fully available.