FullStory Outage History

Started Jul 17, 2025, 09:08 AM UTC· resolved Jul 17, 2025, 09:58 AM UTC

1. investigating Jul 17, 2025, 09:08 AM UTC

   We are currently investigating an issue with mobile capture for EU orgs.

2. identified Jul 17, 2025, 09:11 AM UTC

   An issue was identified with mobile capture for EU orgs. We have determined the cause and rolled back the change that resulted in this issue.

3. monitoring Jul 17, 2025, 09:52 AM UTC

   An issue was identified with mobile capture for EU orgs. A fix has been implemented and we are now monitoring.

4. resolved Jul 17, 2025, 09:58 AM UTC

   An issue was identified with mobile capture for EU orgs. This incident has now been resolved.

5. postmortem Jul 18, 2025, 05:14 PM UTC

   # Postmortem From Wednesday July 16, 8:05 PM UTC through Thursday July 17, 09:42 AM UTC an update to the Native Mobile asset upload service caused 50-70% of all asset management requests to fail due to an issue in routing requests for `rs.eu1.fullstory.com` and `mr.eu1.fullstory.com`. * All native mobile builds targeting the EU environment would likely have failed during the incident * iOS Native Mobile session capture in the EU was impacted at the point that an asset upload failed * Android Native Mobile session capture in the EU environment were not impacted * Native Mobile builds targeted at the NA environment were not impacted * Native Mobile session capture the NA environment were not impacted * Web session capture in NA or EU was not impacted # Customer Impact During the incident window any new Native Mobile build that performed asset uploading or checking would have failed due to 404 errors when hitting the necessary endpoint. As multiple calls are made during the build process almost all builds were likely impacted during the partial outage. Retries of failed builds would have resulted in the same failures until the underlying issue was resolved. Additionally, session capture was disrupted for applications using the iOS SDK that perform asset uploads at runtime, as 404 errors during capture result in capture being stopped immediately. Sessions in progress were interrupted and new sessions could not be created, and this capture data is not available. Android applications were not interrupted, however assets uploaded during these sessions are likely not available during playback. # Root Cause An update to our asset management service was deployed on July 16th to improve the reliability of asset uploads for Native Mobile builds. To mitigate a source of failures during asset uploads, the service was divided into two portions to handle asset uploads in a robust manner to separate these endpoints from the resource crawling and fetching functionality. To support this change, a routing update for our public endpoints was necessary to direct asset upload requests to the appropriate service. While the necessary routing changes were programmatically applied in our NA production and staging environments, the update was not successful in our EU production environments. As a result, subsequent asset upload requests were failing 50-70% of the time because they were not delegated to the appropriate service. # Resolution The issue was addressed on July 17 at 9:20 AM UTC by manually applying the necessary routing changes. As soon as these changes were completed, all requests were properly routed to the correct asset upload service, and normal functionality was restored. # Process Changes and Prevention **We are committed to preventing this incident in the future. We’ve completed the following action items:** * We have increased the frequency of checks that detect differences in configuration between our environments and the expected configuration and between our NA and EU production environments directly. * We have added additional public monitoring checks for all public-facing asset management endpoints used by the Native Mobile SDK, expanding upon our existing coverage of capture and API endpoints. ‌ **In addition, we taking the following steps:** * We will update the iOS SDK so that asset upload failures will not disrupt session capture, aligning with the current behavior of the Android SDK. * We will add additional prompts and alerts when production infrastructure changes are applied to ensure that all environments are thoroughly validated. * Add additional automated integration testing that covers all the expected uses of our exposed asset management endpoints to ensure proper functionality. ‌ We deeply regret this incident and invite any Fullstory customer who was materially affected to contact [[email protected]](mailto:[email protected]). We stand by ready to fully address all of your concerns.

Read the full incident report →

Started Jul 14, 2025, 12:51 PM UTC· resolved Jul 14, 2025, 02:37 PM UTC

1. investigating Jul 14, 2025, 12:51 PM UTC

   Yesterday at 10:00pm EDT, we began experiencing an elevated error rate displaying data in app for a subset of orgs (NA1 Data Center). Metrics, Segments, Funnels, and Dashboards may be slow to load. They may timeout with an error. Configured alerts may be delayed. Data capture is not affected. We are actively investigating the issue and will update this status as soon as we know more.

2. resolved Jul 14, 2025, 02:37 PM UTC

   We've seen a decrease in errors and Fullstory objects should be returning to normal behavior. We do not expect any further impacts in the coming hours or days and will make changes to prevent these errors from causing such performance impacts in the future.

Read the full incident report →

Started Jun 12, 2025, 06:21 PM UTC· resolved Jun 13, 2025, 01:36 AM UTC

1. investigating Jun 12, 2025, 06:21 PM UTC

   We are currently investigating a site outage which is impacting logging in and loading the Fullstory UI. We are actively working to identify the root cause and will provide regular updates as they are available.

2. identified Jun 12, 2025, 06:57 PM UTC

   This outage is due to a current GCP incident that we are closely monitoring here: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW#RP1d9aZLNFZEJmTBk8e1 We'll continue to make updates as we evaluate the full impact that this outage has on Fullstory customers.

3. monitoring Jun 12, 2025, 09:02 PM UTC

   The GCP incident is resolved and we're actively monitoring the results. Sessions captured during the outage are being indexed, but may not yet be available. We'll provide a final update on impact as soon as this is available.

4. resolved Jun 13, 2025, 01:36 AM UTC

   All backlogged sessions are now indexed. If you notice any persisting issues please reach out to [email protected] for assistance.

Read the full incident report →

Started Jun 05, 2025, 03:47 PM UTC· resolved Jun 05, 2025, 04:11 PM UTC

1. investigating Jun 05, 2025, 03:47 PM UTC

   We are currently experiencing delays in loading in the Fullstory Web Application. The issue started around 10:25am ET. We are actively investigating the issue and will provide an update as soon as possible.

2. resolved Jun 05, 2025, 04:11 PM UTC

   We've identified and resolved the issue causing a performance impact to the Fullstory UI. The Fullstory Web Application is now fully operational.

Read the full incident report →

Started Mar 18, 2025, 08:29 PM UTC· resolved Mar 19, 2025, 12:09 AM UTC

1. investigating Mar 18, 2025, 08:29 PM UTC

   At 4:19pm ET, we noticed users experiencing slow loading or failure to load Fullstory. We are rolling back recent changes and continuing to investigate and fully resolve this issue.

2. investigating Mar 18, 2025, 09:13 PM UTC

   We are continuing to investigate this issue.

3. monitoring Mar 18, 2025, 10:09 PM UTC

   At 4:11 PM EDT, we detected an issue that resulted in degraded performance on Fullstory. The service has been restored, and we are actively investigating the root cause. We will provide further updates as our investigation progresses. If you encounter any additional issues or have questions, please contact [email protected].

4. resolved Mar 19, 2025, 12:09 AM UTC

   We have implemented a fix that has resolved the issue and our team is still actively investigating the underlying cause to prevent future occurrences. Please reach out to [email protected] if you experience any further issues.

Read the full incident report →

Started Mar 02, 2025, 11:30 AM UTC· resolved Mar 02, 2025, 11:30 AM UTC

1. resolved Mar 02, 2025, 04:13 PM UTC

   From 6:30am ET until 9:00am ET we were experiencing delays in session processing for accounts in our EU data center. New sessions may not have appeared until 9:30am ET but no data was lost. The issue is now fully resolved and all sessions captured during this time have been processed.

Read the full incident report →

Started Feb 20, 2025, 09:05 PM UTC· resolved Feb 20, 2025, 09:22 PM UTC

1. investigating Feb 20, 2025, 09:05 PM UTC

   We are currently investigating a Fullstory web application outage for some users. Note: This outage does not affect session capture.

2. monitoring Feb 20, 2025, 09:13 PM UTC

   We've rolled back some changes that caused this outage and Fullstory should be back up and running again. Please reach out to [email protected] with any further issues/questions.

3. resolved Feb 20, 2025, 09:22 PM UTC

   All systems are back to normal operations and are working as expected.

Read the full incident report →

Started Jan 08, 2025, 03:15 PM UTC· resolved Jan 08, 2025, 03:35 PM UTC

1. resolved Jan 08, 2025, 03:41 PM UTC

   From 10:15 - 10:35am EST today, we identified an issue with logging in to the Fullstory application. This issue has since been resolved and access to Fullstory has been restored. As a result of this issue, there is a slight delay in session processing that will be caught up within 30 - 60 minutes following the resolution.

Read the full incident report →

Started Dec 11, 2024, 04:42 PM UTC· resolved Dec 11, 2024, 11:10 PM UTC

1. identified Dec 11, 2024, 04:42 PM UTC

   We are experiencing an elevated latency for data loading for a subset of orgs. Users may receive errors while attempting to load Dashboards, Metrics, and Segments. Some pages may take longer than normal to appear.

2. identified Dec 11, 2024, 09:03 PM UTC

   We are continuing to work on a fix for the elevated data display latency. Thanks for your patience.

3. resolved Dec 11, 2024, 11:10 PM UTC

   Things are back to normal now and users should no longer be seeing latency in data loading. If you continue to have issues, please reach out [email protected].

Read the full incident report →

Started Dec 10, 2024, 09:30 PM UTC· resolved Dec 10, 2024, 11:34 PM UTC

1. identified Dec 10, 2024, 09:30 PM UTC

   We are experiencing an elevated error rate displaying data in app for a subset of orgs. Charts may be slow to load. They may timeout with an error. Data capture is not affected. Segment alerts are also delayed for all orgs. We are actively investigating the issue.

2. resolved Dec 10, 2024, 11:34 PM UTC

   The delays have subsided and this incident is now resolved. Please write into [email protected] if you experience further trouble.

Read the full incident report →

Started Nov 14, 2024, 12:11 AM UTC· resolved Nov 14, 2024, 10:12 PM UTC

1. monitoring Nov 14, 2024, 12:11 AM UTC

   Our team has identified an issue impacting data capture for web and mobile sessions that started October 31st, 2024 at 4:03pm ET. This issue impacted 4,770 Fullstory accounts, but only impacts about .5% of total captured sessions per account. Due to our internal retention timeframe, sessions impacted by this issue prior to November 6, 2024 at 5:28pm ET are nonrecoverable. We increased our retention to ensure no further impacted session data would be lost. A fix for the issue has been deployed and we're monitoring this further.

2. resolved Nov 14, 2024, 10:12 PM UTC

   This issue is now resolved. Most accounts will notice a small percentage of sessions affected by this issue. This would result in entirely missing sessions, or pages in sessions, which would also impact Search, Conversions, Metrics, Funnels, and Dashboards. We’re actively remediating the sessions from November 6th - 13th. After remediation, sessions and analytics will be fully recovered. Data sent to a data warehouse or via a webhook for this time period will be delayed. If you watch a session from October 31 - November 5th and experience skipping of playback, it’s likely caused by this issue. If you would like to confirm that your account was impacted, please reach out to [email protected].

3. postmortem Dec 10, 2024, 09:16 PM UTC

   # Postmortem 2024/11/13 Failed Tasks Not Retried / Pages Not Initialized On October 31st, 2024, an issue introduced in our service caused a small percentage \(< 0.05%\) of sessions to not process a portion of data within them. This data was effectively “stuck” in our system. The issue lasted until November 13th, when it was fixed for all future sessions. # Customer Impact The issue caused the affected sessions to be missing data, causing gaps/disruptions when loading playback, webhooks not firing, and events not being exported via the Data Direct feature. Affected sessions captured after Nov 6th were identified and frozen so that we could re-process the data. Starting the week of Dec 9th, we ran all the frozen data back through our pipeline and the majority of it was recovered successfully for sessions with enough data to process. Affected sessions before Nov 6th will remain in this incomplete state, as the data that had been unprocessed got deleted as it was misidentified as something that our system needed to clean up and remove, due to being stuck for some time. We apologize for any impact this may have had on your business, and are working to better understand how to identify issues of this nature more quickly, and resolve them with even less disruption. # Root Cause Due to a code change that altered the behavior of an uncommon failure condition, part of our system marked the data in question to be “completed” when it had actually failed. Normally in this case, it automatically retries this operation and the data gets processed, but did not do so during the incident period. Due to this being a rare circumstance, the issue was not immediately noticed by our typical monitoring systems. # Resolution After noticing some odd system behavior and finding that we had unprocessed data that should have been successful, we quickly traced it back to the code change, and deployed a fix immediately that would ensure no future data gets into this state. # Process Changes and Prevention **Actions Taken:** * Fix the issue for future sessions. * Freeze and reprocess all affected sessions still in our system. * Added more automated testing around this failure case. * Adjusted our monitoring and alerting to better identify “stuck” data that’s legitimate, even when it’s a very small percentage of overall processing. ‌ **Ongoing Improvements:** * Applying lessons learned to other parts of the system, including better detection systems and testing for any other components that process data in a similar way. * Better automated observability triaging so that rare-circumstance issues don’t blend in with general service metric variability. ‌ We deeply regret this incident and invite any FullStory customer who was materially affected to contact [[email protected]](mailto:[email protected]). We stand by ready to fully address all of your concerns.

Read the full incident report →

Started Oct 31, 2024, 06:30 PM UTC· resolved Oct 31, 2024, 06:30 PM UTC

1. resolved Oct 31, 2024, 09:33 PM UTC

   From October 31, 2024 at 2:38PM ET through October 31, 2024 at 4:07PM ET, the Fullstory APIs were unintentionally unavailable. The majority of requests to Fullstory APIs would have returned an error. Fullstory customers that attempted to send an API request during the outage window will need to process these requests again. If you believe your account was impacted or have any questions, please reach out to [email protected].

2. postmortem Nov 07, 2024, 09:07 PM UTC

   On October 31, 2024, our API service experienced a disruption that resulted in a significant number of dropped requests and errors. The incident lasted for approximately 1.5 hours and impacted all users of our public server API. # Customer Impact Most public server API requests sent during the incident time frame \(18:37:29 UTC to 20:12:39 UTC\) were unsuccessful and should be re-submitted. We acknowledge the inconvenience and disruption this caused our customers and sincerely apologize for any impact on your operations. # Root Cause The root cause of the incident was a decrease in the capacity of our API service due to an unsuccessful deployment. This decrease in capacity, combined with higher than usual request volume, led to the service being unable to handle the incoming traffic. # Resolution Upon discovery of the issue, our on-call operations team was immediately notified and responded swiftly. The team's actions included re-executing the failed deployment to restore normal API service functionality. # Process Changes and Prevention **Actions Taken:** * We re-executed the unsuccessful deployment. * We permanently scaled up the capacity of our API service to handle the increased traffic. * We are implementing additional monitoring and alerting to detect and prevent similar issues in the future. **Ongoing Improvements:** * We are reviewing our deployment processes to minimize the risk of similar incidents. * We are investigating ways to improve the resilience of our API service to handle unexpected spikes in traffic. ‌ We deeply regret this incident and invite any Fullstory customer who was materially affected to contact [[email protected]](mailto:[email protected]). We stand by ready to fully address all of your concerns.

Read the full incident report →