Swapcard Outage History

Started Oct 09, 2026, 12:30 PM UTC· resolved Oct 09, 2026, 03:04 PM UTC

1. monitoring Oct 09, 2026, 12:30 PM UTC

   A data replication delay on one of our database replicas may cause some users to see outdated or inconsistent information in the interface. Some users may also run into occasional errors. No data has been lost. Our team has taken action, and we are monitoring closely while the replica catches up. If you see inconsistencies, refresh the page. We'll post another update once this is fully resolved.

2. resolved Oct 09, 2026, 03:04 PM UTC

   The incident is resolved. Replication is fully back in sync, and the platform is working normally.

Read the full incident report →

Started Oct 07, 2026, 10:21 AM UTC· resolved Oct 07, 2026, 11:45 AM UTC

1. investigating Oct 07, 2026, 10:21 AM UTC

   We are currently investigating an issue affecting push notifications. Some users may not receive push notifications. All other platform features are working normally. Our team is working on a fix, and we will post an update here as soon as we have more information.

2. monitoring Oct 07, 2026, 10:31 AM UTC

   A fix has been applied and push notifications are being delivered normally again. We are monitoring to make sure everything is back to normal.

3. resolved Oct 07, 2026, 11:45 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 07, 2026, 03:20 AM UTC· resolved Oct 07, 2026, 10:25 AM UTC

1. investigating Oct 07, 2026, 03:20 AM UTC

   We are investigating an issue where the Developer API may return intermittent errors on some requests. Most requests are succeeding normally. Failed requests can be safely retried. We will post an update as soon as we have more information.

2. resolved Oct 07, 2026, 10:25 AM UTC

   Between 03:00 and 10:25 UTC, the Developer API may have returned intermittent errors on some requests. Requests that failed during this window can be safely retried. The issue has been resolved and the Developer API is operating normally.

Read the full incident report →

Started Sep 11, 2026, 10:07 AM UTC· resolved Sep 11, 2026, 10:29 AM UTC

1. investigating Sep 11, 2026, 10:28 AM UTC

   We are currently investigating this issue.

2. resolved Sep 11, 2026, 10:29 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Sep 08, 2026, 09:01 AM UTC· resolved Sep 08, 2026, 09:04 AM UTC

1. identified Sep 08, 2026, 09:01 AM UTC

   The issue has been identified and a fix is being implemented.

2. monitoring Sep 08, 2026, 09:02 AM UTC

   A fix has been implemented. We are working on restoring the analytics data, there are no data loss during the process.

3. resolved Sep 08, 2026, 09:04 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Aug 31, 2026, 12:01 PM UTC· resolved Aug 31, 2026, 12:01 PM UTC

1. resolved Aug 31, 2026, 12:01 PM UTC

   On Monday 31 August, exhibitors and organizers scanning badges began seeing an error after a scan, with the scan then stored as an offline scan on the device. This happened on devices with a working internet connection, which is why it was initially mistaken for a venue network problem. On iOS, some devices showed only a red banner that did not open when tapped. No lead data was lost. Two mechanisms protected the data at the same time. Server side, each lead was saved to our primary database at the moment of the scan, before the step that failed, so the lead already existed even when the app reported an error. Device side, the app's offline mode captured the scan and synced it once scanning recovered. Because a repeat submission of the same scan updates the existing lead rather than adding a new one, this did not create duplicate leads. The cause was specific to the badge scanning path. Registration, event pages, meetings, messaging, exports and the organizer back-office were not affected during this window.

Read the full incident report →

Started Jul 01, 2026, 05:37 PM UTC· resolved Jul 01, 2026, 07:48 PM UTC

1. identified Jul 01, 2026, 05:37 PM UTC

   Sherlock is currently experiencing degraded performance due to an inference capacity issue. We are working with our inference providers to restore full capacity as quickly as possible.

2. identified Jul 01, 2026, 07:08 PM UTC

   A fix to increase capacity is currently under testing.

3. resolved Jul 01, 2026, 07:48 PM UTC

   Inference capacity has been fully restored

Read the full incident report →

Started Jun 19, 2026, 02:59 PM UTC· resolved Jun 19, 2026, 03:35 PM UTC

1. identified Jun 19, 2026, 02:59 PM UTC

   The issue has been identified and a fix is being implemented.

2. resolved Jun 19, 2026, 03:00 PM UTC

   This incident has been resolved.

3. resolved Jun 19, 2026, 03:35 PM UTC

   We are providing a detailed post-mortem report regarding the service disruption that affected Swapcard Login on June 19th, 2026. This issue was caused by an upgrade of our cache cluster, which led to temporary login failures. The goal of this post-mortem is to share insights from our assessment and the steps taken to resolve the issue while providing transparency to our customers. Summary As part of a planned upgrade to increase the capacity of our cache cluster, we carried out a zero-maintenance-window upgrade, a rollout designed to require no scheduled downtime. During the rollout, a misconfiguration that was not identified in our pre-production environment affected the login service in production, resulting in a period during which users were unable to sign in. Timeline (UTC) [16:35] — Migration of production services to the upgraded cache cluster proceeds as part of the zero-downtime rollout. [16:38] — The login service begins failing to connect to the cache; sign-in errors start. [16:42] — Incident detected and investigation begins. [16:45] — Root cause identified: corrected configuration deployed to login. [16:48] — Login fully recovers; error rates return to normal. Mitigation We updated the login service to connect to the upgraded cache cluster using the correct configuration, restoring sign-in functionality. As a precaution, we reviewed all other production services and corrected one additional service that carried the same latent misconfiguration before it could cause further impact. Root cause During a planned upgrade to increase the capacity of our cache cluster, a misconfiguration was introduced that left the Login service unable to connect to the upgraded cluster. The issue was not identified in our pre-production environment and surfaced in production during the rollout, causing sign-in to fail until the configuration was corrected. Next steps We are adding automated checks to (1) detect any reference to a cache cluster scheduled for decommission and (2) verify configuration parity between our pre-production and production environments, so this class of issue is caught before reaching production in future upgrades.

Read the full incident report →

Started Mar 19, 2026, 12:19 PM UTC· resolved Mar 19, 2026, 12:21 PM UTC

1. resolved Mar 19, 2026, 12:19 PM UTC

   This incident has been resolved.

2. resolved Mar 19, 2026, 12:21 PM UTC

   Summary Approximately two weeks ago, users began experiencing disruption when accessing analytics data within App Studio. Initial investigation did not immediately identify the root cause; a deeper analysis ultimately traced the issue to a routine infrastructure consolidation that was capping database capacity below the threshold required to handle specific event analytics queries spanning long date ranges. Full service has since been restored following a targeted increase in database resources. Timeline • 2026-03-11 19:32 UTC — Incident created. Initial investigation launched following a rise in user-reported loading issues on analytics, despite no prior performance degradation. The initial investigation did not surface a clear root cause at this stage. • 2026-03-18 10:45 UTC — Automated monitoring triggered alerts for elevated database error rates, corroborated by incoming user reports of analytics failures. • 2026-03-19 09:30 UTC — Deeper analysis by the Engineering team identified the root cause: high latency and query failures isolated to specific event analytics queries over long date ranges, linked to the earlier infrastructure consolidation. • 2026-03-19 09:40 UTC — Remediation plan executed: database resources scaled up to meet query load demands. • 2026-03-19 10:05 UTC — Database performance stabilized; analytics functionality verified as fully restored. Resolution To restore service, database resources were immediately scaled up to accommodate the throughput required by analytics queries. Once additional capacity was provisioned, query timeouts ceased and application stability was reestablished. The system remained under active monitoring until no further degradation was observed, at which point the incident was marked resolved. Root Cause Analysis The incident was caused by a capacity mismatch introduced during a routine infrastructure consolidation. Such consolidations are a standard part of our infrastructure lifecycle and are conducted with active monitoring in place to ensure continued performance across all services. In this instance, however, the capacity allocated to the Analytics Database was capped at a level that, while adequate for standard workloads, could not sustain the computational demands of specific event analytics queries executed over long date ranges. The initial investigation did not immediately surface this connection; it was only through deeper analysis that the consolidation was identified as the underlying cause. When these queries were executed, they exceeded available processing capacity, resulting in database timeouts and repeated failures of the analytics service; conditions that persisted until capacity was restored. The investigation also surfaced a broader pattern: a gradual uptick in user-reported loading issues over the preceding two weeks, despite analytics having previously performed reliably. This indicated the capacity cap had been quietly eroding performance well ahead of the full outage. Conclusion We are straightening our processes to prevent recurrence. Monitoring coverage applied during database consolidations will be strengthened to detect query latency degradation earlier, and capacity planning will be refined to better account for the demands of specific event analytics queries over long date ranges.Looking further ahead, the team is actively working on delivering a fully revamped analytics pipeline to production later this year. This is not an incremental improvement; the new pipeline is being built to deliver deeper, richer metrics that give event organizers a comprehensive view of their event performance.

Read the full incident report →

Started Mar 11, 2026, 01:30 PM UTC· resolved Mar 13, 2026, 02:02 AM UTC

1. resolved Mar 11, 2026, 01:30 PM UTC

   This incident has been resolved.

2. resolved Mar 13, 2026, 02:02 AM UTC

   Summary On March 11, 2026, we experienced a service disruption affecting the visibility of speaker and attendee profiles across multiple events. The incident was triggered by a timing issue during a scheduled infrastructure update — a data refresh was initiated before new field definitions had fully propagated to our Search Engine, causing the indexation system to be built from an outdated configuration. Our Search Engine relies on an indexation system to serve profile data quickly across events. When the refresh completed against the stale configuration, new fields were rejected due to strict schema enforcement, leaving profiles absent from search results and listings. The goal of this post-mortem is to share our assessment and the steps taken to resolve the issue, while providing full transparency to our customers. Timeline Time \(UTC\) Event Mar 11 – 11:04 Infrastructure update deployed to production. Changes confirmed as applied. Mar 11 – 13:21 Data refresh initiated as part of the planned release process. Previous indexation remained fully functional during this period. Mar 11 – 13:42 Release step complete. Data refresh running, expected duration ~3–4 hours. Mar 11 – 14:49 Configuration mismatch errors first logged internally. Mar 11 – 15:33 Data refresh reported as complete. Spot checks passed. Final release step initiated. Mar 11 – 16:52 Reports of missing profiles across multiple events surfaced. Incident response team immediately activated. Mar 11 – 17:29 Root cause identified in under few minutes: indexation system had been built before the new fields were fully available. Internal alerts corroborated the finding. Mar 11 – 18:08 Analysis completed confirming the refresh did not produce an indexation with the expected new fields. Mar 11 – 18:39 Corrective data refresh triggered with the correct configuration fully in place, within 30 minutes of initial reports. Mar 11 – 18:43 New indexation confirmed to contain the correct fields. Full refresh in progress. Resolution Once the root cause was identified, the team immediately initiated a new data refresh against the correct, fully propagated configuration. Profile visibility and search behavior were monitored throughout the process. Once the refresh completed, all profiles returned to normal search and listing behavior. Root Cause Analysis The root cause was a race condition between the deployment of new Search Engine field definitions and the triggering of a data refresh. The refresh started before the new configuration had fully propagated, causing the indexation system to be built from a stale schema. Due to strict schema enforcement, subsequent write operations referencing the new fields were rejected, leaving profiles absent from search results and listings. Several factors contributed to this incident: • Ambiguous deployment signal: A pipeline warning created uncertainty around whether the deployment had completed cleanly. • Insufficient spot checks: Post-refresh verification passed at a surface level, masking the underlying schema mismatch until live traffic exposed it. Short-term improvements: • Add a mandatory pre-refresh validation step to confirm the Search Engine schema matches the expected configuration before any refresh is initiated. • Update the release process to include explicit schema validation checkpoints for sub field, as a required step. Conclusion This incident stemmed from a timing edge case in our release process. The improvements listed above reflect our commitment to preventing similar incidents and maintaining a reliable platform for all our customers. For any questions or concerns regarding this incident, please reach out to our support team.

Read the full incident report →

Started Feb 12, 2026, 09:00 AM UTC· resolved Feb 13, 2026, 03:51 AM UTC

1. resolved Feb 12, 2026, 09:00 AM UTC

   This incident has been resolved. It lasted from 09:00 UTC to 09:30 UTC and is now resolved.

2. resolved Feb 13, 2026, 03:51 AM UTC

   We are providing a detailed post-mortem report regarding the service disruption that affected Swapcard customers on February 12th, 2026, from 09:00 UTC to 09:30 UTC. This issue was caused by an unprecedented traffic spike that exceeded the capacity of one of our internal services, leading to temporary degraded performance across badge printing \(SwapAccess\) and Studio access. The goal of this post-mortem is to share insights from our assessment and the steps taken to resolve the issue while providing transparency to our customers. Summary On February 12th, 2026, Swapcard experienced a 30-minute service disruption affecting badge printing via the Check-in App \(SwapAccess\) and access to the Studio interface. Customers with live events during this window experienced printing delays, degraded Studio access, and intermittent error messages. The incident was caused by an unusual and unprecedented volume of concurrent traffic hitting one of our internal services. This traffic pattern had not been observed before and exceeded the scaling thresholds configured at the time. The sudden load caused elevated response times that cascaded to downstream services, including badge generation and Studio access. The platform self-recovered as traffic levels normalized around 09:30 UTC. Timeline 09:00 UTC | An unprecedented spike in concurrent traffic began hitting one of our core internal services, exceeding previously observed traffic patterns. 09:00–09:15 UTC | The service could not scale fast enough to absorb the sudden load. Elevated latency cascaded to dependent services, causing badge generation timeouts and degraded Studio access. 09:15–09:30 UTC | Traffic levels began to normalize. The platform progressively recovered as request queues cleared. 09:30 UTC | Full service restoration. Badge printing and Studio access returned to normal operation. Onsite report | Our infrastructure team conducted a thorough investigation and immediately applied improvements to prevent recurrence. Root Cause Analysis The root cause of this incident was an unprecedented and sudden spike in concurrent traffic that exceeded the scaling capacity of one of our internal services. This traffic pattern had not been encountered before in production, and the service's auto-scaling configuration was not tuned to react quickly enough to absorb such a rapid increase. As response times on this service climbed, the impact cascaded to dependent features — including badge generation and Studio, which rely on it for real-time operations. This cascading effect amplified the user-facing impact beyond what the initial traffic surge alone would have caused. Remediation Following this incident, our infrastructure team immediately took action to strengthen the resilience of the affected services: Scaling improvements: We have significantly increased the resource capacity and improved the scaling configuration of the affected internal service to handle traffic spikes well beyond the levels observed during this incident. The service can now absorb sudden surges much more effectively. Resource optimization: We have optimized the resource usage of the service to ensure it operates more efficiently under load, reducing the likelihood of capacity issues even during unexpected traffic peaks. We have deployed additional monitoring and alerting specifically targeting the failure patterns observed during this incident. This ensures that if a similar traffic surge were to occur, our team would be automatically notified within seconds and could intervene proactively before any customer-facing impact. Conclusion This incident was caused by an unpredictable traffic pattern that had not been previously observed on our platform. While the disruption was brief, we understand the impact it had on customers running live events during that window, and we take that seriously. The scaling, optimization, and alerting improvements we have put in place significantly reduce the risk of a similar incident occurring in the future. Our infrastructure team continues to monitor the situation closely. If you have any questions or concerns regarding this incident, please don't hesitate to reach out to our support team.

Read the full incident report →

Started Feb 04, 2026, 02:58 PM UTC· resolved Feb 04, 2026, 02:59 PM UTC

1. resolved Feb 04, 2026, 02:58 PM UTC

   This incident has been resolved.

2. resolved Feb 04, 2026, 02:59 PM UTC

   We are providing a detailed post-mortem report regarding a production change that caused continuous emails to be sent again \(duplicates\) to some attendees. The impacted scope was limited to events created before February 4, 2025 that still had continuous emails enabled at the time of the incident. The goal of this post-mortem is to share what happened, why it happened, how it was resolved, and what we are doing to prevent a recurrence. Summary A background task runs every 30 minutes to process continuous email campaigns: • Retrieve all enabled continuous email campaigns • For each campaign, verify whether each targeted attendee already received the corresponding email • Send the email only to attendees who have not received it yet The incident was caused by the "already received" check relying on a database that only started being populated after February 4, 2025. For events created before that date, the delivery history existed in a different \(legacy\) datastore. As a result, the system incorrectly concluded that certain attendees had not received the email yet, and re-sent it. Timeline • 2026-02-04 07:56 UTC | A change was released that caused continuous emails to be incorrectly sent again \(duplicates\) to some attendees. • 2026-02-04 11:10 UTC | The change was reverted, stopping further duplicate sends. Resolution We resolved the incident by reverting the change introduced at 2026-02-04 07:56 UTC, restoring the previous behavior and preventing additional duplicate emails from being sent. Root Cause Analysis The root cause was an incorrect assumption in the delivery-history lookup logic: • The deduplication check queried a datastore that only contained delivery records populated starting 2025-02-04. • For events created before 2025-02-04, delivery records were stored in a legacy datastore. • Because the check did not account for the legacy source, it returned false negatives \("not sent yet"\), resulting in duplicate emails being sent on subsequent background-task executions. Immediate Actions Completed • Reverted the problematic change to stop duplicate email sends. • Identified and confirmed the gap between legacy and current delivery-history datastores for pre-2025-02-04 events. Short-Term Improvements • Update the "already received" check to consult the correct source for both legacy and current events. • Add safeguards to prevent re-sending when delivery history is missing/ambiguous \(fail-safe behavior\). • Add targeted tests covering pre-2025-02-04 events with continuous emails enabled. Long-Term Improvements • Consolidate delivery-history storage to a single source of truth \(including backfill/migration strategy for legacy data\). • Introduce idempotency guarantees at send-time \(e.g., campaign\+attendee idempotency keys\) to prevent duplicates even if checks regress. • Add monitoring and alerting on abnormal duplicate-send rates for continuous email campaigns. Conclusion We recognize the impact that duplicate emails can have on attendee experience and event organizer trust. We are implementing the preventative measures above to ensure legacy data paths are consistently handled and duplicates are prevented by design. If you have any questions or concerns regarding this incident, please reach out to our support team.

Read the full incident report →