Some search functionality not available

Update timeline

1. identified Feb 16, 2025, 04:40 PM UTC

   During routine maintenance to our search infrastructure, an update to the indexed data did not apply correctly, resulting in the inability to search for some resources in the admin application, such as candidates.

2. identified Feb 17, 2025, 10:41 AM UTC

   We have identified the cause of the issue and are steadily rolling out the fix.

3. identified Feb 18, 2025, 11:02 AM UTC

   We have restored functionality to 95% of searchable resources and continue to roll out the fix for the remainder.

4. resolved Feb 18, 2025, 04:29 PM UTC

   We have finished rolling out the fix for our search backend, and all search components are 100% functional again.

5. postmortem Feb 27, 2025, 03:32 PM UTC

   We apologise for last week’s disruption to our typically swift and flexible search functionality. This was caused by routine maintenance of our search infrastructure over the weekend of February 15th and 16th, which encountered unexpected issues during data migration. The resolution of the issue was further complicated by the larger-than-anticipated size of some datasets and the difficulty of certain technical processes in keeping up. On Tuesday, February 18th, we also deployed several bug fixes that sped up our data processing and ensured that search failures did not make the UI unusable in some circumstances. Rest assured, the team has thoroughly reviewed the incident, and changes are already in the pipeline to prevent similar occurrences.