Bloomerang Outage History

Started Jan 09, 2025, 03:02 AM UTC· resolved Jan 09, 2025, 05:11 AM UTC

1. identified Jan 09, 2025, 03:02 AM UTC

   We have been experiencing a Bloomerang service outage since around 9:15pm EST. Customers may see errors when attempting to login and interact with the application. Our team is currently working to restore fucntionality. We apologize for any inconvenience. An update will be provided in 15 minutes.

2. identified Jan 09, 2025, 03:26 AM UTC

   We are actively working to restore functionality for our users. All components have been affected including online giving forms. Embedded, Hosted, and Self-Hosted online forms are unable to load during this outage and may display a "Donation Form could not load. Please try again later." An update will be provided in 15 minutes.

3. identified Jan 09, 2025, 03:44 AM UTC

   As of 10:38pm EST we have identified root cause and functionality has been restored. Online forms are loading as expected now. Users will now be able to login but may see degraded performance in the application until further notice. An update will be provided in 15 mins.

4. monitoring Jan 09, 2025, 04:07 AM UTC

   A fix has been implemented and we are monitoring results.

5. monitoring Jan 09, 2025, 04:27 AM UTC

   We are continuing to monitor the application for any further issues.

6. resolved Jan 09, 2025, 05:11 AM UTC

   This incident has been resolved.

7. postmortem Jan 15, 2025, 09:59 PM UTC

   ## **System Outage Postmortem** **Date and Time of Incident:** January 9, 2025, 9:15 PM to 10:35 PM Eastern **Incident Summary:** On January 9, our system experienced an outage from 9:15 PM to 10:35 PM Eastern, impacting several critical services. During this time, some users may have been unable to log into the application and/or experienced errors throughout the platform. Donors attempting to complete transactions via donation forms encountered errors, and API services were disrupted. The outage was caused by an improper configuration value for our CRM services. This misconfiguration prevented CRM services from starting correctly, resulting in disruptions across the system. To resolve the issue, we corrected the named configuration value, which restored service performance. **Next Steps:** To prevent similar incidents, we are improving alignment on configuration changes across teams to ensure thorough review and verification prior to implementation. We sincerely apologize for this disruption and appreciate your patience as we work to strengthen our systems and processes.

Read the full incident report →

Started Nov 07, 2024, 07:46 PM UTC· resolved Nov 08, 2024, 08:45 PM UTC

1. investigating Nov 07, 2024, 07:46 PM UTC

   We are investigating an issue with Acknowledgement Emails not sending after a user adds an acknowledgement to an existing transaction. Any new donations that are added and acknowledged will receive their acknowledgements. We apologize for any inconvenience and will provide updates as more information becomes available.

2. identified Nov 07, 2024, 10:37 PM UTC

   The issue has been identified. Our team is determining a timeline for resolution and will provide an update tomorrow.

3. identified Nov 08, 2024, 03:59 PM UTC

   Our team is planning to implement a fix this afternoon to restore functionality. We will provide updates when that work is completed.

4. monitoring Nov 08, 2024, 07:41 PM UTC

   A fix has been implemented and we are monitoring results.

5. resolved Nov 08, 2024, 08:45 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 31, 2024, 01:06 AM UTC· resolved Oct 31, 2024, 02:32 AM UTC

1. investigating Oct 31, 2024, 01:06 AM UTC

   We are currently investigating reports of Intermittent degraded performance across Bloomerang, these issues may be impacting multiple areas of the application. We apologize for any inconvenience this issue may cause and will provide an update in approximately 30 minutes as we learn more.

2. investigating Oct 31, 2024, 01:35 AM UTC

   Some error pages may be occurring for users attempting to log in or add constituent and transaction data. Issues are intermittent and may be resolved by refreshing the browser and trying again. We continue investigating this issue and will provide another update within 30 minutes.

3. monitoring Oct 31, 2024, 02:16 AM UTC

   We have identified the issue and implemented a fix. Performance is now restored and we are continuing to monitor to confirm.

4. resolved Oct 31, 2024, 02:32 AM UTC

   Monitoring confirmed performance is restored and this incident is resolved. We apologize for any inconvenience.

Read the full incident report →

Started Oct 28, 2024, 09:10 PM UTC· resolved Oct 29, 2024, 02:03 PM UTC

1. identified Oct 28, 2024, 09:10 PM UTC

   Beginning Friday, October 25th, Automatic Email Receipts have not been sending for manually entered cash, check, and in-kind donations. Please note, credit card and EFT donations have not been affected by this issue. We apologize for any inconvenience. Our team is actively working to resolve this issue. In the interim, customers can acknowledge cash, check, and in-kind donations using an email with the Purpose set to 'Acknowledgement'. Please reach out to [email protected] for assistance.

2. monitoring Oct 29, 2024, 01:52 AM UTC

   We have implemented a fix and Automatic Email Receipts created after 10/28 9:30 PM EDT will now function as expected. Manually entered cash, check, and in-kind donations entered between October 25th and October 28th 9:30 PM EDT would not have been sent. Customers can acknowledge impacted cash, check, and in-kind donations using an email with the Purpose set to 'Acknowledgement'. Please reach out to [email protected] for assistance.

3. resolved Oct 29, 2024, 02:03 PM UTC

   This incident has been resolved.

Read the full incident report →