Memsource incident
Performance Disruption of All Phrase TMS (EU) Components between March 27, 2025 9:01 AM CET and March 27, 2025 9:17 AM CET
Resolved after 18 minutes, affecting Analytics and API and 1 more, on March 27, 2025.
### **Introduction** We would like to share more details about the events that occurred with Phrase between 9:01 AM CET and 9:17 AM CET on March 27, 2025 which led to an outage of all Phrase TMS \(EU\) components and what Phrase engineers are doing to prevent these issues from reoccurring. ### **Timeline** 09:01 AM CET: The number of processed requests began to slowly decline. 09:07 AM CET: Monitoring systems alerted our on-call teams about the instability of the environment. 09:12 AM CET: Support tickets started to appear. Our engineers begin investigating the situation. 09:17 AM CET: A faulty component was identified and restarted, which stabilized the environment. **Root Cause** One of the caching components nodes became overloaded with incoming traffic. This caused dependent services to experience significant delays while waiting for responses from the cache middleware. **Actions to Prevent Recurrence** We are upgrading the caching system to a newer version and increasing the number of cache nodes to better spread the traffic and prevent network congestion.
Affected components
Update timeline
- investigating Mar 27, 2025, 08:18 AM UTC
We are currently experiencing a performance disruption of all Phrase TMS (EU) components. Our engineering team is investigating the issue.
- identified Mar 27, 2025, 08:26 AM UTC
The issue has been identified and a fix is being implemented.
- monitoring Mar 27, 2025, 08:27 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Mar 27, 2025, 08:36 AM UTC
This incident has been resolved.
- postmortem Apr 07, 2025, 12:47 PM UTC
### **Introduction** We would like to share more details about the events that occurred with Phrase between 9:01 AM CET and 9:17 AM CET on March 27, 2025 which led to an outage of all Phrase TMS \(EU\) components and what Phrase engineers are doing to prevent these issues from reoccurring. ### **Timeline** 09:01 AM CET: The number of processed requests began to slowly decline. 09:07 AM CET: Monitoring systems alerted our on-call teams about the instability of the environment. 09:12 AM CET: Support tickets started to appear. Our engineers begin investigating the situation. 09:17 AM CET: A faulty component was identified and restarted, which stabilized the environment. **Root Cause** One of the caching components nodes became overloaded with incoming traffic. This caused dependent services to experience significant delays while waiting for responses from the cache middleware. **Actions to Prevent Recurrence** We are upgrading the caching system to a newer version and increasing the number of cache nodes to better spread the traffic and prevent network congestion.
Your product runs on more than Memsource
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services