Resolved after 33 minutes, affecting Engine API, on November 6, 2025.
We determined that because of recently adding thousands(!) of tools, one of our production components required a longer than expected startup process, meaning its health check endpoint was unavailable before the timeout expired. As a result, orchestrator was constantly restarting nodes, leading to poor availability of this critical component. Going forward, we plan to move these tasks out of component startup so this doesn't happen again and we can continue to support even larger numbers of tools.
Affected components
Update timeline
- investigating Nov 06, 2025, 11:20 PM UTC
We are currently investigating an issue with our production deployments. We are attempting to rollback to a stable version. Investigation of the issue continues.
- investigating Nov 06, 2025, 11:22 PM UTC
We are continuing to investigate this issue.
- monitoring Nov 06, 2025, 11:30 PM UTC
Our prod services are now recovering. We are continuing to monitor and have identified the root cause of the issue.
- resolved Nov 06, 2025, 11:54 PM UTC
We determined that because of recently adding thousands(!) of tools, one of our production components required a longer than expected startup process, meaning its health check endpoint was unavailable before the timeout expired. As a result, orchestrator was constantly restarting nodes, leading to poor availability of this critical component. Going forward, we plan to move these tasks out of component startup so this doesn't happen again and we can continue to support even larger numbers of tools.
Your product runs on more than Arcade.dev
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services