Is Coveralls down?

User in Europe and US east coast with large repos (5k+ repos) experienced slow build completion, starting around 2am PST / 5am EST. We have identified the issue and allocated resources to clear a backlog of background jobs. Jobs should clear and builds should complete in the next 45min - 1hr.

The root cause of issues with large repos has been fully addressed. We are continuing to clear some backlogged background jobs. We will continue to monitor and report updates here.

This incident has been resolved.

# Update: Major Performance Optimization Released for Heavy Repo Builds Triggered by this latest incident, we've released a major optimization to our code that processes large \(aka. “heavy”\) repos \(>=5K source files\). Tests over the weekend of Fri, Oct 9-Sun, Oct 11 show a near 20x improvement in performance per heavy repos build at lower, weekend volumes. While we expect the optimization to continue to drastically improve build times for these repos \(and reduce instances of increased latency for them\), we won’t know what the aggregate improvement is until we are at full volume this week. We are monitoring to get a sense of where the performance improvement lands, but are hopeful we’ve crossed a significant milestone in how efficiently these builds are processed given the unique challenges many of them pose.

We are currently investigating this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

Operational in all regions. We are continuing to monitor for any further issues.

We are continuing to monitor for any further issues.

This incident has been resolved.

### Resolved: elevated error rates, Monday September 28, 08:00 to 09:15 PT **Timeline**: Between about 08:00 and 09:12 PST on September 28, some requests to Coveralls returned HTTP 500 errors. Coverage uploads, GitHub webhook events, badges, and coverage pages were affected. Failures were intermittent at first and peaked toward the end of the window. Service was fully restored at 09:12 PST. **Cause**: a capacity issue with our primary database. We have have added capacity, enabled automatic scaling, and changed our monitoring triggers to be alerted well before this kind of issue can recur. **What to do**: If you were affected: re-run any CI job that uploaded coverage to Coveralls between 08:00 and 09:15 PT on September 28, or simply push a new commit. No data was lost or corrupted; those uploads simply did not land. We are sorry for the disruption.

We are currently experiencing occasional "Website under heavy load" warnings. We believe this is associated with an army of scraper bots we experienced and successfully mitigated this weekend. We believe those bots may have gotten around an initial phase of defense involving managed challenges at our edge provider. Separate from a DDoS attack, these are millions of individual requests coming from over 200K different IP addresses across 10 different countries, so our standard DDoS defense of IP quarantines at our firewall do not apply. We are implementing a second phase of defense and will report updates here.

This was the second wave of the attack we experienced this weekend, broadened and tripled in volume. We implemented a second phase of defense which appear to be working and will continue monitoring for the next hour.

This incident has been resolved.