This is resolved: the outage was caused by alerts w/very large text fields not fitting into our background job queue. We increased the column size so its far greater than the possible alert text.
Posted Jun 17, 2015 - 12:18 MDT
Monitoring
We've addressed the corrupted jobs and the queue length has dropped back down. We're working on a fix to prevent this from occurring again.
Posted Jun 17, 2015 - 10:06 MDT
Investigating
Our worker for sending external notifications (Pagerduty, VictorOps, Webhooks, etc) has died. We've identified a corrupted job causing the issue and and are investigating.