AWS DNS was experiencing issues (see http://status.aws.amazon.com/) and our webhooks that contact sites residing on AWS were causing a backup in our queue.
We're looking at options to prevent jobs that hit outside services from impacting our job queues.
Posted Dec 18, 2014 - 13:52 MST
Monitoring
The backup has been cleared. The root cause seems to be connectivity issues between our servers and some 3rd party services, such as PagerDuty - which in turn caused our work queues to back up. We're monitoring to see if connectivity is still a problem.
Posted Dec 18, 2014 - 13:45 MST
Identified
The issue has been identified and a fix is being implemented.