We're back to steady state, with no additional incidents overnight. Closing this issue.
Posted Jan 28, 2016 - 10:01 MST
Update
Update: ingestion is still running well, and we've processed all the outstanding failures. Your charts should look complete, with no historical data lost. If you're seeing anything strange or have questions, please get in touch.
We'll be monitoring the situation overnight, and will post an update in the morning.
Posted Jan 27, 2016 - 22:43 MST
Monitoring
Metric ingestion has caught up - we'll be monitoring the situation through the night. We do have a number of ingestion failures we'll be retrying that largely occurred between 4:30 - 6 pm MT.
Posted Jan 27, 2016 - 20:36 MST
Update
We're down to a 16 minute backlog.
Posted Jan 27, 2016 - 19:51 MST
Update
We're making a bit of progress on metric ingestion - we're about 45 minutes behind right now.
Posted Jan 27, 2016 - 18:48 MST
Update
We're working with our timeseries database provider to resolve the increase in import times.
Posted Jan 27, 2016 - 18:18 MST
Investigating
We're seeing elevated insert rates into our time series database, which is increasing the amount of time to import the metric backlog. We're working to identify the cause.
Posted Jan 27, 2016 - 17:45 MST
Identified
The database is back online. We're working through the metric backlog.
Posted Jan 27, 2016 - 16:37 MST
Investigating
A backlog triggered high throughput on our database. We are upgrading our database, during which time, the Scout UI won't be accessible.
All incoming metrics are preserved, and will show up after the upgrade is complete.