We found the root cause of the issue, and fixed it by adjusting Kafka partition configuration.
Ultimately, the impact is: (all times MST): 10:51am - 12:50pm: small amount of data dropped; 12:50pm-3:06pm: incoming data delayed by about 10min; 3:06PM & onward: data is current; operations normal.
Posted Sep 19, 2016 - 15:56 MDT
Update
Data is no longer being dropped, but you may still see a few minutes latency for new data coming in.
Posted Sep 19, 2016 - 15:12 MDT
Identified
The issue has been identified and a fix is being implemented.
Posted Sep 19, 2016 - 13:04 MDT
Investigating
Not all customers are affected, and for those customers affected, only a subset of reporting nodes are having issues. It appears to be related to clock skew -- we are investigating.