Canvas Data ETL Problem (May 11th, 2017)

ccoan
Instructure
Instructure

On May 11th our ETL Engine which is the process responsible for processing all the data from Canvas, and uploading them to the Customers Portals have run into series of failures causing the dumps to be severely delayed. This is the second time ever Canvas Data has been fully delayed, and the first time since GA that we've been unable to comply with getting a dump out in 36 hours.

These changes were completely unexpected, and were not caused by a recent deploy. Moreover they were caused by some new data we started picking up that has found numerous bugs that we failed to catch before because it didn't exist in any production environment.

We will be providing a full RCA to all customers publically in this space when this is all over. We'll also have multiple people checking the support tickets at canvasdatahelp@instructure.com , in order to notify people this way.

We thank you for your patience, and we're extremely sorry in this major disruption in service. We know how important your data is for you, and we know how time sensitive it is.