Canvas Data 2 retrievals mostly failing for us in the past hour - is any maintenance work being done by Instructure?

[pgo586]

I've noticed that most of our table retrievals have been failing in the past hour (using CD2 REST api to retrieve tables. We've been getting 503 responses for most tables ('service unavailable'). Are others experiencing this too? Is Instructure aware of problems and working on a solution to them? 

Here's an update to this problem: my script stopped after 80 minutes. During this time, it retrieved successfully about 50% of the tables, with the other 50% failing with the same type of 'service unavailable' error. NOTE: the script attempts to re-retrieve failed retrievals a maximum of 3 times. 

This makes me wonder about the readiness of CD2 for prime time - can Instructure please update us as to what is happening? 

[AlexShin]

We noticed this error starting around 4AM EST today (9/11/23) when our automated jobs failed. We've been periodically attempting to refresh our data and have mostly received the error below:

This error was encountered with our home built process. 

('error', {'uuid': '8314e2fb-4021-4495-998c-de290f2c1da8', 'message': 'DAP Querying service is overloaded or under maintenance.', 'type': 'Overloaded'})

We also set up a Postgres DB to test the DAP client and encountered the same error:

root@240f218ec931:~# dap initdb --connection-string postgresql://canvas:DartBoard23@localhost/cd2 --namespace canvas --table accounts
2023-09-11 19:32:25,674 - WARNING - Received error in response:
2023-09-11 19:32:25,677 - ERROR - An exception occurred while executing the command: DAP Querying service is overloaded or under maintenance. (1efdec98-e3f4-48f7-9e07-9fb3464e2be0)
2023-09-11 19:32:25,677 - ERROR -

From our end, we're looking at a much worse success rate than 50%. We attempted the DAP initdb command around 10 times before we hit a successful result for a single small table. 

[MikeRichards]

I feel like Canvas Data 2 should be included on (https://status.instructure.com/). Having a major outage that went on for multiple hours without word from Instructure is extremely concerning. 

Yesterday we made multiple attempts spread out throughout the day and were unable to get a stable session with CD2. Thankfully today it does appear to be back to normal.

[Edina_Tipter · Accepted solution]

This is to say that we were receptive to your feedback and have included Canvas Data 2 service on the Instructure status page under Canvas LMS. In case, next time an outage happens, this will be flagged and we will provide more thorough communication as well.

[J-J-Jason]

Hi Edina-

What are the scenarios where the Canvas Data 2 status would not be green? The times that I've had issues I've noticed that the status does not ever change.

Thanks!
Jason

[Edina_Tipter]

HI @J-J-Jason ,

Right now, the rule is that it changes if the service is not available at all. Yet, it does not change if the issue is sporadic or if it does not affect all tables or if it's affecting just a subset of customers.

Based on the feedback we received, we started to evaluate the existing process as we find ourselves in a new context with CD2 and this requires reaction to new circumstances that did not exist before. Yet, I am asking for some patience in these regards.

Edina

[J-J-Jason]

Thanks, Edina.

On a positive note - I'm finding the DAP library easy to use both from the command line and as a way to extend my own scripts. I appreciate Instructure providing scripts and help with the cumbersome API tasks.

Jason

[jsimon3]

One note that I would to share with everyone that I learned from this experience that we baked into the nuget (https://github.com/uvadev/PullCanvasData2/pkgs/nuget/PullCanvasData2) package:
---   the JWT expires in an hour after authentication...  ---
Its in the API documentation here:  https://api-gateway.instructure.com/doc/ 
But shame on us we didn’t include a catch for it, then it bit us in the ass: 

 

But now:
-  It dynamically retries auth whenever the token is close to expiring

-  In that object is the duration until the token expires

-  So when a request is made and the token is set to expire in under 10 minutes from that point, it async yields while the authentication function gets called again

[mclark19]

Yeah, we found that the token timing out was causing many of our issues when we had tables fail. (We're currently doing a rerun of the job rather than refreshing the token per se, but we might look into that.) That said, I think there is still the underlying issue of the length of time that it takes dap to return a completed job status when the eventual payload is just a single file at the end of the day (we're grabbing each table as a separate job).