New Quizzes Oops Error with Blueprint Sync

[JamesSekcienski]

We use blueprints to manage the initial course content for the majority of our courses. We have migrated the majority of our Classic Quizzes to New Quizzes because we were originally told that we were going to be forced to use New Quizzes and we needed to start the process early to finish in time for the deadline.

In the past the New Quizzes copied into the associated courses without issue. We recently completed our bulk content load for the Spring semester and started receiving reports from instructors that the New Quizzes showed an oops error message that something went wrong with copying the content and there are buttons to Retry or Cancel. The blueprint syncs didn't report errors with New Quizzes failing to copy. However, after running a custom report using API calls, we found that about 40% of them failed to copy based on the assignment workflow state. If we use the "Retry" button it is able to copy the quiz, but it is no longer associated with the blueprint and gets removed from its placement in the Modules. Performing a re-sync also doesn't solve the problem.

We are still awaiting a response from Canvas Support, but we wanted to see if anyone else has experienced a similar issue with New Quizzes and performing bulk blueprint associations/syncs. If so, did you find any efficient ways to resolve the issue? We were given a suggestion for manually fixing the issues for one course, but it isn't scalable for the number of quizzes and courses that need to be fixed. We would prefer to avoid needing to reset the courses as some instructors started personalizing their courses for the upcoming term.

[JayParekh]

Hi James, We are exactly in the same situation and trying to resolve it by working Canvas support and our CSM.  Please share with me if you know a work around or have come up with a solution since you've raised it a while ago. Thank you
Jay

[JamesSekcienski]

@JayParekh 

We had to wait for Canvas Support to prioritize it as something for their engineers to fix.  They fixed the majority of the issues, but didn't provide us with a way to fix it like they did.  Based on the activity in the blueprints, it looks like they must have done something to allow the blueprint sync to fix the issue if it was still in an error state. 

We had a few instructors later report issues with quizzes that were empty, but we suspect it may have been due to someone clicking the "Cancel" on the quiz error rather than "Fix" so it looked like it was fixed, but really the quiz was empty.  For these we deleted the quiz that was missing questions and would re-copy the quiz from the blueprint and then add it to the appropriate module.  This wasn't ideal, but worked well enough for the few remaining errors that were reported by instructors.

We are still waiting on more information about the cause of the issue and how we can ensure it doesn't happen again during our next content load. I'm wondering if there is some type of timeout/throttling issue occurring with New Quizzes that isn't reported to the Canvas API and is separate from the Canvas API Rate Limit.  I have checks built into my script to monitor the Canvas API rate limit to avoid exceeding it and all the blueprint syncs that had errors with New Quizzes reported as completing successfully.  Through some additional testing, I did find that even though the blueprint sync will report that it is finished successfully, the New Quizzes may still be in the process of being copied.  Therefore, it seems they are running on different processes that are not fully communicating with each other to accurately report the status.

We wish the New Quizzes API was ready so that we could audit more to see if there are any other quizzes that don't have any questions in them so we can resolve the issue without having to wait for an instructor to report it.  Using the Assignments API, we were able to find assignments where "is_quiz_lti_assignment" was true (meaning it was a New Quiz) and check the "workflow_state" to see if it was still in an error state ("failed").  However, this doesn't find the quizzes where someone already clicked "Cancel" and the quiz now looks to be fine, but doesn't have questions in it.

[JayParekh]

Hi James,

Thank you very much for the detailed response. Again, we are having very similar experience.  We just heard back from Canvas Engineers that the issue has been resolved for FUTURE New Quizzes sync but for the existing New Quizzes that are not synced well or had errors like no content in them (just like quizzes with no questions in them in your case) will have to be fixed manually. We have about 100 associate courses or more cumulatively which we need to go in and make manual fixes for the multiple New Quizzes within those courses. We are discussing our options with Canvas team to work on the solution without any unwanted repercussions.  They have provided us with two options so far: One to use course export tool and Second is the course reset option both of which are very manual processes and simply too time consuming. 

I am new to the API world and have not used it. Having said that I understand the overall process you've undertaken to monitor the issue and would like to make a guess here that since NQ still operate a bit on the external tool side of the Canvas interface, the BP sync shows that the sync is complete when the New Quizzes '"shells" are created and then it lets the external app make copies inside the "shells" after the Sync process has finished from the BP course. i.e., the sync progress bar coming to a stop and the BP course registers that the sync is complete. 

While this copy is happening, any changes made to the NQ such as editing it from the Modules page or hitting of "retry" or "cancel" button affects the quiz by breaking its link (probably treats the NQ as an edited Quiz) and prevents it from receiving further updates or finish the copy process.

I found that the while the NQ were getting synced, teachers of the associate course or admins had the opportunity to click on any of the NQ from Modules page. If they did that,  (I tested this too) the Quiz will appear to have just the title and no content or questions in them. This opening/editing of a New Quiz from the Modules page also prevents it from getting the content copied across from the blueprints and somehow breaks the link. Which would then also prevent it from receiving further updates from future course syncs and gets listed in 'Exceptions'. 

I am now thinking in the direction of how to fix the existing NQs that are in error state if Canvas does not come up with a time saving fix rather than manual individual associate course fixes.  I'll update here if I come up with a better solution or if Canvas provides a good workaround.  

[JamesSekcienski]

@JayParekh 

I'm sorry to hear that they said the existing issues have to be fixed manually, especially with that many issues needing to be resolved.  It would have been nice if these bugs were resolved and New Quizzes was no longer still in development before telling schools they were going to have to make the switch to New Quizzes.  We were happy with Classic Quizzes and were hesitant to use a tool that was still in development with issues being reported.  However, their original timeline forced us to begin the switch because we had too many blueprints to update in time for the original deadline.  So now we are forced to deal with these unexpected issues that didn't occur with Classic Quizzes.

Thanks for sharing the additional information about the potential problems caused by accessing it through the Modules page.  We didn't discover that issue, but that may have been the cause of some of the issues we saw too.

I hope you are able to come up with a solution that isn't too time consuming.  If you go with the option of performing course resets, perhaps Canvas Support could assist you with doing this for the courses that had a New Quizzes error.  As a side note, there is an API call to reset a course which you could use with Postman Runner to reset your courses in bulk.  The Developer group in the Canvas Community has some old posts sharing how to use Postman to make Canvas API calls.  Be sure to test these on your Beta or Test server first to make sure it is working before using it on Production. I haven't reset courses before with an active blueprint association, so I'm not sure what happens with the association and/or sync when doing this. If it automatically re-syncs the course with the blueprint after the reset, that would be really convenient.

[JayParekh]

@JamesSekcienski Hi James,

I came up with a workaround which was to simply duplicate all New Quizzes in a course and replace the new ones with old ones in the modules. Then resynced the course which made all the students able to start working on the New Quizzes atleast. This workaround led to all New Quizzes to duplicate in the course but Canvas team supported this work around and said that they'll help us cleaning up the old remnants New Quizzes if they do not have any submission attempts.     

I'm still waiting for their help on this as it has not been cleaned up but this work around has helped us get going with a downside that teachers have to manage the markbook/gradebook with duplicates of New Quizzes until it gets resolved. 

Thank you for suggesting about API calls. That's something I've started to learn and hopefully use it in future to troubleshoot such issue as well as use it productively to GET/POST/PUT 😉 

[JamesSekcienski]

@JayParekh 

I'm glad to hear you found a workaround and that Canvas Support is going to assist with the clean-up.

We are preparing for our bulk content load next week for our summer courses.  Canvas Support has said the issue that caused the problem has been resolved and we shouldn't run into any of these same issues, but I'll let you know how it goes.  Fortunately this time around, we are aware of this potential issue so we know to check for it.

[JayParekh]

@JamesSekcienski 

Good luck with the new content load. I'd recommend to just lock all the New Quizzes. In case something goes wrong, atleast teachers from associates won't be able to make any changes and you can push out new ones or just amend the ones that are there. Later on, after the checks are performed, you can unlock the New Quizzes.  

[JamesSekcienski]

@JayParekh 

I wanted to let you know that we only ran into 1 error during our Summer content load.  While we have fewer courses in the summer, I'm hopeful that the issue has been mostly resolved.  However, I'll still be monitoring for issues again with the Fall content load when we have more courses receiving content.

[sandy_griffin]

We are running our summer copies and have had about 20 courses with the error. We are only about halfway done, so there may be more. I reopened our support ticket with Canvas but haven't heard back yet. This is so frustrating.

[DanielStorage]

We're running into this error as well. "New Quizzes" have been very frustrating from the start. I miss Classic Quizzes.