We’re Listening: New Quizzes Features, Classic Quizzes Timeline

[shirenv]

One of the things that made me want to join the Instructure team as the chief product officer was the culture of transparency that it has developed over the years. When I joined Instructure earlier this year, I felt it was really important to hear from our customers as we prioritize development of our products. I’ve spent the last several months meeting with our customers and hearing their unique needs. I want to thank you for being so open with me about the challenges you face! I sincerely value the amount of trust you place in us to get it right. 

We Hear You

Many of our customers wanted to talk about New Quizzes. I heard loud and clear that our rollout of the tool has been… uhh… less than perfect. We hear you and I want to assure you that the development of New Quizzes is now our top priority. Many of you expressed a desire to engage with us on product decisions and rollouts before they happen. We’re committed to transparency throughout this process.

Elimination of the Deadline for Switching to New Quizzes

During a recent call with some of our university customers I heard, “reasons to switch to New Quizzes should be compelling, not compulsory.” I couldn’t agree more. We want our customers to use New Quizzes because it is the best solution available for achieving positive assessment outcomes, not because of arbitrary deadlines. So we are removing the deadline to switch to New Quizzes. This will give each customer the ability to consider what they need to be ready to make the most successful transition. Since we have a very diverse customer base, not all institutions will have the same priorities. We’ll continue to partner with each institution to meet their unique needs as they plan to transition to New Quizzes. 

The Path Forward for New Quizzes

Our customers have told us what they need to transition from Classic Quizzes to New Quizzes successfully, and we’re making those features a priority. We understand that our customer base is diverse, and we’ll continue partnering with them to meet their unique needs as they transition to the new product. In addition, we’ll partner closely with institutions that have chosen to adopt and migrate to New Quizzes and ensure we account for their emerging needs in our ongoing product development. 

We’re Giving Everyone More Time

Our goal is to facilitate a seamless transition to New Quizzes when customers are ready—not because of an arbitrary deadline. So, we’ve eliminated the end-of-life date for Classic Quizzes for now. We hope that will ease the pressure on customers and give them time to plan an implementation of New Quizzes when it makes sense for their unique needs. 

However, we will focus our development energy on New Quizzes. It’s our top priority, so we will not be adding any new features to Classic Quizzes. However, we will continue to fix critical bugs and maintain the product.

We’re Here To Help

To help everyone migrate and transition to New Quizzes, we’ve created a New Quizzes Hub, where customers can find up-to-date roadmaps, release notes, and other resources to aid in the transition to the new product. We also encourage our customers to join The Community, where they can connect with their peers and learn from their experiences as they plan and deploy New Quizzes. 

We aim to meet our customers’ needs with every new feature in Canvas, and we believe New Quizzes is the best assessment tool on the market. We’ve listened to customers’ concerns about the product and migration deadline. So we’ve made changes to ensure everyone can move to New Quizzes when they’re ready. 

If you’ve transitioned to New Quizzes, we’d love to hear about your experience, and if you aren’t ready, we’d love to hear from you too.

[BethCrook]

Thank you!! Off to spread the news! 🐼❤️

[HeatherMDowling]

Best. News. Ever. Thanks, Canvas for being awesome and listening to our concerns. We can look forward to New Quizzes with excitement not concern now. 

[AJ07]

Thanks

[knoxd]

This is fantastic news!  Thank you SO much for continually listening and responding to the concerns/needs of your customers! 🐼❤️

[DanBurgess]

We greatly appreciate this decision and recognize this has its own implications for Instructure. I can image managing two codebases is a challenge. Far better to continue to manage that complexity than to risk the reputation of Canvas on a lamentable push of the community onto a critical component that needs more work. 

To Shiren and the rest of the leadership at Instructure, this was the right move.

To the New Quizzes project team, keep up the good work. Yes NQs has a reputation, but I think with  more development, folks will begin to see it is as the tool that want to go to. 

[atcarver]

Thank you, @shirenv and Instructure. This plan gives us the greatest flexibility institutionally to account for a varied user base, and allows for greater buy-in on New Quizzes in the long run. Much appreciated.

[APreimesberger]

Excellent!  Buy in for Canvas/online testing is difficult for many as it is.  With the new quiz setup/formatting as-is, I was greatly concerned about its adoption among staff.  This is especially true do to migration issues of old test banks to the new quizzes as it would alienate many who already use Canvas but who are not strong on computer-usage.  I can't wait for faster report times in the new quizzes section.  The long wait for analysis is difficult in tight schedules when you need to analyze a test in the same day to determine next actions for the students.  But overall, the look/feel is positive for my experience so far.  I look forward to what is coming.

[bdye]

Thank you for sharing this and your presentation at EduCause!

[mskoch]

Having the flexibility to transition after implementation of important features such as the New RCE, migration of Question Bank to Item Bank (via Bulk Migration), timely quiz data reports, etc., will help make the process go smoother and the experience less stressful for our admins and faculty. Thank you for listening and willingness to make this adjustment. 

[IvoryUnderhill]

Thank you!

[crlinder]

I'm going to be blunt here.  Managing question banks is ridiculously convoluted and needs to be reworked from top to bottom.  Here's just one instance of awfulness.  I import material from a previous course and even the question banks from that earlier course are locked from adding new questions to them in the present course.  Instead I have to import those old question into my most recent version of the course.  

Here's another.  Instead of having the question bank pop up window be moveable outside the current Canvas window, it can only be moved within that Canvas window, blocking my view of the questions already in a quiz.  Plus the already created quiz questions are grayed out.  

I waste way too much time trying to get the system to work for me and my productivity is low because of it.  Surely others have these issues.

[philip_gregory]

compelling, not compulsory. What a great mantra!

So, really looking forward to the compelling part of the equation, So far all I can see is what we are losing with new Quizzes. I want to be more positive as I like what is there so far, we just need a lot more to bring it back to the level of functionality of the classics.

But best of all you are affirming your willingness to listen. Grateful for that indeed.

Imagine if we made the same commitment to our students every day. 

Philgg

[lizjf22]

I don't currently use Canvas but have heard that it's a wonderful platform.

[mathieu]

Canvas New Quizzes

• Seven years in the making
• Fortunes spent
• Multiple failed transitions
• Scores of dissatisfied users

Nailed it!

All jokes aside, we were able to fix Canvas New Quizzes by doing the following:

We used Moodle as an LTI provider

Here's how we did it:

LTI Tool Provider Setup

1. In your institutions Moodle** instance go to Administration > plugins > Enrolments > Publish as LTI tool > Tool Registration
2. Click Register a Platform
3. Enter the name of your Canvas instance or any name that makes sense to you
4. Keep the resulting page open since you will need the links provided in the next steps.

Setup Canvas as an LTI Consumer

1. In Canvas Go to Admin > Developer Keys
2. Create a new Developer Key > LTI Key
3. Use the "Manual Entry" method
4. Give your tool a Title, description, and key name that makes sense to you
5.  Copy the "deep linking URL" from the first step above as the "Target Link URI"
6. Copy "Initiate Login URL" from the LTI provider tool setup above.  It should look something like this: https://YourMoodleInstance/enrol/lti/login.php?id=LONG_Alphanumeric_key
7. Past the above URL from #6 into the "OpenID Connect Initiation URL"
8. Set your JWK Method to "Public JWK URL"
9. Set your Public JWK URL to https://YourMoodleInstance/enrol/lti/jwks.php
10. Add both the launch and deep linking URLs to your Redirect URIs
11. Toggle all options under LTI Advantage Services to on
12. In the "additional settings" section, add an icon URL (or leave it blank) and put "force_embed=1" into the custom fields parameter (sans quotes)
13. Set Privacy level to Public
14. In placements, set "link selection" and remove the other prepopulated options
15. In the link selection" section put https://YourMoodleSite/enrol/lti/launch_deeplink.php into the Target Link URI field.
16. Set SelectMessageType to "LTIDeepLinkingRequest"
17. Set an Icon URL or leave blank.
18. Save
19. Set your key state to ON
20. Copy the number in the details column just above the "show key" button

Add Moodle as an App in Canvas

1. Canvas Admin > Settings > Apps
2. Click View App Configurations
3. Click +App
4. Select "By ClientID"
5. Paste in the number you copied in step 20 above and click submit
6. Click Install
7. In the newly created app, click the cog > Deployment ID
8. Copy the Deployment ID

Add the Deployment to Moodle

1. In Moodle LTI Tool Registration > Edit your tool > Deployments
2. Add a Deployment
3. Give it a name
4. Paste your deployment id from # 8 above
5. Go Platform Details Tab
   1. "Canvas" as the platform name
   2. Enter your canvas instance URL as the PlatformID (issuer) (e.g. https://school.instructure.com)
   3. Copy the ClientID from the Developer key you created above (step 20)
   4. Paste into ClientID
   5. Enter https://YourCanvasURL/api/lti/authorize_redirect into Authentication request URL
   6. Enter https://YourCanvasURL/api/lti/security/jwks into Public Keyset URL
   7. Enter https:/YourCanvasURL/login/oauth2/token into Access token URL
   8. Save changes

Your Tool should now have an "active" badge listed below:

Now you can create quizzes in Moodle, with all the many quiz types and options you always wished Canvas had.  To add them to Canvas:

1. create a quiz in Moodle
2. Click "more" in the course menu > Published as LTI Tools
3. Click Add
4. Give it a name that makes sense to you
5. Select the quiz or activity you want to use (you can use any Moodle activity or the entire course)
6. You can leave most of the other options as they are by default

Now, in canvas:

1. add an assignment > Submission Type > select "External Tool"
2. Click Find
3. Select the Tool you created earlier (e.g. MoodleQuizzes or whatever name you gave it)
4. Select your quiz in the popup deep linking dialogue (pictured below)
5. Click "Add Content"
6. Click Select
7. Complete the setup as per normal

These quizzes load directly in Canvas. For a seamless experience, have your Moodle admin match the Canvas styling by adjusting the site CSS.  Finally! A Canvas Quiz tool that actually works.  Nearly all the user complaints in the community forums have been answered with this one tool.

Here are a few of the question types available:

• Everything Canvas already has plus,
• Concordance of judgment
• concordance of reasoning
• drag and drop onto text
• drag and drop onto image
• drag and drop markers
• Autograde Essay
• Autograde short answer
• Gapfill
• Ordering
• Pattern Match (like Regexp but much easier for non-programmer types)
• Random Select Answers
• Record Audio/Video (student can upload an audio or video response)
• Select Missing Words
• Set Splitting
• Short answer with similarity match
• Word select
• Many many more

In addition to question types, numerous question behaviours will allow you to use quizzes in a formative fashion:

• you can program your questions with unlimited incorrect answers with custom feedback for each response so students can get JiTT responses relevant to the mistake they might have made.  Students may then have an option of returning a different answer with or without penalties

** Your institution will need a moodle instance to serve as an LTI provider.  This can be done by Dynamic registration as well, making the process much easier, but that didn't work for me.  YMMV.  Try Dynamic registration first and if that doesn't work use the manual method documented above.

[HananKuzat]

Thank you for sharing. 

[Shiv-Shiv]

I am finding this platform poorly sign posted

[David_chem]

@shirenv @SuSorensen 

It says you are listening, but one of the louder complains is Partial credit WITHOUT penalty for multiple answers for over two years! Supposedly, it was considered higher priority in Roadmap 2021. But as of now, I can't find it anywhere on the Roadmap. It as if it is either not important or too difficult to implement. Neither seems reasonable, you talk here as if you want to help us implement the New Quizzes and start a whole webpage for New Quizzes. The difficulty is no better, as I have previously provided a formula for calculating the grade and so have others. Either option is better then nothing. Thank you for your consideration.

Edited: 11-15-21

[nwilson7]

@David_chem This is in no way meant to justify anything but curious if it was removed from the roadmap because Canvas is seeing it as done.  The link you included is partial WITHOUT penalty which is different than allowing partial points in general.

Here is an excerpt from the from the Q1 statement on New Quizzes where they released partial points for matching which seems to indicate that Canvas feels partial points for multiple answers is developed:

We released partial credit for matching questions to production environments in the beginning of the quarter. Matching questions include the option to select partial credit, similar to the Multiple Answer questions. Partial credit is the default option for this question type. Newly created matching questions default to the partial credit option. The blog post YESSS Partial Points on Matching Questions! provides additional information.

Again, not meant to justify just to hope help maybe explain why you can't find it anywhere anymore.

-Nick

[David_chem]

@nwilson7 

You are absolutely correct. I did mean partial points WITHOUT penalty. I edited my previous comment to be clearer. Thank you for pointing that out.

The problem is that partial credit with penalty for multiple answers has been in existence for two years. The blog post (link here as well) above was started AFTER the implementation. In that blog, I suggested a solution and was told that they were looking for suggestions and saw the topic as important. Most teachers want the option to adjust settings to suit their teaching style and preferences. Some may like the implementation now, and others (myself included) do not. I would just like to have the option to change to something else, and others use what they prefer.

To be honest, I don't understand the difficulty in implementation. Although I am not a programmer, it is just a different calculation from the current method. I really would like it added as an option.

[Nhadley12]

Thank you, an area of focus can and should be a smooth transition for Question Banks to Item Banks. We need a way to migrate those banks that is easy and straightforward, just as it is to migrate a quiz from Classic to New Quizzes. 

[DeanZepick]

Scared to use new quizzes. I have grade 12 exams with a lot of sources that did not show up for students. As I catch up and create for my grade 10 and 11 courses I am doubtful about proceeding with New Quizzes

[mhemenway2]

Thank you! We've had nothing but problems dealing with the migration to new quizzes.

• Recording option was not available when accommodations were needed for questions and answer options. This is a BIG DEAL when you teach primary elementary kids!
• The pictures didn't migrate - and then they did! This took hours of work to fix in all of my courses. What a mess!
• Surveys are not supported. I have several courses that use surveys.
• These problems are causing our school to consider switching to a different LMS.
• Our curriculum provider (Accelerate Education) has all quizzes in classic format.
• PLEASE do not eliminate classic quizzes until every single bug is worked out!

[sernekos]

I have no intention of switching to New Quizzes until there is an easy, seamless way to migrate Question Banks. I use Question Banks extensively in developing quizzes/exams and would have to migrate or rewrite dozens of them.

[mhemenway2]

I hope that the feature of inserting an audio or video recording can also be included in the new quiz format. I teach primary grades, and I sometimes use this feature for accessibility. While it is still possible to insert a recording using new quizzes, the process requires several steps and takes a great deal of time if you are creating a recording for every question and every response.

[audra_agnelly]

@mhemenway2 this feature is coming this summer. Canvas released it in December but had to turn it off due to a bug in course copying. the most recent update about this feature has been posted here: https://community.canvaslms.com/t5/The-Product-Blog/Re-enablement-of-the-Rich-Content-Editor-RCE-in-... 

[StephanieAshley]

I tried making the transition and it was a nightmare!  Besides how slow it is, all the weird glitches, and that you can't navigate between the different parts without backing out of the quiz, the worst is the fact that it doesn't supply the quiz analytics feedback.  When it works (which it hasn't for me), it takes over two days to get results.  That is simply too long to be useful for most teachers.  And the fact that it doesn't update with updates to scores makes it's functionality marginal even if it functions properly.  As far as I can tell from the discussion boards, instructors have been complaining about this for three years now and not only has it not gotten better, it's gotten worse.  Until that's fixed, there are no more second chances from me.

[hesspe]

The only thing you have wrong is that New Quizzes was announced in 2017, and there have been complaints about it for at least 5 years. What follows is my opinion:  Canvas made a huge mistake trying to implement their "new" quiz tool -  a function which should lie at the core of an LMS - as an add-on - an LTI.  I would guess that millions of dollars and hundreds of thousands of person-hours have gone in to trying to compensate for that initial bad decision.  They should have tossed this attempt in the dumpster long ago, taken out a clean sheet of paper, and started over, but they fell into the sunk cost fallacy hole and have been trapped there ever since.  There are some significant improvements in new quizzes that I wish had been implemented instead as improvements to the original quiz engine.   But for every advantage New Quizzes offers, there is a countervailing drawback.  In addition to that, New Quizzes presents a big learning challenge for anyone wanting to make the switch.  Although we turned on the ability to use it in our Canvas instance a few years back now, and our instructors are presented with the "choice box" every time they create a quiz, and we offer full support to those who chose to use it, few have given New Quizzes a serious trial, and, more concerningly, like, you most  who have,  have gone back to Classic.  I'm confident that under 5% and probably closer to 1% of our faculty members now use New Quizzes.  It's a sad tale really.

[l_lucas]

@hesspe - hit the nail on the head - 100%
We started panicking about the need to migrate by deadlines - but now that classic quizzes will remain available - 'new quizzes' is completing off the table for us - as it remains unusable..

[kettlebarm]

Maybe just junk New Quizzes and make Classic Quizzes better?  Know when to take the L...

Everyone at Canvas is opaque when it comes to Classic Banks to New Item Banks migration.  Are they really working on that??  Is that possible?  Is it going to happen??

I've never seen a definitive response on that...and it is THE absolute must have before most of us even THINK about switching over.

I teach at a school over soon to be 3000 students.  No one here is using New Quizzes for anyting other than a one-off activity.  All of our actual assessment runs through Classic.  We have may improvements we'd like to see in Canvas Quizzes, but Instructure is not making any improvements to Classic....which is what (I'm guessing) over 90% of their customers actually use.  Beyond frustrating....

[cdoherty]

@kettlebarm The Stanford CSM confirmed that there are no plans to handle quiz bank migration (unattached to quizzes). The only way to get Classic Quizzes question banks to migrate is if they are attached to a random draw question group in a quiz. Most of Stanford's question banks are not used in that way, so we will have to create quizzes with random draw question groups, attach the banks, then migrate to get them to the new format. I believe the reason is that Classic Quizzes question banks don't have an API built, so it would be difficult to do this and Instructure doesn't want to do this work. That's just one of the many ways Instructure has poorly built their migration tool to require their customers to do manual work to make existing content work in the new format. Examples: Text blocks migrate as stimulus questions and require manual editing to attach a question or they won't display to students. Practice quizzes will show up in the gradebook unless a particular setting is checked (something not currently done by migration). The migration tool is far from seamless and Instructure doesn't appear to care that the experience will be a nightmare for users and require them to do A LOT of manual editing to make their existing quizzes work in the new format.

[hesspe]

Recommendation to instructors: Using Question Banks?  Stay with Classic Quizzes -- unless you're feeling up to: 1. Migrating all of your quizzes 2. Rebuilding your Item banks, one question at a time.  

[james_whalley]

@cdoherty, I was just set to start testing the bulk migration feature in our dev instance (as it does not work in test and was also not working when I tried in beta). I am glad you put this up though, as I am rethinking even exploring this feature now. Without the ability to migrate question banks that are not connected to quizzes, this feature will be useless for a great number of users at Cornell. 

@shirenv, is there anything that can be done here? Not handling Quiz Bank migration means that you are really not handling Quiz migration. It's hard to believe that you are not planning to handle this. I for one will not be recommending we turn on the migration tool until this is resolved and will continue to tell current Quiz users that they should stick with Classic Quizzes unless feature disparity causes them to need New Quizzes.

[danaleeling]

The Stanford CSM confirmed that there are no plans to handle quiz bank migration (unattached to quizzes).

That is the most depressing news I have heard yet. I have stalled migration pending the ability to directly migrate Question Banks to Item Banks. That leaves me in the position of clinging on to Classic Quizzes until I am dragged, kicking and screaming, into New Quizzes against my will. Not all of the questions in my question banks are used every term, thus I always have unattached questions. And these questions are not part of random draw question groups.

[shirenv]

Hello all - Thank you for engaging in conversation in the Community with others using New Quizzes, and I especially appreciate the thoughtful inquiry. I noticed some recent activity on this blog, and I wanted to take the opportunity to share some clarifying information. 

@cdoherty, you and your CSM are correct. At the moment, the full migration of question banks, including banks not attached to a Classic Quiz, is not yet possible. We have prioritized other functionality that was more desired. However, we do expect to tackle it in the future. There is however a workaround solution available today if you need it. You can create a single ‘temporary’  quiz, include all the questions in your bank, and then migrate the quiz. All banks will then show up in new quizzes. @MPioRoda outlined this in detail in her recent blog, and @james_whalley, it’s likely you’ll also be interested in the information Marissa shared.

We’re aware of the limitation of text blocks migrating as a stimulus with questions attached. We have a solution already identified, and it is included as an item under the Later tab of the New Quizzes roadmap located in the New Quizzes Hub.  The roadmap is our way of being open about our development process. We’re looking forward to sharing more.

While not published yet on the roadmap, I’d like to share that we're also already looking into the option for Practice Quizzes to be migrated with the Zero Point Quiz settings enabled. That said, we need to do more discovery around this one as we want to ensure we don’t confuse users who may not automatically expect that behavior.

While we have over 6,800 institutions and 6.6 million instructors have moved to New Quizzes, we know what readiness looks like will differ from institution to institution. We recognize that there are features that need to be fully migrated, and that was one of the reasons why we removed the deadline to switch to New Quizzes. We want our customers to use New Quizzes because it is the best solution available for their assessment needs, and are best positioned to determine that. We continue to have our teams working their hardest to keep making New Quizzes better, and we hope you’ll continue to thoughtfully engage in conversations, leave feedback on Prioritized Themes, and follow along on the Public Roadmap.

[cdoherty]

@shirenv I am glad you are working to correct the issues with New Quizzes migration. Until that migration works seamlessly and doesn't require any amount of manual editing to make quizzes work in new format, we wouldn't consider New Quizzes ready to use in production. It would make for a very bad first experience for instructors to find out they had to spend time editing their existing quizzes to be able to use them in the New Quizzes tool.

I am aware of the workaround for migrating question banks, as I described exactly that in my post. It would be a lot of work for an institution to do if we are talking about year's worth of quiz question banks. It wouldn't be fair to ask instructors to do this work.

For practice quizzes, we were hoping that the 'Do not display in gradebook or the student's grade page' option could be selected when Classic practice quizzes are migrated, so that they won't appear in the gradebook. That would match the existing and expected behavior of practice quizzes.

[cdoherty]

I made a document of my findings testing New Quizzes Bulk migration, as well as the feedback I received from Instructure folks on their plans to address these issues.

New Quizzes Bulk Migration Issues Summary Oct 2023

[bdye]

I appreciate the work Shiren is doing on this for the community on New Quizzes. It is something that was inherited.

Cdoherty, Thanks for sharing the detailed summary.

After all of these years, New Quizzes feels clunky with little practical use for the average faculty member. Its really a hard sell to transition. It's simply slower and most faculty use very basic quizzes.

Honestly, it should just always remain an option and keep classic quizzes, which the majority of our faculty want to use.

[cdoherty]

I have conducted a more detailed review of the New Quizzes migration process, to include all settings and question type details in Classic Quizzes, and I have found some additional issues to share.

Updated New Quizzes Bulk Migration Issues Summary document
New Quizzes Bulk Migration - Detailed Testing Oct 2023, spreadsheet with detailed notes from testing

[ddemille]

Why do we have to eliminate classic quizzes at all?

I like classic quizzes better. I'd rather see a couple of features added to classic quizzes and call it good. 

[DeletedUser]

After unpleasant experiences with both Classic and New Quizzes, I have concluded that it would be a better use of my time to create a competing product than to continue to waste time using Canvas.

[Pinkivy94]

Classic Quizzes Intro Info for Class

1. I'm at the high school level and it's much more effective for me to be able to project the screen pictured in between classes or before I begin an assessment. My students can walk in the room and get started immediately, eliminating the need for me to spend 5 minutes explaining what's already posted. This feature does not exist in this form in New Quizzes and is one of the many reasons why I (along with most of my colleagues) will not use the NQ format. 
2. it's easier to access questions (tab) in the CQ format, however, when editing an assessment, it's absurd that you can't see the question number. 
3. Why is it so tedious to change the point value of the questions?? Should be able to do it without having to open each question. Ex. All questions listed, add box to the right that can be easily changed by clicking.
4. Make it possible to not have to assign values to each question -- if I have 13 questions and I don't assign a point value, then the program should just automatically distribute points equally.
5. allow for combining assessments without having to go to the test bank. If I give 2 quizzes in a unit and want to reuse those questions on the test, please make it possible to check a box and include those questions on another assessment. 
6. In Speedgrader -- if I am grading essay or short answer questions, it's easier for me to be able to see the question and answer while grading it. I know the rubric function exists but it is such a chore to create and I still end up having to toggle the screen. I admit that this was a feature of the LMS we used prior to switching to Canvas, but it was much more efficient. 
7. If I click the notification for a new submission on the dashboard, it should take me directly to that submission or all of the ungraded submissions, instead I have to search through all of my classes to determine who it is.