cancel
Showing results for 
Search instead for 
Did you mean: 

Priority Gathering for the New Quizzes Roadmap

Instructure
Instructure
8 32 1,664

Over the course of the last several weeks, I have been synthesizing data from several sources to understand the concerns impacting priorities for New Quizzes. After examining the suggestions posted in my previous blog post, reviewing prior roadmaps, and chatting with the global product team representatives, the customer success team, and various customers representing various segments, the themes I’m seeing are as follows:

 

Content is King

One of the learning opportunities which Covid brought to educators across the globe was creating content in an ecosystem. In a time when online learning was new (and intimidating!) to so many, the ability to share and collaborate on quiz content was a wonderful surprise. The reusability of the content provides a value add over traditional course creation—but there are currently limitations in New Quizzes preventing broad reusability. 

 

Accessibility & Accommodations are Critical

Accessibility is non-negotiable; we cannot leave any students without the opportunity to learn and demonstrate their understanding. Many customers use third party tools to assess if a specific quiz has been designed as a11y. Additionally, the audio tools of RCE and quiz printing are needed to support some students for accessibility/ accommodation.

 

Don’t make me move Quizzes with a Uhaul

Customers would like to just wake up and see all Classic Quizzes available as a New Quiz they can enable. Legitimate concerns relate back to content and that it would be catastrophic to lose content. Additionally, customers transitioning from Classic to New Quizzes require lead time of approximately 6 months to test any migrations and handle the change management of preparing educators.  Rollout needs to be coordinated to harmonize with their next round of course development while also not impacting current student assignments.

 

These break down into the following big buckets of items to prioritize:

  • Item Banks at Account Level
  • Item Bank Permissions
  • Quizzes Public APIs (Integrate with third party tools)
  • Integrate New RCE
  • Migration tooling (Quizzes and Quiz Banks)
  • Blueprint: Item Update/ Quiz Locking/ Item Banks
  • Print New Quizzes
  • Partial Credit: Question Type Expansion
  • Analysis CSV: Teacher/ Student
  • To Do: Teacher/Student

 

It can be challenging to meet the needs of such a diverse user set, but it’s actually one of the things that I love the most. 

How do we make sure that we can harmonize the needs of those who are using New Quizzes already, those who want to be using New Quizzes but can’t, and those who don’t particularly care where their content lives but want it all to be there—no matter what! Our costumers are also varied across North America, South America, Europe and the Asia/ Australia markets. Across these regions, customers are also segmented by the needs of K-12 or Higher Education. I’ve been gathering as much feedback as possible to make sure these needs are all considered. If we can’t do everything, this data will help make sure we’re doing the most important things.

 

If you feel that a critical barrier is preventing you from using New Quizzes and is not represented on this list, I’d like to know about it. Please add a comment below, unless you added on my previous TL;DR post already. (If you’ve left a comment on a prior post, please just point me in the right direction!) 

 

Someone mentioned that it would be nice to allow the community to provide priority feedback. I completely agree. It would be great to hear your thoughts. I still have to play mix master and figure out timing/capacity and associate that with the most knowledgeable people to perform a particular task. However, it is wonderful to be able to pair that with the community sentiment. Please take a moment to let me know which of the above priorities are most important to you by choosing a selection on this  Poll .

32 Comments
Explorer II

A critical barrier that is preventing us from using New Quizzes is that the outcomes data does not populate in the Learning Mastery Gradebook. We need the Learning Mastery Gradebook to sync with New Quizzes. Our hope is that at some point, all outcomes information from any source in Canvas will report to one place that is easily accessible for teachers and students.   Thank you!

Adventurer

@SuSorensen ,

This is fantastic!  I'd love for this to become a model for future development at Instructure (give customers a way to provide feedback on Instructure's ideas before putting in a lot of development time and finding out the things aren't actually what the community wanted.

I previously added some feedback to the other thread, so I know you've seen that.  The list you provided here seems like a good start.

-Chris

Surveyor II

Thanks so much for this! One absolutely critical barrier for my institution is the inability to export and import entire quizzes—currently, it can only be done by banking questions manually one by one and then "unloading" them manually one by one into the new course. We have dozens of instructors who keep hundreds of quizzes on shared, unpublished Canvas courses so that they can coordinate. They used to be able to move them from course to course with a few clicks; they can't anymore. It's a decrease in efficiency with no major advantages to make up for it.

Surveyor

I am unable to use New Quizzes because it does not support older QTI versions. I use TestGen (a Pearson test generator) which exports a QTI file that is not compatible with New Quizzes but IS compatible with Classic Quizzes. Bulk Import of QTI files would also be a helpful feature.

Explorer III

@SuSorensen While it may sound counterintuitive, something I'd like to see as part of the New Quizzes priorities is a clear commitment to maintaining (and maybe making some small improvements to) Classic Quizzes over the next 18 months.

Some institutions may have a few specific boxes to be checked before they can confidently move forward with their campus-wide migration. For others supporting diverse users and needs, there may never be a precise point when New Quizzes meets everyone's needs, so they may choose to keep Classic Quizzes around as long as possible, so as to give instructors as much time and flexibility as possible to plan and adapt. Given that I expect to find myself in the latter camp, seeing a small amount of resources dedicated to the maintenance and upkeep of Classic Quizzes will be critical to the migration.

We've been told for years that New Quizzes is just around the corner, and for years, requests for fixes and improvements to Classic Quizzes have been declined in order to focus on New Quizzes. As New Quizzes' development time stretched on, we've found ourselves at a place where dozens of small issues that might individually be ignored add up to a product with significant issues for many of our users, so successfully bridging them to New Quizzes begins with the false assumption that their needs are already being met.

In summary, I hope that Instructure can find some resources to dedicate to delivering impactful improvements to Classic Quizzes. This will build good faith, and help our instructors successfully make the transition to New Quizzes.

 

To more directly answer your question, my top 3 picks are:

1. Reporting parity with classic quizzes

2. Ability to regrade question bank questions

3. A complete re-working of availability windows/time extensions. The mental gymnastics required to set up a time constrained quiz are staggering (I spend so much time explaining how availability windows override time extensions via quiz moderation that I dream about it).

Navigator II

@SuSorensen  One of the biggest issues I see is with the integration to the gradebook, but I do not see where that fits in your list. I mentioned it in your previous TL:DR so I won't go into great detail about the problem. But to emphasize why it matters:

The existing system (manual grade posting) only identifies new quizzes that are pending manual grading if all students turn in their work before grading begins.  That just isn't what happens in normal semesters.  There are always students with excuses that warrant extensions.  So, grading is complete, and a student sees that a zero has been posted as their grade, and out comes a doctor's note.  I can reprogram the quiz to give them another day or two, but since a zero is already published there is no indication in the gradebook whether or not the quiz has been taken. 

New Quizzes set to do not count towards grade, show up in the gradebook, and are subject to the same late penalty as graded assignments.  That directly interferes with their use to measure student progress using the score at least requirement of modules - which is how we are using practice quizzes.  (It also lowers the perceived retake value for students studying, which is a second purpose for our practice quizzes.)  Showing up in the gradebook is fine (and showing in the student to-do list is essential) but there needs to be a way to exempt practice quizzes from the late policy if they are displayed in the gradebook.

Adventurer

Thank you for acknowledging that we need several months to transition to NQ after it is ready.  This will be a huge project for us, almost as big as when we migrated to Canvas, if we have to recreate every bank.

Without more details, I can't be sure what you mean by

  • Item Banks at Account Level
  • Item Bank Permissions

but a critical barrier for us is that the banks belong to people. Even with the ability to manually share a quiz with someone else, that will not work with our work model. We need the banks to belong to the course,  so that anyone with permission to edit the course can edit the banks. 

I'm going to guess this is what you meant by Item Bank Permissions, and I'll fill out the poll accordingly, but I wanted to mention it, just in case. 

Adventurer III

What seems like a quick win for little effort would be adjusting the Moderate screen to only show active students (or add a filter for active/inactive) and allow for sorting by name. I can't figure out what kind of order the names are in but it doesn't seem to be alphabetical. 

 

*EDIT* Deploy notes for Jan 06, 2021 show the moderate screen will no longer display inactive students! Whoo hoo!

Adventurer

Thank you for posting this! The Blueprint needs are huge for us, so thank you for that. I don't know about the expansion of test settings for displaying the answers. Classic Quizzes lets you show answers one time after each attempt, as well as giving time bounds for viewing quiz results. That is not there yet that I've seen. That is a very important priority for us but doesn't show up on the feature parity lists and I wasn't able to bring it up at the UEN Consortium call. Thank you for your transparency and for collecting feedback in this way!

Surveyor II

@epulham 's post makes a really good point.  A key feature we use is that we allow students to view their old quizzes after the class deadline has passed - a balance between test security and enabling students to learn.  Unfortunately, while it is possible to toggle this manually, it would be very helpful to have this be toggled automatically. (ideally, under settings, the Restrict student result view can have a "restrict until" date field that can be toggled).

Learner II

Thanks for posting this @SuSorensen!  I'll echo everything people above said, but was wondering if data from New Quizzes be included in Canvas Data?

Priority wise, I'd say the features and user experience are the most critical parts, but also having New Quizzes data in Canvas Data is important as well.  Focusing on the user experience / functionality parts first makes sense, I could see adding New Quizzes data to Canvas Data taking place after other features are in place. 

Lamplighter II

Thank you @SuSorensen 

My view of new quizzes is not good at this moment. I am seeing errors with new quizzes after a course import using copy a Canvas course option. If I had know about this was issue, I would never have encouraged faculty to use new quizzes. This needs to fixed ASAP ( I already have a ticket on this). You cannot expect faculty recreate quizzes they have already created. This is going to extremely irritating issue for me as our classes start on Monday. 😬

Other issues I have with new quizzes. 

QTI imports - Still can't import content textbook publishers directly to new quizzes. And yes I know I can run them through the current quiz tool but I should be able to import directly into the new quiz tool. 

Editing Quizzes - This is confusing for faculty. There are two sets of settings in different locations. I have to carefully explain the difference between assignment settings (due dates and points) and quiz settings. I would prefer that new quizzes worked like other LIT tools so that the assignment settings button displays while editing the quiz. 

Integration in a Canvas Course - There are several issues here with quizzes not displaying in instructors' To Do List, module flow for students, adding quizzes to modules (listed as assignments). Also if I duplicate a quiz, the title is not updated in the quiz settings. If I update the title in Canvas, it doesn't update the title in the quiz and vice versa. 

CSV report - Instructors need to be able to download the data as CSV. 

Quiz settings with release option - The options are poor in classic quizzes and are no better in new quizzes. Instructors want the ability to  set a time frame when results are available for students to view. This means the results only displays to students during the time frame selected. Instructors just don't time to remember to change these settings manually. 

Integration with the Canvas Commons - Currently there is no portability of quizzes created in the new quiz tool. Instructor don't have time to recreate quizzes. 

Rich content editor - Most faculty want to have access to content in files and some also want students to be able to record video content using our Studio integration. 

Better control of item banks - I am happy that faculty can share item banks but there needs to be more options. I worry that giving read/write access can be an issue so more granular permissions would be nice. It would nice to have an option to allow people to copy the question bank. As admin I can't see the questions banks unless I masquerade as the person. 

Select All - Please allow questions to selected using a select all option in quizzes and item banks. This is one of the most annoying things about Canvas in general

I understand that COVID-19 has delayed the timeline but new quizzes has been in the works for a very long time and I am exhausted trying to keep updated on the changes. It makes it very hard to prepare faculty for the changes. Please be more transparent with development and make sure our CSM knows what is going on as well!

Surveyor II

The CSV export mentioned by @snugent is critical in my opinion.  Frankly, New Quizzes has promise particularly with creative question types, but I think the integration with the rest of Canvas is sorely lacking.

Surveyor

Is there a way to get notified on the Dashboard when a student completes a New Quiz? (Especially one that still needs graded.)

Is there a similar way to utilize "Speed Grader" in the New Quiz? Currently it takes exponentially longer to grade a New Quiz than a class full of Classic Quizzes. 

In Classic I would use speed grader from the Dashboard notification to work through each student. Now with the New I go to the quiz, select moderate, grade one student, back out of the quiz after updating the grade, select moderate again, then grade the next student. 

Also our school does not have the grades synced to PowerSchool so when a student does complete a New Quiz, I do not get notified and I may record their grade as a "0" missing in PowerSchool and not know when to go back into the quiz when they have submitted the New Quiz, when the Classic Quiz notified me on the Dashboard. 

Navigator II

The course copy errors mentioned by @snugent for new quizzes are a huge issue for anyone creating a master course to be used by other instructors.  Even if the retry button works, the retried import is not necessarily linked to its module, and all new quizzes are imported as unpublished, and even though lockdown browser will still be mentioned in the quiz title the prior settings do not copy with the course the way they did with classic quizzes.  Now instead of clicking the lockdown browser once to reactivate the prior settings, we need to manually turn on lockdown browser for every quiz, and then open advanced settings and turn on chromebook and ipad access.  Individually for every single quiz.  Communicating set-up instructions to new faculty needs to be as simple as possible.

Surveyor

I would love to be able to switch between question types after I have started the question. This is one feature of old quizzes I love. Sometimes the question type doesn't work as I thought it would.

I teach math and use the dropdown option INSIDE a table often for quizzes. I would love to have this feature in new quizzes.

Learner II

A top priority - in association with the QTI - is to be able to IMPORT questions / question banks from OTHER sources.   We use Respondus quite a bit for this and it would be great if that continued, but also was improved.   Why cannot New Quizzes have ADDITIONAL options to import questions?    Many faculty may like to create a quiz in Word first and then import all the questions or some of those questions - so they can keep a separate record of their possible content if they choose to.  It would be nice if there were easier ways to import questions - from a variety of sources.  

Surveyor

I would like to see the option for students to be able to annotate on the text we include as an image in assignments and quizzes.  We teach how important it is to underline, circle, write connections, etc.  The students would greatly benefit from being able to do this right there on the text image in order to prove their answers by citing text evidence.

Surveyor

It would be extremely helpful to be able to organize the students in the various columns while in the Moderate screen. For example, click on the Student Name header and it alphabetize the student list. 

Surveyor

I would like for the students to be able annotate on their computers. I like how the "new" quizzes have the option to have the text on one side, but it would be beneficial if the students had a tool to highlight, circle, or underline the answers from the text. They have learned valuable close reading skills and they should be able to use them when taking tests or completing assignments. Thank you!

Surveyor II

Adding my voice to the chorus of those who need to see REPORTING called out as a priority here. Our organization has piloted New Quizzes in one course for a high-profile client of ours and reporting has been a disaster. As I understand it, Instructure rolled this out without proper API. As a result, our Instructure-built custom report, specifically designed a year ago to chart course progress by quiz completion, DOES NOT SHOW ANY QUIZ RESULTS and we have had to routinely dedicate extra staff time to integrate data from other sources for a rolling enrollment of now close to 500 learners. We have seen no progress from Instructure to correct this, or the functionality of our expensive report which we purchased as custom dev to specifically automate this function. 

Learner II

Definitely need to be able to export data in a CSV like you can in Classic.  

First thing I need though is those APIs to release.   We have 3rd party tools we rely on that we cannot use until they have access to the API to develop their tool.  We need those products, like Proctorio, before we can go completely over to New Quizzes.  They need time to develop.

Surveyor

I would like to see the students names in alphabetical order when moderating a new quiz. I would be a lot easier to cross reference who has completed an assignment.

I would like an option in New Quizzes for the students to make the text on side of stimulus full screen so it is easier to read. Then they can minimize to go back to the questions. Annotation options would be plus as well. 

Surveyor II

Wow, thank you so much to everyone for your feedback. I clicked the option to enable the New Quizzes on my course because I'd read they would be enforced in the summer and figured I'd get a jumpstart on converting (though it appears that has been postponed to next summer). However, with all the shocking issues with them seen in the comments, I will absolutely not be enabling them.

I played around with them a little to see if they were really that bad, particularly the "Modules" integration issue as I will be relying heavily on Modules organization to orient students in online learning. The fact that it essentially takes students to some nigh-unrecognizable version of Canvas, with the course sidebar gone and the previous/next arrows missing, and that it doesn't display number of questions/points on the info header are major showstoppers for me. There are probably more (certainly the many issues presented in this topic and Su's intro post), but these were enough to get me to quit. I'm glad the enforcement date has been postponed a year!

Surveyor

App integration is critical for our organization. We use MathType, a math equation editor, which also works nicely for accessibility concerns. New Quizzes currently only support Mathquill, which is lacking in terms of accessibility (i.e. screen readers). A simple expression like 7 reads "backslash backslash s q r t left brace seven right brace backslash backslash", when it should be "the square root of seven".

Explorer III

Hi Susan! Thanks for pointing me here! The following seems like core features that are critical in a modern LMS and for alignment with other features in Canvas:

  • Item Bank Permissions
  • Quizzes Public APIs (Integrate with third party tools)
  • Integrate New RCE
  • Print New Quizzes
  • Partial Credit: Question Type Expansion
  • Analysis CSV: Teacher/ Student
  • To Do: Teacher/Student

The following are important but less critical

  • Blueprint (it's awkward when some parts of a course are covered and some are not)
  • Migration tooling (Quizzes and Quiz Banks)

The following are nice to have:

  • Item Banks at Account Level

UBC would also like to see improved reliability and integration with the core components of Canvas.

Thanks!
Tim

Navigator II

There are several comments on this original post highlighting a lot of the main issues with New Quizzes, and I wanted to expand on a couple here. (These are in no particular order.)

Building a core function as an LTI is a bad experience. If quizzes are first-class tools within Canvas, then it should be part of the core tooling. Having New Quizzes run as an LTI introduces a host of issues (the teacher to-do list doesn't show new submissions, scores show in the gradebook even if the quiz has unscored questions, getting to the quiz to score in the SpeedGrader is a steeplechase) that make it really unpleasant to use.

Building a core function as an LTI puts us on edge as customers. The main benefit of LTI tools is that they're pluggable. When Instructure started building Quizzes and the new Assignments tools as LTIs, to us it's signalling a revenue source. Will we be hit with upsells to continue to use Quizzes? It's not out of the question because by building these as LTI tools, you can granularly turn them on and off for those who can afford to keep them over those who cannot. It's bad optics and makes us talk more seriously about whether or not we can continue to invest in a platform.

Question banks have been gutted. One of the most powerful tools in Canvas is having course-level question banks that can be used to randomly generate quizzes for students using Question Groups. I know we can pick questions in new quizzes, but by having banks pulled out from the old quiz engine, I've now lost that main feature. Allowing for reassessment through the banks gave me control over which standards to assess and when. I knew that each time a quiz was loaded, whether it was the first or ninth, it was aligned to what I wanted to assess. With new quizzes, that is much harder to do. Which leads to...

Not sending Outcome scores back to the Learning Mastery Gradebook is devastating. Being able to get granular reports on student progress at the standard level is immensely powerful. Since new quizzes doesn't talk to the LMG, it's a dealbreaker. I can do one-off assessments, but not having that body of data with the rest of the student information fragments the system so much that I recommend my staff do not take the time to invest in new assessments. In the end, this is the straw that will break the back of quizzing in general through Canvas.

I'm hoping that since the New Quizzes project was started with Instructure was fragmented and all over the place that some of these issues are symptoms of the larger organizational issues. On the other hand, the Thoma Bravo acquisition is concerning, considering the history of private capital organizations picking up investments only to dismantle and sell off assets for profit. 

Thanks for leaving this open for comments. I'm hoping to see some of these tackled head on over the next year.

Surveyor

On the classic quiz you just put in a new editor as of Jan 2021 which superior to the editor you have in new quizzes. Will this new editor be added to the new quizzes?

Surveyor II

As an admin at my school and being part of a distance education schools network, the biggest issue we have with new quizzes is the lack of ability to export from one Canvas instance to another. We have networks of teachers from different schools that share content and course writing responsibilities which has been a huge positive of all schools using Canvas as their LMS. Currently if one course needs to be exported from one instance to another teachers need to create each new quiz in their own instance. For some courses this results in days of work if there are a large number of quizzes.

As a positive our students all love the look of new quizzes and are happy with the product as a student. Incorporating the RCE, ability to print and partial credit are features I hope start to be enabled as these will make my teachers more happy to use new quizzes.

Surveyor II

In addition to my above comment I'd like to add that the way the New Quizzes "fill in the blanks" question type is programmed, with no rich content editor and the "return" key being reserved for creating blanks (thus eliminating the ability to create line breaks), has caused my institution many, many problems.

I recently migrated almost a hundred quizzes, and all of the fill-in-multiple-blanks and multiple-dropdowns from Classic Quizzes are now run together in a way that is confusing to students, looks unprofessional, and makes certain basic language test question types impossible (dialogues, conjugation tables, "put the underlined word in past tense," etc.). They are unusable in this format.

I'm now spending hours (I anticipate it taking at least 20 hours, if not more, and I'm in the very rare position of it being my job to do things like this) reformatting all the "fill in the blanks" questions from scratch, because it is in fact faster to rebuild them from scratch than to make adjustments from what's already there, and I'm also spending my time finding creative solutions to my new inability to format dialogues and tables. At this point, New Quizzes have taken more time and capabilities away from us than they've added, and I hope that these issues can be addressed soon.

Instructure
Instructure

I am watching your comments and tracking these requests. When I started investigating New Quizzes, one of the first things I heard a lot of concern with are issues raised here by @snugent  and @and-hu  related to the experience of adding and editing New Quizzes. Using the blogs area, the Quizzes team was able to solicit and meet with customers to get feedback on workflow designs. I hope you will all consider supporting us in this way in the future as well. Changes will be coming and I’ll post an update in the coming weeks. 

Some comments also reference buggy behavior with Copy and Grade pass back. These are undoubtedly significant to your users and the engineers are actively researching and resolving issues, so you should see improvement shortly. 

On the reporting front, Instructure as a whole is working on data, reporting and analytics. @dave-long  Canvas Data2 is coming along. There is a User Group here: https://community.canvaslms.com/t5/New-Analytics-Users/gh-p/analytics

Additionally, publicly available and documented APIs will help meet these needs. As @froelich notes, this was helpful in the Classic Quizzes API because you can manipulate the data for reporting. 

@kentmiller   Precisely, the New RCE editor added in Canvas this month. I am interested in hearing from customers how important this is to be done soon versus some of the other critical priorities identified. 

Thank you all for your participation and providing feedback. I understand you have been stretched beyond patience with regard to some of these issues, which makes it even more critical for me to make sure we’re delivering the most valuable improvements first.

I view this as a collaborative effort and we cannot be mutually pleased with the outcome without working together. This specific post will be closed before March. However, to continue our open conversation, I’ll start a new post to let you know which of these priorities discussed will be worked on first and ask for feedback on what the Quizzes team believes should come next. I’ll also be requesting help in discovery phases and again with design feedback to make sure we provide true value to you and your users.

Surveyor

Personally, my #1 frustration at the moment is the inability of students to type "math" in their responses.  As the teacher I have access to the rich content editor and can type the question however I like using the equation editor.  But if I want a student to do the same thing in a fill-in-the-blank or numeric answer I am out of luck.  It's so frustrating to want to ask a NUMERIC answer question and not have mixed number or exponents as possibilities.  I understand that I can ask it as short answer then they have the rich content editor available to them, but that increases the grading over head.  

 

It would also be fantastic to have more control over blank placement in a fill-in-the-blank question like you do in Old Quizzes.  In New Quizzes all you can do is in line blank placement with no rich content editing.  (I can have rich content in the question stem, but not the actual fill-in-the-blank structure).  I would like to be able to have a table and have students fill in blanks in a table.  Not possible in New Quizzes.  (I can build the table in the question stem, but not put editable answer blanks).  I would love to find out I'm wrong on this issue, but if there is a way I don't know it.

One more, make numeric answer an option for fill-in-the-blank.  I can use numbers, but only as text.  Having a numeric option means I don't have to worry about all the different ways a student types a number (0.5 vs .5 vs 0.50, etc...).