Grade By Question in SpeedGrader for New Quizzes

[MPioRoda]

Grade By Question in SpeedGrader has been enhanced with more efficient navigation and access to action buttons. We have also added accessibility features to ensure a better experience, particularly for keyboard only and screen reader users.

We added several features that keep the experience consistent with other areas within Canvas such as the Question Navigator Bar which is used when building a new quiz, the use of indicators next to the question numbers on the navigator bar to indicate which ones need manual review, and the thick, blue focus bar on the left of the question to indicate which question is focused on for Grade By Question. The question navigator bar will appear in SpeedGrader whether or not a user chooses the Grade by Question option.

Getting started

To turn on Grade By Question mode, users will click on the gear in the top left corner of the screen and choose Options from the dropdown. A pop-up for SpeedGrader Options will appear so the user can check off Grade By Question and then Save Settings.

SpeedGrader Options Window

Grade By Question Banner

When a user turns on the option for Grade by Question, a banner will appear at the top of the screen to remind them that they are in Grade By Question mode.  This banner can be dismissed using the x on the side of the banner.

Grade By Question Banner

Navigation to questions that need review

There are three ways to navigate to the desired question.

• Summary box - In the top section of the screen where the results are displayed, a summary box appears indicating questions that need review. The links in this summary box will take the user directly to the question and will disappear once those questions have been updated.
• Question Navigator bar -  With Grade By Question on, a half-moon icon is displayed to the left of the number that requires manual grading. The user can click on any of the numbers in the Navigator bar to get to their desired question.
• Scroll and Click - A user can scroll to the desired question and click anywhere within the question box to focus on that question. 

New features on questions

New Grade By Question Buttons on Questions

Focus on the question is indicated by a thick, blue bar on the left of the question.  This focused question will now have three new buttons which were added to reduce scrolling to different areas of the page to move between students or update the grades.  

The Previous Student and Next Student buttons now within the question make it easier to go on to the next student or check back on grades given to previous students. Our earlier improvements on the back-end related to a single session make navigating between students much more efficient.

The Update button is also now within the question so that users do not have to scroll down to the bottom of the screen to update the grade. Reminder that once you click on Update, it will update the score for the entire quiz

When an instructor adds a grade, either through the check mark, x, or by point, they would then click on Update to update the quiz score in the right side panel. After that, they could click on Next or Previous Student. Until the page is refreshed, the scores will be retained so that an instructor could go back and review, but they would still need to click on update in order to save and update the overall quiz score for each student.

Question Not Available for All Students

When quiz questions are pulled from an item bank and a set number of questions is randomly selected, not every student will have the question that an instructor is focusing on.  In those cases, a warning will pop up on the screen with an option to move to the next student that does have that specific question.

Question Not Available Pop-up

Quiz set to Shuffle Questions

If a quiz setting is set to Shuffle Questions, the quiz questions will appear in a different order for each student. While in Grade By Question, the specific question will stay in focus as a user navigates between students regardless of which order the question appears within a particular student’s quiz. 

Accessibility

We’ve paid special attention to accessibility when adding this feature, such as adding visual indicators like the half moon icons on the navigator bar or the focus bar along the left side of the chosen question.  We’ve also added a hidden button to jump to question Navigator Bar within the question box to make it easier for keyboard only users to navigate back more easily.  Lastly, we’ve made sure that when a specific question is chosen, that we send focus to the entire question for screenreader users to be able to begin reading the contents of the entire box.

Release Notes for Grade By Question:

https://community.canvaslms.com/t5/Canvas-Releases/Canvas-Release-Notes-2023-04-15/ta-p/560530#toc-h... 

Screencast:

https://community.canvaslms.com/t5/Canvas-New-Feature-Screencast/Publish-Release-Screencast-2023-04-...

[Charles_Barbour]

I suspect I'm confused but when grading, why not always navigate to the next student who received the question you're grading?

It sounds like the process will take the grader to the next student (even though they may not have received the question), telling them "Question Not Available", and making them click Next Student?

Wouldn't defaulting to always moving to the next student who received the question be more efficient?

This seems like an unnecessary step so I'm assuming I'm not understanding the workflow correctly or there's some technical limitation of which I'm unaware.

[rmartini]

I've been testing this a bit over the last day or so and this is a welcome improvement for New Quizzes. However, I do agree with @Charles_Barbour that it would be much more efficient to just go to the next student who actually received the question rather than making the instructor/grader select "Next Student" every time. My expectation is that you'd look at a question, grade it, then move to the next person who got the question so you don't have to cycle through the entire gradebook.

Here are a couple other things I’ve noted:

• This is partially user error, but it took me a moment to realize that even though I could see the “Grade by question” banner at the top, I needed to select one of the questions for the Previous Student / Next Student buttons to actually appear. I initially thought this wasn't working because I couldn't see the new buttons.
• If the next student has not submitted an attempt, the instructor/grader sort of exits the “Next Student” button workflow and you have to go to the top right to arrow to the next student. This can get a little confusing.
  • In my example from this screen recording, I used a quiz with 2 questions (both pulled from item banks) and you’ll see this occur around the 0:35 mark: Testing Grade By Question
• The gray half moon icon in the navigator bar wasn’t as obvious to me and doesn’t draw my attention as much as the orange icons in the name drop down in SpeedGrader (see screenshot: New Quiz Grade By Question - Half Moon). It would be helpful to make the half moons bigger or have a different color to draw more attention to it and quickly scan if more questions need to be manually graded. 

I’m not sure if there are plans to work on this, but it would be really helpful to see a list of the questions that need grading (across all submissions), especially when item banks are used. We have some large courses where TAs are assigned to grade specific questions (for example, one TA will grade all of Version 1 of an essay question; another TA grades Version 2 of the essay question, etc.). There isn't an efficient way to do this when grading Classic Quizzes when the questions are randomized, so I'd consider this a slight advantage for New Quizzes; but even with this update, TAs still have to cycle through the entire gradebook with extra clicks.

[gmorgan62]

Has anybody not seen the option to turn on "Grade By Question" for New Quizzes? I have it on for old quizzes but don't see it as an option on my new quiz.

[rmartini]

@gmorgan62 : I tested the Grade by Question feature for New Quizzes in our beta instance. According to the 4/15 release notes, it looks like it isn't in production yet (but will be on the 15th).

[cdoherty]

When instructors/teaching teams ask for the ability to grade by question, they want to see ALL responses in ONE page. Having to click through pages to see one response at a time does not meet their needs. This new feature is just as useless as it was in Classic Quizzes. Why is Instructure insistent on keeping this design the same when it was such a clunky workflow in the first place? Did anyone actually consult teachers about this design?

[audra_agnelly]

Is something happening with New Quizzes in Beta? I have been trying to test the grade by question feature for the past several days I am experiencing a combination of different issues after I click Build.

The past few days the build screen was taking a long time to load and eventually timing out (504 error). Beta can be slow sometimes so I waited to try later in the day/next day and still ran into the issue. This morning I opened an existing quiz and saw that alert message "Sorry, the request was unauthorized." and the page is just stuck on the Build screen with the loading message and progress circle; the questions/editor never loads.

I also tried to create a brand new quiz in Beta. After clicking Build I get the "I'm sorry something went wrong. If you are the application owner check the logs." message.

Have others been able to get to the New Quizzes build screen in Beta this week?

[slwilso3]

My instructors have had the same request, @cdoherty   And it should be a file we can download.

[audra_agnelly]

I was finally able to get into beta and have 2 pieces of feedback on the workflow:

I found it to be a bit of a nuisance that the next/previous student buttons will advance to a non-submission (blank screen) where those buttons are not available. I fumbled a second before realizing I needed to click the arrows in the top right of Speedgrader. It's a disjointed workflow to have to switch where you are clicking to advance to the next quiz. If the teacher sorts by submissions you don't run into this, but not everyone is going to have that setting selected. It would be a better flow if the previous/next buttons in the grade by question UI skipped to the next submitted quiz.

I am also concerned that teachers are going to grade responses and neglect to click the update button. In the right panel in speedGrader you do not need to select Update when you enter an overall score, it is autosaved when you advance to the next student, so requiring Update in one area and not another is confusing. You do need to Update when you enter a SpeedGrader comment and get a warning message when you don't do that. I think we'll run into far fewer distraught instructors that none of their grades were saved  if there is either a similar warning to update or an auto save function when the teacher advances to the next student.

[MPioRoda]

Thank you for testing and providing helpful feedback on Grade By Question.  In developing this feature we focused on being able to bring the functionality of Grade by Question with some enhancements to ease the flow and accessibility. 

• Navigate to Next Student who received the question

• The warning and workflow in question was created to address questions that were randomized and do not appear for every student. When a user clicks on the next student button or on the arrow at the top right of the SpeedGrader screen, it sends a request for the next student. The functionality to navigate between students is separate from the ability to identify whether a particular question appears in a given quiz, so advancing to the next student does not query if that question appears for that student. The displayed warning serves to bridge that gap so that teachers are aware of which students have the specific question or not. 
• In the Grade By Question (beta) version for Classic Quizzes, the feature did not work for questions that were shuffle(randomized) in a group, it displayed a message at the top of the screen "The Grade By Question feature is affected by questions that are shuffled in a group". It would show the chosen question for each student, regardless of if it was the same question. For example, if a teacher is working on Question 2, it shows question 2 on each student even if the actual question was different (chosen randomly from a group of questions or shuffled).
• The new functionality improves how SpeedGrader reacts to questions that are randomized or shuffled.  
• For a quiz that has been set to Shuffle Questions, the new functionality that we added will allow the focus to follow the question so that the specific question which the teacher would like to review will always be shown.  
• For a quiz that contains a question randomly pulled from a bank, the added functionality displays the warning that the next student does not have that question which enables the grader to understand it was a randomized question. 

• Students that have not submitted an attempt
• When grading by question and a student has not submitted an attempt, Speedgrader shows a message that “This student does not have a submission for this assignment.”  This behavior is as expected within SpeedGrader for whenever a teacher navigates to a student without a submission. If a quiz attempt has not been submitted, there is no information for SpeedGrader to show for the quiz. We would not add the Grade By Question buttons to this page because this text appears in a number of situations other than just our case. 

In the future, we could look at options to get around this technical limitation in order to skip students that do not have the question or to skip students without a submission, both of which would require a more complex technical plan. Our current goal was to get this functionality into teachers’ hands right away with the current workflow and with some well needed and suggested additions.

Regarding the Update button, we do retain grades while actively grading by question as long as the page is not refreshed. From a teacher perspective, we have heard that they like the opportunity to not have the score update for students until they are done grading all students on a given manually-scored question so that they can ensure they were consistent in their grading across all students.  We did not change any workflow here, we just made it possible to Update from within the question rather than for a teacher to have to scroll to the bottom of the page to click on Update to change the student’s quiz score. 

[Charles_Barbour]

I suspect I'm confused again.

• When grading by question, clicking Next Student does not take you to the next student who recieved the question, but rather just the next Student.
• This is because clicking Next Student, "... sends a request for the next student. The functionality to navigate between students is separate from the ability to identify whether a particular question appears in a given quiz, so advancing to the next student does not query if that question appears for that student."
• When the next student retrieved by the query did not receive the question being graded, you are able to detect this, show a dialog box, then present the grader a button which WILL take them to the next student who did receive the question.

If you can detect the next student didn't get the question so you can show the dialog box with a button linked to the more appropriate query, can't you just run the more appropriate query for the grader? It seems like this could be done with something like a greasemonkey script or Sikuli.

"If you see a box with this text, automatically click this button."

Alternately, couldn't the Next Student query be updated so if a grader is has selected the Grade by Question option it operates in the expected/desired manner?

I also find the Update functionality confusing. It sounds like it would be possible to grade all the students by clicking Next Question, never click Update, then lose the scores? Does Update need to be clicked for every student? Because that could be an immense amount of unnecessary clicking. At the very least it seems like there should be a button which offers to simultaneously Update and navigate to the Next Student.

I'll also echo the comments made by @cdoherty .

When grading by question, I can obviously see the need to display individual responses. For essays, complex answers, or when you want to provide detailed feedback, showing responses individually  makes a lot of sense.

But for some question types and responses, it would be much more efficient to display all the responses on a single page as opposed to individually. (The page would obviously contain only students who had received the question you're grading.) Graders could simply grade each response as they scroll down the page. No need to click a button after grading every student and waiting for the next student's response to be retrieved. This would also make it easier to compare student answers since you could scroll up and down to view them (as opposed to having to click Next/Previous Student multiple times).

When all responses have been graded you could click the Update button once and move on to the next question. (I would imagine a background save process could be enabled to prevent the loss of data if the browser crashes.)

That sort of workflow seems much more efficient than clicking Next and/or Update for every single student.

[cvalle]

I’m curious for the anonymous mode are the student identifiers static? For example is “Student 3” always the same student throughout the semester or is this randomized every time we launch Speedgrader? 

[cvalle]

I understand the need to warn teachers when the next student does not have the same question, but I use large Item Banks for each question so for me this will almost always be the case. It would be great if I could turn off this warning as it adds an extra click to every question I grade.

[cvalle]

Also let me say thank you! This has greatly improved my workflow in Speedgrader!

[wca_tim]

I find it very frustrating that just clicking on "next student" after grading a question does not save the current students grade on that question when I move on.  you must click on update each time if you want the work to be saved.  

The extra step is wasted time and energy, and it is very easy to miss hitting "update" and wind up having to go back through all of the exams from top to bottom again to be sure every question was graded.  This limitation alone makes it far less useful than if it would update each score automatically when the "next student" button is pushed.  (I've forgotten a few times since starting to use this feature and had to regrade a number of exams in order to find the ones that weren't saved...)

Surely this an option that can be added in the future?