Canvas Feedback: Canvas UX risks result in late submission

0 Likes
(1)
Hello there IT Team! Below is friendly feedback hoping to be taken into consideration for possible Canvas dev work.
 
Context: I had completed an "unlimited attempt" assignment & received 100% for it on the third attempt. However, I accidentally began another attempt and I was unaware of this, and Canvas does not tell the student an attempt is underway (it only records the previous attempt). Per "Grades" on the left nav, Canvas was showing me the Attempt 3 score as the recorded score (50/50), which is misleading if that is not going to be the actual recorded score at the end of the due date. Knowing something would result in 0/50 at the cut-off date for the assignment could lead to correcting the error without escalating to the professor and manual correcting the issue.
 
Per the attached, Attempt 4 was ongoing in the background for 11,830 minutes, about 8 days. Canvas does not tell the student this is ongoing.
 
Friendly potential options/suggestions:
  1. On the "Graded" list, add Status column to the user interface Recorded/Not Started/Attempt In Progress
  2. Not show the graded score while another attempt is in-progress - this is very misleading when the user does not know another attempt had been selected.
  3. Set limits for attempts/prevent "unlimited" time for an attempt. Cut an attempt off after 18 hours (for example).
  4. Send frequent "CAUTION" automated email reminders that an attempt is underway
  5. End all ongoing attempts after logging out of EWU
Thank you for kindly listening 🙂
4 Comments
KristinL
Community Team
Community Team
Status changed to: Open
 
James
Community Champion

@CBell19 

You bring up some interesting aspects of the quizzing. Are you using classic quizzes or new quizzes? Canvas has stopped developing classic quizzes, but lots of institutions will keep using them until they force us to stop. It sounds like you might be using classic quizzes and the instructor has set it to keep the latest score rather than the highest or the average.

If you are taking the quiz in a browser (I cannot recommend taking quizzes with the Canvas Student App), then when you go to retake the quiz, it will tell you what score will be kept (it is in the details of the last attempt). The instructor should clearly communicate to the student how the grading is done (I do that, but just because I tell people that in the directions doesn't mean they read it). Just because Canvas shows you what will happen doesn't mean students will find it.

That's where some of your suggestions come in. Is there something that Canvas could do better?

A student should not start an attempt if they are not going to finish it if the average or latest score is kept. While some of your suggestions are aimed at improving Canvas, some of them are designed to remove that responsibility from the student, when that is where it lies. I've tried forcing students to be more organized, it really doesn't work.

Let me address each of the five points you made. My thoughts range from interesting to downright dangerous.

  1. Potentially this might have merit. I'm not sure where it would go on the grades page, though. I don't like the idea of an extra column there. It's hard enough for students to find things as it is. Perhaps retaking a quiz should add something back to the calendar or the To Do list. That would seem to reach a bigger audience than something on the grade page. However, I'm not sold on that idea. One time is all that it has to be taken -- if you take it again, it's up to you to finish it.
  2. You definitely should show the current grade. Students need to know what they have already, not what they will have at some point in the future. However, this could be a place where an asterisk or other note goes that there is a pending attempt that may affect the grade. But if you put it on the grades page, then it should also go on the assignments page, the To Do list, the Syllabus page, and anywhere else that a student might find it.
  3. Canvas allows the instructor to set the time limit. If the instructor wanted it to cut off after 18 hours, they could set it that way. On the other hand, I want my students to have unlimited time and encourage them to start the quiz, walk away if necessary, perhaps print it and work it out on paper, just remember to come back and complete it.
  4. One or two reminders is plenty sufficient. Frequent reminders would lose their effectiveness. Still, though, it is the student's responsibility to know that they have started an exam. I would rather see reminders for assignments that are nearing their due (or close) date without having any submission more than students who open the quiz and then decide not to finish it. It really only affects those who keep the latest or average.
  5. This is a dangerous idea. Canvas serves way more than just EWU. Setting that aside, that would break the time limit that the instructor has set for the quiz. A common scenario is that a student starts a quiz and gets interrupted, so they quit and come back to finish it. If the quizzes are autosubmitted as soon as they log out, you're going to get lots of half-finished quizzes with poor scores. That hurts everyone and benefits no one. Even if you keep the highest grade so it doesn't directly replace the current score, that means that a student has to come back and start over. For those quizzes that only allow one attempt, this is devastating.

I've used average scores on my quizzes for years. At the end of every semester, I go through every single quiz and every single attempt, looking for students who started an attempt and didn't do anything with it. If they answer even one question, then I count it as an official attempt. If they don't answer any questions, then I take it as they opened it by mistake and I throw it out. Over time, the number of students needing the adjustment has dropped and last semester, I didn't have to make a single adjustment. This is why I say that good instructions from the instructor can go along way. Canvas tries to let the student know how that last attempt will be counted, but they cannot make instructors give good instructions.

Something else that Canvas considers is how popular a feature is. I anecdotally suspect that not many instructors use the average (like I do). I also suspect that few use latest, but most use highest. If most instructors are using the highest score, then the last attempt is irrelevant and you're adding a bunch of stuff for a small audience. I don't necessarily like that approach as I teach a subject that doesn't get much love from Canvas because it's not as popular and the demands we make won't effect most people. But I've come to accept that I should not expect Canvas to automatically grade a foreign language essay (a math problem).

That was anecdotal, but I am also a Canvas administrator and have access to data about the quizzes for every single instructor for our institution. I queried how many quizzes keep the highest, latest, or average grades for all published courses in all of our courses that have students in them. Here's what I found: Of the 86,742 quizzes I looked at, 98.6% of them kept the highest grade. That number is a little misleading as it includes all of the quizzes with single attempts. Unfortunately, the data I have doesn't specify whether multiple attempts are allowed or not. 0.9% (less than 1%) kept the latest score. 0.5% kept the average. Still, Canvas would be adding complexity for something that might be an issue less than 1.5% of the time. They normally target the middle 60% of users, while the top 20% or bottom 20% of users may have to deal with what works for the majority. 1.5% is nowhere close enough for them to consider it a major problem.

Note that the 1.5% is based on the number of quizzes, not on the number of students taking the quiz.

While I agree in principle that some things could be different. For example, maybe the "here's what will happen" should be directly under the "Take the Quiz Again" button instead of the link on the side. I don't see much happening on the Canvas side because it doesn't benefit most of the people. I do think instructors should do a better job of informing the students of the rules for the quizzes, but I also think students should not open a quiz that they don't plan on finishing. If it works best for your organization to make sure you complete it in a single session or within 18 hours, that's up to you to keep track of, not Canvas. It's going to be too different for each individual for Canvas to manage.

TiffanyStull
Community Participant

I think it should be clear to the student which score will be the recorded score for multiple submissions (last/highest/average) on the start screen for the quiz and after it is submitted.

Neither "Classic Quizzes" nor "New Quizzes" indicate to a student when starting a quiz attempt what the recorded score will be in cases where multiple submissions are allowed.

Classic Quizzes start screen for a quiz with 3 attempts, recording Last Score:
classic-quizzes-quiz-multiple-attempts-no-scoring-indicator.png

New Quizzes start screen for a quiz with 3 attempts, recording Average Score:

new-quizzes-quiz-multiple-attempts-no-scoring-indicator.png

When multiple attempts or a retake are allowed, it should also display information regarding the recorded score, e.g.: 

  • Your highest score will be recorded.
  • Your latest score will be recorded.
  • Your average score will be recorded.
ProductPanda
Instructure
Instructure
Status changed to: Archived
Comments from Instructure

As part of the new Ideas & Themes process, all ideas in Idea Conversations were reviewed by the Product Team. Any Idea that was associated with an identified theme was moved to the new Idea & Themes space. Any Idea that was not part of the move is being marked as Archived. This will preserve the history of the conversations while also letting Community members know that Instructure will not explore the request at this time.