Late policy with multiple attempts on a quiz

[jlbryan]

When using the late policy, If a quiz has been allowed to have multiple attempts and is set to take the highest score, If the newest submission is late the late policy overrides the highest score function and applies the policy to the newest score on the test. Has this been caught in the beta testing? I would think that if the teacher allows the highest score grab on the multiple attempts, that it would apply the late policy only if the newest score was the highest, or override the late policy.

[erinhmcmillan · Accepted solution]

Hi, @volzk 

The idea you referenced has a note at the top as commented by Instructure; it links to the known issue, which states that the ticket has been shelved by the product team. More investigation will be required as the Gradebook is not compatible with the existing functionality.

Thanks,

Erin

[jw2546]

A year later, this still isn't working right, and now Quizzes will be around an extra two years. I would do a facepalm emoji, but this content editor lacks one.

[morrisj1 · Accepted solution]

I recently received word from Canvas L2 support that this programming error will officially not be addressed, saying that "they [the engineers] have decided to shelve the issue at this time because the gradebook's late policy and multiple quiz attempts are not currently compatible". It's interesting that Canvas uses the exact description of the programming error that they made as their reason for not fixing it.

So to summarize, Canvas was first made aware of this programming error in February 2018. According to this thread, it took them many months to even acknowledge that it was an error and not just a new feature idea. Now, after more than 2 and half years, and despite assurances that a fix was right around the corner, their final verdict is that they won't be fixing it. In other words, Canvas is content to sit by and do nothing while their program continues to incorrectly score thousands of quizzes across the country.

When our university signed on to Canvas, I was enthusiastic due to its many features and excited about how this would benefit my students. Having a lot of features isn't helpful, though, when many of them don't function properly and no effort is made to fix them. Unfortunately, this programming error with the late penalty is only one of many examples of bugs and design flaws that seriously compromise the functionality of the program and which Canvas declines to take any action to fix. I am honestly starting to think that our university made a big mistake in adopting Canvas, and I can only hope we make a better decision the next time around.

[CurtisHartling]

I feel your pain. We just switched from D2L to Canvas and I'm missing D2L more each and every day. When I saw that this issue was marked as "solved", I said to my self before jumping to the solution, "I wonder if the solution is simply, 'you can't do that'". I was right!

[jw2546]

As an update, we are experiencing the same bugs with late penalties and multiple submissions for Quizzes, New Quizzes, and have a related, but different, bug with Kaltura Quizzes and how they interact with marking late attempts when there are multiple attempts.

We have also heard that L2s don't think this needs to be worked on (despite affecting student grade integrity). Our customer service rep promises to take this as far as she can. She recognizes that it should not be on students or teachers to monitor all grades for accuracy and that the product should work.

I am no engineer, but, it seems that it's a simple order of operations issue. First, apply late penalty to late attempts. Second, compare attempts to determine highest score. Finally, display the proper score. How hard is that?

[bxf13b]

I would rather address this issue case by case.  

You can adjust the status in the gradebook as shown in this workaround video:

https://youtu.be/vJfFe2Fx3Gw

[jw2546]

If they fixed it right and applied the penalty to all late assignments before determining which is first, it wouldn't create any loophole. Simple order of operations. Plus, looking at it case by case sucks for the instructor I am supporting that has an average of 30 of these cases per quiz (with at least 12 quizzes per semester). She shouldn't be punished because Instructure cannot apply the grading rules in the correct order. Apply penalty, then compare. Simple

[volzk]

Hi bxf13b, 

Are you an instructor who uses these features?  It seems odd that you'd even find this discussion if you are happy with the features as they are.  Is Canvas trying to garner support for their inability to fix an obvious bug in their code?

For those of us who use these features (multiple attempts, late penalties, and "highest score"), it seems silly to apply a late penalty to a perfect score that was earned before the late penalty applied.  In fact, it seems very commonplace in teaching nowadays to allow students to re-take online quizzes (which many of us have worked hard to create!) as extra "practice" before an exam.  This is good reason for matching up the multiple attempts with highest score in combination with a late penalty.  

It is well known from previous conversation here that we can manually change a status for a student, but this does NOT fix the problem.  It's a temporary workaround.  (Some of us have been using the workaround for a few years now.)  When an instructor has, say 15 students contact them to change a single grade, it is quite a hassle.  (This happens over and over...)

I am not using the late penalty on Canvas this semester because I have had way too many hassles with the manual changes.  It's a disappointment that Canvas won't fix this coding error.

Katie

[bxf13b]

Hi Katie, 

I am an instructional designer and I also teach. When I teach, I make adjustments individually. It has worked for me. When asked by one of our faculty members about the same issue, I checked to find if there are other solutions because I also wonder what others do in such situations. After reading y'all's responses, I have revised my response to indicate mine as only a workaround. If we can have Canvas update its features that would be great. I apologize for the confusion I might have caused.

However, beyond feature updates, I still have my reservations about allowing late attempts past the due date without a penalty if the same penalty is applied to students who tried only once after the due date if all students are allowed to turn in assignments late. Wouldn't that be unfair for other students? If that should be done, why not move the due date forward to give everyone an equitable chance? That way, everyone can have multiple attempts before the due date and the highest score would be kept. What scenario would make it desirable to not implement the late policy for people who make attempts after the due dates to get a higher score? We cannot have it both ways. If exceptions have to be made, we can set a different due date for (a) student(s) in special circumstances, or as I suggested earlier, adjust case by case.

In terms of features, I would rather that Canvas allows individual late policies for assignments rather than a global change (just like the post grade policies). I can understand that we may want to be more lenient in some assignments and stricter in others. However, it could cause a lot more confusion for students as well.

[jw2546]

Hello bxf13b,

I don't think anyone is talking about not applying a late policy for a late attempt (or at least I am not). I just don't want late policies applied to on time work or late work being selected as the highest scoring submission and then having the late policy applied. I've had both happen and neither makes any sense to me.

I agree that having a coursewide policy and then individual policies OR maybe having late policies assigned by assignment group (since this is generally how late policy differentiation is done in a class) would be great, but I think that is a different topic. 

--James

[volzk]

Hi bxf13b,

It seems like you haven't actually seen the problem/programming error here.  No one is asking for late attempts to not be penalized.  We are asking for early attempts to not be penalized!  We want the late penalty to apply to each individual attempt.  The issue here is that a student can earn, say, 90% (on time), and then choose to go back into the quiz and use an extra attempt to say, study/practice for their exam.  That same student might earn 100% on this studying (yet late) attempt.  But - if my late penalty is, say, a 25% reduction, then Canvas will give the student 75% as their final grade (since the 100% was the higher than 90%)!    We want the final grade to compare the 90% (first attempt) to the 75% (second/late attempt after penalty), giving the student the 90% as the final grade. 

Note that if I choose to manually ignore the late penalty, then this student would earn 100%.  Ugh!  This is definitely not a fix.  (In fact- this is what you are arguing against, and I agree with you!  We don't want that student to get the 100% as a score, as they didn't earn it "on time".)  Thus, I have to manually look at each attempt and then decide what final grade I want to choose.  It can be quite a hassle, especially with more attempts in the mix.

Thanks for your interest in the conversation.  We hope you can understand what the issue is here, and support our efforts to get Canvas to fix the error.  We aren't disagreeing with your desire to penalize lateness!  But unfortunately, at this point, we can't use this feature with the current coding errors.

Katie

[bxf13b]

I do support your effort to get Canvas to make changes! More flexibility means power to the teacher! I have made an earnest effort to clarify that in my previous response. 

However, I feel annoyed that I am being talked down to when you say that I don't seem to see or understand this or that.  I understand it well. I just disagree about the solutions. I just have reservations about the message one could send. I also worry about potential loopholes the suggested change creates.

Let's say that the student made 80 the first time, and the second attempt was late and this student made 100, and the late policy took away 10% leaving 90. And the solution some in the thread seek would keep the 90, since the sequence sought is to apply the penalty first and then compare to award the highest score. This WILL be a loophole unless Canvas can change their algorithm to still take the first attempt even if the late attempt earned a student a higher score with the late policy applied! That would be ideal if Canvas can allow that. Again, I would hope that Canvas provides that kind of flexibility. Otherwise, you substitute one problem with another or more if you request Canvas to simply reverse the order.

More importantly, what are you saying to students here? Oh, don't be late, or you will be penalized. Oh wait.  Maybe not, if your score is higher with the late policy applied. We'll take the highest score anyways, late or not. If I were that student, I would say, make up your mind! Or I would tell my buddy: Hey, I found that if you take it late, and you made a higher score, you get the higher score and the professor doesn't even notice or care.

A better solution would be to make late policy locally applicable to grade columns instead of globally for everything.

Before that happens, I still would do any of these:

• Change the due date for all or some students when situations warrant such changes (such as preparation for exam);
• Adjust lateness manually for students if you decide to accept it for certain situations; I thought students deserve the kind of individual consideration. If you want to automate everything and avoid hassle, maybe rethink the assessment method.
• Stick to the due date and the penalty policy. Create a practice quiz, especially when you use question banks, to help students study for the exams.

[jw2546]

I'm sorry if you feel talked down to. I do not understand what loophole would be created if they did what I am asking.

Currently:

   Example 1:

• On time attempt score: 80
• Late attempt score: 100 (before penalty), 90 (after penalty)
• Current order of operations: 100 > 80, 100-.1x100=90, final score=90 (accurate)

   Example 2

• On time attempt score: 80
• Late attempt score: 85 (before penalty), 75 (after penalty)
• Current order of operations: 85 > 80, 85-.85=75, final score=75 (inaccurate)

Our requested fix:

   Example 1:

• On time attempt score: 80
• Late attempt score: 100 (before penalty), 90 (after penalty)
• Current order of operations: 100-.1x100=90, 90>80, final score=90 (accurate)

   Example 2:

• On time attempt score: 80
• Late attempt score: 85 (before penalty), 75 (after penalty)
• Current order of operations: 85-.85=75, 75<80, final score=80 (accurate)

If you could provide me an example of a loophole here, I'd be glad to read it.

The message I am sending the student with my recommendation is:

   Don't be late or you'll be penalized. But, if you happen to do SO much better on the late attempt that, after the penalty, your score is higher than your on-time attempt, then I'll bump your score up to your new high score.

Your list of things you would do are:

- Things we do when applicable and manageable.

or

- NOT solutions that easily allow instructors to allow multiple attempts, have a late policy, and then award the highest post-penalty grade to the student (which is the whole thing we are asking for).

[morrisj1]

As jw2546 points out, this solution is impractical at best. Looking at every quiz for every student takes an enormous amount of time, and it's impossible to tell from the gradebook page which scores Canvas has inappropriately penalized. Therefore, the only option is to examine every quiz that has been marked late to see if Canvas has incorrectly discarded the student's highest score. Most students are not even aware that they have lowered their grade by reattempting a quiz after the due date that they previously took on time because they assume, as any reasonable person would, that when Canvas tells them it will keep their highest score, it will.  

But let's assume that an instructor is willing to spend the time to manually sort through every quiz with a late tag, check it, and change the grading status from late to none as the video suggests. Even *that* doesn't fix the problem in many cases. For example, suppose a student took a quiz on time and got 80%. Then, the student takes the quiz late for extra practice and gets all of the questions right. Due to the late penalty, the second score of 100% is correctly reduced to 50%, the first score is incorrectly counted late and also reduced to 50%, resulting in an incorrect final score of 50% for the quiz. Changing the grading status from late to none to "fix" this problem awards the student 100% because the 2nd attempt is now incorrectly regarded as on time, creating a situation where there's no point to having a late penalty at all and removing all motivation for students to complete quizzes on time. The correct score on this quiz should of course be the original 80%, but the only way to make that score appear in Canvas is to go into the individual quiz attempts and manually fudge in negative points to reduce the 100% to the correct score of 80%, which is an unbelievable waste of instructors' time.

Canvas's year long refusal to fix a simple order of operations programming error is truly mind boggling.

[morrisj1]

There seems to still be some misunderstanding about this programming error. The basic problem is that Canvas tells students that it will keep their highest quiz score when instructors choose that option, but this is not always the case when late penalties are applied.

To illustrate, suppose that an instructor has asked Canvas to keep the highest quiz score and has chosen to apply a penalty of 50% for late work. Also suppose that a student takes the quiz once on time and earns a score of 80%, and then takes it late and gets 100%. Now, the highest score needs to be recorded.  Here's a comparison of how it should be calculated, according to what Canvas says it will do, compared to what Canvas actually does:

Correct Calculation (What Canvas Says It Will Do)

(1) First, look at the two scores and decide if late penalties apply to any of the scores. The 80% was on time, so it should remain unchanged. The 100% score was late, so it should be reduced to 50%. This results in scores of 80% and 50%.

(2) Now, take the highest of the two scores from the previous step, which were 80% and 50%. The highest score is 80%, so that's the student's final grade.

---------------

Incorrect Calculation (What Canvas Actually Does)

(1) First, ignore the fact that the second quiz attempt was late. Then, compare the two quiz scores of 80% and 100% and take the highest one, which results in a score of 100%.

(2) Now, assume that every quiz attempt that the student made was late, regardless of whether or not this was the case, and apply that late penalty to the score from step 1. The 100% from Step 1 is reduced to 50%, resulting in an incorrect highest quiz score of 50% for the student.  50% is the score that Canvas actually records as the "highest" score.

--------------

It should be emphasized that the student in the above scenario earned an 80% score on time, so when Canvas tells the student that it will keep their highest score, the 80% score should be the absolute lowest final score for the student on this quiz, regardless of what the student earns on future attempts or when these future attempts are made. Asking Canvas to fix this problem is not a request for Canvas to simply change a feature that we don't like, as Canvas support would have us believe; it's a request for them to fix a genuine error in their programming. Their program is incorrectly telling students that it will keep their highest score when this is clearly not the case.

Also note that source of this error is simply the fact that Canvas compares the quiz scores first and then applies the late penalty second. All Canvas would have to do to fix this error is to reverse the order of these two steps, and yet they refuse to do so.

[bxf13b]

@morrisj1 

I appreciate your explanation and the thinking behind it.

This is a loophole that I tried to explain with one of your examples.

"Example 1:

• On time attempt score: 80
• Late attempt score: 100 (before penalty), 90 (after penalty)
• Current order of operations: 100 > 80, 100-.1x100=90, final score=90 (accurate)."

To you this is accurate because this is the message you try to convey: "Don't be late or you'll be penalized. But, if you happen to do SO much better on the late attempt that, after the penalty, your score is higher than your on-time attempt, then I'll bump your score up to your new high score."

To another professor, the method to compare two scores and take the highest (even with the penalty applied) could be a problem. Here is one scenario, by slightly adjusting some of what you have said:

• Student makes first and on time attempt score: 80
• The teacher releases the correct responses upon the due date, so everyone became aware of the correct answers.
• Equipped with knowledge about the correct responses, the student makes a late attempt, after having seen the correct responses, and scores 100 (before penalty), and 90 (after penalty)
• Using what you proposed, to apply the penalty and take the highest of the attempts, the student earns the final score of 90. The teacher and other students may not think that is a fair approach. 

The message you try to send might help you and your students, but it may create issues for others.  Therefore, as I said earlier, I still support a change, but insofar as the change

1. gives a teacher the flexibility to choose which columns the late policies can apply (just like the post grade policy), or
2. gives a teacher to choose which sequence to use.

[jw2546]

@bxf13b 

Why would someone release the correct answers while it is still possible to make attempts? If they want them to practice, they should do what you said and make a practice test. If they genuinely want to allow late attempts, they should release the answers after the attempt window closes. Fixes would not create a loophole, they would just bring to light currently poor quiz policy design by some instructors.

[volzk]

@bxf13b 

The loopholes you mention are all related to pedagogy.  I would never set my courses up with the structure that you have mentioned, and I don't see any reason any intelligent instructor would do so!  But - we aren't here to debate pedagogy.  We are just asking for a programming error to be fixed.  It seems like you are simply trying to support the folks at Canvas by coming up with reasons why they shouldn't fix their error.

[bxf13b]

You do acknowledge these are loopholes. Reverse the sequence is not a fix, it creates a different set of issues.  You cannot talk about the use of educational technology without talking about pedagogy as well. They are not entities that can be separated to live in mutually exclusive spheres. I would not encourage anyone to send confusing messages to students that maybe being late is held accountable and then maybe it doesn't, as long as you make a higher score after the penalty. That is NOT a good way to design or teach a course. 

I am just a user as you are. There are plenty of ways Canvas can change in so many other areas, and even regarding this interesting issue. I have suggested a few alternative ways to make changes. If you think what I said gives Canvas the support not to change in the way you want, perhaps your suggestion needs some further thought. 

[morrisj1]

Hi bxf13b,

Thanks for your reply and your good points. I completely agree that it would be better if late policies could be applied to individual assignments. Thanks also for giving the example to illustrate the loophole by which a student could earn a higher grade by looking at solutions and then taking a quiz late; I have experienced this issue myself. When I was still using Canvas quizzes, I avoided this problem by setting the closing date for the quiz to be the same date on which solutions were made available to students so that students could no longer take the quiz for credit after solutions were posted.

There was one part of your response that I didn't quite understand, though. I agree that our proposed fix would figure the grade as you said; that is, if a student earned an 80% on time and then a 100% late (after looking at the solutions, and reduced to a 90% due to the late penalty), then our proposed way of figuring the grade would assign a best score of 90% to the student. However, this is exactly what Canvas would do now without changing its programming. Canvas would make believe that the 100% score was on time, compare the 80% and the 100%, take 100% as the best score, and then apply the late penalty to come up with the same final score of 90%. Since our proposed change and Canvas's current programming result in the same best score of 90% in this particular situation, I'm not sure how keeping Canvas's current programming would prevent this loophole that you described.

My objection is that Canvas currently treats all quiz attempts as if they're late, even those that were submitted on time, when figuring the best quiz score. This doesn't make any sense to me or to my students, and it is inconsistent with how Canvas tells the students that it will figure their grade. Finding and manually fixing every incorrectly calculated quiz score takes so much time that I have stopped using Canvas quizzes altogether.

[bxf13b]

@morrisj1

You are right, these two sequences both result in students getting 90. I am not saying that the current method is fine. Simply penalizing late attempts would be an improvement in the situation you described, to allow students another attempt at a later time, with the assurance that their score can go up, stay the same, but not go down. I am not yet sure whether to accept the late attempt with a higher grade serves the purpose of encouraging on-time completion. I just hope that Canvas will provide more choices for faculty to cater to different teacher preferences. I appreciate your responses and wish you the best.