[ARCHIVED] Why are quiz penalties (fo wrong answers) not calculated the same way as points for correct answers?

[haug]

This question concerns New Quizzes with Multiple Answers Questions and explicitly not to Multiple Choice Questions where questions cannot have more than one correct answer.

This may indeed turn out to be a feature request, but I'm putting it as a question, as I want to make sure I'm not missing anything (it seems strange to me that Instructure has not yet considered the problem I am describing).

So, here is the thing: I think the way in which "Partial Credit with Penalty" scoring is done in Canvas New Quizzes doesn't make sense. In a nutshell, the way it currently works (at least with the default settings) is that Canvas awards a fraction of the points available for each correct answer and subtracts an
equivalent fraction for incorrect answers.

This works fine when the number of correct and incorrect alternatives is the same. But let's consider a question that has one correct alternative and three wrong alternatives. Let's say the total number of points for that question is 3. Canvas will award 3 points for the correct answer and penalize each chosen wrong alternative with -3 points. In other words, the student will get zero points if they select a single wrong alternative, even when they have correctly selected also the correct alternative. To me this score seems like a gross underestimation of the students knowledge.

I don't understand why does canvas not calculate the penalty for a wrong answers the same way as it calculates points for correct answers. What am I missing?

Points for correctly chosen alternative = Total points available / number of correct alternatives

so why not

Penalty for incorrectly chosen alternative = Total points available / number of wrong alternatives?

In the example above, this would mean that the penalty for a wrong answer would be 3 / 3 = 1. So the student who chose the one correct alternative and one of the three wrong alternatives will get 3 -1 = 2 points.

Does this make sense to anyone else? Is there currently any way of grading Multiple Answer Quizzes in this way?

[haug]

Maybe one thought to add to my original post above: assuming that we calculate penalty points the same way as we calculate positice points (as I propose), what happens if there is only one wrong alternative but three correct ones, i.e. the reverse of the above example?

In that case each correct answer gives 1 point and the wrong answer gives 3 penalty points. So we have a somewhat similar situation as the one I originally compained about: if the student chooses the wrong answer, they will get no points, no matter how many correct answers they choose. The only difference compared to the current situation I complained about above is that there is only a single "loose all" question, whereas in the other example, there are three "loose all" questions and only one "win all" question. But I still think that my proposal is better than the current state exactly because of this difference. Students are much more likely to loose all when they get many "loose all" alternatives than when they only get one.

As I think about it, the difference between the current state and my proposal boils down to what model of knowledge we use. The Canvas' knowledge model seems to give partial credit to partial knowledge: if there is only one correct answer, it assumes that all wrong answers are equally wrong and if student picks one of those, they don't have any of the knowledge required in this question. If there are multiple correct answers, it assumes that whatever wrong answer there is, it is only as wrong as one of the right answers is right. So while it does not allow for partial knowledge in the first scenario, it does so in the second.

In a way it makes sense, becaus if there is only one right answer to a question, that seems to indicate a kind of "black and white" question, without greyzones (i.e. partial knowledge). But the problem is that the answer alternatives in a quiz do not represent the nature of the question. They are chosen by the teacher based on various reasons. If the teacher provides only one correct alternative, that doesn't mean that the other alternatives are totally wrong. In fact, they are often designed to be partially correct, for obvious reasons. 

So what are the assumptions behind my suggestion? (I'm finding this out as I write): 

When there is only one correct answer but multiple wrong ones, I would assume that only all wrong answers combined can wipe out the knowledge indicated by the correct answer (as opposed to a single wrong answer wiping out the correct one). Putting it differently, I'd assume that a student who selects only the correct answer has full knowledge of the subject matter, not just because he/she selected that answer but also because he/she didn't select the other answers. If the student selects one of the wrong answers, this therefore doesn't indicate that the student has no knowledge, but less than full knowledge. Hence he/she loses only part of the points earned with the correct answer. 

Finally, the other scenario, with one wrong and multiple right answers: here, the assumption is obviously that full knowledge is indicated by selecting all correct answers and not selecting the wrong alternative, but if the wrong answer is selected, that wipes out any knowledge indicated by the correct answers. 

TBH, I'm not really happy with that either. My intuition tells me that the wrong answer should only eliminate one of the correct answers, not all of them. 

So the general formula for that would be to determine points and penalties by dividing the maximum points with either the number of correct or wrong altermatives, whichever is greater.

What do you say about that? I'm starting to thing that the best solution would probably be to allow us to manually set the number of penalty points for each wrong alternative. That would allow us to also take the degree of "wrongness" into consideration.

[mskoch]

@haug Thanks for sharing your thoughts. I see where you are coming from with this proposed method of grading multi-answer questions, but I think our instructors would have a hard time with it because it means the calculation used in each question would vary depending on the number of correct vs. incorrect answer choices in the question.

In general, our instructors would like students to receive some credit for selecting any of the correct answers. In the current system, they will need to write questions with more correct answers than incorrect answers to achieve this. If we have an added grading option of counting only correct answers, then they would be able to use this option for questions with more incorrect answers than correct answers. This is the option we had when we were using Desire2Learn and one of the things instructors missed when we moved to Canvas. 

[RussellCGPS]

I totally agree with this. Partial correct doesn't always require penalties. My assumption is that a majority of users who saw that partial correct multiple answer questions would work by only adding the number of correct options chosen until it meets the total points possible.

This solution does not fit the problem. Teachers are still having to manually grade those questions when they shouldn't have to. 

[elhflynn]

I 100% agree with this!!!

I just made a New Quiz for the sole reason of the potential for partial credit awarded for selecting the ONE correct answer, along with any wrong answer(s), in a multiple-answer question. I hoped this changed from the same way it was in Classic Quizzes, but it turns out partial credit is still not an option here.

Please, Canvas, do what we ask and need from you as instructors!!

[elhflynn]

Seriously, why can't/don't they just do everything we ask and need? Does anyone know the true reason?