Understanding Multiple Answers Questions

James
Community Champion
19
80472

This document explains what Multiple Answers questions are, how partial credit is computed for these questions, why Canvas computes partial credit the way it does, and compares the multiple answers offerings of other Learning Management Systems.

What is a Multiple Answers Question?

A Multiple Answers question is one that allows for selection of multiple answers. In the parlance of HTML forms, it uses checkboxes (stylized as squares) instead of radio-buttons (stylized as circles).

Depending on the learning management system, it is sometimes called multiple answer, multiple response, multiple select, or multi-select. Others consider them part of multiple choice questions.

Within a quiz, they are displayed like this:

Quiz Question Example

Other Question Types with Partial Credit

In Canvas, there are three types of automatically-graded questions that allow partial credit: Fill in Multiple Blanks questions, Multiple Dropdown questions, and Multiple Answers questions.

The Fill in Multiple Blanks questions, which also called Multiple Fill-in-the-Blank, and the Multiple Dropdown questions assign points by dividing the total number of points by the number of responses and then awarding points for each correct response. This is a fairly standard way of assigning partial credit and most people understand how points are awarded.

If a 6 point question has 3 parts, then each part is worth 2 points. If a student correctly answers 2 of the 3 parts, then they earn 4 points, which is two-thirds of the possible 6 points.

What's the Big Deal about Multiple Answers?

Multiple Answers questions are graded by a completely different set of rules. By itself, that contributes to some of the confusion. Throwing aside whether it's right or wrong, it's not consistent with the other two types of questions that allow for partial credit.

Some see that as an immediate "it's wrong" while others see it as an "it's an opportunity".

The trouble stems from a little note at the bottom of one of the Canvas Guide pages: How do I create a Multiple Answers quiz question?

Note: This question type awards a fraction of the points available for each correct answer selection and subtracts an equivalent fraction for incorrect answer selection.

I'm not sure whether the trouble is because it's a little note at the bottom, because of the way it works, or because it's not the way people expect it to work. It may be a little of all three.

As a math teacher, I actually understand that sentence and have had my students take quizzes on "How to take Canvas quizzes" so that they would understand it and be able to work it to their advantage. However many people do not understand what that means. When faculty don't understand it, they can't explain it to the students who don't understand it. When people don't understand something, they sometimes bad-mouth it, call it wrong, and demand it be fixed. That doesn't necessarily mean it's broken, but that it needs better explained.

Here are some of those conversations in the Community. There are many others, this is just a sampling to show that the topic comes up frequently.

Ask a Question

Feature Requests

Ways to Assign Partial Credit

I took a look at some of the major players in the Learning Management System (LMS) game and found that there were essentially two ways of assigning partial credit. Note: "All or nothing" isn't really partial credit.

  • Assign points to each response and then add the points assigned to the checked items to get the points awarded for each question.
  • Define a correct response as selecting an item that should be checked or not selecting an item that should not be checked.

There are variations on these themes. Some take the difference between the number of correct marked responses and the number of incorrectly marked responses. Some make sure that the point values never go negative. But essentially, the schemes awarding partial credit boiled down to those two techniques.

The Right Way?

When assigning partial credit, people's understanding of "the right way" depends a lot on what learning management system they're coming from. For these, "the right way" often translates to "the way I've always done it."

Stepping back, the real issue is the assumptions that you're working under.

Assigning Points to Individual Responses

If you are used to assigning points to individual responses, then you probably work under these assumptions.

  • If a student selects all of the items that should be checked, then they should get the full points possible.
  • If a student selects some of the items that should not be checked, then they should be penalized. Some call this a penalty for guessing. This means that assign a negative point value for certain responses.
  • The total points earned for a question is the sum of the points assigned to each item that the student selects.
  • The lowest score a student can get for a question is 0. Although individual responses may have negative points associated with them, if the student misses more than they get right, then they should get a 0 for the question, not a negative score.

Defining Correct Responses

If you are coming from a system that awarded points based on correctly selecting (or not selecting) items, then you probably work under these assumptions.

  • Students are awarded points based on the percentage of questions that are correctly marked. If they correctly mark 80% of the questions, then they should get 80% of the possible points.
  • Each item is weighted equally. Without the ability to go through and assign points to each individual response, there is no way to tell the system, for example, that items that should be selected count twice as much as items that should not be selected. If you don't enter points to each response, then you don't have control over how much each part is worth.
  • No matter how badly a student does, they won't lose points.

Example Scenario

Before you can really understand how Canvas grades Multiple Answers questions, you need to complete a little exercise.

Here are the assumptions that we are going to work under.

  • You will be assigning points to individual responses (this is the first approach).
  • Students should get the full points if they correctly select all items that should be checked and do not select any items that should not be selected
  • Students should incur a penalty for selecting items that should not be checked.
  • Students receive the sum of the points assigned to the items the student selects
  • You have no idea as to the level of correctness of each response. All that you know is that an item should either be selected or it shouldn't be selected (this is similar to the second approach). This also means that a student should lose as much for a choosing a wrong response that they earn for choosing a correct response.

You have a Multiple Answers question that is worth 12 points. There are 6 possible choices: 4 choices that are correct and 2 distractors that are incorrect.

You need to assign points to each item. Remember that points for each item may be positive or negative and that the final score will be the sum of the points for each item that is selected.

Without loss of generality, let's say that the first, second, fourth, and sixth items are the correct ones that should be selected and the third and fifth are the incorrect ones that should not be selected. Furthermore, we'll label the responses as A, B, C, D, E, and F for simplicity.

Challenge: Complete the table below by assigning points to each of the six items. Don't skip ahead to the answer until you've attempted it on your own.

Item

A

B

C

D

E

F

Correct?

yes

yes

no

yes

no

yes

Points

Incorrect Point Distributions

Before we look at the correct answer, let's look at some incorrect ones.

Correct?

yes

yes

no

yes

no

yes

Reason Why Invalid

Points

2

2

0

2

0

2

Only 8 points possible

Points

3

3

0

3

0

3

No penalty for guessing

Points

3

3

-1

3

-1

3

You don't know levels of right or wrong

Points

2

2

2

2

2

2

Awarding points for incorrect answers

Some people might have suggested the last scheme for a different reason. If you are used to defining correctness based on selecting a correct item and not selecting an incorrect item, then it makes sense. However, in this scenario, the points awarded is the sum of the points for each selected item, not the sum of the points for each correctly answered item. If you don't penalize and use this method instead, then a student could simply check all of the boxes and get 100% every time.

Correct Point Distribution

If you understood and followed the assumptions and instructions, then the correct distribution of point values is given in the table below.

Item

A

B

C

D

E

F

Correct?

yes

yes

no

yes

no

yes

Points

3

3

-3

3

-3

3

Here's the explanation of how those values were arrived at.

  • There are 12 points possible and 4 correct answers. If the student is to get full credit for correctly selecting all 4 of the items that should be selected, then each correct response should be worth 12 / 4 = 3 points.
  • Students should lose as much for a wrong answer as they get for a correct answer, the points assigned to each incorrect response is -3.

The example scenario we just worked through and the calculations that we just came up with is exactly how Canvas grades the Multiple Answers questions. You tell it how many points the question is worth and which items are correct and it does all of the calculations for you.

Let's look at that note from the Canvas Guides again.

Note: This question type awards a fraction of the points available for each correct answer selection and subtracts an equivalent fraction for incorrect answer selection.

As a mathematician, I will admit that statement is insufficient to explain the situation. A minor tweak that would help clarify what is meant would be to insert the word "equal" so that "a fraction of the points" becomes "an equal fraction of the points."

When Canvas says, "This question type awards a fraction of the points available for each correct answer selection" what they mean is "Divide the possible points by the number of items that should be selected to find out how many points to award for each correctly selected item." This is necessary to ensure that a student who gets correctly selects everything gets full credit.

When Canvas says, "and subtracts an equivalent fraction for incorrect answer selection" what they mean is "a student loses the same amount for marking a wrong answer that they get for marking a correct answer."

What their page doesn't say is that a student's score won't go negative and the minimum score for the question is 0.

It also doesn't explain why that process is used or that it is exactly what you would do given the assumptions listed at the start of this scenario. Canvas determines the points by adding the points assigned to each selected response. It makes sure the student gets full points for correctly selecting what should be checked, it takes away an equal amount of points for each incorrectly selected response, and doesn't let the score go negative.

Probably the most controversial part of the scenario is that you don't know the level of correctness of the responses. The teacher, who is a content expert, has the knowledge and understanding to assign different point values to each individual response. Most faculty probably would not use that level of granularity and Canvas simplifies things by automatically assigning the points for the user.

Others would say that the most controversial part is how you define "correct" and it should be based on selecting a correct item or not selecting an incorrect item. That is the other major approach to grading multiple answers questions. That doesn't apply to this scenario because Canvas adopted the principle of totaling the points for each selected response.

Example Students

Let's continue with the example scenario and see how some fictitious students would fare. The names of the students were chosen based on the list of popular baby names in 1998 that is published by the Social Security Administration. The names were chosen from those appearing in relatively the same position in both the list of male and female names so that the gender of the names could not be used to imply intelligence.

To summarize: there are 12 possible points, responses A, B, D, and F are correct, responses C and E are incorrect, each selected correct response earns 3 points, each selected incorrect response costs 3 points, unselected items are ignored, the score for the question is the sum of the points for all selected responses, and the score may not be negative.

The table shows the items each of our five students selected, the points obtained from each response, and the total score for the question.

Name

Selected

A

B

C

D

E

F

Pts

Notes

Jordan

ABDF

3

3

3

3

12

Only checked correct responses

Casey

ABEF

3

3

-3

3

6

3 right, 1 wrong

Peyton

ACE

3

-3

-3

0

1 right, 2 wrong. Minimum score of 0

Avery

ABCDEF

3

3

-3

3

-3

3

6

Checked all responses

Riley

A

3

3

Thought it was multiple choice

Other Learning Management Systems

What follows is a summary of what other Learning Management Systems do for their multiple select questions. This list of LMSs is not meant to be inclusive, but is based off the Spring 2016 update to the Edutechnia LMS Data report.

I've never used any other LMS besides Canvas, so all of the descriptions are based off of descriptions from those LMSs or universities using those LMSs.

Angel

Angel isn't really a big player anymore as Blackboard has announced its end of life in October 2016, but many people coming to Canvas may be former Angel users.

Penn State has a help page for the Angel LMS called Add a Multiple Select Question to an Assessment that gives the following admonition.

Multiple select questions that do not provide negative values for incorrect answers provide no penalty to students who select one or all of the incorrect answers. In this case, a student who indiscriminately selects ALL of the available choices (correct and incorrect) would receive the same full credit for the question as a student who answered it absolutely correctly.

Later on the page, there are two methods of grading, which are verbatim from the Angel 8 Instructor's Reference Manual.

Normal: Grading is based on the sum of the point values assigned to each correct choice selected by the student. Students are automatically penalized for selecting incorrect answers, but the final score for the question cannot be less than 0 points.

All or nothing: The student is given full credit for an exactly correct answer. Partially correct answers are given a zero for the question.

The Normal way that Angel graded Multiple Select answers is exactly the same way that Canvas grades Multiple Answers questions, with the exception that Canvas automatically assigns the points for you, including awarding negative point values to the incorrect answers.

Blackboard Learn

The Blackboard Learn LMS allows for negative points on matching, multiple answer, and multiple choice questions whereas Canvas uses it only for Multiple Answers questions. The Blackboard manual says to use negative credit to discourage guessing. However, the option was under question settings and you had to specify both partial credit and negative points separately and you had to specify it as a percentage rather than a raw point value. The default was all or nothing and instructors really had to do a bunch of configuration to set Blackboard up to award partial credit, so many faculty may not have known it was there and just left it at the default. The differences are that Canvas removes all the decision making from the process and automatically assigns points and that you cannot get a negative score with Canvas.

Blackboard Learn allows too many variations and requires the instructor to decide what percentages to award or deduct for incorrect responses, so I can't give a clear example here.

D2L

The D2L LMS allows for three ways of grading multi-select questions: all or nothing, right minus wrong, or correct answers.

The all or nothing method is pretty clear -- no partial credit is given.

The correct answers approach awards points based on the number of items correctly selected, not the number of correct items that are selected. This means that selecting a correct answer or not selecting an incorrect answer is considered to be correct or right, but selecting an incorrect answer or not selecting a correct answer is considered to be incorrect or wrong. Each answer is either correctly marked or incorrectly marked and so you take the total points for the question and multiply it by the fraction of the questions that were correctly marked.

This is probably the most common interpretation of partial credit for a multiple select question because it's the way the other partial credit questions work in Canvas.

Let's revisit our example students and see how they would fare using this approach. There are 12 points and 6 responses, so each item is worth 12/6=2 points. The student earns 2 points for correctly marking the item and 0 points if the item is not correctly marked. Items A, B, D, and F should be selected while items C and E should not be selected. To simplify things, we'll show R for right answers and W for wrong answers and then multiply the number of right answers by 2.

Name

Selected

A

B

C

D

E

F

Pts

Notes

Jordan

ABDF

R

R

R

R

R

R

12

Only checked correct responses

Casey

ABEF

R

R

R

W

W

R

8

Correctly marked 4 items

Peyton

ACE

R

W

W

W

W

W

2

Correctly marked only item A

Avery

ABCDEF

R

R

W

R

W

R

8

Checked all responses

Riley

A

R

W

R

W

R

W

6

Thought it was multiple choice

The right minus wrong approach is the more complicated one. The manual says:

Users receive points equal to the number of right answers they choose minus the number of incorrect answers they choose. To determine how much each answer is worth, the system takes the total number of points assigned to the question and divides it by the total number of answer choices. … Users can receive a minimum of zero on a question: they cannot receive a negative mark

Let's see how our five students would fare. This time, their score is difference between the number of items answered correctly and the number of items answered incorrectly: the number right minus the number wrong.

Name

Selected

A

B

C

D

E

F

Right

Wrong

Difference

Pts

Jordan

ABDF

R

R

R

R

R

R

6

0

6

12

Casey

ABEF

R

R

R

W

W

R

4

2

2

4

Peyton

ACE

R

W

W

W

W

W

1

5

-4

0

Avery

ABCDEF

R

R

W

R

W

R

4

2

2

4

Riley

A

R

W

R

W

R

W

3

3

0

0

Note that Peyton received a 0 instead of -8 because negative scores are not allowed.

Moodle

Moodle treats multiple-answer questions as a variation on the multiple choice question type. They have two sentences that explain how the grading works.

Each answer may carry a positive or negative grade, so that choosing ALL the options will not necessarily result in good grade. If the total grade is negative then the total grade for this question will be zero.

This is more flexible than what Canvas does, but requires additional configuration. Canvas automates the decision making about how many points to award for each choice, but the overall score is the sum of the points for the selected items.

Moodle does not natively support all or nothing questions, but there is an All-or-Nothing Multiple Choice plugin available that will extend the capability.

Pearson

Pearson is another player that has decided to get out of the LMS market. They have a couple of different LMS products, LearningStudio and OpenClass.

LearningStudio provides the infrastructure for MyLabs & Mastering, which will remain,

Learning Studio grades multiple answer questions based on correctly marking the individual items. Their developer site calls them Many Multiple-Choice question types and wording suggests it's an all or nothing approach. Another document says that you can award partial credit for the multiple answer questions that is based on correctly selecting right answers and not selecting wrong answers that appears to be based off the fraction of correctly marked responses.

Sakai

Sakai allows for a multiple correct, multiple selection variation of the multiple choice question. There are two grading methods in Sakai, Right Less Wrong and All or Nothing, both of which mirror those found in D2L. This means that they are based on correctly marking individual items rather than the number of correct choices.

Method Comparison

Let's see how our five students would do under the different approaches.

Remember that items A, B, D, and F should be selected while items C and E should not be selected. There are 12 points possible.

Name

Selected

All or nothing

% Correct

Right-Wrong

Canvas

Jordan

ABDF

12

12

12

12

Casey

ABEF

0

8

4

6

Peyton

ACE

0

2

0

0

Avery

ABCDEF

0

8

4

6

Riley

A

0

6

0

3

Bear in mind that this is a single example. The results will vary depending on the number of responses, the number of correct responses, and the combination of items chosen by the students.

Summary

There are a combination of methods for grading multiple answers questions.

  • All or nothing: Provided by Angel, D2L, Blackboard, Sakai, and Pearson, but not Canvas. Moodle requires a plugin to support this approach.
  • Partial credit based on the percentage of correctly marked items: Provided by D2L and Pearson.
  • Partial credit based on a difference: The way that difference is defined varies depending on the LMS. Provided by Canvas, Angel, D2L, and Sakai.
  • Full control over the point value assigned to each individual response: Provided by Angel, Blackboard, and Moodle.

LMS

All or nothing

% Correct

Difference

Point Values

Canvas

X

Angel

X

X

X

Blackboard

X

X

D2L

X

X

X

Moodle

plugin

X

Pearson

X

X

Sakai

X

X

The Future

Canvas is working on revamping quizzes with their Modern Quizzing Engine. I've heard some really awesome things about it that should fix most of the concerns of people regarding how Canvas grades Multiple Answers questions

In the meantime, Avi Naiman and James Jones have have developed a user script called QuizWiz that will add "all or nothing" and "% correct" grading to Canvas. This will be a user script add-on to SpeedGrader that will allow you to click one button and regrade a quiz submission. There are a lot more features, but that is the portion that is germane to this document. See QuizWiz: Enhancements to SpeedGrader and Quizzes for more information.

19 Comments
curtain
Community Participant

This is one of the most valuable user created pages I have ever come across.  Thanks for putting this together with such insight and careful language.

adamwarecs
Instructure
Instructure

WOW - Just WOW

cholling
Community Champion

I agree, WOW. This is awesome and makes it clear.... finally..... to me how Canvas is scoring these and does so in a much better manner than I've been able to explain it.

THANK YOU!!!!!

cholling
Community Champion

May I share this? Any chance you could put it in PDF format so that we can download it?

James
Community Champion
Author

The community software has an option to convert it to PDF. Go to Actions at the top right and click on View as PDF. It does not come through cleanly for this, though. Any PDF I would make would basically just be printing the page to a PDF, which would suffer from the same malady. You may be able to highlight and then copy/paste into Word or Google Docs and clean it up that way. It's three years old and kind of out of date now -- and Canvas doesn't do it this way for New Quizzes -- they just have all or nothing -- one of the main reasons I cannot use New Quizzes just yet.

James
Community Champion
Author

Oh yeah, I forgot to mention that you may share it, I just won't be putting it into a printable form.

heyitstowler
Community Novice

This is mind-boggling to me. None of this reasoning seems to explain what is wrong with the "Defining Correct Responses" section.

Some people might have suggested the last scheme for a different reason. If you are used to defining correctness based on selecting a correct item and not selecting an incorrect item, then it makes sense. However, in this scenario, the points awarded is the sum of the points for each selected item, not the sum of the points for each correctly answered item. If you don't penalize and use this method instead, then a student could simply check all of the boxes and get 100% every time.

this by no means addresses the method of assigning points based on correct responses. If you are assigning points based on correct responses, then a student selecting all of the boxes would not receive 100% (unless all responses were correct) as the improperly selected responses would not be considered a correct response.

*You do not need to penalize a student for incorrect answers if you are only assigning points for correct answers*! If a student misses out on potential points for a question, then that *is penalizing the student already*!

Remember, unless you make it a course policy, a student has *no idea how many answers are going to be correct*. Thus, every decision to check or leave an answer unchecked is *a deliberate choice*. Thus, each option correctly left unanswered *should be rewarded* just as a correctly selected answer.

Look at every single example from the lens that for each choice, a user must correctly assess whether or not the answer should be selected. This is the only method that provides perfect partial credit based on how correct the users response is.

Using the examples illustrated in your "Let's see how the students would fare" table, let's look at the points awarded to the student as a percentage of the available points and compare it to the accuracy of their answers:

Jordan: 6 right, 0 wrong. Answers are 100% correct. 12 / 12 points, 100% of possible points

Casey: 4 right, 2 wrong. Answer is 66% correct. 4 / 12 points, 33% of possible points.

Peyton: 1 right, 5 wrong. Answers are 16.5% correct. 0 / 12 points, 0% of possible points

Avery: 4 right, 2 wrong. Answer is 66% correct. 4 / 12 points, 33% of possible points.

Riley: 3 right, 3 wrong. Answer is 50% correct. 0 / 12 points, 0% of possible points

 

Under this implementation, students can get *half of the responses right* and still get 0% of the points. Note that the only response where the percentage of points awarded matches the accuracy of the question is the 100% case.

Under a points-per-correct-response, let's see how the students fare:

Jordan: 6 right, 0 wrong. Answers are 100% correct. 12 / 12 points, 100% of possible points

Casey: 4 right, 2 wrong. Answer is 66% correct. 8 / 12 points, 66% of possible points.

Peyton: 1 right, 5 wrong. Answers are 16.5% correct. 2 / 12 points, 16.5% of possible points

Avery: 4 right, 2 wrong. Answer is 66% correct. 8 / 12 points, 66% of possible points.

Riley: 3 right, 3 wrong. Answer is 50% correct. 6 / 12 points, 50% of possible points

Under this method, the students are awarded points *perfectly in proportion with the accuracy of their answer*. I do not understand how anyone could possibly see this as a bad thing.

The arguments for systems that incorporate penalizations for wrong answers as a "check for guessing" do not make any sense. To understand why, it might be easier to think of each response as an independent True or False question. If a response applies to the prompt, then you select "true" (by checking it) and if not, you select "false" by leaving it unchecked.

Under the solution this article advocates, only questions that have "True" as a correct answer where the student also correctly selected "True" should receive points. Questions where the student selected "false" should never receive points, even if the correct answer was "false".

This would not make sense, so why do we apply the reasoning to multiple answer questions?

The argument that it is a check for guess does not make sense. If you want to penalize students for incorrect guessing and only award points for correct answers, the student would need the opportunity to "pass" on an option. Thus, they would forfeit the points they would have received if they answered correctly, but they would not be penalized additional negative points for an incorrect guess.

In the multiple response question, *students do not have this option*. Answers are either checked or unchecked, but *there is no state that counts as a non-response*. Punishing students for "guessing" when the question format literally provides them no option not to guess is unfair for students.

I do not understand how anyone can stand by this method of grading. At the very least, you should offer the option to grade the answers in a by-correct-response method for those of us who are not drinking whatever kool-aid you are drinking.

James
Community Champion
Author

@heyitstowler

As I mentioned, it seems that every person has their own way they think multiple answers questions should be graded.

With New Quizzes, there is a stimulus question that allows grouping of questions with a prompt, so that you can have six true-false questions and have it graded as six independent questions, giving you the percentage correct.

That is not the way that classic quizzes works and that's what this document was written for. Students do not have an option to pass, nor an option to require a mark of one option or another. If there was this option, then the percentage of correct responses would make sense because they would be marking something.

Canvas chose to only count items where students actually selected an item. Saying that students made a conscious decision not to mark it is not always correct. I always have to specify the "select all correct responses" in the instructions or many students think it is a multiple choice question rather than a multiple answers question. Even then, I have those who think they are just looking for a single answer (like Riley).

Your approach wants to award people points for not answering the question. Since Canvas has no way of knowing why the student didn't pick that response, they chose to award points solely for overt action that the student took (selecting an answer), not for leaving an answer unselected.

Let's say that I leave the question blank because I ran out of time. There were 2 questions that should not be selected, so I just got 33% for a question I didn't even look at. If I had a six part multiple answers question where only one part was correct and skipped that question, then I would get 83% of the points. If they carefully considered the question and decided that none of the answers were correct, then the 83% might be justified, but Canvas doesn't know whether that happened or they just didn't answer it. 

You want to give Riley 50% for marking 1 answer (when there were 4 that should have been marked). If I knew that Riley ruled out those other answers, that would be justified. But I don't have that information. If all that you tell someone is "There were 4 correct answers and Riley marked 1 of them, so I gave her 50%", they're going to question your math.

The first time I used multiple answers questions, I was shocked to find out it didn't work as six independent true false questions. The documentation was ambiguous and that's part of the reason I wrote this document.

You write, "At the very least, you should offer the option to grade the answers in a by-correct-response method."  I'm just a community member trying to explain how and why things work. I don't work for Instructure / Canvas and cannot change their product directly, but I have written a lot of tools for people to change little things.

One of those tools does just what you asked. I wrote QuizWiz, which does many things, but  happens to give people alternative ways to grade multiple answers questions. Even though I think correct percentage is not the right way to grade, I realized that there were people who wanted to use it. This document was a side-document I wrote when I created QuizWiz to help people understand the way that Canvas graded things the way they graded and provide them with an opportunity to grade the way they preferred.

As I mentioned, with new quizzes, you can use a stimulus question to get what you want.  You don't want to use multiple answers questions as they work the same way as classic quizzes, but they also allow for an all-or-nothing approach. With classic quizzes, you can create a text-only question and then use a question group to randomly select the true-false questions, but that won't work if you deliver the questions one question at a time. Canvas is not developing classic quizzes and has plans to deprecate them.

I haven't drank the kool-aid (Flavor Aid) since November 18, 1978. Unrelated to any childhood traumas, I haven't had a soda since March 15, 2019.

matthewthomas
Community Participant

This is a super helpful comparison of the different methods of marking these questions. Thanks you for this.

After teaching in Moodle for several years, with it's completely customizable multiple answers scoring, I hit on a method for scoring these that I really like. In sum, choosing one of the correct answers adds the appropriate portion of the point value for the question while choosing an incorrect answer subtracts the amount equivalent of the point value for the question spread over the total number of choices. 

Using your example, I would score the answers as such:

A B C D E F
3 3 -2 3 -2 3

 

Since there are 4 correct answers and 6 total answers, a correct response receives 3 points (12/4), whereas checking an incorrect response deducts 2 points (12/6). Thus, your hypothetical students would receive:

Name

Selected

All or nothing

% Correct

Right-Wrong

Canvas

Matt

Jordan

ABDF

12

12

12

12

12

Casey

ABEF

0

8

4

6

7

Peyton

ACE

0

2

0

0

0

Avery

ABCDEF

0

8

4

6

8

Riley

A

0

6

0

3

3

I like this scheme, because it penalizes for choosing incorrect responses, but those don't totally cancel out a correct response. It'd be great to have that option, but I am aware that this is probably a niche thought of how to do this.

James
Community Champion
Author

@matthewthomas 

Interesting take. I'm not sure how many people think that picking a wrong answer should count less than picking a right answer, but given how popular the other ideas were, it's probably greater than I imagine.

The code for QuizWiz is open source if you want to write a handler for that. I'm in the middle of a major refactor of QuizWiz, but I'm also in the middle of some other major refactors so who knows how long it will be before it's fixed. I'm working on separating the Quiz portion of QuizWiz from the SpeedGrader portion so that when Classic Quizzes are deprecated and we're forced to use New Quizzes, that people don't have to have such a large code base to get just the part they want.

Way back when New Quizzes was first being pushed (about four names ago), there was talk of letting the values for each specific value be specified. This might have only been for multiple choice questions instead of multiple answers questions, but it would have allowed for the ultimate freedom in awarding points.

AndreV
Community Explorer

It seems to me that Canvas' way of grading multiple answers questions is awful. I have a question with 5 possible answers; 3 should be selected, 2 shouldn't (no, yes, yes, yes, no). Let's say I allot 5 points to this question

If, hypothetically, a student selects all answers (yes, yes, yes, yes, yes). This student will have selected 3 correct answers and 2 wrong answers. Logically, it makes sense that the student should get 3/5 (correct answers grading scheme?). 

But Canvas gives this answer 1.67/5--counting 1.67 points x 3 per 'yes' answer, but -1.67 for wrongly selecting a 'no' answer. So 5 points minus (1.67 x2) = 1.67.

This is mind boggling and totally nonsensical. It's also a recipe for student mutiny. And there is no way to change this? At the very least Canvas should offer a few grading options like D2L. 

Those Canvas quizzes are incredibly clunky and maddening. 😞

James
Community Champion
Author

@AndreV 

The first time I used multiple answer questions, I thought the same thing that you did. Then I took the time to figure out what was happening and decided Canvas was doing it correctly for what the question is designed to be. When people try to use it in a different way than intended, then I agree, it doesn't make sense.

Let's say that a student doesn't check any of the boxes, which you would count as no, no, no, no, no. You get them 40% of the points for not answering the question at all? The student who gets confused and thinks there it is multiple choice and answers no, yes, no, no, no would get 60% of the points for only picking 1 of the 3 correct responses.

The thing to understand is that there is no way to for the student to say "no" to a multiple answers response. There is only a way for a student to say "yes." Leaving it blank is not the same as saying "no." They might not have answered because they didn't think it was an answer, but it might also be because they ran out of time, completely skipped the question, or got confused and thought it was multiple choice instead of multiple answers (despite the instructions you put in the question about being sure to select all correct responses).

Multiple answer questions are not five true-false questions with the same stem question. In that situation, a student is required to pick an answer or it is counted wrong. That's what you're wanting it to be, but that's not what it is. Once you realize what they are, understanding why they work the way they do makes more sense. Canvas does not award points for inaction on the student's part, they only look at what the student has checked.

The ability to have five true-false questions with a single stem is coming with New Quizzes and stimulus questions, but it does not exist with Classic Quizzes. With classic quizzes, you would have to offer five true-false questions and repeat the stem, or put in a no-text question with the stem and then put five true-false questions.

Currently, no-text questions used in this way are not transferred in a meaningful way into New Quizzes. They come through as a stimulus question stem, but no questions come with them and they don't display on the quiz because there are no questions to it. You would have to manually add the questions to the stimulus question after the migration.

AndreV
Community Explorer

@James 

I appreciate the thorough reply. I see what you're saying, but it's still not convincing to me. Students can always guess their answers in these objective tests (which is one of the reasons why I dislike them so much). True, they could get points for not checking any boxes at all. But they could also get points by mindlessly checking all boxes. It goes both ways. 

James
Community Champion
Author

@AndreV 

You just made the case for why Canvas grades the way it does. Awarding points for not answering and not taking points off for answering incorrectly are both flawed approaches for the way that Multiple Answers questions are designed and implemented.

Sure, people wish they worked other ways like a series of true-false questions, but they don't. Canvas isn't going to change that for Classic Quizzes as they are in the process of phasing those out and they're not doing any additional development for them.

At this point, if multiple answers questions don't work the way you want them to, then your only Canvas-native option is to not use them. An external solution is to use the QuizWiz script I wrote that provides alternatives for how multiple answer questions with Classic Quizzes are graded. You have to manually go through each quiz in SpeedGrader and apply the fix, which makes it difficult when you allow multiple attempts or keep the average (so that you need to regrade each attempt). There is a button I added that will allow you to quickly move through the quizzes, but you still have to click once for each student.

I really don't recommend using QuizWiz for the multiple answers question regrading because I think Canvas does it right. The exception is when you are required to grade them as All or Nothing and at this point, switching to New Quizzes is a better approach. But there are a lot of other enhancements in QuizWiz that make using SpeedGrader more efficient.

AndreV
Community Explorer

Okay, thanks. I will continue to battle this system and try to figure it out. Unfortunately my institution has not yet enabled the New Quizzes. I really miss giving quizzes on paper. So much more flexible and easy. This online teaching has become the bane of my existence and Canvas sure isn't making it any easier. 😞

ShannonHoste
Community Novice

@James I am still stuck. The discussion is super helpful but I'm struggling with the math. I have a 1 point quiz in canvas that is multiple answers. There are 4 answers and all 4 should be selected to be correct. With this when a student selects 3 of the 4 answers it results in a 0.67 answer. My model would expect 0.75 but, based on your discussion I understand why this is not the way it is calculated. However, with the negative points for incorrect answers then the grade should be 0.5 not 0.67. What am I missing?

James
Community Champion
Author

@ShannonHoste 

Best Guess

Is this Classic Quizzes and your question is part of a question group where each question is worth something other than 1 point? If so, what you specify for the question is irrelevant, it uses what the question group specifies for the points.

The second part of this is that you likely only have 3 of the 4 responses marked as correct.

If I had to put all the evidence together, I would suspect that this question is part of a question group worth 2 points each and that only 3 of the 4 questions are marked as correct in Canvas.

Long Response

The situation you've laid out has three conditions: (1) there are four correct responses to the question, (2) the question is worth 1 point, and (3) the student selected 3 of the 4 responses.

That combination of assumptions does not ever return 2/3=0.67 points. That means that one of the conditions is wrong.

Number of Responses

Four Correct Responses

If there are 4 correct responses, then each response becomes worth 1/4 point (assuming the point value is 1).

If the student marks 3 of the 4 correct responses, then they did not mark a correct response and did not mark any incorrect responses (since there are none). That means that they would have scored 3(1/4)+0(-1/4)=3/4 or 0.75 points.

In order for the student to score 2/3 points, then the question could not be worth 1 point, it would have had to be worth 8/9 of a point. If the question is worth 8/9 of a point, then each part would be worth 1/4(8/9)=2/9. Then the student would have gotten 3(2/9)=6/9=2/3 of a point.

It is very unlikely that you made the question worth 8/9=0.8888888... points. If you did, you would have had to enter 0.89 as the point value, then 3/4*0.89=0.6675=0.67, which is what you're seeing.

Getting 2/3 points with a 1 point question is only be possible if (1) not all responses are marked as correct or (2) the question isn't worth 1 point. I would double check both of those conditions and I suspect that both are issues.

3 Correct Responses

It is much more likely that you have three correct responses than making the question worth 0.89 points.

If the question is worth 1 point and there are 3 correct responses, then each selected correct response would be worth +1/3 and each selected incorrect response would be worth -1/3 point.

If the student answers three out of four responses, then there are two possibilities.

  • The three they selected were all correct and the one they didn't select was incorrect. That is a completely correct scoring. They would get 3(1/3)+0(-1/3)=1.
  • They selected two correct responses and one incorrect response. They would get 2(1/3)+1(-1/3)=1/3.

No other scorings are possible if there are three correct responses marked. 

Even if three of the responses are marked as correct instead of all four, it is still impossible to get 2/3 of a point if the question is worth 1 point and the student selected 3 out of 4 responses.

Thinking that you selected all four as correct when creating the question, but having accidentally only selected three is much more likely to happen than making a question worth 0.89 points.

If the question were worth 2 points, then you would double the outcomes so it would be possible to get 2 points or 2/3=0.67 points, the last one being what you're seeing.

Question Not Worth 1 Point

We have established that is impossible for a student to mark 3 out of 4 items and get 2/3=0.67 points.

That leaves us with a high probability that the question is not worth 1 point.

If the problem was worth 2 points and there three correct responses out of four and the student selected two correct and one incorrect response, then they could get 2/3=0.67 points.

If the question says it is worth 1 point, this may not be how much it is worth.

If the question is included in a question group, then all questions in the question group are worth the same amount of points and the point values on the individual questions are ignored. This is necessary to make each quiz have the same number of points no matter which questions were selected.

If the question says it's worth 1 point, then including it in a question group worth 2 points would make it worth 2 points.

This is the only scenario I've looked at that gets the 2/3 points. You have three responses marked as correct instead of four like you think you do and you have it in a question group where each question is worth 2 points.

Updating Question / Regrading

There is always the possibility that a change to the question or an attempt to regrade has impacted things.

The scoring method for a student's quiz is locked in when a student starts taking a quiz. If there were three correct responses when the student started the quiz and it was worth 2 points, then that's how that student would be scored.

If you realized the mistake and changed the question to have 4 correct responses and changed it to 1 point after the student had started the quiz, it wouldn't matter. For that student, it would still have 3 correct responses and be worth 2 points.

Any students that started the quiz after the change would get the question with the updated point values.

I want to believe that you would have mentioned changing the point values in the original question, so I don't hold this out as a likely possibility, but it is still a possibility.

You can regrade a multiple answers question with Classic Quizzes and there are lots of options about what to do when you regrade. However, that would likely lead to a feasible result and you would have remembered a regrade, so I do not think that is what is going on here.

Other Scenarios

There are other ways to get 2/3 point, such as 1/6 of the points in a 4 point question, but none of them seem reasonable and get further and further away from what you have described what you think the situation is.

If what I've described doesn't fix the issue, then I would need more information to figure out what's going on.

It is always possible that there is a miscalculation on the Canvas side of things, but I haven't seen it in this situation.

ShannonHoste
Community Novice

@James Thank you so much. Your walk thru really helps me understand how this works. You are correct, upon investigation one of answers was marked 'possible answer' and 3 were correct. My assumption was that those were all counted at .25.

joel_ostblom
Community Member

This is a very useful post, thanks for putting it together @James . I would still disagree with the idea that the current format of multiple answers grading is suitable or fair, and at the very least I think Canvas should offer the option to instructors to choose to give marks according to the proportion of correctly checked and correctly unchecked boxes.

As for one of the earlier comments that "Canvas does not award points for inaction on the student's part, they only look at what the student has checked.", I think Canvas actually currently does reward student inaction since students will be penalized for taking action, but not for inaction. So if a student is 50% sure an option is correct, they should definitely not check the box as it could negate their previous correct answers. It now becomes a game theory challenge for the student to figure how sure they should be before checking a box where each student also need to figure out how risk averse they are and how much they would like to gamble in getting more points. It would be much more straightforward to simply check the box if the certainty was >50% and only risk losing out point for that particular option in case the correct answer was.

Even if a student run out of time, I don't really have an issue with them getting some points for unchecked boxes, since they could have blindly checked boxes in a few seconds and it doesn't seem like that behavior is more worthy of reward than just leaving them blank. But if Canvas truly think this is an issue, then they could make multiple answer work more like T/F and have "Unchecked", "True", "False" as the option. Note that T/F is currently not a replacement for multiple answers as it created many more separate questions that are both harder to keep track of in question banks and that cannot be meaningfully grouped while still randomizing the order of questions and answer in the quiz in general.

There is another thread here asking for this feature to be implementedhttps://community.canvaslms.com/t5/Idea-Conversations/Partial-Credit-for-Multiple-Answers-WITHOUT-Pe...