This document explains what Multiple Answers questions are, how partial credit is computed for these questions, why Canvas computes partial credit the way it does, and compares the multiple answers offerings of other Learning Management Systems.
What is a Multiple Answers Question?
A Multiple Answers question is one that allows for selection of multiple answers. In the parlance of HTML forms, it uses checkboxes (stylized as squares) instead of radio-buttons (stylized as circles).
Depending on the learning management system, it is sometimes called multiple answer, multiple response, multiple select, or multi-select. Others consider them part of multiple choice questions.
Within a quiz, they are displayed like this:
Other Question Types with Partial Credit
In Canvas, there are three types of automatically-graded questions that allow partial credit: Fill in Multiple Blanks questions, Multiple Dropdown questions, and Multiple Answers questions.
The Fill in Multiple Blanks questions, which also called Multiple Fill-in-the-Blank, and the Multiple Dropdown questions assign points by dividing the total number of points by the number of responses and then awarding points for each correct response. This is a fairly standard way of assigning partial credit and most people understand how points are awarded.
If a 6 point question has 3 parts, then each part is worth 2 points. If a student correctly answers 2 of the 3 parts, then they earn 4 points, which is two-thirds of the possible 6 points.
What's the Big Deal about Multiple Answers?
Multiple Answers questions are graded by a completely different set of rules. By itself, that contributes to some of the confusion. Throwing aside whether it's right or wrong, it's not consistent with the other two types of questions that allow for partial credit.
Some see that as an immediate "it's wrong" while others see it as an "it's an opportunity".
The trouble stems from a little note at the bottom of one of the Canvas Guide pages: How do I create a Multiple Answers quiz question?
Note: This question type awards a fraction of the points available for each correct answer selection and subtracts an equivalent fraction for incorrect answer selection.
I'm not sure whether the trouble is because it's a little note at the bottom, because of the way it works, or because it's not the way people expect it to work. It may be a little of all three.
As a math teacher, I actually understand that sentence and have had my students take quizzes on "How to take Canvas quizzes" so that they would understand it and be able to work it to their advantage. However many people do not understand what that means. When faculty don't understand it, they can't explain it to the students who don't understand it. When people don't understand something, they sometimes bad-mouth it, call it wrong, and demand it be fixed. That doesn't necessarily mean it's broken, but that it needs better explained.
Here are some of those conversations in the Community. There are many others, this is just a sampling to show that the topic comes up frequently.
Ask a Question
- Problem with Multiple Correct question type in quizzes tool
- Multiple Points on quiz-students earning zero instead of partial credit
Feature Requests
- Multiple Answer quiz question - partial points without penalty (174 votes and currently in phase 3.5 under consideration) with a note that in Canvas Studio: Modern Quizzing Engine there will be full credit, partial credit, no credit, or negative credit allowed.
- All or No Points on Multiple Answer Questions (107 votes before being archived)
- Two Industry-approved ways to score Multiple Answer - neither are the current way (archived and moved into Modern Quizzing Engine)
- Multiple Response Questions where it's "All or None" (NO partial credit) (marked as duplicate)
- Multiple Response Grading (marked as duplicate)
- Full Points for Any One of Multiple Correct Answers (archived with 16 votes)
Ways to Assign Partial Credit
I took a look at some of the major players in the Learning Management System (LMS) game and found that there were essentially two ways of assigning partial credit. Note: "All or nothing" isn't really partial credit.
- Assign points to each response and then add the points assigned to the checked items to get the points awarded for each question.
- Define a correct response as selecting an item that should be checked or not selecting an item that should not be checked.
There are variations on these themes. Some take the difference between the number of correct marked responses and the number of incorrectly marked responses. Some make sure that the point values never go negative. But essentially, the schemes awarding partial credit boiled down to those two techniques.
The Right Way?
When assigning partial credit, people's understanding of "the right way" depends a lot on what learning management system they're coming from. For these, "the right way" often translates to "the way I've always done it."
Stepping back, the real issue is the assumptions that you're working under.
Assigning Points to Individual Responses
If you are used to assigning points to individual responses, then you probably work under these assumptions.
- If a student selects all of the items that should be checked, then they should get the full points possible.
- If a student selects some of the items that should not be checked, then they should be penalized. Some call this a penalty for guessing. This means that assign a negative point value for certain responses.
- The total points earned for a question is the sum of the points assigned to each item that the student selects.
- The lowest score a student can get for a question is 0. Although individual responses may have negative points associated with them, if the student misses more than they get right, then they should get a 0 for the question, not a negative score.
Defining Correct Responses
If you are coming from a system that awarded points based on correctly selecting (or not selecting) items, then you probably work under these assumptions.
- Students are awarded points based on the percentage of questions that are correctly marked. If they correctly mark 80% of the questions, then they should get 80% of the possible points.
- Each item is weighted equally. Without the ability to go through and assign points to each individual response, there is no way to tell the system, for example, that items that should be selected count twice as much as items that should not be selected. If you don't enter points to each response, then you don't have control over how much each part is worth.
- No matter how badly a student does, they won't lose points.
Example Scenario
Before you can really understand how Canvas grades Multiple Answers questions, you need to complete a little exercise.
Here are the assumptions that we are going to work under.
- You will be assigning points to individual responses (this is the first approach).
- Students should get the full points if they correctly select all items that should be checked and do not select any items that should not be selected
- Students should incur a penalty for selecting items that should not be checked.
- Students receive the sum of the points assigned to the items the student selects
- You have no idea as to the level of correctness of each response. All that you know is that an item should either be selected or it shouldn't be selected (this is similar to the second approach). This also means that a student should lose as much for a choosing a wrong response that they earn for choosing a correct response.
You have a Multiple Answers question that is worth 12 points. There are 6 possible choices: 4 choices that are correct and 2 distractors that are incorrect.
You need to assign points to each item. Remember that points for each item may be positive or negative and that the final score will be the sum of the points for each item that is selected.
Without loss of generality, let's say that the first, second, fourth, and sixth items are the correct ones that should be selected and the third and fifth are the incorrect ones that should not be selected. Furthermore, we'll label the responses as A, B, C, D, E, and F for simplicity.
Challenge: Complete the table below by assigning points to each of the six items. Don't skip ahead to the answer until you've attempted it on your own.
Item | A | B | C | D | E | F |
Correct? | yes | yes | no | yes | no | yes |
Points |
Incorrect Point Distributions
Before we look at the correct answer, let's look at some incorrect ones.
Correct? | yes | yes | no | yes | no | yes | Reason Why Invalid |
Points | 2 | 2 | 0 | 2 | 0 | 2 | Only 8 points possible |
Points | 3 | 3 | 0 | 3 | 0 | 3 | No penalty for guessing |
Points | 3 | 3 | -1 | 3 | -1 | 3 | You don't know levels of right or wrong |
Points | 2 | 2 | 2 | 2 | 2 | 2 | Awarding points for incorrect answers |
Some people might have suggested the last scheme for a different reason. If you are used to defining correctness based on selecting a correct item and not selecting an incorrect item, then it makes sense. However, in this scenario, the points awarded is the sum of the points for each selected item, not the sum of the points for each correctly answered item. If you don't penalize and use this method instead, then a student could simply check all of the boxes and get 100% every time.
Correct Point Distribution
If you understood and followed the assumptions and instructions, then the correct distribution of point values is given in the table below.
Item | A | B | C | D | E | F |
Correct? | yes | yes | no | yes | no | yes |
Points | 3 | 3 | -3 | 3 | -3 | 3 |
Here's the explanation of how those values were arrived at.
- There are 12 points possible and 4 correct answers. If the student is to get full credit for correctly selecting all 4 of the items that should be selected, then each correct response should be worth 12 / 4 = 3 points.
- Students should lose as much for a wrong answer as they get for a correct answer, the points assigned to each incorrect response is -3.
The example scenario we just worked through and the calculations that we just came up with is exactly how Canvas grades the Multiple Answers questions. You tell it how many points the question is worth and which items are correct and it does all of the calculations for you.
Let's look at that note from the Canvas Guides again.
Note: This question type awards a fraction of the points available for each correct answer selection and subtracts an equivalent fraction for incorrect answer selection.
As a mathematician, I will admit that statement is insufficient to explain the situation. A minor tweak that would help clarify what is meant would be to insert the word "equal" so that "a fraction of the points" becomes "an equal fraction of the points."
When Canvas says, "This question type awards a fraction of the points available for each correct answer selection" what they mean is "Divide the possible points by the number of items that should be selected to find out how many points to award for each correctly selected item." This is necessary to ensure that a student who gets correctly selects everything gets full credit.
When Canvas says, "and subtracts an equivalent fraction for incorrect answer selection" what they mean is "a student loses the same amount for marking a wrong answer that they get for marking a correct answer."
What their page doesn't say is that a student's score won't go negative and the minimum score for the question is 0.
It also doesn't explain why that process is used or that it is exactly what you would do given the assumptions listed at the start of this scenario. Canvas determines the points by adding the points assigned to each selected response. It makes sure the student gets full points for correctly selecting what should be checked, it takes away an equal amount of points for each incorrectly selected response, and doesn't let the score go negative.
Probably the most controversial part of the scenario is that you don't know the level of correctness of the responses. The teacher, who is a content expert, has the knowledge and understanding to assign different point values to each individual response. Most faculty probably would not use that level of granularity and Canvas simplifies things by automatically assigning the points for the user.
Others would say that the most controversial part is how you define "correct" and it should be based on selecting a correct item or not selecting an incorrect item. That is the other major approach to grading multiple answers questions. That doesn't apply to this scenario because Canvas adopted the principle of totaling the points for each selected response.
Example Students
Let's continue with the example scenario and see how some fictitious students would fare. The names of the students were chosen based on the list of popular baby names in 1998 that is published by the Social Security Administration. The names were chosen from those appearing in relatively the same position in both the list of male and female names so that the gender of the names could not be used to imply intelligence.
To summarize: there are 12 possible points, responses A, B, D, and F are correct, responses C and E are incorrect, each selected correct response earns 3 points, each selected incorrect response costs 3 points, unselected items are ignored, the score for the question is the sum of the points for all selected responses, and the score may not be negative.
The table shows the items each of our five students selected, the points obtained from each response, and the total score for the question.
Name | Selected | A | B | C | D | E | F | Pts | Notes |
Jordan | ABDF | 3 | 3 | 3 | 3 | 12 | Only checked correct responses | ||
Casey | ABEF | 3 | 3 | -3 | 3 | 6 | 3 right, 1 wrong | ||
Peyton | ACE | 3 | -3 | -3 | 0 | 1 right, 2 wrong. Minimum score of 0 | |||
Avery | ABCDEF | 3 | 3 | -3 | 3 | -3 | 3 | 6 | Checked all responses |
Riley | A | 3 | 3 | Thought it was multiple choice |
Other Learning Management Systems
What follows is a summary of what other Learning Management Systems do for their multiple select questions. This list of LMSs is not meant to be inclusive, but is based off the Spring 2016 update to the Edutechnia LMS Data report.
I've never used any other LMS besides Canvas, so all of the descriptions are based off of descriptions from those LMSs or universities using those LMSs.
Angel
Angel isn't really a big player anymore as Blackboard has announced its end of life in October 2016, but many people coming to Canvas may be former Angel users.
Penn State has a help page for the Angel LMS called Add a Multiple Select Question to an Assessment that gives the following admonition.
Multiple select questions that do not provide negative values for incorrect answers provide no penalty to students who select one or all of the incorrect answers. In this case, a student who indiscriminately selects ALL of the available choices (correct and incorrect) would receive the same full credit for the question as a student who answered it absolutely correctly.
Later on the page, there are two methods of grading, which are verbatim from the Angel 8 Instructor's Reference Manual.
Normal: Grading is based on the sum of the point values assigned to each correct choice selected by the student. Students are automatically penalized for selecting incorrect answers, but the final score for the question cannot be less than 0 points.
All or nothing: The student is given full credit for an exactly correct answer. Partially correct answers are given a zero for the question.
The Normal way that Angel graded Multiple Select answers is exactly the same way that Canvas grades Multiple Answers questions, with the exception that Canvas automatically assigns the points for you, including awarding negative point values to the incorrect answers.
Blackboard Learn
The Blackboard Learn LMS allows for negative points on matching, multiple answer, and multiple choice questions whereas Canvas uses it only for Multiple Answers questions. The Blackboard manual says to use negative credit to discourage guessing. However, the option was under question settings and you had to specify both partial credit and negative points separately and you had to specify it as a percentage rather than a raw point value. The default was all or nothing and instructors really had to do a bunch of configuration to set Blackboard up to award partial credit, so many faculty may not have known it was there and just left it at the default. The differences are that Canvas removes all the decision making from the process and automatically assigns points and that you cannot get a negative score with Canvas.
Blackboard Learn allows too many variations and requires the instructor to decide what percentages to award or deduct for incorrect responses, so I can't give a clear example here.
D2L
The D2L LMS allows for three ways of grading multi-select questions: all or nothing, right minus wrong, or correct answers.
The all or nothing method is pretty clear -- no partial credit is given.
The correct answers approach awards points based on the number of items correctly selected, not the number of correct items that are selected. This means that selecting a correct answer or not selecting an incorrect answer is considered to be correct or right, but selecting an incorrect answer or not selecting a correct answer is considered to be incorrect or wrong. Each answer is either correctly marked or incorrectly marked and so you take the total points for the question and multiply it by the fraction of the questions that were correctly marked.
This is probably the most common interpretation of partial credit for a multiple select question because it's the way the other partial credit questions work in Canvas.
Let's revisit our example students and see how they would fare using this approach. There are 12 points and 6 responses, so each item is worth 12/6=2 points. The student earns 2 points for correctly marking the item and 0 points if the item is not correctly marked. Items A, B, D, and F should be selected while items C and E should not be selected. To simplify things, we'll show R for right answers and W for wrong answers and then multiply the number of right answers by 2.
Name | Selected | A | B | C | D | E | F | Pts | Notes |
Jordan | ABDF | R | R | R | R | R | R | 12 | Only checked correct responses |
Casey | ABEF | R | R | R | W | W | R | 8 | Correctly marked 4 items |
Peyton | ACE | R | W | W | W | W | W | 2 | Correctly marked only item A |
Avery | ABCDEF | R | R | W | R | W | R | 8 | Checked all responses |
Riley | A | R | W | R | W | R | W | 6 | Thought it was multiple choice |
The right minus wrong approach is the more complicated one. The manual says:
Users receive points equal to the number of right answers they choose minus the number of incorrect answers they choose. To determine how much each answer is worth, the system takes the total number of points assigned to the question and divides it by the total number of answer choices. … Users can receive a minimum of zero on a question: they cannot receive a negative mark
Let's see how our five students would fare. This time, their score is difference between the number of items answered correctly and the number of items answered incorrectly: the number right minus the number wrong.
Name | Selected | A | B | C | D | E | F | Right | Wrong | Difference | Pts |
Jordan | ABDF | R | R | R | R | R | R | 6 | 0 | 6 | 12 |
Casey | ABEF | R | R | R | W | W | R | 4 | 2 | 2 | 4 |
Peyton | ACE | R | W | W | W | W | W | 1 | 5 | -4 | 0 |
Avery | ABCDEF | R | R | W | R | W | R | 4 | 2 | 2 | 4 |
Riley | A | R | W | R | W | R | W | 3 | 3 | 0 | 0 |
Note that Peyton received a 0 instead of -8 because negative scores are not allowed.
Moodle
Moodle treats multiple-answer questions as a variation on the multiple choice question type. They have two sentences that explain how the grading works.
Each answer may carry a positive or negative grade, so that choosing ALL the options will not necessarily result in good grade. If the total grade is negative then the total grade for this question will be zero.
This is more flexible than what Canvas does, but requires additional configuration. Canvas automates the decision making about how many points to award for each choice, but the overall score is the sum of the points for the selected items.
Moodle does not natively support all or nothing questions, but there is an All-or-Nothing Multiple Choice plugin available that will extend the capability.
Pearson
Pearson is another player that has decided to get out of the LMS market. They have a couple of different LMS products, LearningStudio and OpenClass.
LearningStudio provides the infrastructure for MyLabs & Mastering, which will remain,
Learning Studio grades multiple answer questions based on correctly marking the individual items. Their developer site calls them Many Multiple-Choice question types and wording suggests it's an all or nothing approach. Another document says that you can award partial credit for the multiple answer questions that is based on correctly selecting right answers and not selecting wrong answers that appears to be based off the fraction of correctly marked responses.
Sakai
Sakai allows for a multiple correct, multiple selection variation of the multiple choice question. There are two grading methods in Sakai, Right Less Wrong and All or Nothing, both of which mirror those found in D2L. This means that they are based on correctly marking individual items rather than the number of correct choices.
Method Comparison
Let's see how our five students would do under the different approaches.
Remember that items A, B, D, and F should be selected while items C and E should not be selected. There are 12 points possible.
Name | Selected | All or nothing | % Correct | Right-Wrong | Canvas |
Jordan | ABDF | 12 | 12 | 12 | 12 |
Casey | ABEF | 0 | 8 | 4 | 6 |
Peyton | ACE | 0 | 2 | 0 | 0 |
Avery | ABCDEF | 0 | 8 | 4 | 6 |
Riley | A | 0 | 6 | 0 | 3 |
Bear in mind that this is a single example. The results will vary depending on the number of responses, the number of correct responses, and the combination of items chosen by the students.
Summary
There are a combination of methods for grading multiple answers questions.
- All or nothing: Provided by Angel, D2L, Blackboard, Sakai, and Pearson, but not Canvas. Moodle requires a plugin to support this approach.
- Partial credit based on the percentage of correctly marked items: Provided by D2L and Pearson.
- Partial credit based on a difference: The way that difference is defined varies depending on the LMS. Provided by Canvas, Angel, D2L, and Sakai.
- Full control over the point value assigned to each individual response: Provided by Angel, Blackboard, and Moodle.
LMS | All or nothing | % Correct | Difference | Point Values |
Canvas | X | |||
Angel | X | X | X | |
Blackboard | X | X | ||
D2L | X | X | X | |
Moodle | plugin | X | ||
Pearson | X | X | ||
Sakai | X | X |
The Future
Canvas is working on revamping quizzes with their Modern Quizzing Engine. I've heard some really awesome things about it that should fix most of the concerns of people regarding how Canvas grades Multiple Answers questions
In the meantime, Avi Naiman and James Jones have have developed a user script called QuizWiz that will add "all or nothing" and "% correct" grading to Canvas. This will be a user script add-on to SpeedGrader that will allow you to click one button and regrade a quiz submission. There are a lot more features, but that is the portion that is germane to this document. See QuizWiz: Enhancements to SpeedGrader and Quizzes for more information.
