This document explains what Multiple Answers questions are, how partial credit is computed for these questions, why Canvas computes partial credit the way it does, and compares the multiple answers offerings of other Learning Management Systems. What is a Multiple Answers Question? A Multiple Answers question is one that allows for selection of multiple answers. In the parlance of HTML forms, it uses checkboxes (stylized as squares) instead of radio-buttons (stylized as circles). Depending on the learning management system, it is sometimes called multiple answer, multiple response, multiple select, or multi-select. Others consider them part of multiple choice questions. Within a quiz, they are displayed like this: Other Question Types with Partial Credit In Canvas, there are three types of automatically-graded questions that allow partial credit: Fill in Multiple Blanks questions, Multiple Dropdown questions, and Multiple Answers questions. The Fill in Multiple Blanks questions, which also called Multiple Fill-in-the-Blank, and the Multiple Dropdown questions assign points by dividing the total number of points by the number of responses and then awarding points for each correct response. This is a fairly standard way of assigning partial credit and most people understand how points are awarded. If a 6 point question has 3 parts, then each part is worth 2 points. If a student correctly answers 2 of the 3 parts, then they earn 4 points, which is two-thirds of the possible 6 points. What's the Big Deal about Multiple Answers? Multiple Answers questions are graded by a completely different set of rules. By itself, that contributes to some of the confusion. Throwing aside whether it's right or wrong, it's not consistent with the other two types of questions that allow for partial credit. Some see that as an immediate "it's wrong" while others see it as an "it's an opportunity". The trouble stems from a little note at the bottom of one of the Canvas Guide pages: How do I create a Multiple Answers quiz question? Note: This question type awards a fraction of the points available for each correct answer selection and subtracts an equivalent fraction for incorrect answer selection. I'm not sure whether the trouble is because it's a little note at the bottom, because of the way it works, or because it's not the way people expect it to work. It may be a little of all three. As a math teacher, I actually understand that sentence and have had my students take quizzes on "How to take Canvas quizzes" so that they would understand it and be able to work it to their advantage. However many people do not understand what that means. When faculty don't understand it, they can't explain it to the students who don't understand it. When people don't understand something, they sometimes bad-mouth it, call it wrong, and demand it be fixed. That doesn't necessarily mean it's broken, but that it needs better explained. Here are some of those conversations in the Community. There are many others, this is just a sampling to show that the topic comes up frequently. Ask a Question Problem with Multiple Correct question type in quizzes tool Multiple Points on quiz-students earning zero instead of partial credit Feature Requests Multiple Answer quiz question - partial points without penalty (174 votes and currently in phase 3.5 under consideration) with a note that in Canvas Studio: Modern Quizzing Engine there will be full credit, partial credit, no credit, or negative credit allowed. All or No Points on Multiple Answer Questions (107 votes before being archived) Two Industry-approved ways to score Multiple Answer - neither are the current way (archived and moved into Modern Quizzing Engine) Multiple Response Questions where it's "All or None" (NO partial credit) (marked as duplicate) Multiple Response Grading (marked as duplicate) Full Points for Any One of Multiple Correct Answers (archived with 16 votes) Ways to Assign Partial Credit I took a look at some of the major players in the Learning Management System (LMS) game and found that there were essentially two ways of assigning partial credit. Note: "All or nothing" isn't really partial credit. Assign points to each response and then add the points assigned to the checked items to get the points awarded for each question. Define a correct response as selecting an item that should be checked or not selecting an item that should not be checked. There are variations on these themes. Some take the difference between the number of correct marked responses and the number of incorrectly marked responses. Some make sure that the point values never go negative. But essentially, the schemes awarding partial credit boiled down to those two techniques. The Right Way? When assigning partial credit, people's understanding of "the right way" depends a lot on what learning management system they're coming from. For these, "the right way" often translates to "the way I've always done it." Stepping back, the real issue is the assumptions that you're working under. Assigning Points to Individual Responses If you are used to assigning points to individual responses, then you probably work under these assumptions. If a student selects all of the items that should be checked, then they should get the full points possible. If a student selects some of the items that should not be checked, then they should be penalized. Some call this a penalty for guessing. This means that assign a negative point value for certain responses. The total points earned for a question is the sum of the points assigned to each item that the student selects. The lowest score a student can get for a question is 0. Although individual responses may have negative points associated with them, if the student misses more than they get right, then they should get a 0 for the question, not a negative score. Defining Correct Responses If you are coming from a system that awarded points based on correctly selecting (or not selecting) items, then you probably work under these assumptions. Students are awarded points based on the percentage of questions that are correctly marked. If they correctly mark 80% of the questions, then they should get 80% of the possible points. Each item is weighted equally. Without the ability to go through and assign points to each individual response, there is no way to tell the system, for example, that items that should be selected count twice as much as items that should not be selected. If you don't enter points to each response, then you don't have control over how much each part is worth. No matter how badly a student does, they won't lose points. Example Scenario Before you can really understand how Canvas grades Multiple Answers questions, you need to complete a little exercise. Here are the assumptions that we are going to work under. You will be assigning points to individual responses (this is the first approach). Students should get the full points if they correctly select all items that should be checked and do not select any items that should not be selected Students should incur a penalty for selecting items that should not be checked. Students receive the sum of the points assigned to the items the student selects You have no idea as to the level of correctness of each response. All that you know is that an item should either be selected or it shouldn't be selected (this is similar to the second approach). This also means that a student should lose as much for a choosing a wrong response that they earn for choosing a correct response. You have a Multiple Answers question that is worth 12 points. There are 6 possible choices: 4 choices that are correct and 2 distractors that are incorrect. You need to assign points to each item. Remember that points for each item may be positive or negative and that the final score will be the sum of the points for each item that is selected. Without loss of generality, let's say that the first, second, fourth, and sixth items are the correct ones that should be selected and the third and fifth are the incorrect ones that should not be selected. Furthermore, we'll label the responses as A, B, C, D, E, and F for simplicity. Challenge: Complete the table below by assigning points to each of the six items. Don't skip ahead to the answer until you've attempted it on your own. Item A B C D E F Correct? yes yes no yes no yes Points Incorrect Point Distributions Before we look at the correct answer, let's look at some incorrect ones. Correct? yes yes no yes no yes Reason Why Invalid Points 2 2 0 2 0 2 Only 8 points possible Points 3 3 0 3 0 3 No penalty for guessing Points 3 3 -1 3 -1 3 You don't know levels of right or wrong Points 2 2 2 2 2 2 Awarding points for incorrect answers Some people might have suggested the last scheme for a different reason. If you are used to defining correctness based on selecting a correct item and not selecting an incorrect item, then it makes sense. However, in this scenario, the points awarded is the sum of the points for each selected item, not the sum of the points for each correctly answered item. If you don't penalize and use this method instead, then a student could simply check all of the boxes and get 100% every time. Correct Point Distribution If you understood and followed the assumptions and instructions, then the correct distribution of point values is given in the table below. Item A B C D E F Correct? yes yes no yes no yes Points 3 3 -3 3 -3 3 Here's the explanation of how those values were arrived at. There are 12 points possible and 4 correct answers. If the student is to get full credit for correctly selecting all 4 of the items that should be selected, then each correct response should be worth 12 / 4 = 3 points. Students should lose as much for a wrong answer as they get for a correct answer, the points assigned to each incorrect response is -3. The example scenario we just worked through and the calculations that we just came up with is exactly how Canvas grades the Multiple Answers questions. You tell it how many points the question is worth and which items are correct and it does all of the calculations for you. Let's look at that note from the Canvas Guides again. Note: This question type awards a fraction of the points available for each correct answer selection and subtracts an equivalent fraction for incorrect answer selection. As a mathematician, I will admit that statement is insufficient to explain the situation. A minor tweak that would help clarify what is meant would be to insert the word "equal" so that "a fraction of the points" becomes "an equal fraction of the points." When Canvas says, "This question type awards a fraction of the points available for each correct answer selection" what they mean is "Divide the possible points by the number of items that should be selected to find out how many points to award for each correctly selected item." This is necessary to ensure that a student who gets correctly selects everything gets full credit. When Canvas says, "and subtracts an equivalent fraction for incorrect answer selection" what they mean is "a student loses the same amount for marking a wrong answer that they get for marking a correct answer." What their page doesn't say is that a student's score won't go negative and the minimum score for the question is 0. It also doesn't explain why that process is used or that it is exactly what you would do given the assumptions listed at the start of this scenario. Canvas determines the points by adding the points assigned to each selected response. It makes sure the student gets full points for correctly selecting what should be checked, it takes away an equal amount of points for each incorrectly selected response, and doesn't let the score go negative. Probably the most controversial part of the scenario is that you don't know the level of correctness of the responses. The teacher, who is a content expert, has the knowledge and understanding to assign different point values to each individual response. Most faculty probably would not use that level of granularity and Canvas simplifies things by automatically assigning the points for the user. Others would say that the most controversial part is how you define "correct" and it should be based on selecting a correct item or not selecting an incorrect item. That is the other major approach to grading multiple answers questions. That doesn't apply to this scenario because Canvas adopted the principle of totaling the points for each selected response. Example Students Let's continue with the example scenario and see how some fictitious students would fare. The names of the students were chosen based on the list of popular baby names in 1998 that is published by the Social Security Administration. The names were chosen from those appearing in relatively the same position in both the list of male and female names so that the gender of the names could not be used to imply intelligence. To summarize: there are 12 possible points, responses A, B, D, and F are correct, responses C and E are incorrect, each selected correct response earns 3 points, each selected incorrect response costs 3 points, unselected items are ignored, the score for the question is the sum of the points for all selected responses, and the score may not be negative. The table shows the items each of our five students selected, the points obtained from each response, and the total score for the question. Name Selected A B C D E F Pts Notes Jordan ABDF 3 3 3 3 12 Only checked correct responses Casey ABEF 3 3 -3 3 6 3 right, 1 wrong Peyton ACE 3 -3 -3 0 1 right, 2 wrong. Minimum score of 0 Avery ABCDEF 3 3 -3 3 -3 3 6 Checked all responses Riley A 3 3 Thought it was multiple choice Other Learning Management Systems What follows is a summary of what other Learning Management Systems do for their multiple select questions. This list of LMSs is not meant to be inclusive, but is based off the Spring 2016 update to the Edutechnia LMS Data report. I've never used any other LMS besides Canvas, so all of the descriptions are based off of descriptions from those LMSs or universities using those LMSs. Angel Angel isn't really a big player anymore as Blackboard has announced its end of life in October 2016, but many people coming to Canvas may be former Angel users. Penn State has a help page for the Angel LMS called Add a Multiple Select Question to an Assessment that gives the following admonition. Multiple select questions that do not provide negative values for incorrect answers provide no penalty to students who select one or all of the incorrect answers. In this case, a student who indiscriminately selects ALL of the available choices (correct and incorrect) would receive the same full credit for the question as a student who answered it absolutely correctly. Later on the page, there are two methods of grading, which are verbatim from the Angel 8 Instructor's Reference Manual. Normal: Grading is based on the sum of the point values assigned to each correct choice selected by the student. Students are automatically penalized for selecting incorrect answers, but the final score for the question cannot be less than 0 points. All or nothing: The student is given full credit for an exactly correct answer. Partially correct answers are given a zero for the question. The Normal way that Angel graded Multiple Select answers is exactly the same way that Canvas grades Multiple Answers questions, with the exception that Canvas automatically assigns the points for you, including awarding negative point values to the incorrect answers. Blackboard Learn The Blackboard Learn LMS allows for negative points on matching, multiple answer, and multiple choice questions whereas Canvas uses it only for Multiple Answers questions. The Blackboard manual says to use negative credit to discourage guessing. However, the option was under question settings and you had to specify both partial credit and negative points separately and you had to specify it as a percentage rather than a raw point value. The default was all or nothing and instructors really had to do a bunch of configuration to set Blackboard up to award partial credit, so many faculty may not have known it was there and just left it at the default. The differences are that Canvas removes all the decision making from the process and automatically assigns points and that you cannot get a negative score with Canvas. Blackboard Learn allows too many variations and requires the instructor to decide what percentages to award or deduct for incorrect responses, so I can't give a clear example here. D2L The D2L LMS allows for three ways of grading multi-select questions: all or nothing, right minus wrong, or correct answers. The all or nothing method is pretty clear -- no partial credit is given. The correct answers approach awards points based on the number of items correctly selected, not the number of correct items that are selected. This means that selecting a correct answer or not selecting an incorrect answer is considered to be correct or right, but selecting an incorrect answer or not selecting a correct answer is considered to be incorrect or wrong. Each answer is either correctly marked or incorrectly marked and so you take the total points for the question and multiply it by the fraction of the questions that were correctly marked. This is probably the most common interpretation of partial credit for a multiple select question because it's the way the other partial credit questions work in Canvas. Let's revisit our example students and see how they would fare using this approach. There are 12 points and 6 responses, so each item is worth 12/6=2 points. The student earns 2 points for correctly marking the item and 0 points if the item is not correctly marked. Items A, B, D, and F should be selected while items C and E should not be selected. To simplify things, we'll show R for right answers and W for wrong answers and then multiply the number of right answers by 2. Name Selected A B C D E F Pts Notes Jordan ABDF R R R R R R 12 Only checked correct responses Casey ABEF R R R W W R 8 Correctly marked 4 items Peyton ACE R W W W W W 2 Correctly marked only item A Avery ABCDEF R R W R W R 8 Checked all responses Riley A R W R W R W 6 Thought it was multiple choice The right minus wrong approach is the more complicated one. The manual says: Users receive points equal to the number of right answers they choose minus the number of incorrect answers they choose. To determine how much each answer is worth, the system takes the total number of points assigned to the question and divides it by the total number of answer choices. … Users can receive a minimum of zero on a question: they cannot receive a negative mark Let's see how our five students would fare. This time, their score is difference between the number of items answered correctly and the number of items answered incorrectly: the number right minus the number wrong. Name Selected A B C D E F Right Wrong Difference Pts Jordan ABDF R R R R R R 6 0 6 12 Casey ABEF R R R W W R 4 2 2 4 Peyton ACE R W W W W W 1 5 -4 0 Avery ABCDEF R R W R W R 4 2 2 4 Riley A R W R W R W 3 3 0 0 Note that Peyton received a 0 instead of -8 because negative scores are not allowed. Moodle Moodle treats multiple-answer questions as a variation on the multiple choice question type. They have two sentences that explain how the grading works. Each answer may carry a positive or negative grade, so that choosing ALL the options will not necessarily result in good grade. If the total grade is negative then the total grade for this question will be zero. This is more flexible than what Canvas does, but requires additional configuration. Canvas automates the decision making about how many points to award for each choice, but the overall score is the sum of the points for the selected items. Moodle does not natively support all or nothing questions, but there is an All-or-Nothing Multiple Choice plugin available that will extend the capability. Pearson Pearson is another player that has decided to get out of the LMS market. They have a couple of different LMS products, LearningStudio and OpenClass. LearningStudio provides the infrastructure for MyLabs & Mastering, which will remain, Learning Studio grades multiple answer questions based on correctly marking the individual items. Their developer site calls them Many Multiple-Choice question types and wording suggests it's an all or nothing approach. Another document says that you can award partial credit for the multiple answer questions that is based on correctly selecting right answers and not selecting wrong answers that appears to be based off the fraction of correctly marked responses. Sakai Sakai allows for a multiple correct, multiple selection variation of the multiple choice question. There are two grading methods in Sakai, Right Less Wrong and All or Nothing, both of which mirror those found in D2L. This means that they are based on correctly marking individual items rather than the number of correct choices. Method Comparison Let's see how our five students would do under the different approaches. Remember that items A, B, D, and F should be selected while items C and E should not be selected. There are 12 points possible. Name Selected All or nothing % Correct Right-Wrong Canvas Jordan ABDF 12 12 12 12 Casey ABEF 0 8 4 6 Peyton ACE 0 2 0 0 Avery ABCDEF 0 8 4 6 Riley A 0 6 0 3 Bear in mind that this is a single example. The results will vary depending on the number of responses, the number of correct responses, and the combination of items chosen by the students. Summary There are a combination of methods for grading multiple answers questions. All or nothing: Provided by Angel, D2L, Blackboard, Sakai, and Pearson, but not Canvas. Moodle requires a plugin to support this approach. Partial credit based on the percentage of correctly marked items: Provided by D2L and Pearson. Partial credit based on a difference: The way that difference is defined varies depending on the LMS. Provided by Canvas, Angel, D2L, and Sakai. Full control over the point value assigned to each individual response: Provided by Angel, Blackboard, and Moodle. LMS All or nothing % Correct Difference Point Values Canvas X Angel X X X Blackboard X X D2L X X X Moodle plugin X Pearson X X Sakai X X The Future Canvas is working on revamping quizzes with their Modern Quizzing Engine. I've heard some really awesome things about it that should fix most of the concerns of people regarding how Canvas grades Multiple Answers questions In the meantime, Avi Naiman and James Jones have have developed a user script called QuizWiz that will add "all or nothing" and "% correct" grading to Canvas. This will be a user script add-on to SpeedGrader that will allow you to click one button and regrade a quiz submission. There are a lot more features, but that is the portion that is germane to this document. See QuizWiz: Enhancements to SpeedGrader and Quizzes for more information. ... View more

Attached Excel workbook is able to download data from Canvas (using Get API calls) to create a progress report of all students in a course with modules to which requirements are added (see How do I add requirements to a module?). (no data means not yet downloaded while download is in progress) Compatibility MS Office 2010 on MS Windows (higher versions work most likely, but are not tested) MS Office 2011 on Mac OS X (version 2016 for Mac barely supports VBA and is therefore not supported) Enable editing and content In case Excel displays the following warnings, enable editing and content for the Workbook to work properly: Settings First fill in the settings: the Canvas url of your institution (e.g. https://yourinstitution.instructure.com) an API access token (check How do I obtain an API access token for an account?) select the desired content for column A and column B click Save Run progress report To run a progress report: find the course ID of the course from which the report needs to be generated in the address bar fill in the Course ID in the worksheet at cell B2 (e.g. 210) click the button Download data from Canvas to create the progress report Depending on the amount of students and module items this may take a some time (about half a second per student). In the top-left corner the download progress is displayed in percentages. To cancel the download progress click the button Download data from Canvas again (only while download progress is less than 100%, otherwise downloading would start again). Versions history: 1.11 (July 29, 2020) Updated the processing of API pagination (Instructure changed the method regarding the enrolments API) to accommodate for both the old and the new method. Changed some Excel formulas to prevent #DIV/0! from being displayed 1.13 (November 9, 2022) Added a row "Requirement" below the module item names If a module item has no requirement, the completion information per student stays empty and a comment is added for explanation (note: When no requirements are configured for a module, Canvas assumes progress as Complete, whether students interacted with the content or not at all) Performance In case you experience extremely slow performance (a minute per row or so) to the point that Excel is freezing, then make sure the Windows Defender Antimalware Client version is at least 4.18.2102.4. You can check the version on your device following this guide How to Find the Microsoft Defender Version Installed in Windows 10. Remarks Disclaimer: use this workbook at your own risk (I cannot be held responsible for any undesired consequences). This Excel workbook only reads from Canvas, never writes to it and should not have any negative effect on Canvas in terms of integrity or performance. First try it on your test environment https://yourinstitution.test.instructure.com (slow performance is expected and may cause time-out errors) before using it on production Check samples from reports with actual data in Canvas or in case your student(s) get accounted for their activities in modules as Canvas is the most reliable source Suggestion: save the workbook as a template (xltm file) after entering the settings so you can always start with a fresh and clean workbook so you will be prompted to save it. Keep in mind that if you don't save the file in a macro-enabled workbook format (xlsm), running the report again will not work any more as it requires the macro functionality to run. I created this workbook largely in my own spare time (for fun, to learn, to share and because I can), so huge amounts of support can't be expected Leave your comments in case you are using it, if it makes your work easier or more fun Download Download the latest version of this file from this Google Drive folder. Do not open in Google Docs nor in MS Office online, because that will not work, just first save the file and then open it in the Microsoft Excel application. Additional tools Automated report of assignments in courses  Automated report of group sets, groups and members in a course  Create course groups from sections using Excel  Download data to Excel using VBA and the API (workbook with code attached)  ... View more

I'm looking for peer review and/or group member evaluation (i.e. students rate each other's performance on group work) tools that others are using. I'm aware of Peerceptiv and PeerGrade. What else is out there? Thanks! ... View more

We’re getting reading to roll out Microsoft Teams, which is super exciting as the LTIs seem very useful. Has your higher ed institution deployed any of the Microsoft LTI’s yet? If so, I’d really like to hear about your experiences! Here’s a few questions that I’m curious about:  -Have these tools been used in a significant majority of your online classes yet? - If you enable OneDrive, do you think it makes sense to disable the built-in Google collaboration feature? - What are your thoughts about FlipGrid as a mobile friendly video-enhanced discussion app? - The Classroom Notebook powered by OneDrive seems incredible for students asking questions and giving answers to/from fellow students in a class. I wonder if students would feel comfortable using it, and what purpose they’d use it for..any thoughts? -Teams seems to do everything that Zoom can do. Does your school license and support both, or just one of these web conferencing tools?  Here’s my initial thoughts without taking the tools for a test drive:  Teams for web conferencing, persistent chat, collaboration tools, and 3rd party apps seems like a great way to foster student-to-student learning (at fixed and open times) The OneDrive external assignment tool type seems like it will be really useful for having students submit PowerPoint, Word, or Excel files through a web browser instead of downloading a Microsoft file, editing it locally, and then uploading the updated file to an assignment file submission for grading. It doesn’t seem like a Teams Classroom can be set on a BluePrint level and ‘stick’ to courses that it’s synced to, and then doing a manual roster sync.  The React add-on to Teams seems super useful for teachers to quickly understand how their students are feeling towards the coursework.  If your school is using Teams now in your Canvas instance, I’d love to discuss how it’s going with your Canvas Admin team.  Cheers, Adam Voyton adam.c.voyton@wilmu.edu  ... View more

Hi, We would like to know of schools that have implemented Ellucian ILP 5 SaaS and how their grade passback has been working.   We have a current Ellucian support case being worked on for grading in courses with pass/fail students since they are failing.  Since instructors at our school do not know if a student is pass/fail, when a grade is put in for the student, the entire passback of grades for the course fails.  We are curious what other schools have done with this issue?  We know other schools are using the Grade passback LTI and have been sending grades from Canvas to Banner but are wondering how they get around this issue of pass/fail students. Thanks   ... View more