Register for InstructureCon25 • Passes include access to all sessions, the expo hall, entertainment and networking events, meals, and extraterrestrial encounters.
Dear All,
John here – new to Canvas and this community and looking for some help understanding how to integrate use of Quizzes, Outcomes and Learning Mastery in Grades.
Please find below a detailed description of what I have done (Steps 1-8) and the questions it has raised (in purple italics).
Apologies for cross posting to both the ´Outcomes´ and ´Quizzes´ group but I wasn´t sure where was the best place to locate this query. (Is there a protocol around placing questions?)
Many thanks for your time on this.
jb
Quizzes – Outcomes – Learning Mastery in Grades
To test this functionality, I followed the process below:
- In Outcomes I represented my Programme Learning Outcome (PLO) as an Outcome Group and called this PLO1- ´Use the four basic operations of arithmetic´.
- Within this Outcome Group/PLO, I created 4 sub-outcomes: LO1.1 ´Add numbers´, LO1.2 ´Multiply numbers´, LO1.3 ´Subtract numbers´ and LO1.4 ´Divide numbers´
- For LOs 1.1-1.4, I used the Canvas default settings (see Figure 1).
Figure 1
My understanding of the rubric in Figure 1 is that in order for a student to demonstrate ´Meets Expectations´ for this LO, they need to accumulate 3 points from course assessments that are aligned with this LO. Is this understanding correct?
4. I next created 4 questions banks in Quizzes. I named each bank per my 4 LOs (see Figure 2)
Figure 2
5. I then created 3 questions (1 point each) that assessed each LO and aligned those questions with the relevant LO (see Figure 3). I set mastery at 60% (the Canvas default).
Figure 3
I am unclear what ´mastery at 60%´ means.
- Does it refer to 60% of questions in a particular quiz?
- If so, does this mean that a quiz can only assess one specific outcome? Is it not possible to design a quiz that contains questions aligned with several outcomes?
- Does it refer to 60% of the points available for a particular LO (in my example 3 points – 3 questions x 1 point each)?
Scenario A: So, to demonstrate Mastery of each LO, a student needs to answer 2 of the 3 questions correctly (60% of 3 points = 1.8 points). Is this understanding correct?
Scenario B: Let´s say that I decide that LO 1.2 – ´Multiply numbers´ is the most challenging outcome for my learners to achieve so I decide to give them additional opportunities during my course to demonstrate their achievement of it. To that end, I offer them 2 Quizzes, each with 3 questions (1 point per question). To demonstrate Mastery of this LO, a student now needs to answer 4 of the 6 questions correctly (60% of 6 points = 3.6 points). Is this understanding correct?
- What does this mean in terms of the rubric in Figure 1 which states that ´Meets Expectation´ requires 3 points?
- Do I need to keep track of how many question points are aligned with each LO and manually update each rubric to reflect this?
- In the case of Scenario A and B above, does this mean that the rubrics for LO1.1, 1.3 and 1.4 would need to be ammended as below
and that the rubric for LO1.2 - ´Multiply numbers´, would now look like this?
Figure 5
6. Next, I created a Graded Quiz. The Quiz had 4 questions – one per LOs 1.1-1.4. Therefore, there are 4 points available for this Quiz (4 questions x 1 point per question).
7. Then I acted as a Test Student and took the quiz, answering all questions correctly.
8. When, as the student, I check my results in Learning Mastery, I see that I have achieved a score of 5/5 for LOs 1.1-1.4 and, therefore, have mastered all of these LOs and by extension PLO1: ´Use the four basic operations of elementary arithmetic.´
Figure 6
For the life of me I don´t understand how this is possible given that each question is only worth one point. Please help!
My sincerest thanks in advance for any guidance you can offer.
Best Wishes,
jb
.png "jcarpenter1"){kind=avatar}
Hi @johnb and 
I am by no means an expert in Outcomes, but I can see where you have misunderstood in a few places based on your thorough explanation. So here's my attempt to clear up a few things.
Outcome Group
Great job in setting up the outcome/PLO group to keep related LO together! You're off to a great start and staying organized with naming conventions and all! 
Mastery Level
As Jennifer Carpenter pointed out, you have mastery set for 3 points out of 5 points total on the rubric (if all LO are set up as LO1.1 Add numbers) which is the 60%. We use a 4-3-2-1 scale for mastery level so there are no skipped numbers versus the 5-3-0 in the default rubric.
If someone scores 80% on your quiz, they've met the 60% threshold for meets expectations. If someone scores 100% on your quiz, they've met the 100% threshold for exceeding expectations.
Calculation Method
You also have selected Decaying average for the Learning Outcome, so if it is "checked" or aligned more than once then there will be some magic. So in Scenario B, if you have a second 3-question quiz for Multiply Numbers, the student still needs to get 60% of the quiz correct for mastery, but Canvas will do some math to compute the overall mastery level. This second attempt/alignment counts more (65%) than the first attempt (35%).
I'm sure some one will come along and show us how the math breaks down. Something like if the second attempt is 80% and first attempt was 100%, then the mastery is 80 * 0.65 + 100 * 0.35 = 52 + 35 = 87. 87% meets the 60% threshold for meet expectations.
All this is to say you do not have to change your LO rubric to accommodate more questions, Canvas will do the math/magic for you. Keep all your LO rubrics consistent. And when you are doing the quiz, you can have any number of questions and do not have to stick to 3 questions.
Outcomes & Quizzes
Again Jennifer Carpenter confirms that it's 1 outcome per quiz, so you while you have a quiz with 4 questions for each of the LOs, you are not actually assessing all of those outcomes in that quiz.
Student Learning Mastery
So when the student report show 1 alignment for each of the LOs, it's for each of those individual 3-question quizzes where the Outcome is aligned. In your practice, did you get all of the questions correct? as in 100%? if so then you mastered each of the LO at 5/5.
Points are labels
And finally, I think some of the confusion comes from the use of the word "points" in the Learning Outcome setup. From my perspective, these are not points as in earned points but more like ranking points. We usually use outcomes on assignments and not with quizzes so it's easier to relate "oh you performed at a level 3 on this assignment" rather than saying "you got 3 points out of 4 on this assignment". It looks like when it comes to aligning with the Quizzes, the "points" become valuable in understanding the percentage needed to show mastery.
I had found a bunch of resources the other day on Outcomes, but I didn't save them after absorbing the knowledge, but there are here in the community. Here's one by @Chris_Hofer Outcomes and Canvas Data with lots of juicy input from others.
Hope some of these explanations are helpful.
Cheers - Shar
Hey @johnb , happy to help:
Only one outcome per quiz
No, you can attach as many outcomes as you like. The big caveat is that if you want to pass results back to the Learning Mastery Gradebook, you cannot use New Quizzes. That's on the development roadmap according to Instructure, but I wouldn't expect that until midyear next year at the easlist, to be honest. Here's out process:
- Set up Question Banks that are outcome aligned.
- Create question banks in your course(s) that are all around a single outcome. This is important because if you have multiple outcomes attached and they get the question wrong, you don't know which outcome they're struggling with. Single-outcome alignments give you more granular data on student understanding.
- In the bank, click on "Align outcome" on the right and choose the appropriate one (I'm assuming you already know how to do this from above).
- Set the percent correct cutoff. In other words, how many of these questions on any given assignment does a student need to have correct to earn your "mastery" score from the rubric. Here's a scenario:
- A quiz has 6 questions from a bank set to mastery at 60% (assuming a 4 point rubric, mastery set to 3):
- Student A: 3/6 --> Rubric score 2 (not mastered)
- Student B: 4/6 --> Rubric score 3 (mastered)
- Student C: 6/6 --> Rubric score 4 (beyond mastery)
- A quiz has 6 questions from a bank set to mastery at 60% (assuming a 4 point rubric, mastery set to 3):
- Note that the rubric algorithm is unknown. I've asked engineers at Instructure and have not gotten a solid answer about where the cutoffs happen internally.
- Also note that you can repeat questions across banks. For instance, you may want to add a DOK level to your banks so you have Outcome A DOK 1 (all knowledge, identification questions) along with Outcome A DOK 3 (synthesis, application). You can pull from those banks independently and have different mastery percentages defined in each.
- Design the Quiz
- When you're designing your quiz, pull questions from the bank instead of writing them directly on the quiz.
- Pulling single questions (Find Question option) gives all students the same question in the same place.
- Using the Question Group allows you to specify n questions from x bank. This is how you assess multiple outcomes on a given Quiz. Create as many groups as you need for each Outcome you want to assess.
Incorporating the Mastery Score with the Assessment Score
Canvas does not combine the mastery rubric score with the assessment score in any way (this is a huge gap, IMO). In other words, the rubric evaluation from the assessment is independent of the assessment score. This is helpful in one particular way, in that you can look at a student score and break down their understanding at the Outcome level. I've actually written an internal tool for our teachers that does this on a per-assignment level. We can highlight students who score poorly overall (test anxiety, dyslexia, ELL, etc) but show mastery on individual Outcomes.
Our teachers look at outcomes as guide posts for designing upcoming instruction and remediation. We do add the Outcome results to the gradebook, but it's something we've designed internally with our teachers via the Canvas API. Since the rubrics are independent of Assignment scores, we've done custom work to get an Outcome to post to the traditional gradebook automatically. Our process is as follows:
- The gradebook is split into two Categories: Outcomes and Assignments
- All assignments - tests, quizzes, homework, etc go into Assignments with their point values.
- Each course Outcome gets a 1 point assignment created in the Outcomes category.
- Raw scores are placed into the Assignments category and calculated as normal (total points, weights, etc)
- Outcome scores (1 or 0) are calculated based on the rubrics. Throughout the semester, teachers are looking at the Learning Mastery Gradebook and scoring the corresponding Outcome assignment appropriately. Our cutoff is 3/4 for a 1/0 in the gradebook.
- For a total score, teachers typically do a 75/25 (Outcomes/Assignments) weighted average.
The reconciliation process is very labor intensive, so the custom tool we use with the rubric breakdown also does the gradebook reconciliation for us. Teachers align a single Assignment from the gradebook with a single Outcome in the dashboard and that manages the scoring. This is what I meant by Canvas missing a big opportunity with using rubrics to affect the gradebook. Hopefully, this gets built in, but again, not any time soon.
This can be confusing to just read, so if you'd like to talk outside the thread, let me know and we can arrange something this week even.
This discussion post is outdated and has been archived. Please use the Community question forums and official documentation for the most current and accurate information.