[ARCHIVED] How do I integrate Quizzes, Outcomes and Learning Mastery?

[johnb]

Dear All,

John here – new to Canvas and this community and looking for some help understanding how to integrate use of Quizzes, Outcomes and Learning Mastery in Grades.

Please find below a detailed description of what I have done (Steps 1-8) and the questions it has raised (in purple italics).

Apologies for cross posting to both the ´Outcomes´ and ´Quizzes´ group but I wasn´t sure where was the best place to locate this query. (Is there a protocol around placing questions?)

Many thanks for your time on this.

jb

Quizzes – Outcomes – Learning Mastery in Grades

To test this functionality, I followed the process below:

1. In Outcomes I represented my Programme Learning Outcome (PLO) as an Outcome Group and called this PLO1- ´Use the four basic operations of arithmetic´.
2. Within this Outcome Group/PLO, I created 4 sub-outcomes: LO1.1 ´Add numbers´, LO1.2 ´Multiply numbers´, LO1.3 ´Subtract numbers´ and LO1.4 ´Divide numbers´
3. For LOs 1.1-1.4, I used the Canvas default settings (see Figure 1).

Figure 1

My understanding of the rubric in Figure 1 is that in order for a student to demonstrate ´Meets Expectations´ for this LO, they need to accumulate 3 points from course assessments that are aligned with this LO. Is this understanding correct?

4. I next created 4 questions banks in Quizzes. I named each bank per my 4 LOs (see Figure 2)

Figure 2

5. I then created 3 questions (1 point each) that assessed each LO and aligned those questions with the relevant LO (see Figure 3). I set mastery at 60% (the Canvas default).

Figure 3

I am unclear what ´mastery at 60%´ means.

• Does it refer to 60% of questions in a particular quiz?
  • If so, does this mean that a quiz can only assess one specific outcome? Is it not possible to design a quiz that contains questions aligned with several outcomes?
• Does it refer to 60% of the points available for a particular LO (in my example 3 points – 3 questions x 1 point each)?

Scenario A: So, to demonstrate Mastery of each LO, a student needs to answer 2 of the 3 questions correctly (60% of 3 points = 1.8 points). Is this understanding correct?

Scenario B: Let´s say that I decide that LO 1.2 – ´Multiply numbers´ is the most challenging outcome for my learners to achieve so I decide to give them additional opportunities during my course to demonstrate their achievement of it. To that end, I offer them 2 Quizzes, each with 3 questions (1 point per question). To demonstrate Mastery of this LO, a student now needs to answer 4 of the 6 questions correctly (60% of 6 points = 3.6 points). Is this understanding correct?

• What does this mean in terms of the rubric in Figure 1 which states that ´Meets Expectation´ requires 3 points?
•  Do I need to keep track of how many question points are aligned with each LO and manually update each rubric to reflect this?

• In the case of Scenario A and B above, does this mean that the rubrics for LO1.1, 1.3 and 1.4 would need to be ammended as below

Figure 4

and that the rubric for LO1.2 - ´Multiply numbers´, would now look like this?

Figure 5

6. Next, I created a Graded Quiz. The Quiz had 4 questions – one per LOs 1.1-1.4. Therefore, there are 4 points available for this Quiz (4 questions x 1 point per question).

7.  Then I acted as a Test Student and took the quiz, answering all questions correctly.

8. When, as the student, I check my results in Learning Mastery, I see that I have achieved a score of 5/5 for LOs 1.1-1.4 and, therefore, have mastered all of these LOs and by extension PLO1: ´Use the four basic operations of elementary arithmetic.´

Figure 6

For the life of me I don´t understand how this is possible given that each question is only worth one point. Please help!

My sincerest thanks in advance for any guidance you can offer.

Best Wishes,

jb

[kbink]

Hi John,

What great questions.  I can't wait to see the official answers. I went to outcomes I've been playing with to see if I could see what "I am unclear what ´mastery at 60%´ means." means.  I'm confused too.  At the subaccount level I've set master to be 4/4 on the rubric item.  I then used "Find" to import them into a specific class.  Why is it even asking for a what mastery means here in the question banks?  Does this mean an instructor could override the meaning of mastery set at the sub account level?

I may play with some of your other questions too and see if I can add more of my confusion. Thanks for starting the thread!

Kalli

[jcarpenter1]

 @johnb  , welcome!  1st question: Since a 3/5 meets expectations on the rubric, and 3/5 = 60%, then your rubric has 60% as meeting expectations.  You would need to change your range in your rubric if you wanted it higher.

2nd question: Last time I tried to use it, you could only link 1 outcome to a quiz.  That had been up for a vote, and I'm not sure how that went.  Have you checked out these directions?  How do I align an outcome to an assessment question in Quizzes.Next? 

Here's a great blog to read: https://community.canvaslms.com/people/Gideon.Williams .

I may not have all of your answers, but maybe it helps some!

[Shar]

Hi  @johnb  and Welcome to the Canvas Community!

I am by no means an expert in Outcomes, but I can see where you have misunderstood in a few places based on your thorough explanation. So here's my attempt to clear up a few things.

Outcome Group

Great job in setting up the outcome/PLO group to keep related LO together! You're off to a great start and staying organized with naming conventions and all!

Mastery Level

As Jennifer Carpenter pointed out, you have mastery set for 3 points out of 5 points total on the rubric (if all LO are set up as LO1.1 Add numbers) which is the 60%. We use a 4-3-2-1 scale for mastery level so there are no skipped numbers versus the 5-3-0 in the default rubric.

If someone scores 80% on your quiz, they've met the 60% threshold for meets expectations. If someone scores 100% on your quiz, they've met the 100% threshold for exceeding expectations.

Calculation Method

You also have selected Decaying average for the Learning Outcome, so if it is "checked" or aligned more than once then there will be some magic. So in Scenario B, if you have a second 3-question quiz for Multiply Numbers, the student still needs to get 60% of the quiz correct for mastery, but Canvas will do some math to compute the overall mastery level. This second attempt/alignment counts more (65%) than the first attempt (35%).

I'm sure some one will come along and show us how the math breaks down. Something like if the second attempt is 80% and first attempt was 100%, then the mastery is 80 * 0.65 + 100 * 0.35 = 52 + 35 = 87. 87% meets the 60% threshold for meet expectations.

All this is to say you do not have to change your LO rubric to accommodate more questions, Canvas will do the math/magic for you. Keep all your LO rubrics consistent. And when you are doing the quiz, you can have any number of questions and do not have to stick to 3 questions.

Outcomes & Quizzes

Again Jennifer Carpenter confirms that it's 1 outcome per quiz, so you while you have a quiz with 4 questions for each of the LOs, you are not actually assessing all of those outcomes in that quiz.

Student Learning Mastery

So when the student report show 1 alignment for each of the LOs, it's for each of those individual 3-question quizzes where the Outcome is aligned. In your practice, did you get all of the questions correct? as in 100%? if so then you mastered each of the LO at 5/5.

Points are labels

And finally, I think some of the confusion comes from the use of the word "points" in the Learning Outcome setup. From my perspective, these are not points as in earned points but more like ranking points. We usually use outcomes on assignments and not with quizzes so it's easier to relate "oh you performed at a level 3 on this assignment" rather than saying "you got 3 points out of 4 on this assignment". It looks like when it comes to aligning with the Quizzes, the "points" become valuable in understanding the percentage needed to show mastery.

I had found a bunch of resources the other day on Outcomes, but I didn't save them after absorbing the knowledge, but there are here in the community. Here's one by  @Chris_Hofer ‌ Outcomes and Canvas Data with lots of juicy input from others.

Hope some of these explanations are helpful.
Cheers - Shar

[johnb]

Dear All,

Many thanks indeed for your help to date on this  @kbink , ishar-uw‌ andJennifer Carpenter   

1. Only one outcome per Quiz

This is a very useful clarification. I must say I find this quite counter-intuitive given that outcomes are aligned with individual questions (not quizzes). My understanding of this was that your result on a specific question related directly to the outcome aligned to that question (as opposed to the quiz, which I understood as a ´shell´ for hosting questions). Does this mean then that you cannot, for example, design a capstone 40 item MCQ incorporating assessment of multiple outcomes?

2. ´Mastery set at X%´ relates to an individual Quiz as opposed to an aligned outcome

This also is a very useful clarification. The question this raises for me then is how does Learning Mastery integrate the result from a Quiz with the result from an Assignment where both are aligned to the same outcome?

For example: LO1.1 - ´Add numbers´

- I design a 10 point quiz (10 questions x 1 point each) aligned with this outcome.

- I also design an assignment worth 20 points  aligned with this outcome.

In my course, there are now 30 points available in connection with this outcome. 

If a student scores 10 points in the Quiz but only 5 points in the Assignment, how does Canvas represent this in Learning Mastery with regard to LO1.1 - ´Add numbers´?

 @bbennett2 ‌ - I´d be particularly grateful for your thoughts on the questions above in purple - your Instructure Con 2017 presentation was a great help in getting started on this road and I know you have been wqorking with these features for several years now.

Thank you!

jb

[bbennett2]

Hey  @johnb , happy to help:

Only one outcome per quiz

No, you can attach as many outcomes as you like. The big caveat is that if you want to pass results back to the Learning Mastery Gradebook, you cannot use New Quizzes. That's on the development roadmap according to Instructure, but I wouldn't expect that until midyear next year at the easlist, to be honest. Here's out process:

1. Set up Question Banks that are outcome aligned.
   • Create question banks in your course(s) that are all around a single outcome. This is important because if you have multiple outcomes attached and they get the question wrong, you don't know which outcome they're struggling with. Single-outcome alignments give you more granular data on student understanding.
   • In the bank, click on "Align outcome" on the right and choose the appropriate one (I'm assuming you already know how to do this from above).
   • Set the percent correct cutoff. In other words, how many of these questions on any given assignment does a student need to have correct to earn your "mastery" score from the rubric. Here's a scenario:
     
     • A quiz has 6 questions from a bank set to mastery at 60% (assuming a 4 point rubric, mastery set to 3):
       • Student A: 3/6 --> Rubric score 2 (not mastered)
       • Student B: 4/6 --> Rubric score 3 (mastered)
       • Student C: 6/6 --> Rubric score 4 (beyond mastery)
   • Note that the rubric algorithm is unknown. I've asked engineers at Instructure and have not gotten a solid answer about where the cutoffs happen internally.
   • Also note that you can repeat questions across banks. For instance, you may want to add a DOK level to your banks so you have Outcome A DOK 1 (all knowledge, identification questions) along with Outcome A DOK 3 (synthesis, application). You can pull from those banks independently and have different mastery percentages defined in each.
2. Design the Quiz
   • When you're designing your quiz, pull questions from the bank instead of writing them directly on the quiz.
   • Pulling single questions (Find Question option) gives all students the same question in the same place.
   • Using the Question Group allows you to specify n questions from x bank. This is how you assess multiple outcomes on a given Quiz. Create as many groups as you need for each Outcome you want to assess.

Incorporating the Mastery Score with the Assessment Score

Canvas does not combine the mastery rubric score with the assessment score in any way (this is a huge gap, IMO). In other words, the rubric evaluation from the assessment is independent of the assessment score. This is helpful in one particular way, in that you can look at a student score and break down their understanding at the Outcome level. I've actually written an internal tool for our teachers that does this on a per-assignment level. We can highlight students who score poorly overall (test anxiety, dyslexia, ELL, etc) but show mastery on individual Outcomes. 

Our teachers look at outcomes as guide posts for designing upcoming instruction and remediation. We do add the Outcome results to the gradebook, but it's something we've designed internally with our teachers via the Canvas API. Since the rubrics are independent of Assignment scores, we've done custom work to get an Outcome to post to the traditional gradebook automatically. Our process is as follows:

• The gradebook is split into two Categories: Outcomes and Assignments
  • All assignments - tests, quizzes, homework, etc go into Assignments with their point values.
  • Each course Outcome gets a 1 point assignment created in the Outcomes category.
• Raw scores are placed into the Assignments category and calculated as normal (total points, weights, etc)
• Outcome scores (1 or 0) are calculated based on the rubrics. Throughout the semester, teachers are looking at the Learning Mastery Gradebook and scoring the corresponding Outcome assignment appropriately. Our cutoff is 3/4 for a 1/0 in the gradebook.
• For a total score, teachers typically do a 75/25 (Outcomes/Assignments) weighted average.

The reconciliation process is very labor intensive, so the custom tool we use with the rubric breakdown also does the gradebook reconciliation for us. Teachers align a single Assignment from the gradebook with a single Outcome in the dashboard and that manages the scoring. This is what I meant by Canvas missing a big opportunity with using rubrics to affect the gradebook. Hopefully, this gets built in, but again, not any time soon.

This can be confusing to just read, so if you'd like to talk outside the thread, let me know and we can arrange something this week even.

[johnb]

Thanks so much  @bbennett2 ‌ - really appreciate your time on this. 

You are right - it is confusing to read! I would be very glad to take you up on your offer to chat outside of this thread if that is still a possibility. I am finished work for the holidays today but am back on January 6th so perhaps we could schedule something for week beginning 6th? What is the best way for us to do that?

Wishing you a very happy Christmas.

jb