Canvas Outcomes & New Quizzes: Pt. 2

The content in this blog is over six months old, and the comments are closed. For the most recent product updates and discussions, you're encouraged to explore newer posts from Instructure's Product Managers.

BenFriedman
Instructure Alumni
Instructure Alumni
13
2959

Product Blog Headers -- 1200 x 200.png

 

In a previous blog, I detailed the 3 types of outcome alignments available in New Quizzes and some use cases for each. I also shared that our Outcomes development team is working on sending New Quiz data with outcome alignments to the Learning Mastery Gradebook and outcomes report exports. We’re very excited to bring this capability to Canvas as part of the transition to New Quizzes

In this post, we’re going to take a look at how outcomes-related scoring and reporting happen in New Quizzes. I’m excited to be joined by Chrystal Langston, a Senior Software Engineer on the Outcomes team as co-author to do that! 

We’re going to take a look at an example of how two students perform on an outcome-aligned New Quiz where each individual item carries an alignment. Once a student completes this quiz the results will be aggregated up across items with a percentage correct calculated that becomes an outcome assessment “score.” We generally do not suggest having only one item that assesses an outcome, particularly if it is multiple choice or a similar item type. 

Quiz Setup

Question

Outcome Item Alignment

Points

1

Outcome 2

1

2

Outcome 2

1

3

Outcome 3

1

4

Outcome 3

1

5

Outcome 3

1

 

Now let’s break this down to understand how a proficiency rating is derived from the quiz’s score percentage. If the quiz’s score percentage is equal to a rating’s score percentage, the student will be awarded the aligned rating. If the quiz result’s score percentage is greater than a rating’s score percentage but less than the next rating’s score percentage, the student will be awarded the lower of the two ratings. For an easy way to remember the scoring, think of the Price is Right game show, the lower price wins! #onedollarbob. 

Outcome 2, and 3 Details (using the Canvas default proficiency rating levels)

Title

Rating

Score Percentage

Exceeds Mastery

4

100%  (i.e. 4/4)

Mastery

3

75-99% (i.e 3/4)

Near Mastery

2

50-74% (i.e. 2/4)

Below Mastery

1

25-49% (i.e. 1/4) 

No Evidence

0

0-24%

Outcome Results 

Next, let’s look at the quiz performance for two example students -  Ally Adams and Steven Smith. Outlined in the below two tables are their results. You will be able to see in the table how a student’s performance on the quiz items leads to a proficiency rating.  

Steven Smith Quiz Results

Outcome

Points and Percentage

Proficiency Score

Outcome 2

1 out of 2 (50%)

Near Mastery

Outcome 3

3 out of 3 (100%)

Exceeds Mastery

 

Ally Adams Quiz Results

Question

Aligned Outcome

Points Possible

Outcome 2

0 out of 2 (0%)

No Evidence

Outcome 3

3 out of 3 (100%)

Exceeds Mastery

Outcome Analysis Reporting

Below are screenshots of how these results are represented through the Outcomes Analysis reports for the two students which currently communicate results by a single quiz. As mentioned above, New Quiz results will soon go to the Learning Mastery Gradebook and .csv reports (and/or combined with assignment results) to give an overall picture of student mastery of outcomes over multiple assessments, a common feature request. Stay tuned for when this will be available as we work towards a release date!

In the first screenshot, we see whether each student met mastery and how many points they received on the quiz questions aligned to each outcome.

BenFriedman_0-1662127214063.png

In the Outcome detail screens, we can see the distribution of students across mastery levels and the average score of students compared to where mastery was set for each outcome.

BenFriedman_1-1662127214154.png

BenFriedman_2-1662127214153.png

 

 

The content in this blog is over six months old, and the comments are closed. For the most recent product updates and discussions, you're encouraged to explore newer posts from Instructure's Product Managers.

13 Comments
ProfGeoMtnGrl
Community Member

I am trying this new feature out in my class this Fall. I can see the student mastery under the outcomes analysis, and I would like for the students to see the same thing. When I run the quiz for a test student, the quiz score shows up, but not their mastery levels on each of the outcomes. How do I set this up from the student side so that they see their outcomes on the quiz.

amyslack
Community Participant

@ProfGeoMtnGrl I think you bring up a great point, will students eventually be able to see this data about themselves? It would be helpful for students to have a view within Grades that would also show them their own progress towards mastery as well, I would think this is in the works but I love that you mentioned this! 

 

I am so excited to see this live and hopefully soon! I love that this new infrastructure will be a way to push more educators towards Mastery based teaching in learning by better supporting something that often feels overwhelming and complex in a simple user friendly way 🙂 

mbmacdonald
Community Participant

@BenFriedman Thanks for your (and the whole teams') hard work on this, and thanks for this clear explanation. Can you just clarify something for me?

I am trying to pull outcomes reports (outcome results reports) at the sub-account level, for subaccount-level outcomes in order to see progress towards outcomes for all students in all courses in a particular academic program.

It looks like right now, those reports (again, account/subaccount level reports) will NOT include results of New Quizzes that have outcomes aligned, correct? Is that what is meant by "Admin Outcomes Report includes New Quiz data" which is on the roadmap under "before july 2023"?

AlanFoucault
Community Member

Really need the Learning Mastery Grade book features to work with New Quizzes. Looking forward to this update. Any timeline for it to be enabled?

adamwarecs
Instructure
Instructure

@ProfGeoMtnGrl  - Whilst student's can't view this for individual quizzes, they can view progression here. Via  Alignments are the assignments and other Canvas items that are linked to an outcome. These alignments show what course assignments you participated in to earn your score.

@AlanFoucault - If the feature option is turned on you should have access to it now.

amyslack
Community Participant

We are seeing issues with the LMGB not loading and have been since the launch. Is there an expected timeline for the not loading issue to be corrected?

jsailor
Instructure
Instructure

Hi, @amyslack,

We're sorry to hear that you are experiencing trouble. If you have not already, please submit a support ticket so that our support team can dig into the issue and escalate to our engineers if needed. 

 

paul_fynn
Community Coach
Community Coach

@adamwarecs 

We are watching this with interest for a number of use cases that we have and very much welcome the development work that has been undertaken..

In the current view we can see, and the students can see,, what they have achieved in mastery outcomes at course level.

  • Given that we are setting sub-account level outcomes, it would be helpful if staff and students were able to see sub-account level mastery aggregated from more than one Canvas course eg where the student as accumulated intra-untit outcomes across their award
  • It would also be helpful to be able to provide access to this from anywhere in the sub account

We understand from the Community forums that where grades are imported for an assignment, Learning Mastery/outcomes are not triggered because they require that the teacher has graded through the rubric - is that correct and are there any plans to address this ?

All insights very much welcomed !

 

nerfphysics2
Community Member

I'm trying to see my outcome-aligned New Quizzes results for students in Learning Mastery Gradebook and is it even possible that this is still not a feature?  Students can't see their quiz results in their LMGB, I can't see them in mine and that's where I do my grades from. What am I missing here? Frankly, until Canvas can align an outcome to a New Quizzes question and then report the student's performance on that outcome in Learning Mastery Gradebook, New Quizzes is useless to me and to students. How is this not fixed/built in by now?  It's insanity. Canvas is a major company. This is a basic feature. Old Quizzes is so 1985. New Quizzes is better but can't do basic grade reporting for SBG after YEARS of development. I love so much about Canvas - I'm in the CCE program right now and I want so much to use Canvas with my classes and use all the cool things I'm learning, but if I can't assess my students and get visible mastery scores for them, then what's the point. 

Either let me attach a rubric to individual New Quizzes questions or report the aligned Outcomes. Please, when is this coming? 

MariahBirgen
Community Participant

I have figured out how to align Outcomes to specific questions in New Quizzes, but I now am trying to do the same thing to questions in the Item Banks and cannot do so because the Outcome is not attached to the whole Item Bank so I can't retrieve it. Help.

jsailor
Instructure
Instructure

@nerfphysics2, outcomes aligned New Quizzes reporting in the Learning Mastery Gradebook was added last November, so this should be working for you. The one exception to that is for outcomes that are added at the sub-account level, which we are currently working to support. If your outcomes are course or account level outcomes and they are not populating the Learning Mastery Gradebook, please submit a support ticket so that we can dig in and provide help if needed. 

@MariahBirgen, outcomes do not need to be attached to the item bank to be added to questions, however, only account level outcomes are supported in item banks currently.   

nerfphysics2
Community Member

@jsailor Thanks for responding. I'm in a school in a large district. We do not have district-level outcomes so I created and am using my own outcomes in my courses. I'm assuming that's what you mean by 'course level' outcomes?  Or would those be sub-account level outcomes?  Because whatever they are, they are not being reported to LMG. I'm happy to put a help ticket in if they are considered course-level outcomes that should be populating the LMG view.

flora_woratsche
Community Novice

Is it possible to correct an outcome alignment and have a quiz re-scored using the new alignments? I am not using new quizzes