Connecting Outcomes to Mastery Paths & Outcomes Best Practices

[acohen11]

Our ID team is pursuing a project to use Outcomes in a course so that Instructors have access to the Learning Mastery Gradebook as well as connecting the Outcomes to Mastery Paths and allowing certain content to be unlocked based on a mastery level of particular outcomes.

Canvas Support has said that this is something is being developed but is not available at this time.

• I am wondering if any of you have be able to create something like this with a work around.

• Also, as we are jumping into using Canvas Outcomes, does anyone have any best practices in regards to Outcomes that they would like to share?

Curious about your thoughts. Thanks for any advice.

[kmeeusen]

Hi  @acohen11 ‌

I don't have an answer for you, because like you our school is finally getting around to outcomes (5+ years after adoption), but you should be able to get some good responses in https://community.canvaslms.com/groups/canvas-focus-group-outcomes?sr=search&searchId=cb0644b3-acb0-...‌.

Kelley

[krenz053]

Hi Kelley,

Can you comment about how your institution is implementing Outcomes? Are you setting them up at the sub-account level for individual departments to manage or do you set them up at the institutional level?

After you've collected Outcomes data, what are you doing with it? On the technical side are you using tools to help interpret and visualize the data? If so, which tools? What groups or committees work with the data and what comes out of their work?

[kmeeusen]

Hi  @krenz053 

Our school is in its outcomes infancy. We have taught faculty how to write good ones, we then taught them how to align outs at various levels (module to course, course to program, program to college wide). We are now teaching them how to align course outcomes with assessment activities, or rather how to align assessments with their outcomes. We have not even begun to teach them how to put all that ground-work into practice in their Canvas classrooms.

However, we have set up our outcomes in Canvas by creating a structure for our program outcomes, and have begun entering them using work-study students. That's as far as we have gotten. Once all outcomes are loaded, we will begin the tedious chore of teaching faculty how to use them in Canvas.

Kelley

[bbennett2]

Old thread, but I just found it. This is going to be long. Apologies in advance!

I've done a lot of work over the last 18 months on training teachers on using Outcomes. We're not implementing them across our district (I'm K12) but we have several pockets playing with aligning Outcomes to assignments and quizzes.

In reply to question 1, the short answer is no. You cannot tie course progress (or anything, really) to Outcome performance automatically. I talked with one of the developers at last year's InstructureCon about some small changes that would really make them more powerful in a course, but nothing has been implemented yet.

The closest you could do to achieve progress based on Outcomes is to set up individual assignments for each Outcome you're assessing. When a student reaches mastery on an Outcome, the grader would mark the traditional assignment as complete (or 100%, etc) which would then toggle any other progress checks in place (Mastery Paths or Module requirements/prerequisites). You cannot automate this yet, but it's something I've submitted to Instructure as an enhancement between the Learning Mastery gradebook and the traditional gradebook.

For best practices, some other things to consider:

1. If you're not on the Quizzes.Next beta (the new Quiz engine) you need to set up Question Banks if you want to tie individual questions to individual Outcomes. Name each bank with the Outcome identifier (I used a alphanumeric system, 1.A.1, etc) and every question in that bank aligns to the Outcome you're assessing. When you're building Quizzes, use the Find Questions option rather than adding them directly to the Quiz to gather that data. The Quizzes.Next update will change this pattern by allowing you to align an outcome directly to a question.
2. Think through your rubric structure. Canvas defaults to 5-3-1, but I've found that it's too easy to mark each student as meeting requirements. I changed all of my rubrics to 4-3-2-1 so I had to distinguish between Meets Expectations (3) and Approaches Expectations (2). Each Outcome was set to toggle mastery at a 3. (A 4 was "Exceed Expectations" and we had specific indicators for that rubric score.) 
3. The Outcome calculation mechanism is also important. I used Decaying Average weighted to 75%. I found, anecdotally, that scoring based on Outcomes closely matched a student's grade on a traditional scale. It's really an arbitrary decision, but the decaying average gave better results over a semester than a single high score or the mode score for the student.

Outcomes really open up opportunities for instructors and students to discuss the difference between doing the legwork and showing what they know. It made conversations much more productive because I could focus in on the content rather than what papers or assignments simply needed to be made up or redone.

Feel free to message me any time for more details or with more questions.

[krenz053]

Hi Brian,

Can you comment about what you are doing with the Outcomes data you are collecting? Specifically, I'm interested in both the technical and process pieces of what you do with the data. 

• Technical: Do you use data analysis or visualization tools to help gain insights into the data? If so, which tools?
• Technical: Does your group utilize any of the custom reports available from Instructure related to Outcomes?
• Process: Do groups meet to discuss the Outcomes data and make changes to courses or curriculum? If so, what groups or committees do you have in place?
• Process: Do you look at year-over-year data to spot trends?

If others reading this post would like to share information from their institution that would be great!

[bbennett2]

Hey  @krenz053 ‌

1. We're not doing anything with analytics right now, but I'm looking forward to Analytics2, just announced this week, for what kind of reports I can pull at the teacher level.

2. We have not set up any custom reports (yet).

3. Since this is only individual teachers, there aren't any committees. As it becomes more widespread (one of our goals for this year) we will certainly take outcome performance into account as we revise curriculum.

4. We're just entering our third year with Canvas, so this is the year I'll be looking for YoY trends now that we'll have several teachers using them again.

[bhauf]

Hi  @krenz053  (and  @bbennett2 ),

This post is a few months old, but  I can jump in here to answer some of your questions from when I worked at my previous institution. It has been a little over a year since I've been there, but as I brush up on Outcomes, it appears not much has changed since I've last used it. 

1. Because of how we layered our Outcomes by School > Program > Dept > Course (e.g. Public Health > Masters of PH > Epidemiology), we were able to run admin level reports to aggregate the data at those levels to provide us with not only course level competency achievement, but competency across a department and program. Setting up a proper naming convention with our outcomes helped us format the data from the CSV export. We didn't get the opportunity to do much with data visuals.

2. There are two admin level reports that can be pulled: Outcomes Results and Student Competency. The latter report will NOT pull quiz results. Courses have visual data charts that were helpful to our faculty for at-a-glance looks.

3. Yes, groups met at the program director levels to meet and discuss (this was at a state university) overall curriculum improvements based on competency performance. Our center would help pull and create the data reports to where improvement may be needed. We could see how competencies were performing across courses to give us some indication if poor performance was a result of the subject matter, content, pedagogy, etc. It also helped us redevelop Rubrics so that students were assessed based on the competencies so that when they received a grade for the assignment, it was highly reflective of their outcome performance.

4. We would look at the data semester-to-semester as many of our courses were offered in spring, fall, and summer. So some courses we had data three times per year, others once or twice.