[ARCHIVED] Best practices in setting up Outcomes

[CWilkerson]

Hi Canvas Community,

Our institution is just getting started in Canvas, migrating over from Moodle.

We are looking to implement Canvas Outcomes in order to gather the Mastery data associated with it. As we do this we see that there could be Institution, Program, and Course Outcomes that could potentially be measured by linking Outcomes and Rubrics and attaching that rubric to an Assessment. What suggestions do you have for a new institution setting up Outcomes? Do we focus on Program outcomes in order to draw the most robust data on that program and student OR should we go ahead and start with Course Level Outcomes and gather data that way? Any suggestions would be greatly appreciated. We want to be sure we gather the most appropriate/useful Mastery information.

Thanks!

[biray]

Looping Deactivated user​ into the conversation. He might have some insights.

[CWilkerson]

Thank you!

[bhauf]

Deactivated user​, I am doing a presentation at InstructureCon15 on this very topic. We (UNT Health Science Center) have been using Outcomes for a little over a year now. The session should be recorded and made available after the conference. I will attempt to write our best practices and experiences; it is just a copious amount of information that may take a bit to put together.

[bhauf]

Deactivated user​ Here is the write-up. I left out using questions banks for scoring outcomes as that will take longer to explain. I will attempt to add screenshots for some of the descriptions below as I am able to.

Overview

Our institution recently realigned program competencies and outcomes to assess student performance across the curriculum. Here is brief description of the process:

One of the schools at our University was searching for a better solution to assess competency through various assessment methods, such as quizzes, papers, oral presentations and examinations. At the course level, faculty outline competencies on their syllabi and map them to assessments that are delivered throughout the semester, enabling students to see how competencies are introduced and reinforced throughout the curriculum.  The school was looking for consistency in how students were assessed across programs, so program level competencies and outcomes were introduced and used for aligning with assessment items. This approach made it easy for us to collect data on competencies and outcomes that were used across the program to see how students are progressing.

The Setup

We started at the highest account level in Canvas for creating outcomes. At the time, this would make it easier to update outcomes and import them into sub-accounts and courses so that all the data rolled up to the highest level.

Structure

We structured, or grouped, the Outcomes as follows:

• School > Program > Concentration > Competency > Outcomes
• Bold indicates a Group/Folder created within the Outcomes Tool
• These Outcomes were then imported into the School’s sub-account in Canvas so that they could be assessed by faculty enrolled in courses organized in this sub-account.

The structure was created for faculty to easily navigate to and select outcomes to import into their courses. We used the competency and outcome alignment matrix developed by the school to define this structure.Group/Folder structure is also important for the Student Learning Mastery gradebook. Depending how you organized your Outcomes in the course level outcomes tool will determine how outcomes are organized in this view. For example, we imported the Concentration folder, competencies folder, and outcomes for each competency so that they were grouped by competency in the student learning mastery view. This enables you to be able to see how many outcomes a student is mastering within a competency.

Nomenclature

Again, the nomenclature was created based upon the structure. Each outcome was named as follows:

• School.Program.Concentration.Competency#.Outcome#
• Example: SPH.MPH.EOHS.C1.LO1
• The reason for this nomenclature is due to the Outcomes report tool in Canvas. The report tool does not include the group structure, so we had to create a name that allowed us to see where this particular outcome resides.

Rubric Scale

For easy implementation, our school agreed upon a 5 point scale with easily identifiable descriptions for every learning outcome. This varied slightly from the default rubric in Canvas (5,3,1) as we saw the need to allow faculty to score students who fell between outstanding, acceptable, and unacceptable. Since learning outcomes in a particular program are assessed across multiple courses, it was important to implement a method that was both easy to read and use for both students AND faculty. We the used the API to update outcomes with the rubric after they were created.

There are also some calculation methods to consider. Best score, highest score, decaying average. See here: What calculation methods are available for Outcomes?

Be sure to decide on a calculation method as this will impact the learning mastery gradebook and cannot be adjusted after an outcome is scored.

Alignment

Since outcomes were created at the account level, they need to be imported into the course. Things to note:

• Make sure your outcomes are grouped so that they are organized in the student learning mastery view.
• Faculty cannot delete outcomes from their course once they are imported (admins can), but they can delete outcomes that are in groups. We decided to import outcomes into the course for the faculty so that the folder structure was correct and persistent through all courses.
• If you import any outcomes that will not be assessed, delete those from the course since they will display as "not met" in the learning mastery areas. We import into the course-level based on concentration and then delete those that are not needed so that the folder structure is retained.
• Outcomes are aligned using the rubric tool that can be attached to assignments, quizzes, and discussion boards. Outcomes can be imported onto the rubric to either grade the student or not impact the grade at all. We chose the latter so that outcome scoring was separate from traditional student grades (A, B, C, D, etc).
• Scoring outcomes occurs in speedgrader.

I hope this is helpful. I'll try to clean this up since it is essentially an information dump from my head.

[benjamin_rodrig]

Hi Bryan, 

You provided a very helpful write up on how your institution set up your Outcome structure. 

I know this is an older post, but here's a question: Does this Outcome group structure mirror your Institution's overall Sub-account structure?

Thanks,

Ben

[CWilkerson]

Oh perfect! Thank you! This is so helpful. I will certainly watch the recording and appreciate you getting me some items in "writing" as well I am hoping to make it to CanvasCon next year as our Institution will be about a year into using Canvas.

[biray]

Deactivated user​ (Thanks for chiming in  @bhauf ​)

If you want more information about outcomes, here is a compilation of all the sessions at InstructureCon 2015 that are related to outcomes and rubrics:

Verifying Student Learning with Outcomes

A Collaboration in Measuring Student Competencies and Outcomes in Canvas

Woodstock via the LMS: Canvas for American Studies

Herding Fish: Encouraging Faculty to Use Subaccount-Generated Outcomes and Rubrics

Orchestrating Instructional Design at the System Level

Also, if you (or anyone else reading this thread is not at InstructureCon this year, no fear... every session is being recording and will be available after the conference for viewing).

Hope that helps!

[CWilkerson]

Thanks a million!!

[emily]

This is very helpful - thank you! And I may need to go to InstructureCon.

A question - how do you do program-level reporting on Outcomes? I know how to find the Learning Mastery report for one course, but we are interested in looking at outcomes across our entire curriculum.

Thanks,

Emily

[scain]

My institution is new to Canvas, and I'm trying to dig into Outcomes as it shows remarkable potential. We've set up a large enrollment course to pilot outcomes, but I have run into one question. Our outcomes are built using a 0, 1, 2, 3 rating system, with mastery at 2. However, the rubrics are used for grading, and we needed to have it be a 5-point value for each of the outcomes. What I've noticed is that the value of 5 is not only used for grading that particular assignment but even passes along into the Learning Mastery grade book. Now, students who received 5 in the rubric are showing 5 in the Learning Mastery for that outcome, which could drastically distort their achievements. Has anyone else reported this? What is the best way to handle outcomes that are used to score assignments? Is there a better way to be handling this process?

[aharkless]

Hello,

I know this post dates back to 2015. However, our university is trying to dive into using Outcomes. We are trying to determine best practices as well as finding someone that has retrieved data from Canvas Outcomes.

[kkhan]

Some best practices that I discovered are: 

1. Start with course goals: Before setting up outcomes, it's important to have clear course goals that you want your students to achieve. These goals can be derived from learning standards, program objectives, or departmental goals.
2. Use outcome groups: Outcome groups are a way to organize related outcomes. For example, you might group outcomes by module, assignment, or learning objective.
3. Use measurable verbs: When creating outcomes, use measurable verbs that describe what students should be able to do. For example, "analyze", "evaluate", "synthesize", and "apply" are all measurable verbs that can help you define clear and specific outcomes.
4. Align outcomes with assessments: Once you have defined your outcomes, align them with assessments. This will help you track student progress towards meeting those outcomes. You can align outcomes with assignments, quizzes, and discussions.
5. Set mastery criteria: Setting mastery criteria helps you define what level of proficiency students need to demonstrate to meet the outcome. For example, you might set mastery criteria at 80% or higher on a quiz or assignment.
6. Communicate outcomes to students: Make sure that students understand the outcomes and how they will be assessed. You can do this by including the outcomes in your syllabus, course objectives, and assignment instructions.
7. Track student progress: Canvas LMS allows you to track student progress towards meeting outcomes. Use this feature to identify areas where students may need additional support or to adjust your teaching strategies to help students meet the outcomes.