cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
Community Coach
Community Coach

Outcomes and Canvas Data

My supervisor, eludwig@morainepark.edu‌, has asked that I do a bit of research on using Outcomes at the Account (or Sub-Account) level and associated rubrics and then wanting to use Canvas Data to pull that information.  Our General Education department would like to start implementing assessment across courses (linked to Core Abilities) and then be able to report out on the results of that assessment.  We are interested how/if other institutions are using them (especially in connection with Canvas Data).

As a Canvas admin at our Technical College, my knowledge of Outcomes isn't stellar, and at times I only use Outcomes for building parts of rubrics in odd and awkward ways, and we don't use Canvas Outcomes at the college level too much...as far as I'm aware.  So, any input would be helpful.  Thanks!

13 Replies
Highlighted
Adventurer III

So, I have zero knowledge of how Canvas Data works because we don't use it. But, I use Outcomes to essentially power standards-based grading in my course. This summer, I'm coaching a cohort of teachers to use the same method in their classes (all 9-12 grade).

There are some setup pieces that can be done either at the teacher (course) level or at the admin (account/subaccount) level to make it work.

1. Outcome Imports

Instead of using the built-in CCSS or state content standards, we're rolling our own. This does a couple things for us:

  • Each department has the same essential standards (guaranteed curriculum) included in their course by default. These standards are written in student-friendly language (I can..., I will...,) so they understand the goal.
  • The GUID for each Outcome is consistent so we don't have import conflicts.
  • All teachers have the same base set of Outcomes to use.
  • We define a 0-4 rubric instead of Canvas' 0-3-5 scale (more on that below).

Teachers are encouraged to add additional Outcomes to their course based on focus, preference, etc. We offer that service through the admin console as well.

2. Rubric Scoring

For standards-based grading to work well, the Canvas 0-3-5 default scale is not descriptive enough. We use the following (based on several research-backed practices):

  • 0 No evidence - there was nothing turned in.
  • 1 Does not meet expectations - Work submitted that does not show evidence of comprehension.
  • 2 Approaches expectations - Work submitted shows comprehension, but with some misconceptions that need to be addressed.
  • 3 Meets expectations* - Work shows comprehension of the topic. This is the baseline for calling an outcome "mastered."
  • 4 Exceeds expectations - Work goes beyond the immediate standard and shows holistic comprehension.

This scale is important for a couple reasons:

  • Mastery does not mean perfection. Meeting standards - showing understanding with the individual, immediate standards - is doable in every assessment situation. Over time, the aggregate of these evidences are used to report mastery.
  • A zero is used if and only if there is no work submitted.
  • The teachers must distinguish between meeting expectations (3) and not meeting expectations (2) in every assessment. There is no neutral response.

3. Grading

We set our mastery marker at a three because it leaves room for going above and beyond the immediate assessment piece. To calculate the aggregate, I suggest a 65/35 decaying average because, in my experience, it is accurate. The last attempt has a little more weight on the score while everything else remains in play. Scoring based on the last attempt only increases the chance for a fluke (positive or negative), so it isn't reliable. The n-times score only accounts for the five most recent attempts.

Individual outcome mastery percentages can be tweaked depending on the assessment. Tagging an Outcome on a question bank allows you to determine the percent correct to be marked a 3 on the rubric. If it's an old standard (one we've used multiple times) I'll sometimes bump that cutoff toward the 80% mark. Other times, if it is difficult, marginal, or new, I'll drop it toward a 50%. The fluidity helps set an appropriate bar to account for exposure time, etc.

The Learning Mastery report does not update the traditional gradebook out of the box, so you either need to update scores line by line or create your own extension or LTI plugin to handle that function (if you want it).

I think that's the bulk of your questions. If you want to do a Hangout or Zoom call to talk more nitty gritty, I'd be happy to set up a time.

Highlighted

This is great, Brian! We use outcomes built at the sub-account level in Canvas to track program-level outcomes in a doctoral program. The key is setting it up correctly and applying the outcome to the assignments. Most of our faculty will use the outcome without adding points to their assignment rubric and this feature allows for quick data collection by the assessment committee. I'd also be more than willing to talk or share our experience with using outcomes!

Highlighted

We have each of our buildings on subaccounts, so we may go that route as well for some of the outcomes that are location specific (elective courses, etc). I agree on the points - translating rubric scores into a letter grade, if necessary, is weedy. It's one thing I'm going to work with a group of ~20 people on this summer, to try and find a method that works well.

Highlighted
Surveyor

One of the major problems with Outcomes is that it uses the same rubric as your assessment rubric. What I mean by that is if you use a rubric to assess your student assignment and you want to use Outcomes within Canvas to track the progress you have to add the points for your Outcome to your assessment. For example, I have an assignment worth 15 points and I am using a rubric to grade it. Now I want to add an Outcome rubric to see if the student is mastering the information. I set the rubric for 20 points. The assignment now has to be 35 points. There is no way to have two different rubrics assigned to the same assessment. 

The simple solution is for Instructure to add some code that would allow for two rubrics assigned to an assessment. One for academic assessment and one for Outcomes.

0 Kudos
Highlighted

John, one thing you can do is to not add points from the outcomes Rubric. There is a checkbox when adding an outcome to a rubric to not have it add points to the assignment. The "points" tracked in the Outcomes still show on reports but will not impact the student's score on the assignment. Our faculty use this exclusively to track program outcomes without adding a bunch of points to the assignment. I hope this helps!

Highlighted

john.r.boekenoogen-1@ou.edu‌, if you un-check the "Use this criterion for scoring" checkbox when importing the Outcome into a course-level grading rubric, it will not add points to your assessment.

Highlighted

Has there been an update to Outcomes? I tried this a few months ago and what we kept seeing was that both the academic Outcomes were totaling up in the rubric. Even checking the box that says not to add the scores to the grade. The grade box was blank and instructors would have to look at the rubric, subtract the Outcome points and then add it to the grade box. If this is still true, I think instructors will be confused still on why the two numbers are there. Plus, I am not sure what the student sees when they look at the rubric. I believe it shows both the Academic and the Outcomes in the same rubric which is also confusing for the students.

The tutorials from Instructure are not very clear, (in my opinion), and if someone has a better step-by-step process I would be happy to look at it.

0 Kudos
Highlighted

john.r.boekenoogen-1@ou.edu‌, the ability to exclude the points for outcomes from the rubric score has been available for quite some time—I can't say exactly how long, but I aligned account-level outcomes with course-level rubrics extensively when I was teaching, and I stopped teaching in early 2015—so I don't know why the academic outcomes are adding points to course-level rubrics in this case. 

The process for bringing outcomes into course rubrics is documented in the Import Outcome section of How do I align an outcome with a rubric in a course? 

In addition, when the Use this rubric for assignment grading checkbox has been enabled in the assignment rubric itself, the rubric scores automatically populate the grade field, eliminating the need for double-entry (i.e., the SpeedGrader grade field will not be blank). This is detailed in How do I add a rubric to an assignment? 

If these resources do not help solve the problem, please ask Canvas Support to look at some specific examples.

Highlighted

I want to echo Stephanie's point. I use Outcomes heavily in my classes to facilitate Standards-based learning (grading). Each outcome is added to an assignment, unchecking the "Use this criterion while scoring" box when I add it to the rubric. The other thing I've done is check the "Remove points from rubric" box to leave only the rubric descriptors. At the end of the semester, I have a list of assignments worth a point value (usually less than 10, essentially used for completion credit) and Outcome feedback for the student.

Outcomes haven't gotten much love lately and I think it's a huge missed opportunity for Instructure. Getting it just right is a little bit of a shell game, making sure the correct boxes are checked, etc. My grade flow turned into completion for turning it in and then giving feedback on each Outcome linked in the rubric with more specific feedback on the assignment comments. It worked really well and made the process manageable.

If you want more detail, I have a blog post in Canvas Developers about an LTI app I wrote to manage updating grades based on aggregate Outcome scores. I'd also be happy to answer whatever specific questions you have directly.

Labels