I direct a program and would like to create an Outcomes rubric to gather data for assessment. It wouldn't have to be used in each class for grading, but could be something an instructor could quickly use to rate their students and gather data. This would be done by different instructors in the program, so I would need to have access to all of their outcomes to generate a report for assessment.
Is anyone doing this? Is it useful? Are there quirks about setting it up that I should watch out for? Or should I look for a better way to collect this data?
Outcomes sounds like a good tool for this, but I'm not sure if it will work the way I want it to.
@kadunkelberg , since you've posed a prompt designed to elicit multiple responses, with no single one likely to stand out as uniquely "correct," I've flipped the format over to a discussion. I've also shared the discussion with the Higher Education and K-12 groups, and have added a few tags, to enhance its visibility and availability in search results.
Do you have access to your programs sub-account? If so, you may want to use it. Creating outcomes/rubrics at the sub-account or account level can allow you to pull the data on the outcomes more easily. Keep in mind that this only works if you are using the same outcomes (and measurement scale for these outcomes) across all the courses, and that once you gather scores for an outcome at the sub-account level, you cannot delete it.
Thank you for replying. Our school is fairly new to Canvas (1-year) and so far I do not have access to our program's sub-account. It's good to know what to ask for, though!
For program assessment, we would use the same rubric across all courses or all that we are assessing, which is why it wouldn't be used for grading but only for assessment. It's okay if I can't delete the scores (though I'd be interested in hearing what problems that could cause that I should watch out for). I'd also like to find out more about how to download the results, if that's possible.
I do think it's a good thing that you can't delete scores, but it also means you can't edit the outcomes. For example, imagine you implement your outcomes, then realize you'd prefer different wording. You can't change it. One way we got around it at my previous institution was to have the outcome descriptions point to an external rubric that we could edit. For example, effective communication pointed to the effective communication rubric. We never had to edit the actual Outcome in Canvas, but it allowed us the flexibility to edit the effective communication rubric.
Another issue can be if you revise outcomes frequently. This could mean having a large number of Outcomes in your sub-account or account that you can't delete and that you can't retire or otherwise keep people from using. This could cause an issue if your 2018 rubric is different from your 2016 rubric, and some faculty use the incorrect rubric. It can also very easily get cluttered or difficult to navigate, so I suggest a very clear structure from the start.
I like using the Canvas Commons to share assignments set up with the rubric attached. This is an alternative way of handling things if you can't get sub-account access, or if you want to have some assignment settings in common.
Downloading the results is pretty straightforward for outcomes at the sub-account or account level (there are two reports that are pre-built in Canvas for this), but it can get messy if the outcomes are just at the course level. Keep in mind that the data provided is very raw, so if you have many outcomes, it can take a bit of time to get it into a friendly format.
Instructure has been reaching out to a variety of institutions to get more information about how Outcomes are used, could be used, and might be re-imagined within Canvas. They seem like they are really doing some good background work to make it work better in the future.
I'm glad to hear Instructor is taking Outcomes seriously, since it seems that it could be used effectively for assessment.
It seems like the biggest problem with allowing revisions would be that you could change the meaning of the rubric (and one that has already been used) by altering it for future use.
The advantage would be that you could fix a typo.
This is similar to the issue with editing a quiz, where changing the wording of a question to correct a typo seems fairly straightforward, but editing a question too much could change the correct answer.
How to strike a balance between allowing enough editing to fix a mistake and not allowing too much that would significantly change the meaning of the rubric once it has been used?
That is a challenge! At my previous institution, the account-level Outcomes were managed by a very limited number of people, and even fewer had the ability to edit typos in the Outcome rubrics. In practice, we only made small tweaks to the wording on the rubrics, but it was a risky approach. I'd love to see some type of versioning so it's clear what version of a rubric someone was looking at when they submitted an assignment.