Linking Outcomes to Assignments with Python

[bbennett2]

I'm trying to make standards-based grading more approachable for my teachers. When I was teaching full time, I held to Frank Noschese's Keep It Simple philosophy. Single standards correlate to single assignments that are scored as pass/fail. Now, I averaged these out on a weighted scale to calculate a 0-100 grade, but that's for another post

Using Canvas, I was able to set up a functional reassessment strategy to aggregate demonstrations of proficiency.

The Learning Mastery Gradebook in Canvas does not translate anything into the traditional gradebook. This mean that every week or so, I would have to open the Mastery report alongside the traditional gradebook and update scores line by line. This was tedious and prone to error.

Using the Canvas API and a MySQL database, I put together a Python web app to do that work for me. The idea is that a single outcome in a Canvas course is linked with a single assignment to be scored as a 1 or 0 (pass/fail) when a mastery threshold is reached.

The App

Users are logged in via their existing Canvas account using the OAuth flow. There they are shown a list of active courses along with the number of students and how many Essential Standards are currently being assessed (ie, linked to an assignment).

Teacher Dashboard

Single Course

In the Course view, users select which grading category will be used for the standards. Outcomes are pulled in from the course and stored via their ID number. Assignments from the selected group are imported and added to the dropdown menu for each Outcome.

Users align Outcomes to the Assignment they want to be updated in Canvas when the scores are reconciled. This pulls live from Canvas, so the Outcomes and Assignments must exist prior to importing. As Assignments are aligned, they're added to the score report table.

Score Reports

Right now, it defaults to a 1 or 0 (pass/fail) if the Outcome score is greater than or equal to 3 (out of 4). All of the grade data is pulled at runtime - no student information is ever stored in the database. The Outcome/Assignment relationship that was created tells the app which assignment to update for which Outcome.

When scores are updated, the entire table is processed. The app pulls data via the API and compares the Outcome score with the Assignment grade. If an Outcome has risen above a 3, the associated Assignment is toggled to a 1. The same is true for the inverse: if an Outcome falls below a 3, the Assignment is toggled back to a 0.

I have mixed feelings about dropping a score, but the purpose of this little experiment is to make grade calculations and reconciliation between Outcomes and Assignments much more smooth for the teacher. It requires a user to run (no automatic updates) so grades can always be updated manually by the teacher in Canvas. Associations can also be removed at any time.

Improvements

To speed up processing, I use a Pool to run multiple checks at a time. It can process a class of ~30 students in under 10 seconds. I need to add some caching to make that even faster. This does not split students into sections, either. 

I've started turning this into an LTI capable app which would make it even easier for teachers to jump in. If you're a Python developer, I would really appreciate some code review. There is definitely some cleanup to be done in the functions and documentation and any insight on the logic would be great.

The source for the project is on GitHub.

[James]

Your experience with outcomes and transferring them to grades seems like mine -- nightmarish. I also wrote code to tackle it and decided not to do outcomes ever again. Okay, "ever" might be too long.

I used the API to pull all of the outcomes into a CSV file and then used Excel to calculate their grades. I then had a single grade that had their class average in it. Each week I would create a new one and move the old one into an assignment group worth 0% so that they could track their progress over time.

All of that was predicated because I had originally chosen one method of grading and found out that it was taking too much class time to assess things and it was difficult getting repeats on the missed material on new quizzes.

Thanks for sharing your code that will help others with their struggles with outcomes. And thanks for sharing the link to that article, it's helped me not have such negative views towards outcomes.

[bbennett2]

Honestly, I think Outcomes are one of the parts of Canvas they're really missing some cool opportunities on. I've talked with the engineers and our CSM several times about some simple ways they could make Outcomes much more powerful, this being one of those ideas. I'm hoping they put some more resources into the component in the near future, especially because of the renewed push for competency based reporting across all levels of education.

[nate_mcclennen]

@bbennett2 Hi - Just came across this post and wondering if you ever were able to pull unscored outcome data out of Canvas if the outcome was on a rubric and the rubric was linked to an assignment and the assignment to a student. We are having trouble getting unscored data on outcomes out via Canvas Data so searching a bit to find expertise...

[bbennett2]

@nate_mcclennen I'm not sure I follow what you mean for that case. This tool pulls outcome results from the Learning Mastery Gradebook, which is the aggregate for that outcome. I'm not doing anything right now for single assessments. 

To make sure I understand, you're looking for information on students who have no outcome score for a specific assignment? 

[nate_mcclennen]

@bbennett2 I see - you are pulling the outcome result (decaying average, most recent, etc.). So, not applicable, I am looking for individual outcome scores on assignments (or more specifically, assignments assigned to students and connected to rubrics with outcomes, but outcomes have not yet been assessed).

[bbennett2]

@nate_mcclennen 

Ah, got it.

I actually do have something like that...though it's not written up anywhere. 

I broke it into two parts, one function to call the rubric and get the outcome IDs and a second function to return structured data (a list of dicts).

Rather than posting it all here, I've put an annotated version in a GitHub gist for easier reading. This will run in the Python interpreter. I use it in an internal LTI to display a table for teachers to look at assignment scores alongside individual Outcome results from that assignment. Hope this helps.