cancel
Showing results for 
Search instead for 
Did you mean: 

Rubric Analysis Using the API

garth
Community Champion
29 7 7,316


Recently I had a faculty member ask if there was a way to get a breakdown of a rubric showing how many students scored in each rating of a criteria.  This would be a useful view to help determine where students were having trouble.  The modified view of the rubric might look something like this:

modified-rubric-view.png

Here you can see in red the number of students that scored in each of the ratings associated with the criteria "Requirements".  This is a simple example, but hopefully illustrates how this might be useful.

I exposed a feature to faculty to access this report through our LTI application.  LTI is a subject for another time, but I'll walk through the API calls that I used to pull this view together.

Note:  There is more than one way to code a solution, and more than one language code it with.  The purpose of this post is to give you general guidance on where to find the information needed, implementation is in your hands.

Get the Course Details

Given the course id, or the sis id, get the course definition so you can have the short name, full name, and any other course details you would like to include in the output.  Documentation on the API call to get course details can be found here:

Courses - Canvas LMS REST API Documentation

If you are using the Canvas course id:

/api/v1/courses/:id

If you are using the sis id the API call will look like this:

/api/v1/courses/sis_course_id:[your_sis_id]

The schema for a course can be found here:

Course Schema

Get List of Assignments

The next step is to get a list of assignments for the course.  You will inspect each assignment to determine if there is an associated rubric.  Documentation on the API call to get a list of assignments can be found here:

Assignments - Canvas LMS REST API Documentation

GET /api/v1/courses/:course_id/assignments

The schema for an assignment can be found here:

Assignments Schema

Where is the rubric definition?

So exactly where do you find the rubric? The Assignment definition contains the variable rubric.  This is the full definition of the criteria and associated ratings that will be used to generate the final report.

If you look at the Assignments Schema​ you will see a property name rubric.  I recommend that you use something like Postman (Postman - Chrome Web Store ) to manually call the API to retrieve details of an assignment that uses a rubric, and inspect the results.  It will be much easier to see the structure of the rubric if you look at the actual results.

Get Assignment Submissions

Now you will need to iterate through the assignments, and load all submissions associated with each assignment.  Documentation on the API call to get submissions can be found here:

Submissions - Canvas LMS REST API Documentation

GET /api/v1/courses/:course_id/assignments/:assignment_id/submissions

IMPORTANT:  Be sure to read the section on "Request Parameters", you will need to include a parameter to tell Canvas to send back the "rubric_assessment".  The API call with the parameter will look like this:

GET /api/v1/courses/:course_id/assignments/:assignment_id/submissions?include[]=rubric_assessment

The schema for a submission can be found here:

Submissions Schema

I recommend that you use something like Postman (Postman - Chrome Web Store ) to manually call the API to retrieve details of a submission that has been graded with a rubric, and inspect the results to see the structure of rubric_assessment.  This variable contains the results of the graded rubric for the submission, which you will need.

If you are using dynamic objects, it will be easy to store a list of submissions with each assignment.

Generate Rubric Data

Now that you have assignments and submissions, you can start to gather the data.

Inspect each assignment for the use of a rubric, look for the value assignment.use_rubric_for_grading to be "true".

Also confirm that the assignment has submitted work, i.e. assignment.submissions.Count > 0.

If there are no submissions you likely want to skip this rubric as there is no associated work.

For each assignment that you are going process:

Create a rubric dictionary

Each criteria in the rubric has a unique id, this will come in handy.  Create a dictionary using the criteria id as the key, and the criteria as the value.  You will see later that the submission references the criteria by id.  Creating the dictionary will allow you to quickly find the correct criteria to update without having to constantly iterate through the items.

As I added the criteria to the dictionary, I added a variable to each rating: rating.students = 0

This is the value we will use to count how many students scored in each criteria rating.

Inspect Submissions

Assuming we have found an assignment with a rubric, and there is submitted work to inspect, start iterating through the submissions.  Each submission will have a rubric_assessment variable containing the rating values for each criteria.

NOTE:  The API documentation was not clear on the structure of the rubric_assessment variable.  I suggest using Postman to make an API call for a submission that was graded with a rubric, and inspect the results to get a clear picture of the structure.

Steps I took to tie it all together:

  • Iterate through each submission and look at the rubric_assessment value.
  • Iterate through each element of the rubric_assessment and use the rubric dictionary to find the criteria:
  • Using the criteria from the dictionary, then you will need to find the rating that matches the points in the assessment.
  • Using the rating, increment the value for rating.students (this is the variable we injected into the rating object when we built the dictionary)

By using dynamic objects to represent the rubric and all of the internal pieces, the rubric object that we started with in the Assignment is now updated with the student count for each criteria rating, and can be used to generate report output.

Rinse and repeat... do this for each assignment that you would like to report on.

I kept it simple, and report on all graded assignments period.

Report the results

The rubric for each assignment is now updated with the student count for each rating in each criteria.  You can dump the rubrics to a simple CSV file, or you can format an HTML report, the choice is yours.

Regardless of what data you are trying to pull out of Canvas using the API, I strongly recommend using something like Postman​  or   DHC REST Client​ so you can manually make the API calls and inspect the results to make sure you are getting the expected results.

If people find this useful, I will work on a write-up on my gradebook analysis report, which allows faculty to request a summary report outlining potential errors that should be corrected before posting grades back to the SIS.

2016.11.14 - Update

If you would like to see source code demonstrating how to make API calls in .NET, I have posted a blog with source code here:

From this example you can derive your own workflows to automate admin tasks or reports.

7 Comments
garth
Community Champion

After writing the post above, I thought a basic algorithm might help to put the pieces together.

I have published a document for the algorithm so indentation can be maintained (html editor doesn't do a good job with indentation).  You can find the document here:

rubric-analysis-pseudocode.txt

I hope it helps : )

Stefanie
Community Team
Community Team

 @garth , I'm enjoying reading your solution, and since so much of what you have described is not yet familiar to me, I am working toward an understanding of it. Smiley Happy To enhance its visibility, I've shared this blog post with a few other spaces in the Community--the Higher Education​, K-12​, and Canvas Admins​ groups.

To help me wrap my brain around this: Have you created the equivalent of a learning mastery gradebook for a rubric, but one that can effectively be run without first having to align outcomes to that rubric?

garth
Community Champion

What I've created is only an analysis of existing rubrics.  The steps that I've outlined will output a report showing how students were assessed for each rubric in a course.

What We Have Today

Faculty approached me and said it would be very useful if they could see how many students scored in each rating of each criteria in a rubric.  Currently you can use SpeedGrader and easily analyze a single rubric for each individual student on each individual assignment.  For example, this screen shot is taken from a test course, showing how a single student was graded using the rubric:

speedgrader-rubric.png

What Was Asked For

What the faculty were asking for is an aggregate view.  With respect to the assignment as a whole, how did the students perform?  How many students scored in each rating for each criteria?  The goal being, can we easily see where students are struggling?  Which criteria rating shows the highest number of students, and which criteria rating shows the least number of students?  Where do we need to focus on the curriculum, outside of "outcomes"?

Hopefully the highest number of students are scoring the maximum number of points.  However, if you could look at the number of students scoring in each rating, and you could see that 10 out of 20 students scored only 2 points for "Word Usage", then you might want to make an adjustment to the curriculum to address how students understand this particular requirement, and attempt to raise that score next time around.

Output of the Algorithm

The results of the steps I have outlined will generate an aggregate view of each rubric in a course that is associated with graded submissions.  That combined view will show how many students scored in each rating for each criteria, to give the faculty a quick overview of how the students understood the assignment.  Using the rubric from above again as an example, the output might look like the following:

example-modified-rubric.png

In this modified view of the rubric, let's look at "Word Usage",

  • 2 students scored 5 points
  • 4 students scored 4 points
  • 4 students scored 3 points
  • 10 students scored 2 points

This aggregate view shows that half the students appear to be struggling with "Word Usage".

Although students may have gotten a passing grade on the assignment, we can see there is a clear problem area.

I hope this helps to clarify my effort, sometimes it's not easy to get ideas from my head into words : )

I am very interested to hear other ideas of how we can use the rubrics to flush out details, or other ideas about reports that would be useful to administration and faculty.  Please share your thoughts.

ilovell
Community Contributor

This is so many levels of awesome! We have several faculty requesting this information. I am interested in learning more how you implemented things as an LTI application. I have been doing something very similar for our faculty using google sheets/script and it is a mess but, it works. This is exactly what they have been asking to do. Especially if something they could run real-time and not have to deal with google sheets.

garth
Community Champion

ilovell​ I posted three articles on how to get started with LTI.

I work with the Microsoft stack, and use .NET, but the steps can be followed using other languages.

Take a look and see if they help you get started:

My rubric analysis report is a button that faculty can click in each of their courses.

I have also provided them witha  gradebook analysis report, highlighting things that may be problematic, such as over grading or missing grades or over due grades.  Once you start working with the gradebook the things you can do start jumping out at you.

I hope this helps.

Boekenoogen
Community Contributor

Will you share your code for your rubric button. This looks great!

susan_oconnell
Community Participant

Looks like many of Garth's helpful links are no longer working. Is the information available somewhere else?

Thanks for any help!