Quiz Submissions API: are inactive enrollments (no longer) included?

[saelterman]

Hi,

Many months ago I started a project to extract quiz results by question and by student. I am quite confident I got a Postman collection working that gave me all the quiz submission events for a particular quiz in a particular course.

There would be quite a bit of work to determine if the event was the final selection a student made for a question and related minutiae, but it seemed this was the way to go.

Then I got sidetracked and just now returned to the project. The API call I had before (GET request URL below) returns a status code of 200, but no submissions are included at all. The UI shows submissions for that quiz in that course, though they all belong to now inactive enrollments.

Is there perhaps a query string parameter I am overlooking to include results from inactive enrollments?

My GET request:

https://troy.instructure.com/api/v1/courses/{{CourseID}}/quizzes/{{QuizID}}/submissions?include=subm... 

JSON response:

{
"quiz_submissions": [],
"submissions": [],
"meta": {
"primaryCollection": "quiz_submissions"
}
}

https://troy.instructure.com/api/v1/courses/

[James]

 @saelterman ,

What your end-game is here? It sounds like you're trying to get how students did on individual questions (John missed question 5 on quiz 2 because he answered "Red" when the correct answer was "Blue."), but there's a couple of other things possibly going on here as well.

If you're after the just the grades, then use the List submissions for multiple assignments from the Submissions API instead of the Quizzes API? Here's a similar thread from 3 years ago that discussed the submissions API and suggested that an alternative for those who need the quiz API is to add the students back into the test or beta instance and then download the results: Quiz submissions API for completed enrollments 

If you're after the actual responses to the questions, then the quiz submission events may not be the way to go. They expire after a time (without double checking, I think it's a year -- I did check 2 years and they're not there anymore). I used to recommend that as the only way to get the student responses, but in October 2018 I discovered that the list submissions methods of the Submissions API will return the actual results of the students if you add the query parameter include[]=submission_history.

If you just need the Student Analysis Report, which gives a CSV for the questions and the student responses and how many points they got for each question, that is available through the web interface. You can also schedule it through the Quiz Reports API. The CSV file isn't very friendly for certain types of questions, but it's a lot easier to obtain.

[saelterman]

Hi James,

Thanks for the quick reply.

I am indeed after the information that will tell me which students answered which questions correctly (broadly speaking, it's related to AoL reporting).

The last time I looked at the CSV files available through the UI, they didn't meet our needs, but I can check that again. I was also trying to avoid having to go in to every course and download the CSV manually, but was I was unaware of the API to schedule the reports. I will look again.

I was afraid they would expire, so I did try to get info for more recent courses, and all their submission events were blank too.

I am trying the Submissions API now to see what that returns. Thanks for the tip!

Sven.

[James]

I completely agree with you about the CSV files. They haven't gotten any better

The Submissions API with the include[]=submission_history will give you their responses, but you'll still likely need to pull in some extra information from the other quiz calls if you want to know what the actual responses were. If you're just after whether it was correct or not and how many points they got, it's included with the call.

When you make that call, there is a submission_data property that looks like this:

[saelterman]

The sample data that you showed is exactly what I need (pending some evaluation where question_id can lead me), however, there is no submission_history and no submission_data again.

The source code for the Submissions API (line 235) uses the same method I discussed in my reply below, meaning that if my analysis of the source code is correct, submissions by inactive enrollments are excluded.

Next, I am going to try to direct my API calls to the test environment and activate the enrollments to see if that will give me what I need.

[James · Accepted solution]

 @saelterman ,

See the Canvas Enrollment Status Comparison pdf

Have you removed these enrollments from the course (deleted them)? If so, then you're not going to be able to get anything. If we delete them, we have to re-add them to get anything. 

If they are inactive, then it says you can see previously submitted assignments. That means that there is an option to get that information. The gradebook fetches the data through the API, so it is possible.

We don't mark our students as inactive, we either delete them or conclude them depending on whether it's within the first 10 days of the course or after that. That means it's going to be a little challenging for me to test.

I have a suspicion about what is going on. In the new gradebook, when I toggle the "Show inactive enrollments", Canvas makes a call to an internal location (not an API) to get the list of users. However, all of my development work with the gradebook was done with the old one and when I toggle it in the old gradebook, it makes a call to get a list of users that should be included and then it specifies the individual users using student_ids[] multiple times rather than using student_ids[]=all.

The call to get the list of users looks like this:

GET /api/v1/courses/2513281/users

?enrollment_state[]=active&enrollment_state[]=invited&enrollment_state[]=inactive&enrollment_type[]=student&enrollment_type[]=student_view

&include[]=avatar_url&include[]=group_ids&include[]=enrollments&per_page=100

Then, it takes a list of those student IDs and make a call to the submissions API (I've replaced the actual IDs by {id#}

GET /api/v1/courses/2513281/students/submissions?grouped=1
&student_ids[]={id1}&student_ids[]={id2}&student_ids[]={id3} ...
&response_fields[]=id&response_fields[]=user_id&response_fields[]=url
&response_fields[]=score&response_fields[]=grade&response_fields[]=submission_type
&response_fields[]=submitted_at&response_fields[]=assignment_id
&response_fields[]=grade_matches_current_submission
&response_fields[]=attachments&response_fields[]=late
&response_fields[]=workflow_state&response_fields[]=excused
&response_fields[]=cached_due_date&exclude_response_fields[]=preview_url

I include all of the parameters because it was helpful for me to minimize the information that I needed. In particular, the exclude_response_fields[] can help cut down on descriptions on assignments to save the information transferred.

Based on that ...

Instead of using student_ids[]=all have you tried using student_ids[]=123&student_ids[]=456 where 123 and 456 are the Canvas IDs of some students who are inactive?

I'm using the documentation from List submissions for multiple assignments (just so we're on the same page)

[saelterman]

Hi James,

Thanks for helping me to research this.

1. I've since confirmed that the reason I am not getting submissions is because enrollments are inactive (not deleted or concluded).
2. I am not actually using the List submissions from multiple assignments API, I am using the List assignment submissions. I suppose I could switch and use your suggestion. My alternative would have been to use the test environment and temporarily re-activate all the students in the course.

While I am considering this, I have moved on to getting some information about the quiz questions. Because the quizzes use quiz groups based on assessment question banks, I am now struggling with the next part. There seems to be no endpoint for listing all the question groups in a quiz. (I may ask a separate question in the community about that; I don't want this thread to get into too many different directions.)

PS: This is just to satisfy my curiosity, but are you indicating that you are deleting all enrollments from your courses after 10 days of the course end date?

[James]

PS: This is just to satisfy my curiosity, but are you indicating that you are deleting all enrollments from your courses after 10 days of the course end date?

No. We have a "census date" that is required for reporting to the state, it's generally the 10th day of the semester. If a student drops before that date, it's like they were never there as far as reporting goes. For those students, we delete their enrollment. We conclude enrollments for any students who make it to the census date and drop later.

We don't delete or conclude any enrollments when the course is over. We let Canvas soft-conclude them for us. That means that if I go back to my Fall 2012 course (first one we used Canvas with), my students and all their work is still there.

I am not actually using the List submissions from multiple assignments API, I am using the List assignment submissions. I suppose I could switch and use your suggestion.

Try my suggestion. The list submissions for a single assignment doesn't allow you to specify the students and it doesn't include the inactive ones. The one I'm recommending allows you to list multiple students including those who are inactive. It allows you to get results for more than one assignment at a time, too, although you can specify just a single assignment if you like. 

You'll still need to make another call to get the list of inactive students, but it sounds like the only way you're going to get to the information is to use the more flexible multiple assignments version. 

What you could do is use yours for the active students (if you use this in the future when they're still active) and then get a list of just the inactive ones and use the multiple assignments version with the inactive list.

Because the quizzes use quiz groups based on assessment question banks, I am now struggling with the next part. There seems to be no endpoint for listing all the question groups in a quiz. (I may ask a separate question in the community about that; I don't want this thread to get into too many different directions.)

Correct. Linking to questions inside banks is problematic as there is no direct API for getting the results out. Getting questions from banks out is problematic in general. You can do is create a quiz that contains all of the questions in the bank by adding them to the quiz rather than linking them. Then export that quiz as a QTI file and parse it. The one part I'm not sure about is whether that will contain the question ID that you need or not.

There may be another way to get to it if you can get the questions that were actually delivered with an exam, possibly using get all quiz submission questions. That call does support an undocumented parameter quiz_submission_attempt. That potentially means a lot of API calls to build a working copy.

I would have to review and test some things before I feel could comment with any certainty. I perceive the problems with linking to question banks to be more than the benefits, so I don't link to them and don't have anything readily available for testing.

[saelterman]

I am at it again, and this solution does indeed work. Thank you for your efforts!

[saelterman]

I've started digging in the quiz submissions API source code. Line 173 seems to indicate that there's some process involved to determine the "visible student IDs." (see canvas-lms/quiz_submissions_api_controller.rb at d0deea460cdc67757fcd4148833a4aebbc584aa5 · instruct... )

This leads to line 2462 of models/course.rb which seems to indicate that 'inactive' must be specified in the include parameter. But the include parameter from the call to get quiz submissions doesn't get passed down to that method and thus submissions from inactive enrollments aren't included.

(My analysis may be flawed, I am not used to reading Ruby code.)