Automatically grading quizzes with an external tool

[vmiletic]

Suppose I want to have an quiz in Canvas, where each question (let's say Essay or File Upload type to start somewhere) is graded by an external software tool. I'm curious about two things for this scenario:

1. Can I get Canvas to call the external software tool via some kind of API upon student's submission and get the response and use it for grading the question? If yes, can I do so without modifying the source code, i.e. by writing an extension of sorts?
2. Should I instead get an external tool to use Canvas API to grade the submitted answers (this Quiz Submissions - Canvas LMS REST API Documentation  if I understand correctly)? Can I use this API on https://canvas.instructure.com/ , i.e. without self-hosting?

Thanks in advance.

[pklove]

I don't think you can do (1) without modifying Canvas.

You should be able to do (2).

You might be interested in some of the discussion at: https://community.canvaslms.com/message/91588-lti-external-assignment-with-canvas-submission-system 

[stuart_ryan]

Hi  @vmiletic ,

I agree with Peter, 2 should be completely doable.

I have seen something recently in the API documentation that ?may? allow you to achieve one. There are now web-hook subscriptions available Webhooks Subscriptions - Canvas LMS REST API Documentation  where you can subscribe to know that a change has taken place. 

You could (conceivably) use the web hooks to have your external tool be notified of a new quiz submitted (using the QUIZ_SUBMITTED Event Type) which you can then have your tool do the rest of the interactions with the API to push the mark back.

I have never authored something like this before, but, conceivably quite doable. I would note that Canvas in this case won't call the external tool and expect a response, it will more call the external tool to say 'hey something has happened' and then rely on your tool to do the heavy lifting once it is told of an event (such as doing the grading, and then using the parameters in the initial notification to mark using the API).

I would be very curious to see how you go if you can get this up and running!

Hope that helps,

Stuart

[pklove]

Oh, I missed the Webhooks.

It would be interesting to know if anyone is using them.

[stuart_ryan]

Agreed! They are only very new, I haven't even had a chance to tinker or heard of anyone that has. Would be very interesting to see how they work and play with a realtime integration. I will keep my ear to the ground at InstructureCarn this year to see if anyone has started using them and might have some examples.

[vmiletic]

Wow, thanks for quick response(s)! I checked the other question and I'll check the Webhooks Subscriptions API.

I have no problem doing the heavy lifting myself, the idea that I could be notified upon submission and don't have to trigger grading manually is already more than I expected Canvas to do.

I plan to open source the tool and will gladly share the code here.

[kyle_tower1]

Is there any updates on this project or tools you've developed or found useful in the venture to automatically grade coding assignments?

[vmiletic]

Unfortunately no, in the end for that course I switch to Moodle and used Preg question type instead of developing something custom.

[pimmel2]

So, let me get this straight.  There are apps out there that will grade essay questions / quizzes?!  What are folks using?   Very Interested.

Pat

[vmiletic]

I assume this question is directed my way.

We do not plan to grade free form essay answers, at least not for now. However, we do plan to grade questions that expect somewhat structured answers, i.e. a shell command, a couple of configuration directives, filling a table, a list of words/numbers, or something similar.

I'm not aware of any tool that would work for our use cases. I'll be developing my own tool (likely in Python because that's what I work with most of the time) and I will post the source code.

[maguire]

For 2, you need somewhere to run your program that uses the API to get the submissions. This could be in a local machine or the cloud.

The key question is when should your program fetch the submissions to grade them;? Alternatives: check periodically, user-triggered (either the user taking the quiz or the teacher triggering a check for new submissions to be graded), or try to use webhooks to get a notification that there has been a new submission. An external script can easily check for submitted but ungraded submissions. For an example of getting the ungraded submission information, see https://github.com/gqmaguirejr/Canvas-tools/blob/master/list_ungraded_submissions_in_your_courses_JS... 

A third approach is actually to do the grading in real-time via a LTI application that realizes the quiz - in this case, the user can get feedback on a per question basis. Additionally, you can have a dynamic path through the questions where you can ask different questions depending upon the user's answer to the previous question. For this approach, you need somewhere to run the LTI tool.

So a lot depends upon what time scale you want to operate on.