Display math multiple choice answers in quiz statistics?

[willi5cl]

I write multiple choice/multiple answer quiz questions where the answers are mathematical formula using the equation editor in the rich content editor.  The options show up nicely in the quiz, but don't display in the quiz statistics.  (See attached screenshot.)  I typically set up the quiz to shuffle the answers.  Is there a way to get the quiz statistics page to show the equations so I can better interpret my students' responses without turning off the answer shuffling?

[James · Accepted solution]

 @willi5cl  

Short answer: Within the current limitations of the Canvas interface, this is not possible.

This looks more like a multiple answers question than a multiple choice question. I'm going to answer the question like it was one, but it wouldn't change the answer much if it were multiple choice.

One thing you can do is download the Student Analysis. It will give you a list of the question and the responses chosen by students. From there, you may be able to get an idea of which questions were answered.

Here, the first attempt (bottom row) was left blank while the second attempt (middle row) had the first three items checked. The a, b, c there are in the unshuffled order.

Unfortunately, there's not an easy way to determine from the the quiz statistics page. The order there does not match up with the order in the original or in the shuffled order. It might be in the order edited (I think I went back to edit one of the answers, but don't remember exactly). But I wouldn't hold out hope on that.

If I use the browser's developer tools to inspect the table, I get this:

From the reactid's, I can tell that this is question 110073156 and that the responses are 4917, 9980, 1646, and 8900 in that order.

If I query the API for the questions on the exam, I can find out what those questions are.

For example, if my path (the part after the hostname) in the browser's location is

 /courses/896851/quizzes/6345859/statistics

I would change it to

   /api/v1/courses/896851/quizzes/6345859/questions

to get the questions for the entire quiz or  

   /api/v1/courses/896851/quizzes/6345859/questions/110073156

for just this question.

The output is mostly unreadable in the browser (this is just part of it).

I use a REST client to query the API and it provides a formatted response. You want to look at the answers section.

Here I see 4917 is the integral of x³√(x²-4) dx and 9980 is the integral of x ln x dx.

You could copy all of the output from the browser, ignoring the while(1); at the beginning, and paste it into an online JSON formatter to clean it up.

But it's a lot of work.

Note that the answers here matched the order presented because I forgot to tell it to shuffle the questions.

On another quiz, that was shuffled, I got

The order of the responses here is 9051, 3573, 4549, and 8362.

The order returned with the API call is 9051, 4549, 8362, and 3573. This is the same order that they appear in the quiz.

You might be able to take the API call results and just look at the order of the IDs without messing with the actual content. That would tell you the order and you could compare that to the order the questions are in when you edit the quiz. Then you would just need to match it up with the order that was presented on the page.

This is all manual interaction on your part.

It won't work if the questions are being pulled from a linked question bank (it's okay if they were found in a question bank and included in the quiz).

If you wanted to have it display the results, you would write a JavaScript program that would listen on that page for the answers to be rendered, then use the data-reactid to replace the content by fetching it from the quiz and injecting it into the table.

That might cause problems with heights of the rows, I didn't go that far into my testing.

And as a final note, Canvas is making a push to move everyone to New Quizzes, so if you write the program to do that, it will be a temporary solution.

[James · Accepted solution]

 @willi5cl  

Short answer: Within the current limitations of the Canvas interface, this is not possible.

This looks more like a multiple answers question than a multiple choice question. I'm going to answer the question like it was one, but it wouldn't change the answer much if it were multiple choice.

One thing you can do is download the Student Analysis. It will give you a list of the question and the responses chosen by students. From there, you may be able to get an idea of which questions were answered.

Here, the first attempt (bottom row) was left blank while the second attempt (middle row) had the first three items checked. The a, b, c there are in the unshuffled order.

Unfortunately, there's not an easy way to determine from the the quiz statistics page. The order there does not match up with the order in the original or in the shuffled order. It might be in the order edited (I think I went back to edit one of the answers, but don't remember exactly). But I wouldn't hold out hope on that.

If I use the browser's developer tools to inspect the table, I get this:

From the reactid's, I can tell that this is question 110073156 and that the responses are 4917, 9980, 1646, and 8900 in that order.

If I query the API for the questions on the exam, I can find out what those questions are.

For example, if my path (the part after the hostname) in the browser's location is

 /courses/896851/quizzes/6345859/statistics

I would change it to

   /api/v1/courses/896851/quizzes/6345859/questions

to get the questions for the entire quiz or  

   /api/v1/courses/896851/quizzes/6345859/questions/110073156

for just this question.

The output is mostly unreadable in the browser (this is just part of it).

I use a REST client to query the API and it provides a formatted response. You want to look at the answers section.

Here I see 4917 is the integral of x³√(x²-4) dx and 9980 is the integral of x ln x dx.

You could copy all of the output from the browser, ignoring the while(1); at the beginning, and paste it into an online JSON formatter to clean it up.

But it's a lot of work.

Note that the answers here matched the order presented because I forgot to tell it to shuffle the questions.

On another quiz, that was shuffled, I got

The order of the responses here is 9051, 3573, 4549, and 8362.

The order returned with the API call is 9051, 4549, 8362, and 3573. This is the same order that they appear in the quiz.

You might be able to take the API call results and just look at the order of the IDs without messing with the actual content. That would tell you the order and you could compare that to the order the questions are in when you edit the quiz. Then you would just need to match it up with the order that was presented on the page.

This is all manual interaction on your part.

It won't work if the questions are being pulled from a linked question bank (it's okay if they were found in a question bank and included in the quiz).

If you wanted to have it display the results, you would write a JavaScript program that would listen on that page for the answers to be rendered, then use the data-reactid to replace the content by fetching it from the quiz and injecting it into the table.

That might cause problems with heights of the rows, I didn't go that far into my testing.

And as a final note, Canvas is making a push to move everyone to New Quizzes, so if you write the program to do that, it will be a temporary solution.

[willi5cl]

James,

This has been happening all semester (I use the quizzes as formative assessment before each class period) and I have been using the student analysis to figure out which answers are which.  But today, this is what I got in the student analysis for that question:

So there was no text, even garbled, for me to look at!  (By the way, I've never seen the student analysis give me letters for the answers chosen.  How did you get it to do that?  Is that part of the New Quizzes?)

The upshot is that I'm just going to stop shuffling the answers on questions where all of the answers are only written in the equation editor so that I can tell which is supposed to be which.  But that's immensely frustrating, given that the equations are built on LaTeX, and there are nice easy tools to display LaTeX in any webpage.  Clearly, nicely typeset mathematics can be displayed in the quiz itself, so I'm disappointed that it can't be displayed properly in the quiz statistics.

Thank you very much for the amount of time you spent generating that answer to my question.  I hope that it helps some future frustrated math professor!

--Cassie

[James · Accepted solution]

 @willi5cl  

I was not using New Quizzes. We don't have that turned on in our institution yet and I have to go to our beta instance to use it.

I didn't do anything special with my quiz. I went in tonight, created a quiz with a single multiple answer question. I only put four responses in there.  Oh wait, I did do something different.

It was something I was testing to see if it would help.

What I did was to put an invisible letter before or after the question. What I mean by invisible was I used white text, which doesn't show up on the white background. It didn't come through on the question breakdown, so I forgot about. It does look like to comes through when doing the quiz analysis. The reason it didn't come through the first time was because I added it between the first and second attempt.

Here I've moused over it to show the invisible a.

Where's what it looks like to the student.

The second time I tried this, I ended up putting the invisible letter in front of the integral. The Canvas editor wouldn't let me put a space after the equation, so I had to type "a", then arrow left and hit space. It was easier to just type "a" then space and then create the equation. Before saving the question, I went through and updated the foreground color.

The reason it was in the same order was because I had put them in a, b, c, d order when I created them, not because I forgot to shuffle the equation.

You cannot save the equation to the Excel file, but it did transfer the letter and the accompanying space.

You're not the first person to be frustrated by the lack of quiz statistics -- or the lack of support for mathematics as a whole inside Canvas quizzes.

[willi5cl]

This is a wonderful solution!  I just employed it in my next quiz, so we'll see how the responses turn out after they submit them.  Thank you for such a simple work-around!