Classic Quiz statistics issue

[rlbrown21]

In the statistics report from quizzes, you can see how many students missed a question.

The issue is that if a student gets multiple attempts, only the attempt kept is showing in the statistics.  

So if a student misses the same question twice, but gets it correct on a third attempt (which ends up being the graded attempt), then the stats only show that the question was answered correctly.  more accurately it should show two incorrect and one correct attempt so faculty know where students are struggling.

am I missing something?  is there a setting or a different report that is more accurate?

[James · Accepted solution]

@rlbrown21 

For Classic Quizzes, you have Analysis Reports (Student and Item) links at the top of the Quiz Statistics page.

Neither one will exactly give you what you want -- statistics for all attempts.

• The Item Analysis uses the first attempt but has several limitations.
• The Student Analysis gives you all attempts, but does not do any kind of statistics on them.

To get the multiple attempt situation, you would need to use the Student Analysis and then do your own review of the data.

On the back end of Canvas, not available through the web interface, there is a Create a quiz report endpoint in the Canvas REST API. This allows the user to request that all submissions should be used. That sounds promising until you read the whole description of what it does: "Whether the report should consider all submissions or only the most recent. Defaults to false, ignored for item_analysis." This makes it totally useless for what you need.

• This is kind of technical and cannot be done through the Canvas web interface.
• It defaults to false, but if you get the student analysis through the web interface, it does include all of the submissions. There's no need to use the API if you're a teacher just looking at the data for your class. It is helpful for people who have to download a lot of these reports (say there are 20 sections of a course that gave this quiz).
• The most important part is that it is "ignored for item analysis". In other words, even if you told the item analysis to include all of the submissions, it's not going to do that.

That brings us back to the Student Analysis report and creating your own statistics.

Analyzing the Student Analysis isn't the easiest thing to do, especially depending on the question and whether it's part of a question group. I can go in and create a Pivot Table, put the student names as the rows and then pick a question for the column. I then take the name or ID and put it in the values and change it to be Count (if it defaults to Sum). It will then show how many times each student gave that particular answer.

This image shows that the last student answered "a minimum of 12 hours" three times and did not get the question on one of their four attempts. Getting results are not reliable. I use question groups so there are a lot of blank entries for me. You also cannot tell the order in which the responses came in.

You would have to change the columns for each question, so it's a slow process.

You could make a different table. This time use the numeric column that follows the question that contains how many points the student achieved. Those columns don't have useful names in the CSV file, so you may want to rename them before creating the pivot table.  Use the student name for the rows, the attempt for the column, and the sum of the point column for that specific question as the values.

In the image, the question was worth 0.5 points, so that means the student got it correct. I can see that my second student didn't get the question on the first or second attempt, they missed it on the third, but got it right on the fourth attempt.

This technique does not show me what choices the student gave. I would have to look at the data in the file to find that -- the pivot table is meant to summarize.

[James]

I wanted to mention that in my second pivot table, you could have used average instead of sum for the points column. Then anything less than the point value would mean that the student missed the question at least once.

You could also make your own table.

Here, I've used the COUNTIFS() function to count the number of times a student got 0 points and the number of times they got more than 0 points. I then copied those two columns over and because each question in the spreadsheet took two columns, it worked well and I didn't have to adjust. This shows all of the questions (I've only done the first three) in the quiz.

I see that one student got the first question correct 6 times without missing it, but then missed the second question once and got it right twice. I do not know what order they did that in, but hopefully they didn't get it wrong after getting it right.

I can also see the totals for the question. My second question was missed 6 times and answered correctly 20 times. Because of the question grouping I use, other questions have different number of attempts (question 1 was asked 31 times and question 3 was asked 33 times).

I could go through and add another calculation. For example, I could divide the number correct by the sum of the number missed and the number correct to get a rate of success. Then when I look through them, I see that question 1 had 96.8% correct, question 2 had 76.9% correct, and question 3 had 96.7% correct. That lets me focus on question 2.

Question 2 happened to be the same question I analyzed with my pivot tables -- about how much time students should expect to put into my course each week. This was a "Course Introduction quiz (syllabus quiz with some other things thrown in). The answer is given in multiple documents that the student should have read before taking the quiz. It gives you an insight into students who want to do the minimum or who are used to high school where they didn't have to study for things.

The bottom student missed question 2 and never got it right. When reading the tables, it is important to understand what they say and what they do not say. Do not read more into that than what is there. This table does not tell me how many attempts a student made. One might think that bottom student was doing the minimal work, she took the question once and missed it and never retook it. However, when I look at the data, I see the student made four attempts, but only received that particular question one time. I do not know whether that student figured out how much time was needed, but she knew her answer was wrong and she ended up with an "A" in the class.

I could have added another column to the table that uses COUNTIF() to see how many attempts each student made. I just didn't do it here. The idea here is just to give people who stumble across this an idea of what can be done with common tools.

Finally, when I say question 1, I'm referring to the order they appear in the Student Analysis CSV file, not the order they were given in the quiz. They are likely the same if there are no question groups being used. If question groups are used, then they appear in the order the questions were answered in, so the first student taking the quiz mostly determines the initial order and then new questions are added in the order they get asked to other students.

[rlbrown21]

Thank you James, you are amazing as always.  I do wish canvas had an easier solution for non-technical faculty.  I will pass this on to my faculty anyway