Improve quiz analysis through robust reporting

[MPioRoda]

Theme Overview

Teachers need reports that are easy to manipulate, full of useful information, and efficient to access

What value could this provide to users?

• Quicker responses when accessing quiz and item analysis reports
  
• Item Analysis, Student Analysis, and Outcomes reporting would be easy to download and in a format that is easy to manipulate
  
• More information would be available in reports to help track progress or gaps

Why was this theme chosen for prioritization?

January 2024 • In support of this Canvas theme, the Data & Insights team plans to add metrics for New Quizzes pageview & participation usage within Admin Analytics.

Why was this theme chosen to open for voting?

April 2023 • Quiz reports are key components to help customers understand outcomes and student performance. Reporting should be responsive, easy to manipulate, downloadable, and include the data necessary to help teachers and administrators make sound decisions on effectiveness and mastery.

Why was this theme chosen for prioritization?

June 2023 • While we will not be able to prioritize all ideas within this theme in the second half of the year, we will focus our efforts on improving the Item and Student Analyses.

[ProductPanda]

Note from Instructure

April 2023 • This theme was chosen to Open for Voting. 

How do Ideas and Themes work in the Instructure Community? 

How do I participate with themes? 

[JThead]

I would like to see a column for duration included in the quiz reports, as well. This information is available under SpeedGrader, but is not captured in any of of the quiz statistics.

[nathanboettcher]

Canvas does already have some great quiz analysis, but if this featured more prominently in Canvas for faculty, that might increase usage. Further statistical analysis would be wonderful, but an easy summary of results should be readily available for faculty to become interested in the data enough to "drill down" to more detailed results.

[JamesSekcienski]

One of the highest priorities for this theme should be providing live data reporting in New Quizzes like there is with Classic Quizzes.  The quiz and item analysis reports shouldn't potentially take days to generate.

[LMacaulay]

The item analysis data in New Quizzes under Reports is useful, but needs to be able to be exported as a CSV or Excel spreadsheet file for easier analysis. Additionally, many faculty don't even realize the data that is on screen since there are so many clicks to even see the data.  

[CraigOgden]

I would also like to see item analysis of bank questions.  It would be nice to see all the questions from a bank and see which questions are missed the most and which questions seems to be the most difficult so we can remove them or move then to a more challenging bank. OR review those questions with our students more.

[lbach]

I would very much like to see an option to download all student responses in New Quizzes. Getting a summary statistics report is not sufficient for program review and assessment purposes, we need to be able to export the essay responses submitted by individual students.

[KathyPalm]

This is something our institution has been missing since the move to Canvas, being able to immediately view statistics on a New quiz after the quiz has been taken. We are very interested in this theme!

[ProductPanda]

Status changed to: Post Voting Review

[KristinL]

Status changed to: Prioritized

[james_whalley]

Can you tell us, of the 20 feature ideas listed here, which led to the focus of "improving the Item and Student Analyses"?

[CaitG]

I agree this feature needs to be available.  We need to be able to easily track a student's progress over time.  Also, many of us conduct research on our educational interventions and I used to use the old quizzes as an easy way to do likert-scale questions and surveys. Now I have gotten myself in a pickle where i need to hand-transcribe 200 responses! The new canvas quizzes are so difficult to use, I think Im going to start using qualtrics.  Disappointing since the technology is clearly there. Please add this feature ASAP!

[TamasBalogh]

@james_whalley As stated, this time we are focusing on the ideas related to Item Analysis and Student Analysis:
- Faster Quiz and Item Analysis
- Y axis for the charts in both Quiz and Item analysis
- Explanatory descriptions for some metrics
- Answer Frequency summary to more question types
- Add student names to Answer Frequency Summaries
- Student Analysis report (in CSV format)
- Add time spent column to Student Analysis
- CSV downloadable for Quiz and Item Analysis

[JuliaCF]

Thanks for the update, Tamas! We have a faculty member who is wondering when the Student Analysis report (in CSV format) for new quizzes will be deployed - do you all have a time estimate for this? Thanks again!

[TamasBalogh]

Hey @JuliaCF,

Unfortunately, I don't have a timeline for this now, but I understand the Student Analysis is essential for you and a lot of customers so we are keeping it high on the priority list. 

[leward]

Why is it that of the four themes with the status of "in development", only one idea, [New Quizzes] Quiz and Item analysis needs to be faster, has been completed, while at the same time 73 ideas associated with themes that are not in development were delivered in 2023 (Canvas Ideas Featured in 2023 Releases).  Seems like the prioritized themes are not getting a lot of love. 

[JamesSekcienski]

@TamasBalogh 

I saw on the New Quizzes Hub that the Student Analysis report looks like it won't be released until Q2 or later.  Is that accurate?  I was hoping to see it (or at least API endpoints to get the submissions to build our own reports) as part of this Theme development.

[TamasBalogh]

@JamesSekcienski 
Please keep following the New Quizzes Hub. It's not fully clear yet what our prioritizes will be in Q1. We'll update our roadmap as we progress. This theme includes a lot of ideas so it takes several months to develop.

Currently the team is working on adding Answer Frequency Summary tables for more question types and the Item Analysis CSV download, both of them are aiming to improve quiz reporting. 

[JamesSekcienski]

@TamasBalogh 

Thank you for the update.  I know you mentioned the Student Analysis report as one of the items your team was going to focus on developing with this theme, but I've also seen your regular updates with all the improvements that are being made to New Quizzes in general so I know you have a lot on your plate with New Quizzes.  Unfortunately, we are now stuck trying to figure out an efficient solution for department/program chairs that relied on the Student Analysis report for reporting needs.

[TamasBalogh]

@JamesSekcienski hopefully, we will be able to provide the Student Analysis soon 🙂 I'd love to have a chat about the student analysis with your institution. Let me send you a DM to see if you'd be open for a discussion.

[TamasBalogh]

Hi all, 

As we continue to work on the robust reporting, I'd like you to take a look on my recent blog post about a possibly way for the CSV download for Quiz and Item Analysis. Your feedback would be valuable and useful!

[mbmacdonald]

The Student Analysis report must include submission date/time info. I checked, and that information *was* included in the counterpart report for Classic Quizzes.

This is one possible solution to an issue that has been plaguing my users: complete inability to sort/filter submissions by submission date, which is especially challenging for Quizzes that have large numbers of submissions that are submitted at different times across a long time period. I have summarized all of this in a recently submitted idea that I hope can still be added to this theme: https://community.canvaslms.com/t5/Canvas-Ideas/New-Quizzes-Sort-Filter-by-submission-date/idi-p/596...

Including this info in the Student Analysis report is probably the best and most robust way to address this issue, but there are other ways this could be addressed, as described in this user's idea: https://community.canvaslms.com/t5/Canvas-Ideas/New-Quizzes-Show-Submission-Timestamp-For-All-Submis.... This idea seems to involve including submission date info in the gradebook so that users can take advantage of sorting and filtering in the gradebook. As such, this idea was sorted into a different theme: "Improved UI Controls and Usability - Additional filtering and view options ". But these issues are related. I don't really care which angle we choose to attack this problem, as long as we get some form of this functionality as soon as possible.

[RSimonson]

I rely HEAVILY on the .csv downloads available in Quizzes > Survey Statistics > Student Analysis. Item Analysis does NOT serve the same purpose of drill-down molecular detail. I NEED each student's answer to EACH question. I download these results 100+ times each semester for 200+ students. Without this feature that is currently available in Classic but not yet available in New Quizzes, I would have to redesign more than half of the coursework I have and potentially remove some of the most insightful work and incentives for my students. I have already contacted Canvas Support and have been assigned this Case number: 10650650