We have discovered a significant issue with Canvas LMS regarding the inclusion of test student exam submissions in the item analysis for New Quizzes. After a lengthy follow up with Instructure support we were told that this test student data cannot be removed. This is problematic for our faculty of nursing for the following reasons:
- Accuracy of Item Analysis: Nursing colleges rely on accurate item analysis to maintain the validity of exam items. The inclusion of test student data skews the results and compromises the reliability.
- Accreditation Standards: Our institution must align with external accreditation standards that mandate fair testing practices. The inclusion of non-real student data undermines our compliance with these standards.
- Contradiction with Canvas Documentation: According to Canvas's very own documentation, submissions made in Student View are supposed to be excluded from item analysis. However, this is not happening, suggesting a discrepancy between the intended functionality and the actual system behavior. (See Quiz and Item Analysis Report Data Exclusions here: https://community.canvaslms.com/t5/Canvas-Resource-Documents/New-Quizzes-Quiz-and-Item-Analysis/ta-p...) Use
Case Example: An instructor uses the Student View feature to preview an exam. The test student’s submissions are then included in the item analysis, leading to skewed data that affects the validity and fairness of the exam results for real students. This has already happened at least twice in the past month alone from faculty who were simply trying to ensure that their exams were set up properly from a student point of view.
Canvas should ensure that any exam submissions made in Student View are excluded from the item analysis. This adjustment will align with the intended functionality described in Canvas's documentation and uphold the integrity of exam processes at our institution and others that rely on accurate data for their assessment policies.
