In the Q1 quarterly update post, I mentioned our ongoing efforts to enhance the speed of Item Analysis generation. I am now pleased to provide more comprehensive information on this development.
New Approach to Generate the Report
Our primary focus is to significantly reduce the time required to generate the Item Analysis report. To achieve this, we have implemented a new on-demand method for report generation. Users will be able to navigate to the reports tab in New Quizzes and promptly generate the report for a specific quiz or view an existing - previously generated - one. With this new approach, users can obtain the report when needed, without waiting for the 24-hour scheduled generation.
Changes in the Report
During the discovery phase, we actively sought feedback from numerous customers and collaborated closely with Instructure's psychometrician to enhance the precision and utility of the report.
In addition to the mean score, we are introducing the median score in the new report. The standard deviation will be represented as a percentage, aligning with the mean score's format. To ensure the accuracy of Cronbach Alpha, we exclude randomly selected quiz items from its calculation.
In addition, the Item Analysis will now utilize the Corrected Item-Total Correlation Coefficient instead of the Basic Pearson Correlation coefficient method, making it a more sophisticated calculation. Furthermore, we have refined the Discrimination Index calculation to improve its accuracy.
What Can You Expect Before the Release?
We acknowledge that these are significant changes, and we are committed to supporting our users in understanding and leveraging the new report effectively. Prior to the production release, I will publish a comparison document, shedding light on the differences between the old and new versions.
To further assist users, we will update the user guides with comprehensive explanations of the formulas used to calculate the various metrics. Our aim is to ensure a seamless transition and provide ample resources to aid in utilizing the enhanced Item Analysis effectively.
You will notice a minor alteration in the report cards soon to correspond with the upcoming implementation of the report card for the new Item Analysis:
Further Work
After the faster Item Analysis process is completed and released, our focus will be on further enhancing the report. We plan to introduce a CSV download option for users who wish to have more flexibility in manipulating the data. If you or your institution would benefit from the CSV format and would like to discuss your specific use case, please feel free to leave a comment, send me a private message, or reach out to your CSM. Your feedback and input are highly appreciated.
The content in this blog is over six months old, and the comments are closed. For the most recent product updates and discussions, you're encouraged to explore newer posts from Instructure's Product Managers.