[ARCHIVED] Can someone answer questions about quiz item analysis - the help documents are inconsistent

[doig]

I am conducting a training for college instructors on the use of item statistics to improve grading fairness and improve test quality.  When possible, I am referring to a variety of LMSs since audience members will vary in the products they use.  I wanted to include Canvas as one of those LMSs.

I have found two sources of information on the quiz item analyses provided in Canvas.  One is Canvas 2019 | guides.canvaslms.com | updated 2018-10-06 Test Quiz Item Analysis.  The second is  the following webpage https://community.canvaslms.com/docs/DOC-15054-4152780620  I have attached these.

These two items would appear to discuss the same content; that is, the test item statistics calculated in Canvas and their interpretation.  I first read the guide document which made no sense to me based on my knowledge of item analysis.   I could not believe that what it described was correct.  So I explored the website and found the webpage referenced above.   Here too, I find confusing explanations as well as inconsistencies with the guide document. So I believe that these two sources of support for users are inconsistent and that each has different errors.

I have attached each of the sources with my questions/comments.  I would appreciate some help from someone knowledgeable about item analysis, perhaps a psychometrician, to help me understand what is really calculated and provided in Canvas.    Thanks for your assistance.

Kathy Doig PhD

Professor Emeritus

Michigan State University

[chadscott]

There is probably some discrepancy due to the fact that in old quizzes, you can only attach outcomes to the quiz as a whole and not to individual questions, where as in Quizzes.Next, you can attach outcomes to individual questions. With the ability to individually attach outcomes to questions, you are able to have multiple questions of the same outcome, at different depth of knowledge levels, to get a better sense of student mastery of the objective. Therefore, I would assume that the calculations would have to be different. I'm no psychomajig, so can't address the actual math involved.