Re the use of question groups to randomize the questions on a quiz: what is the underlying probability function associated with the randomization function when Canvas chooses questions from a pool/group? For example, assume 5 different questions/variations/options are in a group (or pool) and available for item # 1 on a given quiz. Assuming a uniform distribution, all five options would have an equal (20%, or 1 in 5) probability of being selected by Canvas, and in 20 previews/attempts, one would expect each of the five questions/variations/options to be selected by Canvas approximately 4 times, yes?
But that's not what we're experiencing, based upon our admittedly unscientific testing/previewing of an exam we've created and hope/expect to use many times for many students: in 20 previews, 2 of the options (of the 5 available) were chosen 6 or 7 times by Canvas, and the other 3 available options were each chosen by Canvas only approximately twice. And this happened multiple times over on the 25 or so questions on the quiz that rely on question groups and randomization by Canvas - some options were much more "popular" and chosen more frequently by Canvas than other options. On questions where there are 8 options in the group/pool, two of the options were not selected once by Canvas in 20 attempts/previews. Even allowing for variation on the expected outcomes (choices by Canvas), the options chosen by Canvas seemed inconsistent with a uniform distribution. On the question where there were only two options in the group, there *was* a uniform distribution: each was selected by Canvas about ten times.
So we're trying to get a handle on how Canvas selects from a group of questions, and our initial conclusion is that the probability distribution associated with the randomization function in Canvas is perhaps not a uniform one, but some other probability distribution? Ideally (for our purposes), we would see all of the options more evenly distributed (equally likely to be chosen/selected by Canvas). What is the probability distribution used? Is there a manner of using Canvas or creating question groups on a quiz that will accomplish our goal? Or have we merely tested an insufficient number of times for the probability distribution to properly manifest (assuming a uniform probability distribution), or committed some other mistake?
Many thanks for any insights.