Hello,  @awerner  Welcome to the Canvas Community! The instructions in How do I create a quiz with a question group to randomize quiz questions?  will help you accomplish this in Old (Classic) Quizzes.

New Quizzes does not use question groups, and has a single checkbox setting that allows instructors to randomize questions across the entire quiz (How do I manage settings for an assessment in New Quizzes?). If you don't see the ability to enable the New Quizzes feature option in the settings area of your course, it most likely means that your school hasn't yet enabled it at the account level: please ask the local Canvas admin or eLearning department for your college for additional assistance.

Thanks, Alicia. Stay safe, and be well.