Quiz Settings to Maximize Security

kona
Community Champion
92
238307

The following document was created to help our faculty better understand quiz settings and how they can be used to enhance quiz security. In many cases locking down a quiz as much as possible (ex: not allowing students to review what they got wrong on a quiz) can hinder student learning, so be sure to think through the positives, negatives, and the goals of the quiz before securing it. There is no way (in class or online) to guarantee that students aren't cheating, but to maximize quiz security there are various ways you can apply your quiz settings and availability.

Below are is a list of quiz settings and information on how they can be used to maximize your quiz security. For more information on this topic see the following CanvasLIVE webinar -

Question Groups - Create question groups to randomize quiz questions. The larger the number of questions in the question group (compared to the number randomly being picked for the quiz) the more secure the quiz. If there are a limited number of quiz questions and all of them are needed for the quiz then question groups can still be used to randomize the order of the questions. By using question groups it is unlikely that students will get the same questions or the same questions in the same order.

Question Groups

Shuffle Answers - This is a checkbox option under quiz settings and will randomize the order of the question's answer choices. By using “shuffle answers” students may get the same quiz question(s), but their answer options probably won't be in the same order. **If your quiz question has an answer like “all of the above/below” or “answer A & C” do not use the “shuffle answer” setting.**

Shuffle Answers

Time limit - This is a checkbox option under quiz settings and limits the amount of time students have to take a quiz. When deciding on a time limit try to estimate the average time a student should need to answer each question - taking into account different types of quiz questions. If quizzes are given in a non-proctored environment the time limit can be used to cut down on the amount of time students have to look up question answers and/or share the questions with other students. If a student needs extended test time this can be added per individual student under the Moderate Quiz settings.

Time Limit

Allow Multiple Attempts - This is a checkbox setting that allows students to have multiple attempts to take a quiz. From the multiple attempts Instructors can decide which quiz score to keep: Highest, Latest, or Average. If the number of allowed attempts is left blank, then students will have unlimited attempts that would only be stopped by the "Until" date/time. In general unless there is a large question bank of questions and/or your goal is mastery of the content, multiple quiz attempts does not create a secure quiz environment.

Allow Multiple Attempts

Let Students See Their Quiz Responses - This is a checkbox option under quiz settings and after a student submits a quiz allows them to see (1) the quiz questions & answer options, (2) the answer options the student selected, (3) if they got a question incorrect or partially incorrect, and (4) ALL question feedback. This information will be shown regardless of the due & until dates and even if the correct answers are NOT made available (via the “Let Students see the Correct Answers” setting). For enhanced quiz security leave this box unchecked altogether or leave it unchecked until after the “Due’ and ‘Until’ date(s).

The following options only appear if the "Let Students See Their Quiz Responses" box checked:

Let Students See Their Quiz Responses

Only After Their Last Attempt - This checkbox option only shows up if the quiz is set to allow multiple attempts with a specific number of attempts; this option does not show up if there are unlimited attempts. If this box is checked students will NOT be able to see any quiz results other than the score they got on each quiz attempt until they take the quiz for the last time (as designated by the number of allowed attempts). After they take the quiz for the last time the student will then be able to see (1) the quiz questions & answer options, (2) the answer options the student selected, (3) if they got a question incorrect or partially incorrect, and (4) ALL question feedback. If you are using multiple attempts, this is the best setting to use to keep the quiz secure. Yet, this information can't be controlled by a date/time, so a student could use all their attempts, get the question information, and then share the information before the due/until date(s).

Only Once After Each Attempt - This checkbox option  allows students to see (1) the quiz questions & answer options, (2) the answer options the student selected, (3) if they got a question incorrect or partially incorrect, and (4) ALL question feedback, but only ONCE after they have submitted a quiz. In a proctored environment this is a useful setting because students will get to see feedback from the quiz, but it is only visible immediately after submitting the quiz. If the student goes back to the quiz they will see their score for the quiz, but none of the other quiz information (1-4 above) will be visible.

Let Students see the Correct Answers - This checkbox option allows students to see the correct answers to quiz questions. If this box is checked students will have access to (1) the quiz questions & answer options, (2) the answer options the student selected, (3) which answer option was correct, and (4) all question feedback. For enhanced quiz security the best option is to leave this box unchecked (along with the “Let Students See their Quiz Responses” option) until after the “Due” and “Until” dates and then also limit how long the students can view the correct answers.

Only After Their Last Attempt - this checkbox option under "Let Students see the Correct Answers" only shows up if the quiz is set to allow multiple attempts with a specific number of attempts; this option does not show up if there are unlimited attempts. If this box is checked students will NOT be able to see which answer options were correct until after their last attempt (as designated by the number of allowed attempts). After they take the quiz for the last time the student will be able to access (1) the quiz questions & answer options, (2) the answer options the student selected, (3) which answer option was correct, and (4) all question feedback for as long as designated. For enhanced quiz security the best option is to leave this box unchecked (along with the “Let Students See their Quiz Responses” option) until after the “Due” and “Until” dates and then also limit how long the students can view the correct answers. Note: This setting also overrides the "Show Correct Answers at" date. If you want correct answers to be released or hidden on a certain date/time or do NOT check this box.

Show/Hide Correct Answers at - this option allows students to see the correct answers to quiz questions starting at the designated "show" time and hidden again at the designated "hide" time. The show and hide options can be used independently of each other or together to create a window of when correct answers are visible to students. If one or both dates are provided students will have access to (1) the quiz questions & answer options, (2) the answer options the student selected, (3) which answer option was correct, and (4) all question feedback - during the time frame designated by the show/hide dates/times. For enhanced quiz security the best option is not use this until after the “Due” and “Until” dates and then also limit how long the students can view the correct answers. Note: If you want to use the show/hide correct answers option then do NOT check the "only after their last attempt" box directly above it, it will override your date settings and make quiz answers visible to students after their last attempt.

Show One Question At a Time - This is a checkbox option under quiz settings and allows students to only see one quiz question at a time. It is recommended to use this feature in conjunction with the "Lock Questions after Answering" option. Used together, it requires the student to answer each individual question before moving on and prevents students from going back and changing their answer. **Students do NOT like getting one question at a time and especially if they are prevented from going back and changing their answers, so make sure you really want to use this option before implementing.**

Show One Question At a Time

Require an access code - This is a checkbox option under quiz settings and prevents students from taking the quiz without the correct code. This feature can be used to make sure students are taking the quiz at a specific time or in a certain location; this works especially well for students who need to take a test in the Testing Center or with a proctor. This setting can also be used if students have started taking an untimed quiz in class (or a proctored environment) and don't finish it in one class period and need to come back to class to finish it. The Instructor can add (or change) the access code after the quiz has started and it will completely lock the students out of the quiz until the student enters in the new access code.

Require an access code

Filter IP Addresses - This is a checkbox option under quiz settings that can be used to make sure students are taking the quiz in a specific location. This works especially well for students who need to take a quiz in the Testing Center or with a proctor in a secure environment. **To use this option you will need to talk to your Canvas Admins so they can coordinate things.**

Filter IP AddressesQuiz Availability

In general to maximize quiz security it is best to have a quiz open the least amount of time as possible. Below is information on the quiz availability options and how they can be used to enhance quiz security.

Quiz AvailabilityAssign to - In courses with multiple sections this feature allows instructors to designate the availability settings by course section or individual student(s). In general, unless you have students in different sections taking the quiz at different days/times, this would be left as “Everyone” (which is the default). This feature can’t be used with students who need testing accommodations or who are in different groups. For students who need different availability than the rest of the class it is recommended to manually unlock the quiz using the Moderate Quiz settings in conjunction with an Access Code.

Due – This is the day and time that the quiz is due. This does not stop students from taking a quiz, but quizzes submitted after the Due date will be marked as late in Speedgrader and in the Gradebook. Recommended practices for quiz security are to not allow students to take a quiz after the Due date; this can be done by making the Due date and Until date the same. If students are allowed to take quizzes after the “Due” date, then leave the "Until" date blank or set to the furthest day/time late submissions are allowed.

Available From – This is the day and time the quiz becomes available to students. Students will not be able to access the quiz before this day and time unless the quiz has been manually unlocked; manually unlocked is available under the Moderate Quiz settings. In general if a student or students are taking the quiz early the time will need to be set to when the first group of students is taking the quiz. This is also when it's a good idea to use the "Require an Access Code" option to keep the rest of the students out of the quiz until they should have access to it. **If a quiz is not published (has a gray cloud instead of green) then students WILL NOT be able to access or even see the quiz, even after the "Available From" date.**

Until – This is the day/time that the students will no longer be able to take the quiz. If the Until day/time is set beyond the "Due” date" students will still be able to take the quiz until the “Until” date, but it will show up as "late" in SpeedGrader and the gradebook. If a student is in taking a quiz Canvas will auto-submit the quiz at the "Until" date/time regardless of whether the student is in taking the quiz or not. If there is no "Until" date set then the quiz will not auto-submit until the end of the course. For enhanced security instructors should set the “Until” date to be the same as the “Due” date because it prevents students from taking the quiz after it is due. Yet, this can also cut students off in the middle of taking the quiz because the "Until" date will automatically submit the quiz.

Important Message: Quiz availability (dates/times) have NOTHING to do with whether a student can view their quiz results. The ONLY way to stop students from viewing their quiz results is to have the “Let Students See their Quiz Responses” and “Show which answers were correct” boxes UNCHECKED. If you want students to initially see their quiz results, these boxes can be checked and set to close once you no longer want students to view the quiz results.


Preventing Students from Viewing Quiz Scores

The only way to prevent students from viewing their quiz score at the end of a quiz is to mute the quiz before the students take the quiz. Please note though, muting an assignment will override your other quiz settings related to quiz feedback. Students will not see any quiz feedback including their responses to questions or the correct answers. Instead students will only get this message (see image below).

Muted Quiz

Once the quiz is unmuted, students will be able to see/access quiz feedback based on the the quiz settings.

For more information on how to mute or unmute quizzes see the following guides - Not authorized to view the specified document 2952 & Not authorized to view the specified document 2823

Do you use images as part of your Quiz?

If a course has the Files link enabled on the course navigation menu then students will be able to use that to browse all the files and this includes by default, any images you've uploaded to Canvas to embed in your quiz.

 

To prevent this, it is a good idea to either:


Useful Canvas Quiz Guides

Useful Canvas Quiz Guides

What options are available for Quizzes? (Reviews all the Quiz Settings)

Not authorized to view the specified document 2900

Not authorized to view the specified document 2628

Not authorized to view the specified document 2971

Not authorized to view the specified document 2979

Not authorized to view the specified document 2983

Not authorized to view the specified document 2984

92 Comments
Stef_retired
Instructure Alumni
Instructure Alumni

 @sstumpf ‌, are these email notifications? If so, would you please forward those notifications to community@instructure.com along with a screenshot of how you currently have your email notifications set here in the Jive Community so we can investigate further?

cdollar
Community Explorer

Regarding "Important Message: Quiz availability (dates/times) have NOTHING to do with whether a student can view their quiz results. The ONLY way to stop students from viewing their quiz results is to have the “Let Students See their Quiz Responses” and “Show which answers were correct” boxes UNCHECKED. If you want students to initially see their quiz results, these boxes can be checked and set to close once you no longer want students to view the quiz results."

I want students to initially see their quiz results right after they take the quiz but never again. I am not sure what is meant by "these boxes can be checked and set to close once you no longer want students to view the quiz results" .  It's the "set to close" part that I am confused about. Do you mean I have to manually uncheck the boxes after my selected date? Thanks!

kona
Community Champion

 @cdollar , if you check the box for "only once after each attempt" then the students will see the answers and feedback information right after they submit the quiz, but after that will no longer be able to view the results. Does this help?

cdollar
Community Explorer

Yes! Thank you!

Carrie Dollar

Professor of Biology

Saint Clair County Community College

323 Erie Street

Port Huron, MI 48061

810-989-5647

cdollar
Community Explorer

After a bit more thinking, what will the students see when they attempt to access the quiz after having taken and reviewed the quiz if I check the box for "only once after each attempt" ? Will that only see their score?

kona
Community Champion

Yep. It will give them a message that the results have been locked or something like that. I don’t remember the exact wording. 

dejonghed07
Community Champion

Today we tried un-checking the box to "Let Students See Their Quiz Responses". Unfortunately, students are still able to view the quiz and the correct answers. Did this setting work for you or anyone else who tried it (after first allowing students to see their responses)?

Quiz Responses Option

kona
Community Champion

 @dejonghed07 , you scared me with this!! I just tried it on one of my own quizzes and it worked as expected (see below) so I'm not sure why you are experiencing the opposite. 

In my own course I leave my quizzes open (so the box is checked to let students see their quiz responses). I just went to our quiz from last week, unchecked that box, saved the quiz, and then went in as one of my students (since I'm also an Admin). When I navigated to the quiz page (as the student) it told me the "Quiz results are protected for this quiz and are not visible to students." It then said, "Correct answers are hidden."

If you go in and try this and can replicate that the students ARE able to see the quiz results then I would definitely report it to Canvas Support because it's not functioning as intended/expected.

dejonghed07
Community Champion

Hi  @kona  - Thank you for checking and for providing more details! The teacher was selecting a student to view that student's grade book. For some reason, I thought viewing the individual student grade book as a teacher was viewing what students see (and what they print when they print their grades). Now I realize that the teacher's view of the individual student grade book is different than the individual student's view of the grade book! 

After reading your response, I selected the student to "Act as User" (as an administrator) and it is working exactly as you described. So, it was user error on my part! For teachers, using Student View to test their quiz first might be the best way to later check the Student View of the quiz settings, too. 

Thanks again!!!

kona
Community Champion

Whew! Glad to hear things are working as intended! Thank you for follow-up on this and letting us know what you found!

ayanni
Community Novice

I was helping an instructor set up quizzes and for some reason when it asks if you want to 'Let Students See Their Quiz Responses', this was grayed out for the instructor and she was not able to select this as an option. Is there a particular reason why this option might have been grayed out for her or why she would not have access to it?

kona
Community Champion

 @ayanni , hmm... nothing I can think of off the top of my head. Do you have any third party software that might be interfering with these settings? Can you replicate this with other quizzes for the same Instructor? 

ayanni
Community Novice

Kona, I am looking to see if the instructor had set it up with Respondus to not allow students to see their quiz responses. Other than that, no other third party software should be interfering with it because we both have the same LTIs and mine works just fine. When she tried a different quiz, the same thing happened. I will look into Respondus first and hopefully that will work. Thanks!

kona
Community Champion

 @ayanni , thanks for the extra information and please let us know what you find out with Respondus. We also use Respondus so it would be good to know if it somehow interferes with this setting. 

ayanni
Community Novice

Kona, I have solved the issue! Turns out when I was showing the instructor how to create quizzes, she was using a past course shell, which explains why this was grayed out for her. When, she went into a course shell to be used in the fall, the feature was enabled. Just in case, though, I did try Respondus and had absolutely no issues. 

kona
Community Champion

Whew, very good to know! Thank you for providing an update on this!

ayanni
Community Novice

 @kona ‌ Thanks for all of the helpful advice!

ProfessorBeyrer
Community Coach
Community Coach

 @kona  Thank You for sharing all this expertise with us. A colleague of mine asked a question about academic integrity and Canvas quizzes and I first thought of this document, which I shared with him. I also thought of some additional ways that Canvas tools could answer his direct question and created a blog entry: Adding Academic Integrity to Quizzes.

bphilip
Community Novice

Is there any way to put a date/time restriction on "let students see their quiz responses?" The current options allow me to do that only for the correct answers. It looks like the only way to restrict "see quiz responses" is by manually checking the box when the time comes, which is obviously kludgy.

(In one of my courses, I allow students to take a 5-minute  quiz anytime in a 4-hour window. I don't want to give them any chance to review anything until the 4-hour window is done.)

kona
Community Champion

 @bphilip , unfortunately for now, no. Manual is the only way to keep a quiz totally locked down before releasing results to students. 😞

Kona

ksutter
Community Explorer

If students are allowed to see their results, will that screen be visible as long as that tab is open? We had a test where I think kids who took the test earlier were leaving their results tab open and could then share that screen with other students, so we aren't going to use that option until we figure it out. Muting may be a solution.

kona
Community Champion

 @ksutter , yes, you are correct, as long as they leave the answer page open, they would be able to share the answers with other students or even take pictures of it with their phone. My recommendation is to either (1) mute or (2) remove all options for students to view their quiz responses. 

Hope this helps! Kona

ahardesty
Community Participant

This is such a great document! I have had it bookmarked for some time now. Thank you so much for posting it! With Canvas forcing us into New Quizzes, is there a version of this posted somewhere that has been updated for New Quizzes? 

chris_bacchus
Community Novice

Thank you. With Question Groups you also have to allocate number of points per question. What if each question in the group has a different score/points allocated to it in the question bank? Will those points still be kept, or will each question default to the number of points you allocate to each question with the grouping function.?

cdollar
Community Explorer

Chris: It will default to the points you allocate in the group itself; it ignores the point value from the bank.

chris_bacchus
Community Novice

Thanks so much, I am assuming then that there is no way to allocate a different amount of points to each question when they pulled into the group setting?

Chris

Lecturer, Physiotherapy

Faculty of Health

University of Canberra, ACT, 2601

Phone: +61 (0)2 6201 2806

Fax: +61 (0)2 6201 5727

Location: Building 12 Level D Room 27

Australian Government Higher Education

Registered Provider CRICOS: #00212K

NOTICE & DISCLAIMER: This email and any files transmitted with it may contain confidential or copyright material and are for the attention of the addressee only. If you have received this email in error please notify us by email reply and delete it from your system. The University of Canberra accepts no liability for any damage caused by any virus.

chris_bacchus
Community Novice

Thank you.  I have just realized that I can still award any point value when I mark each question in Speed Grader so that is still OK. I have another question if you don't mind. The quiz instructions seem to be appearing on every page when a new quiz question loads. My instructions take the entire page so students are not seeing the next question unless they remember to scroll down. Can that be changed? Thanks so much

lsigsbury
Community Explorer

I have the same question, looking for a guide for New Quizzes.

akoticha
Community Novice

Quick question. Is this possible in Canvas? I want students to take a 60-minute quiz. I want them to start between 1 pm and 2 pm tomorrow. If they don't start by 2 pm, then they miss the quiz. However, if someone starts at 1:58 pm, then they still get a full 60 minutes. There is no "auto-submit" at 2 pm, so that someone who starts at 1:58 only gets two minutes. Basically, is there an auto-submit flag for the "Until" time that can be turned on or off? If it is turned on, then at 2:00 pm, every quiz is auto-submitted at 2 pm, and no quiz can start after 2 pm. If that flag is turned off, then no quiz can start after 2 pm, but if one has started, it will not be auto-submitted. (The auto-submission will come from the Time Limit variable.) And, btw, what does the Time Limit variable do? I assume it auto-submits at the end of the time interval, but is that correct? (I didn't see this in the description above - it simply said, "limits the amount of time a student has," but didn't say what happens afterwards.)

ProfessorBeyrer
Community Coach
Community Coach

How's this: Give the quiz a 60-minute limit and make it available from to 1 pm to 3 pm. Then at 2 pm change the quiz settings to require a password. From that moment no new students will be able to begin the quiz (unless in a fit of pique you give them the password and then they will have fewer than 60 minutes to complete the quiz), and any students who began it during the hour before you added a password will still get the whole hour you give them.

akoticha
Community Novice

Hi Gregory: That sounds like a good idea. There is one potential problem. I'm using the Respondus Lockdown Browser (+ webcam) to administer this test. The issue therefore is that when I change some settings on the quiz, the settings for the browser get temporarily screwed up (one has to go back and reset the settings in Respondus). The issue is that between the time I change the setting on the quiz and I reset the settings in Respondus, someone can start taking the quiz on a regular browser. (Once they do, you can't kick them out.) But also, not every change in the quiz settings requires a reset in Respondus. I don't know if changing the password in the quiz is one that does or not require is reset in Respondus. I will try to do a test case and report back here. I definitely like your idea, so thanks.

And that aside from you about my possible "fit of pique" - LOL!

Meanwhile, is there a "enhancements" board on Canvas where I can request an "auto-submit" flag as I described above?

ProfessorBeyrer
Community Coach
Community Coach

Yes, any of us can submit an idea to improve Canvas. The process is detailed in the page https://community.canvaslms.com/docs/DOC-14904-75187841184 Once your idea is submitted then those of us who find it will read and decide whether to vote it up. You've got my vote already and I'm following you in the community, so when you create the idea I'll get a notification and will know when to vote.

akoticha
Community Novice

Hi Gregory:

Thanks for the idea.

So let me tell you what didn't work. I did what you suggested. As I expected, when I changed the Quiz settings to require a password, then it gave an error in the Lockdown Browser settings (Btw, when one is in this mode, with an error on the lockdown browser settings, students can get into the quiz on any browser, so it is to be avoided.) Anyway, I went and fixed the error in the Lockdown Browser quickly (this would minimize the likelihood that someone would get in through another browser in the interim). The problem, it seems to me, is that when I fixed the lockdown browser, it lost the "require a password" setting on the canvas quiz, so that probably doesn't work.

However, your idea gave me a subsequent idea, which I think might work. In the lockdown browser itself, I can require a password. So, I can change the setting in the lockdown browser at 2 pm, and that should solve the problem. I haven't tried it yet - will do so soon. Thanks for the idea.

tcrump
Community Explorer

If I set the quiz to allow students access from Monday the 1st until friday the 5th, I set the quiz availability as available Monday the 1st 8am, UNTIL Friday the 5th 4 pm, and DUE Friday the 5th 4pm. Then I set to see the quiz answers at a certain date (like the day after it is due, Saturday the 6th). I receive a message that the answers aren't available Saturday the 6th because the quiz was only available until Friday the 5th. 

But if I don't set the UNTIL part, students who failed to take the quiz on time (thus receiving a zero), can still go in and take the quiz and I have to go through manually after each quiz and check for any late submissions and manually change their grade to a zero, which then results in exchanging extra messages to explain why I did this (even though it should be apparent).

Is there a workaround for this? How can I let the whole class see all the correct answers after the due date without opening access to the quiz for students to actually take it?

crafte
Community Champion

Hi tcrump‌, you will not be able to enter a date for correct answers if you also checked an option 'Only once after...'.

hasti
Community Champion

Usually I give my students 3 attempts on each quiz, I check the "Let Students See Their Quiz Responses" box (but none of others), and then after the deadline go back in and check "Let Students See The Correct Answers". I have one student who has an extension on a quiz, so I'd like that student to take the quiz under the usual circumstances but (since there is an exam coming up very quickly), I also don't want to continue to delay the release of the correct answers to the remaining 391 students in the class.

If I check "Only After Their Last Attempt", that will allow everyone who used all 3 attempts to see the correct answers but what about everyone else who did not use all 3 attempts and for whom the quiz deadline has passed? Are they out of luck or will the fact that the quiz deadline has passed mean they are considered to have used their last attempt? (I have ~20 students who did not use all attempts and did not get everything correct.)

asilbernagel
Community Novice

I am also curious about how to limit feedback.  I offer math quizzes with two attempts.  If the student gets one wrong, I have feedback in the red/wrong comment box that shows the correct answer and an explanation of how to get it.  However, my students have discovered that if they take the 1st attempt without answering any questions correctly, all the feedback boxes pop up and provide the answers!  Clever!  How do I get the feedback to appear only after the 2nd attempt.

I would also like to avoid a date based solution as I have some students working at different paces.

cindi_ward
Community Member

I have created a New Quiz. My students have taken the quiz. I need to grade the short answer and fill in the blank questions that the computer didn't grade correctly. However, the quiz is not showing up under my TO DO List. There is NO Speedgrader button for me to push. The ONLY option I have is to click on each student name in Moderate, make changes, and click back out. I have to scroll to the next student's name and so on. That is VERY LABORIOUS!! HELP ME, PLEASE!!! How can I fix this issue or work around it?

RebekahBlack
Community Member
cindi_ward
Community Member

YES!!! Thank you SO much!! That was a lifesaver!!!

hallale
Community Member

@kona I've been using New Quizzes for years, for both at-home open book homework assignments and for exams, and it's seeming right now that some of my students are copying and pasting the Quiz questions for their homework assignments into ChatGPT or some other AI, as students who do not perform well on the proctored exams and who at the beginning of the semester got 70-80's on the quizzes are now getting 100% on an almost weekly basis.  (Even my top students don't regularly get 100%, even though they are open book, so I'm quite sure these students are using AI to answer the questions.)  Is there a way to make it so that students cannot copy and paste the text from the quizzes?  That would do much to fix the issue, as I doubt they'd be willing to hand type it all into their AI.  

kona
Community Champion

@hallale -- unfortunately, there isn't a great way around this as long as you do online quizzes in an unproctored environment. One thing we use at my school is Proctorio, which can lock down the quizzes so students can't copy and paste text from the quizzes. Yet, they can still take pictures or ultimately, they could still type out the answer in ChatGPT.