Can "Stopped viewing the Canvas quiz-taking page" in the quiz log reveal cheating?

[bjl11100]

Hi, 

I told my students that moving away from the quiz to other windows/tabs would be cheating. Can you help me interpret the quiz log to see if the evidence there establishes this was happening? (I know Canvas itself is not a lockdown browser, but I had so much trouble requiring Respondus...) 

It's a traditional Canvas quiz. For some students, there is no report that they "Stopped viewing..."; but for others, the log reports look like the attached images (showing what appears to be a pattern of navigating away from and back to a quiz surreptitiously). 

Here's what Canvas docs say: 

Viewed (and possibly read) a question [1] displays when a student is active on the quiz page, but has not answered the question.

Answered a question [2] displays when a student has answered the question. If questions are shown being answered multiple times, the student either changed their answer(s), or the answer was generated by the quiz autosave feature.

Stopped viewing the Canvas quiz-taking page [3] displays when a student navigates away from the quiz (closes the browser tab, opens a new browser tab, or navigates to a different program). According to a "Community Champion," "Not viewing the quiz-taking page can occur from being inactive within Canvas for more than 30 seconds (including navigating to another Canvas page), or clicking out of the quiz (such as on another browser tab or window) for more than 15 seconds." 

Resumed [4] displays when the student has returned to the quiz.

It sounds to me like any log event reporting that a student "Stopped viewing" for between 15 and 30 seconds before another event clicked away from the quiz. 

Here are more of my considerations: According to the above-quoted "Community Champion," the words "Stopped viewing the Canvas quiz-taking page" come to appear in a log under these two conditions: 

1. The first condition is met when a student is inactive within Canvas for more than 30 seconds. (So a 30-second interval before those words would not prove cheating, as it might be that the student is thinking. Hence, I set aside as inconclusive the cases in which the logged interval is more than 30 seconds.)  
2. The second condition is met when, after an interval of less than 30 seconds but more than 15, a student clicks out of the quiz.

I'm focusing on cases in which the second condition is met as potential cases of cheating. But are there any conditions in which there's a 16-30 second interval in which a student did not click away from the quiz? (If not, these conditions seem to establish cheating in my cases.) 

Thank you for the clarification! 

Brendan 

[chriscas]

Hi @bjl11100,

First, I do want to note something that Instructure states in the blue box as the top of the documentation: "Quiz logs should not be used to validate academic integrity or identify occurrences of cheating."  I think it's important to lead with that, because academic integrity is a high-stakes item and something you probably want to be 100% right.  Based on Instructure's statement, I (a Canvas admin) will not use these logs at all for academic integrity purposes.  I think there are a lot of variables where some information may be missed in the log and perhaps some extra info is actually in the log.

One scenario to consider is whether a notification may have popped up on the student's screen while taking a test.  It could be from an incoming email, an incoming text message, or other similar event (depending on the device and operating system).  It could be that the student inadvertently clicked on the notification when it popped up, causing them to launch over into another program, and it realistically could take some number of seconds to get back to the quiz.  I know some instructors will consider this cheating, saying students are responsible for silencing notifications on their device while taking the quiz, but that seems a bit harsh to me.

On the flip side of this, some browsers have GenAI assistants built in now that are accessible right from a sidebar, and those cam work without a student ever leaving the quiz.  So it's possible that some students are utilizing those and wouldn't necessarily be flagged in the quiz log at all.  Similarly, if a student has a second device available, they could use that to look up info and never be logged.

I think some of these scenarios probably went into the thinking behind Instructure's statement not to use the logs to investigate academic integrity, and I hope some of the examples here help you with this.  I'm sure there are some other example cases others may be able to provide as well around this if needed.

Please do reply back if you have additional questions or concerns or if I've misinterpreted your question.

-Chris

[bjl11100]

Hi, and thanks for the response.

Like you, I didn't plan to use quiz logs as any part of assessment of academic integrity. I discovered the contents of the logs during an exam, noticing that some students were apparently(?) clicking away, and others not. So the approach I’m taking is forensic (and limited only to detecting a single, albeit common, form of cheating). The answers I’m looking for would have to be pieced together based on knowledge of how Canvas log reports are generated. I'm asking the community because I believe it includes at least some developers who know enough about how Canvas logs are generated to answer this question. 

Maybe there is no sure interpretation of the logs, for the kinds of reasons you very plausibly present (about interference from browser plugins or OS-level events). They give traction to your speculation that "these scenarios probably went into the thinking behind Instructure's statement not to use the logs to investigate academic integrity."

Still, in light of the documentation I cited, I have trouble seeing how such repeated, short intervals in the logs should/could be taken as unproblematic: 

Still, unless there's a response that allows us to disambiguate "false positives" from "true positives" in the quiz logs, my plan is to proceed as if Canvas documentation is not complete on this question, and that there is thus no sure way to use the information Canvas routinely and systematically generates to do anything other than what it is officially intended for. I think it would be too bad if the community never has an answer to this question about a common form of cheating, given that people who know the software (esp. the designers) might be able to confirm or deny that the documentation on this question is complete. 

[VickiHanes]

Can this happen if a student is given permission to backtrack among questions? I found this happened if the students were answering questions out of order. Also, not every question they answered is noted in the log. Does this indicate a glitch in the system. 

[healing24]

Hi Chris, 

Thanks for your reply. Regarding your post, let's say a student received an incoming email or text message or accessed Chrome from their phone, and the text/chrome icon appeared in the docket. Is it fair to assume that this wouldn't be considered navigating away from canvas if the apps were not opened on the computer (even though the icon appeared on the dock)? You have to open the app on your computer for the event to register that you navigated away from Canvas, right?

[AllanNishimura]

I would definitely agree, but what about cases where this occurs with every other question?

[VickiHanes]

Update: After communicating with students and going over their activity with them, I made sure that on our next quiz to remind them to close all windows and also to put their devices in "Do Not Disturb" mode. I also requested that if they were going to review their questions that they do so at the end so I could check if it was backtracking that was causing the notation. As a result, no activity of going out of the test window was noted, even when students did some backtracking and changed answers. So, the initial reports on the other quiz were most likely accurate as to whether or not they went out of the testing window, or got notifications, or were running other programs/windows in the background. 

[healing24]

Hi Vicki, thanks for sharing. Canvas wouldn't log the activity even if the message/chrome icon popped up on the student's docket, but they did not open either text or chrome, right?

[VickiHanes]

I didn't go that far to ask. Most students are on iPads, on Safari, so not sure if the impact of notifications is different depending on those variables. I just had them put them on "Do not disturb" so there isn't the possibility of a mistake. 

[harnold]

I have noticed the 15 second is usually navigating away and 30 seconds or more is pausing while in quiz. The answering question and the first stopped activity is only 12 seonds with subsequent 15 second interval tells me it is more than likely a navigate away situation. You can also get a message is the student was force logged-out and you will see "Session Started" a second time in the quiz log. In the example below, the student started a really intense math problem #5 , and got "booted" or accidently clicked the "x". The student was able to log back into the quiz a few minutes later after putting log-in info, which is where the second "Session Started" message comes from at 49:17

Prior to the stop he scrolled the whole test and went back to work on the harder problem at the top. Because he did not navigate away and did not stop his mouse movement for more than 30 seconds, there is a 19 minute gap with no stoppage. (see attached)

I also like to compare classes as a whole. One test I thought a student was cheating and 4 people had the "stopped" message, which was about 16% of the class. 3 out of the 4 only had the stop messages only once in the log, but one person had 53 stop messages. Because there are cameras in the testing area we were able to verify screen switching at the time of the test. 

In summary, use the log as a tool to raise your suspicion, but always have a way to verify. I have also found Respondus LockDown to be a deterent especially in brick and mortor school. Time limits also help. Put a reasonable time limit where they can not search every answer. 

[AllanNishimura]

I have two questions;

1. If the 'stopped viewing the Canvas quiz-taking page' comes up on almost every other question, for between a 30 sec - 1 min, doesn't that raise suspicion? (about 100 question exam).

2. Does the `stopped viewing...' occur if screen saver comes up?

[harnold]

Yes, it would raise suspicion especially if it is an online "quiz" where you are not physically there to proctor. My institution uses LockdownBrowser, which has stopped most of the flags during a test. My suggestion would be to take the quiz in student view and have various things happen on purpose so you can view what happens in the log like scrolling in the scroll bar, scrolling to go to multiple questions many questions apart, pause on a question for 1 minute, pause on another question for 5 minutes etc... this will help you determine what is triggering flags in your activity logs and put your mind at ease with what the flags could mean for your students. Usually, stopped viewing every question gives you something to discuss with the student about academic integrity and honesty policies. You can also talk to your institution about add-ons like LockDown Browser. 

[AllanNishimura]

Thank you for these suggestions. My institution discontinued with LockDown Browser because of the cost (~$3000/year). College-wide policy is that all exams must be proctored. I walk around the classroom every 10 minutes or so, but that's not enough. We don't have remote cameras, and of course, the students' computer screens face away from me when I'm in the front.  I'd like to sit behind the class, but the physical arrangement of the classroom doesn't make that possible. The 5 students who confessed to cheating said they were using AI to answer the questions; just wanted to pass that along. I may have to go back to paper exams as I had been before Covid. If you have suggestions for me, please let me know. This whole affair really affects the joy of teaching.