cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
Surveyor

How accurate are Canvas Access Reports?

Jump to solution

I'm working with a faculty member who uses Canvas Access Reports extensively. The issue is she has had a couple instances where the Access Report indicates a students didn't access course material (a lecture for example), but the student insists that they did. In one of the instances the student's discussion post on the lecture also seems to indicate they watched the lecture.

This has led me to ask you: How accurate are Canvas Access Reports in your experience? Have any of you seen anything that would indicate they are not 100% accurate? I'm aware of the Canvas Mobile app being an issue, but in this case the student indicated they were using a laptop. We are just rolling out access reports for entire courses (Obtaining and using Access Report data for an entire course ) so I'll be curious to see what that looks like for the course and student in question.

Is there anything else I'm missing? Thank you in advance!

1 Solution

Accepted Solutions
Highlighted
Navigator

dlavalle@wisc.edu,

The mobile is the number #1 thing.

My script returns for the entire class what Canvas makes you do one student at a time, plus some extra information that Canvas has but doesn't expose. There is one exception that I found out about but haven't rectified yet and I'm not sure if I'm going to. There is code in Canvas to subtract the number of participations from the number of views for quizzes. When a student takes a quiz, they technically have to view it, but they were double counting it as a view and a participation. My code takes the number returned by their AJAX call and doesn't try to make any of those alterations to it, it just delivers the data as-is.

There are periodically bugs that will affect things, like the time that opening the Canvas (student) app on a mobile device would set the last activity for every since class, even ones that hadn't been over for a couple of years. I don't know if that affected the Access Report data or not.

If you've made offline content available, then the student could download the entire course and read it offline. I have not looked at whether or not that changes the accesses. It's also possible that the student was with a friend and read it on her computer.

In general, I would look for patterns rather than absolutes and say that mobile aside it's a good indicator of how many times an asset has been accessed by someone using that student's login.

The bigger picture for me is that even though they may open a page, there is no way through the Access Report of knowing whether or not they actually read the content on the page. I allow multiple attempts on my quizzes and have several students who do poorly on the first 3 attempts without any real improvement and then magically get a perfect score on the last one. Suspicious -- yes, enough evidence to call them out for cheating -- no. But it bothers me enough to change my grading from "highest" to "average" this semester. The point being that just because what they turn in online makes it sound like they know what they're doing doesn't mean anything, it may not have been them doing it.

If you're trying to track something down about a particular student, the teacher may need to get the Canvas Admin involved to check the page views data, which I believe has been updated to incorporate mobile traffic. It's not as easy to work with, but should be more definitive.

View solution in original post

8 Replies
Highlighted
Navigator

dlavalle@wisc.edu,

The mobile is the number #1 thing.

My script returns for the entire class what Canvas makes you do one student at a time, plus some extra information that Canvas has but doesn't expose. There is one exception that I found out about but haven't rectified yet and I'm not sure if I'm going to. There is code in Canvas to subtract the number of participations from the number of views for quizzes. When a student takes a quiz, they technically have to view it, but they were double counting it as a view and a participation. My code takes the number returned by their AJAX call and doesn't try to make any of those alterations to it, it just delivers the data as-is.

There are periodically bugs that will affect things, like the time that opening the Canvas (student) app on a mobile device would set the last activity for every since class, even ones that hadn't been over for a couple of years. I don't know if that affected the Access Report data or not.

If you've made offline content available, then the student could download the entire course and read it offline. I have not looked at whether or not that changes the accesses. It's also possible that the student was with a friend and read it on her computer.

In general, I would look for patterns rather than absolutes and say that mobile aside it's a good indicator of how many times an asset has been accessed by someone using that student's login.

The bigger picture for me is that even though they may open a page, there is no way through the Access Report of knowing whether or not they actually read the content on the page. I allow multiple attempts on my quizzes and have several students who do poorly on the first 3 attempts without any real improvement and then magically get a perfect score on the last one. Suspicious -- yes, enough evidence to call them out for cheating -- no. But it bothers me enough to change my grading from "highest" to "average" this semester. The point being that just because what they turn in online makes it sound like they know what they're doing doesn't mean anything, it may not have been them doing it.

If you're trying to track something down about a particular student, the teacher may need to get the Canvas Admin involved to check the page views data, which I believe has been updated to incorporate mobile traffic. It's not as easy to work with, but should be more definitive.

View solution in original post

Highlighted
Learner II

We tested about three months ago and found the reports to be fairly accurate, with exception of when the students use the app. 

Highlighted
Learner II

james@richland.edu‌ - very nice response. We are trying hard to gauge access and have found that there are indeed variables. The biggest eye opener to us was a course on social media where half the class had not accessed the prereadings a full week into the course (according to the Canvas analytics) They were in fact using mobile devices and the app and it did not report back to be included. So, the Instructor is freaking out and the students are insisting that they read the material. The disconnect was with the mobile experience and was confirmed by Instrcuture that the analytics will be skewed in those cases.

Highlighted
Surveyor

I just had to hop back in here and say thank you to everyone for your responses - james@richland.edu‌, floresr@smccd.edu‌, and hallmans‌. They are very much appreciated!

Highlighted
Adventurer

As an additional comment, it looks like fulfilling Module requirements doesn't work from the App either.  We have a situation where all students in a class but one fulfilled the requirement of submitting an assignment, and yet the one student had submitted as well. Checking page views, it was confirmed she viewed and submitted using the iOS app.  Do any others have this experience?  This is not good when there are prerequisites on the next module.

Highlighted
Community Coach
Community Coach

Hello dlavalle@wisc.edu...

I noticed that there hasn't been any new activity in this particular discussion topic for about two months, so I wanted to check in with you.  It looks like you've received some good help from members of the Canvas Community.  Have you also had a chance to review the posting from nwebb@ccsf.edu‌ that she posted in early September?  Also, is there a response above that you'd be able to mark "Correct"?  Or, if you are still looking for some feedback from the Community, please feel free to post a message below.  For now, since we've not had any new activity in this thread since September 1st, I'm going to mark your questions as "Assumed Answered", but that won't keep you or others from posting additional questions/comments below that are related to your question.  I hope that's alright with you.  Looking forward to hearing from you, Dan!

Highlighted
Surveyor

While researching a little bit about the accuracy of Access Report for Instructors, I realized that Access Report does not show correctly the interaction that an Instructor has done in a discussion when the discussion is set up as a group discussion. Maybe I am missing something but this way is very complicated to evaluate faculty engagement. Does anyone know a solution to this problem? I heard that Canvas is working really hard to update their analytics (finally!), but I am not sure whether Canvas is going to provide data to evaluate instructors (i.e. do they answer students questions on time, do they engage with them, are they educating students after all?). Do you know of a program that can do that? 

I am working on setting up Google Analytics but having these inaccurate issues am not very confident of it. 

I would appreciate any help you may provide. 

Highlighted
Explorer II

I just posted a question in the New Quizzes group: https://community.canvaslms.com/thread/44764-access-report-not-showing-new-quizzes-submission 

I'm not sure if the New Quiz submission was not recorded because of the tool or if it happened from the student app. Sigh.