Accuracy of Data for Mobile-First Students

ken_cooper
Community Participant
1
1386

Hi All,

 

I thought I would post a blog to expand on the comment I made in the Ask Technology & Engineering Leadership‌ blog started by  @Renee_Carney ‌.   @James ‌ was awesome (as usual) and linked to a three year old thread that had new comments by Instructure's Director of Products - Data and Analytics,  @kturco .  I never would have seen those posts in a three year old thread if not for James, so it's great to see someone in leadership comment on the issue.

 

Now having seen Kevin's posts in Canvas Data Requests count vs. Canvas Analytics Page Views‌, I would echo what many others have said and say that it's critical we are getting accurate mobile data reported to us.  I'll share one of his responses specifically to the question of "Are Mobile App page views included"?

 

Mobile page view data was added to the Course Analytics reporting last year. It has not, however, been added to the Access Report. This may be why you see mixed messages on whether or not it is included. While our plan is to  continue adding mobile data to our reports where appropriate, it is critical to understand the limitations of mobile page view data, and the reason for us to start with Course Analytics and not the Access Report.

 

Even more so than Canvas request data, mobile request data is best used in aggregate to identify trends of activity, and not for audit purposes or to get exact counts. Mobile page view data is inherently tricky to capture, since cell phones can lose service, apps can be force-quit, and  phones can lose power and be shutdown mid request. As such, mobile page view might be delayed from getting to our servers, or even dropped. This is true  for all mobile usage data, across all applications with mobile analytics data. It's a very tricky data set to work. This is why we focused on including mobile data first with our aggregate reporting (Course Analytics), as this data is valuable in showing overall activity trends.

 

With the growing use of mobile devices, we understand the need to include mobile data wherever page view data is included. This is something  we  will continue to work towards. I appreciate, and acknowledge this is not an ideal answer to address your immediate question, however transparency is something I take seriously, so want to make sure you have all the details.

 

We have Instructure's Amazon Redshift Hosted Canvas Data Services here at Los Rios (shoutout to Jason Rock jrock@instructure.com for being so great to work with), and mainly use Tableau for visualizations of that data.  The requests table aside it's great to work with, and running queries such as Device Usage gives us a good idea of how many students are using mobile devices (even if they aren't fully mobile-first).  We have tens of thousands of students using mobile devices for tens of millions of page hits every semester here (we had roughly 4.5M page hits per day on the request table last semester), so it seems that having equitable representation of that data should be a pressing concern, and needs to be more accessible than pulling from the requests table in Canvas Data (and that isn't even fully accurate).  

 

As I mentioned in my post, there is an entire online college (our 115th Community College) launching in California this upcoming academic year that is being designed as mobile-first: Online Community College - California.  These students need have their work counted as equitably as students using a PC, and the reports and analytics should not be any different from students using a PC.  This would help our students as well, as there are so many courses being designed to be mobile-first, and that was a huge topic of conversation at the Online Teaching Conference 2019‌ I just attended last week.  I'm very grateful to Kevin for posting and keeping the discussion going, as it is certainly a concern for those of us in charge of pulling those numbers.

 

Oh....and I won't even get into folks like me that use Photon on their iOS devices in order to use a User Agent and spoof everyone so they think I'm on a PC using Chrome or Firefox rather than an iPhone or iPad, since we know those folks are just trying to use something that plays Flash on iOS ( <--- this was my poor attempt at humor for those nice enough to read to the bottom).  Smiley Happy

 

Thanks for taking the time to read this.

 

Ken

1 Comment
rseilham
Community Champion

Thanks, Ken Cooper‌ - 

And yes...I read to the bottom, love the joke. (: 

I agree that a few things are going on here and I wish they all could be solved. At UCF, we use Canvas Data (and a little bit of Google Analytics) to give us aggregate data on the number of unique users, devices, and activity in Canvas on the app and mobile web. I am lucky enough to have direct access to a data engineer that is in Canvas Data almost every day. When I talk about mobile-specific data, he generally shakes his head because it's so noisy compared to what you get on the web. A big reason is a mobile app simply pulls all the information from the APIs, so checking the inbox can produce a bunch of calls and rows in the Canvas Data compared to a simple Page view. We currently track some KPIs for the Canvas mobile apps, but they are very general around unique users and the total number of request produced, but we can't compare the requests with the web because it's not apples to apples. Over time, our data engineer has found patterns and reduced the noise with Canvas Data with the mobile apps, but there is still work to be done to display good data in Canvas Data, and other portions of Canvas to teachers and admins. 

I know I'm preaching to the choir, but it's good to know we both have work to do with Canvas to solve this.