Ken Cooper

Accuracy of Data for Mobile-First Students

Blog Post created by Ken Cooper on Jun 27, 2019

Hi All,

 

I thought I would post a blog to expand on the comment I made in the Ask Technology & Engineering Leadership blog started by Renee CarneyJames Jones was awesome (as usual) and linked to a three year old thread that had new comments by Instructure's Director of Products - Data and Analytics, Kevin Turco.  I never would have seen those posts in a three year old thread if not for James, so it's great to see someone in leadership comment on the issue.

 

Now having seen Kevin's posts in Canvas Data Requests count vs. Canvas Analytics Page Views, I would echo what many others have said and say that it's critical we are getting accurate mobile data reported to us.  I'll share one of his responses specifically to the question of "Are Mobile App page views included"?

 

Mobile page view data was added to the Course Analytics reporting last year. It has not, however, been added to the Access Report. This may be why you see mixed messages on whether or not it is included. While our plan is to  continue adding mobile data to our reports where appropriate, it is critical to understand the limitations of mobile page view data, and the reason for us to start with Course Analytics and not the Access Report.

 

Even more so than Canvas request data, mobile request data is best used in aggregate to identify trends of activity, and not for audit purposes or to get exact counts. Mobile page view data is inherently tricky to capture, since cell phones can lose service, apps can be force-quit, and  phones can lose power and be shutdown mid request. As such, mobile page view might be delayed from getting to our servers, or even dropped. This is true  for all mobile usage data, across all applications with mobile analytics data. It's a very tricky data set to work. This is why we focused on including mobile data first with our aggregate reporting (Course Analytics), as this data is valuable in showing overall activity trends.

 

With the growing use of mobile devices, we understand the need to include mobile data wherever page view data is included. This is something  we  will continue to work towards. I appreciate, and acknowledge this is not an ideal answer to address your immediate question, however transparency is something I take seriously, so want to make sure you have all the details.

 

We have Instructure's Amazon Redshift Hosted Canvas Data Services here at Los Rios (shoutout to Jason Rock jrock@instructure.com for being so great to work with), and mainly use Tableau for visualizations of that data.  The requests table aside it's great to work with, and running queries such as Device Usage gives us a good idea of how many students are using mobile devices (even if they aren't fully mobile-first).  We have tens of thousands of students using mobile devices for tens of millions of page hits every semester here (we had roughly 4.5M page hits per day on the request table last semester), so it seems that having equitable representation of that data should be a pressing concern, and needs to be more accessible than pulling from the requests table in Canvas Data (and that isn't even fully accurate).  

 

As I mentioned in my post, there is an entire online college (our 115th Community College) launching in California this upcoming academic year that is being designed as mobile-first: Online Community College - California.  These students need have their work counted as equitably as students using a PC, and the reports and analytics should not be any different from students using a PC.  This would help our students as well, as there are so many courses being designed to be mobile-first, and that was a huge topic of conversation at the Online Teaching Conference 2019 I just attended last week.  I'm very grateful to Kevin for posting and keeping the discussion going, as it is certainly a concern for those of us in charge of pulling those numbers.

 

Oh....and I won't even get into folks like me that use Photon on their iOS devices in order to use a User Agent and spoof everyone so they think I'm on a PC using Chrome or Firefox rather than an iPhone or iPad, since we know those folks are just trying to use something that plays Flash on iOS ( <--- this was my poor attempt at humor for those nice enough to read to the bottom). 

 

Thanks for taking the time to read this.

 

Ken

Outcomes