New vs. Old Analytics

[Katrina-Hess]

Hello Canvas Community!

We will continue to iterate on improvements to course-level analytics. In the meantime, we've created a guide of the differences. Please let me know in the comments if we've missed anything and I'll be sure to add it to the list!

	New Analytics	Old Analytics
Aggregate Period	Weekly—per student, per section, per course*	Daily—per student and course
Hourly Aggregate	No*	Yes—per student
Category Aggregates	No*	Yes
Account Aggregates	No*	Yes—across multiple courses in a sub-account
Data Display	Table view In development: Submission Date column in Student View data	Graphic Illustration view
Data Comparisons	Course, Section, Student	N/A
Data Customization	Filter by assignments, quizzes, and graded discussions and enable shape representations	N/A
Activity View	View specific students who viewed, didn't view, participate, or didn't participate View specific students with missing or late assignments Message students regarding status	View number of students with page views or participations View number of students with missing or late assignments
Assignment Grade Statistics	Mean score for the class, sections, and students In development: median, high, and low scores per assignment grade	Median, high, and low scores per assignment grade
Rollup Resource Data	View number of resources viewed with list of resources	View number of resources viewed
Rollup for Student Assignment Status	Status per assignment In development: number of missing, late, on time per student	Missing, late, on time
Mobile Page View Data	Included	N/A
Student Messaging in Score Ranges	Yes—via group message sent to students individually	No
Student Messaging for Missing/Late Assignments	Yes—via group message sent to students individually	No
CSV Data Download	Yes	No
*improves overall performance; this functionality will continue to be improved

Best, Katrina

Update 1-30-2020: New analytics provides a better representation on user activity than the old analytics. We removed extraneous requests to better align the data with student behavior.

[vanzandt]

Thank you for putting this together. I notice that it is primarily listing things from the instructor perspective. I feel like I've also read about some of the student view differences/additions, that may not be captured here, but to be honest the specifics of those posts aren't coming to mind.

Looking forward to this being kept up to date as additional development occurs. Thank you again!

[Maeve_McCooey]

Hi khess@instructure.com, thank you for posting this, it is very informative.

I am currently putting together a gap analysis between the two versions of analytics and if I find anything else I will post here. My initial thoughts on new analytics are that it provides greatly improved, meaningful stats and functionality and that the various CSV report downloads will be super beneficial to teaching staff.

Do you have any timelines around when the developments in italics above will be released? And/or any indication of when New Analytics will be fully refined? Could I also ask that you please consider the following ideas/questions raised please?

https://community.canvaslms.com/ideas/15553-new-analytics-filter-identifydifferentiate-sections-stud... 
https://community.canvaslms.com/ideas/15501-submission-date-missing-in-new-analytics  
New Analytics, Student View Submissions feature 

Eagerly awaiting the further iterations/updates of New Analytics and again, thank you for sharing!
m.mccooey@qub.ac.uk 

[dennis_foung_wo]

Thanks for this! Very useful! 

I wonder if there are differences in counting page views / participation between new analytics and old analytics. It seems that the pageview in the old analytics is not the same as the new one, for the same course, same period of time and same student. The two numbers are highlighly correlated but they are really different. The old one is always higher than the new one too. 

[Katrina-Hess]

Hi vanzandt, we include differences/additions with the student view in the list above but please let us know if we missed something!

Best, Katrina

[Katrina-Hess]

Hi m.mccooey@qub.ac.uk‌, please do post anything we may have missed here. 

Do you have any timelines around when the developments in italics above will be released?

• Submission Date column in Student View data -- releasing soon
• Median, high, and low scores per assignment grade -- releasing in 2020
• Number of missing, late, on time per student -- releasing in 2020

And/or any indication of when New Analytics will be fully refined? 

Our plan is to iterate and refine New Analytics on regular basis. We will add new features based on requests and usage. 

Could I also ask that you please consider the following ideas/questions raised please?

• New Analytics Filter - Identify/Differentiate Sections, Students, Assignments -- request received

• Submission date missing in new analytics -- releasing soon 

• New Analytics, Student View Submissions feature -- we are not replacing the submissions visualization but will be adding the Submission date column which will be releasing soon

Hope this helps! 

Best, Katrina

[Maeve_McCooey]

Fantastic khess@instructure.com ! Thank you so much for the update, so helpful

[schmi071]

Hi Katrina, I am not sure this one is on the list. It is the VIEW SUBMISSIONS graph found on this website.( How do I view analytics for a student in a course? )

It is the most useful tool I know for quickly assessing how a student is performing in class. I can quickly see number of late assignments and more importantly HOW LATE these assignments were. I can also see what assignments are missing and quickly count them up (count all red squares). I can assess a students performance in about 3 seconds on this graph. I NEED THIS BACK!.

[ataylor30]

I second this. This graph has been really helpful and it seems like a big lost function with New Analytics.

[thompsli]

I am also a major user of this feature and very concerned about its absence from New Analytics. I regularly use this view during parent-teacher and student-teacher conferences to talk about student habits around late versus on-time submissions, and it's the clearest way to get a visual of who is doing steady work versus working in bursts every few weeks. 

[Katrina-Hess]

Hi everyone, thanks for your feedback. We added a Submitted column in the Student View. You can sort the column headers. To see the Missing assignments, click on Status.

[Katrina-Hess]

Hi dennis.foung.work@gmail.com‌, new analytics provides a better representation on user activity than the old analytics. We removed extraneous requests to better align the data with student behavior.

[schmi071]

Hi Katrina, This is a step in the right direction, but not nearly as much information or as quickly reviewed as the graphic we are looking for. (View submissions graph in this website). I still have to do the math on how late each assignment was (submission date minus due date) and then try to determine if the late submissions were were clumped together during a stressful life situation or throughout the semester. I have about 75 assignments in the class so doing all this takes time. I teach two classes with 400 students in each class another factor in why I am so passionate about streamlining good information on my students.

I am not trying to be difficult here but why not just put in the old graphic in the new analytics as an option?

Thanks. _David

[thompsli]

Here's what I'm missing with New Analytics that I value with the current system. I can make a quick distinction between a student who "works in bursts" like this:

versus a student who is just sort of gradually falling behind the pace and dawdling along like this:

versus a student who is just always, for whatever reason, turning everything in about one day late and has been all term: 

Those all lead to different conversations about work habits, and the visual is really useful for those conversations.

[Katrina-Hess]

Thanks thompsli@hsd.k12.or.us‌, your examples are helpful! I will take this back to the team to discuss. We see the value of this visualization.

[Katrina-Hess]

Hi schmi071@umn.edu, thank you for sharing details of your workflow! The old graphic isn't connected to the new data source so it's not as straightforward as putting the old graphic back in. I will share your feedback with the team as we continue our exploration on improving this chart. 

[thompsli]

I thought of one more example of something that was easy to spot with this visualization, but hard to spot in most other cases: The person who does the "burst work" thing, but well ahead of the deadlines. Example here:

This isn't always a problem, but can be if the quality of work or retention of ideas suffers. (The ones without tails for this particular student were my "check-in" assignments with very narrow open to/from dates - all actual math tended to be done in bursts and then the student would have a few weeks off while they waited for me to open up the next batch of work.)

In this particular student's case, this worked out just fine, but if a student comes in with lower grades on exams as a result of a strategy like this on online assignments then it's something I'd bring up in a "study habits" conversation. (My classes are not self-paced so the exams happen on a schedule to do with the term.) I wouldn't be able to see this pattern at all if I only looked at late versus not late assignments, so the tails were very useful here.

[BradMoser]

What about outcomes? Any chance we can view this inside of the new analytics? 

[Katrina-Hess]

Thanks thompsli@hsd.k12.or.us‌, will add to your examples!

[kirsten_ryall]

Hi there, has there been any requests so far to add 'type of resource' to the 'resources' csv download? I can see resource name which is good, but would also like to see type as well in the report. I think it could be a really valuable report for learning designers. If I had a course that included loads of content pages for example and a few ungraded discussions, and I could see that those ungraded discussions were high in views and participations, perhaps I would include more discussions in future.

Kirsten

[nancy_webb]

Hello, our account rep mentioned that page views and participation metrics  include an aggregate across all devices, including Canvas mobile apps.  So, we will be able to get a count of which tools students accessed using specific devices (desktop, laptop, and mobile)?

Thanks,

Nancy Webb

[griggsda]

David Schmidt hit the nail on the head. I found the graph most useful in 'old analytics'. 

I find 'new analytics' utterly useless.

[ah2274]

Yes, PLEASE take this back to the development team! I agree 100% with Linnea Thompson. This kind of visualization is incredibly useful!  I was shocked to learn it was dropped in the New Analytics.

[anamraicu]

It would be very helpful if we could find in New Analytics:

1.date of each submission is exceptionally important; personally I don't need the Due date in New Analytics, I already know it since I posted the Assignment

2.how many hours each student spent in CANVAS each day

3.how many hours each student participated in the online conferences each day

4.last day of activity in CANVAS, either attending online Conferences, Reading documents or posting assignments; our college expels students after a certain period of nonattendance; this date is critical both for students and for teachers since we would like to keep them enrolled

5.the number of words per each submission

If these functions are already in place, please advise where would be the best place to get instructions

Thank you for your efforts to improve CANVAS

Like • Show 0 Likes0Reply

 Actions 

[Jeff_F]

I am concerned with the aggregate period being weekly vs. more frequently so that may stall our adoption.   If we are to use an analytics tool for the purpose of instructors seeing what is happening then a weekly update is simply too long.  This is probably long past observed and debated.  

If this is not a big deal, would someone please explain?  

[kathleen_gonoud]

I do NOT like New Analytics. It does NOT show the info I need to track student activity. Can we have the choice to use "old" analytics?

[LloydH]

Hello,

I'm wondering if there has been any consideration for giving instructors the ability to apply analytics to daily activity. In the current academic environment, being able to determine what students have done on daily basis is an essential tool. The weekly activity report is just not very useful at all.