On Data Differences Between Analytics and New Analytics

[dlecocq]

In 2019 we introduced New Analytics to replace and improve upon the course- and user-analytics offering in Canvas. We did so because we saw an opportunity to make an update to a valuable product to provide expanded and more accurate information. Since that original Analytics offering was first written, Canvas has grown and the difference between Canvas as it was and Canvas as it is now has led to disparities between what was presented in that tool and what we would expect to see.

All this to say, we know that there are discrepancies between the two. We tried very hard to understand and reconcile as many of those differences as possible, but in some cases we had to make the conscious choice to provide more accurate and relevant data. The first analytics offering was written five years ago, and in the intervening time Canvas has had a long path of changes - from the big to the small-and-nuanced, leading to subtle differences between Canvas’ internals and analytics’ understanding and presentation. This means subtle changes between the two in terms of the interpretation of grade statuses as well as activity. In the case of activity data, this also means slicing and dicing that data in new and different ways (in particular breaking it down to per-week numbers).

Our guiding star with this new work was always to do our best to accurately reflect the reality provided in Canvas.

We'll take this opportunity to try to describe our interpretation of activity information in particular in hopes of clarifying what is being presented now.

The activity information surfaced in New Analytics is derived from request logs from Canvas itself. How a user's action is presented and counted depends on the route/controller used, the status code, and the HTTP verb on that request. For example, loading the URL for an assignment will count as a Page View under the title of that assignment. Below is a table of the URLs that we count and consider in New Analytics and how they are presented (the P column is whether or not it's considered participation)

Name	P	Method and Path
<Assignment Title>		GET /courses/:course_id/assignments/:id(.:format)
<Assignment Title>	✓	POST /api/v1/courses/:course_id/assignments/:assignment_id/submissions(.:format)
<Assignment Title>	✓	POST /courses/:course_id/assignments/:assignment_id/submissions(.:format)
<Assignment Title>	✓	POST /courses/:course_id/submissions(.:format)
<Attachment Title>		GET /courses/:course_id/files/?preview=:file_id
<Attachment Title>		GET /courses/:course_id/files/:file_id/download(.:format)
<Attachment Title>		GET /courses/:course_id/files/:file_id/download.:type(.:format)
<Attachment Title>		GET /courses/:course_id/files/:file_id/preview(.:format)
<Attachment Title>		GET /courses/:course_id/files/:id(.:format)
<Attachment Title>		GET /courses/:course_id/files/:file_id/file_preview(.:format)
<Attachment Title>		POST /courses/:course_id/files/:file_id/inline_view(.:format)
<Calendar Event Title>		GET /courses/:course_id/calendar_events/:id(.:format)
<Calendar Event Title>	✓	PATCH /courses/:course_id/calendar_events/:id(.:format)
<Calendar Event Title>	✓	PUT /courses/:course_id/calendar_events/:id(.:format)
<Collaboration Title>	✓	GET /courses/:course_id/collaborations/:id(.:format)
<Conference Title>	✓	GET|POST /courses/:course_id/conferences/:conference_id/join(.:format)
<Outcome Title>		GET /courses/:course_id/outcomes/:id(.:format)
<Page Title>		GET /courses/:course_id/pages/:id(.:format)
<Page Title>		GET /api/v1/courses/:course_id/pages/:url(.:format)
<Page Title>	✓	POST /api/v1/courses/:course_id/pages(.:format)
<Page Title>	✓	PUT /api/v1/courses/:course_id/pages/:url(.:format)
<Quiz Title>		GET /courses/:course_id/quizzes/:id(.:format)
<Quiz Title>		GET /courses/:course_id/quizzes/:quiz_id/history(.:format)
<Quiz Title>	✓	GET|POST /courses/:course_id/quizzes/:quiz_id/take(.:format)
<Quiz Title>	✓	GET /courses/:course_id/quizzes/:quiz_id/take/questions/:question_id(.:format)
<Quiz Title>	✓	POST /api/v1/courses/:course_id/quizzes/:quiz_id/submissions(.:format)
<Tool Title>		GET /api/v1/courses/:course_id/external_tools/:external_tool_id(.:format)
<Tool Title>		GET /courses/:course_id/external_tools/:id(.:format)
<Topic Title>		GET /courses/:course_id/discussion_topics/:id(.:format)
<Topic Title>		GET /courses/:course_id/discussion_topics/:id/:extras(.:format)
<Topic Title>	✓	POST /courses/:course_id/discussion_entries(.:format)
<Topic Title>	✓	POST /api/v1/courses/:course_id/discussion_topics(.:format)
<Topic Title>	✓	POST /api/v1/courses/:course_id/discussion_topics/:topic_id/entries(.:format)
<Topic Title>	✓	POST /api/v1/courses/:course_id/discussion_topics/:topic_id/entries/:entry_id/replies(.:format)
<User Name>		GET /courses/:course_id/users/:id(.:format)
Course Announcements		GET /courses/:course_id/announcements(.:format)
Course Assignments		GET /courses/:course_id/assignments(.:format)
Course Collaborations		GET /courses/:course_id/collaborations(.:format)
Course Conferences		GET /api/v1/courses/:course_id/conferences(.:format)
Course Conferences		GET /courses/:course_id/conferences(.:format)
Course Discussions		GET /api/v1/courses/:course_id/discussion_topics(.:format)
Course Discussions		GET /courses/:course_id/discussion_topics(.:format)
Course Files		GET /courses/:course_id/files(.:format)
Course Grades		GET /courses/:course_id/grades(.:format)
Course Grades		GET /courses/:course_id/grades/:id(.:format)
Course Home		GET /courses/:id(.:format)
Course Home		GET /api/v1/courses/:id(.:format)
Course Home		GET /api/v1/accounts/:account_id/courses/:id(.:format)
Course Modules		GET /courses/:course_id/modules(.:format)
Course Outcomes		GET /courses/:course_id/outcomes(.:format)
Course Pages		GET /courses/:course_id/pages(.:format)
Course People		GET /courses/:course_id/users(.:format)
Course Quizzes		GET /courses/:course_id/quizzes(.:format)
Course Syllabus		GET /courses/:course_id/assignments/syllabus(.:format)
SpeedGrader		GET /courses/:course_id/gradebook/speed_grader(.:format)