On Data Differences Between Analytics and New Analytics

dlecocq · ‎05-07-2020

In 2019 we introduced New Analytics to replace and improve upon the course- and user-analytics offering in Canvas. We did so because we saw an opportunity to make an update to a valuable product to provide expanded and more accurate information. Since that original Analytics offering was first written, Canvas has grown and the difference between Canvas as it was and Canvas as it is now has led to disparities between what was presented in that tool and what we would expect to see.

All this to say, we know that there are discrepancies between the two. We tried very hard to understand and reconcile as many of those differences as possible, but in some cases we had to make the conscious choice to provide more accurate and relevant data. The first analytics offering was written five years ago, and in the intervening time Canvas has had a long path of changes - from the big to the small-and-nuanced, leading to subtle differences between Canvas’ internals and analytics’ understanding and presentation. This means subtle changes between the two in terms of the interpretation of grade statuses as well as activity. In the case of activity data, this also means slicing and dicing that data in new and different ways (in particular breaking it down to per-week numbers).

Our guiding star with this new work was always to do our best to accurately reflect the reality provided in Canvas.

We'll take this opportunity to try to describe our interpretation of activity information in particular in hopes of clarifying what is being presented now.

The activity information surfaced in New Analytics is derived from request logs from Canvas itself. How a user's action is presented and counted depends on the route/controller used, the status code, and the HTTP verb on that request. For example, loading the URL for an assignment will count as a Page View under the title of that assignment. Below is a table of the URLs that we count and consider in New Analytics and how they are presented (the P column is whether or not it's considered participation)

Name	P	Method and Path
<Assignment Title>		GET /courses/:course_id/assignments/:id(.:format)
<Assignment Title>	✓	POST /api/v1/courses/:course_id/assignments/:assignment_id/submissions(.:format)
<Assignment Title>	✓	POST /courses/:course_id/assignments/:assignment_id/submissions(.:format)
<Assignment Title>	✓	POST /courses/:course_id/submissions(.:format)
<Attachment Title>		GET /courses/:course_id/files/?preview=:file_id
<Attachment Title>		GET /courses/:course_id/files/:file_id/download(.:format)
<Attachment Title>		GET /courses/:course_id/files/:file_id/download.:type(.:format)
<Attachment Title>		GET /courses/:course_id/files/:file_id/preview(.:format)
<Attachment Title>		GET /courses/:course_id/files/:id(.:format)
<Attachment Title>		GET /courses/:course_id/files/:file_id/file_preview(.:format)
<Attachment Title>		POST /courses/:course_id/files/:file_id/inline_view(.:format)
<Calendar Event Title>		GET /courses/:course_id/calendar_events/:id(.:format)
<Calendar Event Title>	✓	PATCH /courses/:course_id/calendar_events/:id(.:format)
<Calendar Event Title>	✓	PUT /courses/:course_id/calendar_events/:id(.:format)
<Collaboration Title>	✓	GET /courses/:course_id/collaborations/:id(.:format)
<Conference Title>	✓	GET\|POST /courses/:course_id/conferences/:conference_id/join(.:format)
<Outcome Title>		GET /courses/:course_id/outcomes/:id(.:format)
<Page Title>		GET /courses/:course_id/pages/:id(.:format)
<Page Title>		GET /api/v1/courses/:course_id/pages/:url(.:format)
<Page Title>	✓	POST /api/v1/courses/:course_id/pages(.:format)
<Page Title>	✓	PUT /api/v1/courses/:course_id/pages/:url(.:format)
<Quiz Title>		GET /courses/:course_id/quizzes/:id(.:format)
<Quiz Title>		GET /courses/:course_id/quizzes/:quiz_id/history(.:format)
<Quiz Title>	✓	GET\|POST /courses/:course_id/quizzes/:quiz_id/take(.:format)
<Quiz Title>	✓	GET /courses/:course_id/quizzes/:quiz_id/take/questions/:question_id(.:format)
<Quiz Title>	✓	POST /api/v1/courses/:course_id/quizzes/:quiz_id/submissions(.:format)
<Tool Title>		GET /api/v1/courses/:course_id/external_tools/:external_tool_id(.:format)
<Tool Title>		GET /courses/:course_id/external_tools/:id(.:format)
<Topic Title>		GET /courses/:course_id/discussion_topics/:id(.:format)
<Topic Title>		GET /courses/:course_id/discussion_topics/:id/:extras(.:format)
<Topic Title>	✓	POST /courses/:course_id/discussion_entries(.:format)
<Topic Title>	✓	POST /api/v1/courses/:course_id/discussion_topics(.:format)
<Topic Title>	✓	POST /api/v1/courses/:course_id/discussion_topics/:topic_id/entries(.:format)
<Topic Title>	✓	POST /api/v1/courses/:course_id/discussion_topics/:topic_id/entries/:entry_id/replies(.:format)
<User Name>		GET /courses/:course_id/users/:id(.:format)
Course Announcements		GET /courses/:course_id/announcements(.:format)
Course Assignments		GET /courses/:course_id/assignments(.:format)
Course Collaborations		GET /courses/:course_id/collaborations(.:format)
Course Conferences		GET /api/v1/courses/:course_id/conferences(.:format)
Course Conferences		GET /courses/:course_id/conferences(.:format)
Course Discussions		GET /api/v1/courses/:course_id/discussion_topics(.:format)
Course Discussions		GET /courses/:course_id/discussion_topics(.:format)
Course Files		GET /courses/:course_id/files(.:format)
Course Grades		GET /courses/:course_id/grades(.:format)
Course Grades		GET /courses/:course_id/grades/:id(.:format)
Course Home		GET /courses/:id(.:format)
Course Home		GET /api/v1/courses/:id(.:format)
Course Home		GET /api/v1/accounts/:account_id/courses/:id(.:format)
Course Modules		GET /courses/:course_id/modules(.:format)
Course Outcomes		GET /courses/:course_id/outcomes(.:format)
Course Pages		GET /courses/:course_id/pages(.:format)
Course People		GET /courses/:course_id/users(.:format)
Course Quizzes		GET /courses/:course_id/quizzes(.:format)
Course Syllabus		GET /courses/:course_id/assignments/syllabus(.:format)
SpeedGrader		GET /courses/:course_id/gradebook/speed_grader(.:format)

mary_anderson · ‎05-08-2020

So when it uses GET is that when it is being clicked on in the Module page or somewhere specific, or from several possible locations.

James · ‎05-08-2020

@mary_anderson

The table is a little techy for those unfamiliar with the language. They also refer to the routes used within Canvas, which may not be familiar to people not versed in the API.

GET, POST, PUT, PATCH, and DELETE refer to HTTP request methods. They generally (not always) related to the CRUD acronym for HTTP action verbs. PATCH is similar to PUT, it varies in the way the information is supplied.

POST is Creating something new / overwriting with new values
GET is Reading something
PUT/PATCH is Updating something that already exists
Delete is DELETING something

The GET could happen through the web interface or through the API. It doesn't have to be a modules page, it could be that you linked to a file from a content page and it would recognize when students viewed the file from there. Mobile applications use the Canvas REST API to make their calls since they cannot use internal calls.

In the past, there have been some issues with Mobile apps not counting stuff accurately. Then they started logging too much and the Page Views available to the Admins became flooded with crap (that's not an acronym) from API calls and it was difficult to get to the good stuff for all the noise. Hopefully this is a movement towards getting things counted consistently.

The PUT, PATCH, and POST require the student to do something other than just view the content.

For example, let's consider this entry:

<Page Title> ✓ POST /api/v1/courses/:course_id/pages(.:format)

A POST to the /api/v1/courses/:course_id/pages is the Create page endpoint of the Pages API. In English, that means that someone created a new content page.

The one right below that

<Page Title> ✓ PUT /api/v1/courses/:course_id/pages/:url(.:format)

maps to the Update/Create a page endpoint, and happens when someone changes the content or title of a page.

The two entries before that

<Page Title> GET /courses/:course_id/pages/:id(.:format)
<Page Title> GET /api/v1/courses/:course_id/pages/:url(.:format)

Map to the Show page endpoint that can be accessed by pretty-name url) or by a numeric ID. These involve GET statements and don't count as participations. The first one is not an API call, but I'm pretty sure that it will work with an /api/v1 in front of it.

The Canvas LMS API Documentation provides more information about what the calls represent. In many cases the information is there even if there is no /api/v1 in front of the pathname.

l_lucas · ‎05-15-2020

I am not sure how this post helps those of us who have been wondering why the new analytics data is very different to the old analytics data - and for those of us who have been reporting to those above us on the old data - and will now have very conflicting data to report.

All i can take from above is that - over time the old data became wrong (or perhaps misleading) and Instructure has done its best to make the new data correct.

Perhaps I am missing something ...