How far behind does data lag?

[MikeBrinkman]

Yesterday I was looking at the number of published courses for our Summer 1 term. When I looked at yesterday's course_dim workflow_status, it showed X number of courses as Active -- assuming that is the indicator that a course is published. However, I manually counted Y courses that are published on the website, which is greater than X. I did this before Canvas Data showed the last update was completed yesterday afternoon.

I downloaded the updated data this morning, and the data shows X courses being published as opposed to Y courses. Is there a significant lag in Canvas data, or is course_dim workflow_status not a good indicator of how many courses are published in the system.

[ben_hudson]

Hey Mike,

Per the data portal in our Canvas environment:

For all file downloads, please note that the dates specified do not reflect the actual dates of the data, but instead when the data was finished exporting. The most recent data in a given export is generally 24-36 hours older than the date given. An exception is that the request table is 48 hours behind due to the large amount of data that needs to be processed. All dates are in UTC.

The documentation states the following for workflow_state in course_dim, it should be a good indicator:

Workflow status indicating the current state of the course, valid values are: completed (course has been hard concluded), created (course has been created, but not published), deleted (course has been deleted), available (course is published, and not hard concluded), claimed (course has been undeleted, and is not published).

Last note I have is I either read or have been told that the files provided are "best attempt" to provide the data and should be used for large aggregation, not targetting certain things so this could also be a contributing factor.

Thanks,

Ben

[Jeff_F]

@ben_hudson  - I am thinking the note about 'best effort' applies to the Requests table only.  

https://portal.inshosteddata.com/docs#requests 

[MikeBrinkman]

Thanks, Ben! Now I feel dumb for asking, since it was right there. I have a friend that says "Obvious means overlooked". The additional information was valuable though. It was up to date with Monday when I ran it this morning.

[ben_hudson]

No problem, glad I could help!