CD1 Requests vs CD2 web_logs: completeness and correctness

[craig_thompson]

The CD1 Requests table included a significant disclaimer: "The data in the requests table is a 'best effort' attempt, and is not guaranteed to be complete or wholly accurate. This data is meant to be used for rollups and analysis in the aggregate, _not_ in isolation for auditing or other high-stakes analysis involving examining single users or small samples. As this data is generated from the Canvas request log files, not a transactional database, there are many places along the way data can be lost and/or duplicated (though uncommon)."

However the CD2 web_logs table does not have a comparable warning. This blog post also suggests significant improvements have been made to the data quality of the web logs.

So, can users now be confident that the CD2 web log data is complete and correct, and suitable for "auditing or other high-stakes analysis involving examining single users or small samples"?

[sgergely]

Hey @craig_thompson!

Thanks for your question! While we’ve made significant improvements to web logs data quality, it’s important to note that this data still isn’t at the level of audit log quality. The improvements mentioned in the blog post aimed to enhance the web log data in CD2 compared to CD1. We wanted to have at least the same level of quality or even better quality of data in CD2.

The primary purpose of the CD2 interface is to provide analytics data, which is why our dataset documentation still includes this disclaimer:

“Disclaimer: The data available is a ‘best effort’ attempt and is not guaranteed to be complete or wholly accurate.”

[craig_thompson]

Thanks for pointing out the disclaimer in the introduction to the Datasets, I’d missed that.

Could you please provide some additional guidance regarding suitable analytics uses for CD2? I’m having a hard time reconciling CD2 being intended to provide “analytics data” with the disclaimer.

Many of the questions our analytics team receives require precision responses. We’re frequently asked to provide yes/no responses to “did they or didn’t they?” questions about student activity. As one example, an instructor may need to know which students have (not) viewed a resource. This level of precision is already supported by New Analytics, and is something we expected to be able to replicate in custom reports/dashboards/tools build on top of CD2.

[sgergely]

Our goal with web logs is to offer highly granular data, which may appear similar to audit log functionality. However, our data pipeline is currently optimized for analytics rather than precise audit tracking. By ‘analytics,’ I mean the process of examining aggregated data to identify patterns, trends, and insights that support broader decision-making. Our focus is on empowering institutions to gain insights into areas like student engagement, course effectiveness, and overarching trends that can inform strategic decisions.

That said, we are continually working to enhance our system, and I’ve already added web logs as a potential audit-level data source to our roadmap. I’d be interested in learning more about your specific yes/no questions to see how we might further optimize our platform to support these needs.