New dataset in Canvas Data 2: Catalog data is coming

[Edina_Tipter]

Great news for Catalog users (and future Catalog users) 😍: I am happy to announce that as per our commitments, Catalog data is available, labeled as BETA, via the CD2 API and CLI. Isn’t it great? 🤗

This new dataset enables users to write queries for Catalog data and join them with existing Canvas tables for broader comparisons beyond Canvas LMS. Additionally, it expands the spectrum of available options and offers insights into Catalog utilization, encompassing details such as items added to carts, purchase records, course completion rates among students, and the issuance of certificates.

 

What datasets will be available?

As part of the release, 28 tables* will become available to customers. Users will be able to retrieve them by either using the CD2 API or CLI. In order to differentiate these tables, which can have the same name as some of the already exposed Canvas tables (e.g. users), we have introduced a new namespace, ‘catalog’. Thus a sample table list or schema query will look as follows:

API example:

Table list: GET:: {{BASE_URL}}/dap/query/catalog/table

Table schema: GET: {{BASE_URL}}/dap/query/catalog/table/{{TABLE_NAME}}/schema

CLI example:

dap list --namespace catalog

dap schema --namespace catalog --table orders

With this, we have released a new version of the CLI as well, 0.3.14, that supports the following operations:

• Retrieving datasets (downloading snapshot/incremental query results) from the catalog namespace.
• Loading the downloaded datasets into a Postgres DB under the catalog namespace.
• Loading the downloaded datasets into a MySQL DB by adding the catalog prefix to each (i.e. catalog__order), in order not to overlap with Canvas or other tables in the future. (The namespace as a prefix is only added for catalog tables).

Example: dap --initdb --connection-string mysql://scott:password@localhost/testd --namespace catalog --table orders

For more release details, please visit the API/CLI change log.

 

New schema documentation tool

Additionally, we are trying out a new schema documentation tool which gives the engineering team greater flexibility to make adjustments if needed. For example, with this tool, foreign keys are clickable and take you directly to the referred table. We’d love to get your feedback about any changes that would make it more usable for you. You can use the comments section under the post to provide feedback. 

As the product grows and matures, you can expect new datasets coming, such as mobile logs as well as improvements to our API and CLI as we learn from customer feedback.

*Disclaimer: Kindly note that when joining Catalog and Canvas tables using the foreign key relations, there is a possibility of encountering data disparities, more likely in historic data, as a result of potential synchronization gaps between Canvas and Catalog. The symptom is not new, it has been present in CD1 as well yet there is no quick and easy way to fix it.