Catalog Analytics Improvements Teaser

[KataKeri]

In this and the following quarters (Q3 and Q4 in 2022 and continuing in 2023) the Catalog team will be working on improving Catalog Analytics. Let me break it down to you in detail what this means.

But first, why?

We’d like to provide a solution for all the data needs our users have, since most of it requires a lot of manual work from admins currently. Based on our research we hope we can provide you with a solution which will help you:

• see which courses perform well
• gather and strain user defined field data more easily
• do troubleshooting (enrollments and payments)
• have a detailed overlook on your revenue and create reports
• check on overall completions and certificates

and make adjustments/decisions on your courses based on these data.

We have broken down this improvement into 5 phases, and we will roll them out to Catalog Beta in phases also. And in each phase we’d like to hear your feedback, so please share!

Phase 1 - We’re doing some database changes. This won’t affect the UI/UX yet. Our goal is to adjust our database in a way that it will be more reliable and less prone to bugs and defects. It will also be able to serve more comprehensive reports.

Phase 2 - Adjusting the tabs - In this phase we will say goodbye to the Reports tab, and only keep the Analytics tab. Under the Analytics tab you will see three sub-categories (tabs): Enrollments, Revenue and Registrations. There will be a 4th tab also, called ‘All data’ but this will be implemented in Phase 5. More on that later.

Phase 3 - Filters - In this phase we will add a comprehensive set of filters to the tabs, so you can strain your data in a more efficient and meaningful way.

Phase 4 - Download data - In this phase we’ll introduce an option to download your data in a CSV format. Filters will also apply to the CSV so you will be able to download more polished reports.

Phase 5 - All data - As I mentioned in Phase 2, this will be the 4th tab added under Analytics, so you can have an overview of all data in one place.This table will provide every single data point for the root catalog. Filters will be applicable to this table, too, so you will be able to narrow down the data based on your preferences.You will also be able to download this table.

Additional information: We are not changing the current access level for Analytics, sub-catalog admins can see data which belongs to the sub-catalogs they are admins for.

Similar to Shopping Cart and Bulk Purchase / Enrollment, we are going to have multiple Beta releases before releasing the new version to Production.

With each beta (and prod) release I will keep this post updated, so stay tuned!

UPDATE 1 (2022. November 22nd) - We have released Analytics Improvements Phases 1-2 to beta today. You can find it under 'Analytics (New)' next to the old 'Analytics' tab. Let us know what you think.

UPDATE 2 (2023. January 2nd) - We have released Analytics Phase 3 to beta on December 23rd. In this phase we added a comprehensive set of filters to the tabs, so you can strain your data in a more efficient and meaningful way.

UPDATE 3 (2023. March 23rd) - We have released Analytics Phase 4 to beta on March 23rd. In this phase we added the ability to export and download Catalog Analytics data in a CSV format. 

UPDATE 4 (2023. April 4th) - Production release date got delayed, and there are some changes in concept too.
See more here: https://community.canvaslms.com/t5/The-Product-Blog/Update-on-Catalog-Analytics-Improvements/ba-p/56...

UPDATE 5 (2023. April 25th) - We have released Analytics Phase 5 to beta on April 24th. This was the last phase, so all of the features of Improved Analytics is now available to test out. See more here: https://community.canvaslms.com/t5/Canvas-Catalog-Releases/Catalog-Release-Notes-2023-05-02/ta-p/567...

UPDATE (2023. May 2nd) - We have released all phases to production today!

[jkayew]

Thanks, Kata, I really appreciate the update, inclusive of some sneak-preview screenshots. This is very helpful. I'm especially excited about better and more detailed filter abilities. This should be huge and go a long way to cut back on the manual API reports needed to better view the data, discover and analyze trends, and extrapolate and manipulate more meaningful reports for management and future planning! 

That being said, there are two large pieces I am hoping are a part of this project to make those filter and reporting flexibilities truly meaningful, but I did not see them explicitly mentioned in this update.

1) Filter of Course Listing Field Data: From your screenshot, the only added course filter I see is if it is open for enrollment or not, and a mention that we would be able to "gather and strain user-defined field data more easily." These are great, but as there are so many setting options per course listing (and some at the subcatalog level), adding the listing fields are key to include in filters to be able to see course effectiveness, trends, and policy adherence at-a-glance, especially at the root administrator access level.

- Currently, we have no way of efficiently and effectively performing oversight of the platform since we have many administrators per department and unit across a large campus. Being able to filter for all fields would help not just with reports, but with monitoring proper administrator use of the platform, or if a course needs attention and has been neglected by its administrator. All fields being: Credit/CEU/Points, Bulk feature use, Course Costs, Alt Text use for ADS requirements, Course Dates (& expirations), Days to Complete, Open/Close Enrollment, Visibility (link/storefront/hidden), Catalog Listing & Canvas Course IDs, Tags, List Order, Student Cap, Waitlist, which courses are offering a certificate, and even if/where "customization code" is detected or in use, etc.

- What would be additionally helpful is the ability for an admin to search the platform for keyword use. For example, we only use Catalog for non-credit courses, so it would be helpful to search course listings for words that may indicate a violation of policy or cause confusion like "credit," "CEUs," "PDUs," etc.

- Basically, it would be invaluable for any admin to be able to select from all data fields in Catalog or the related Canvas course in order to truly "analyze" the courses hosted by Catalog, without the manual API effort only performable by our select IT admins. This filter ability could be its own "Course Listings Data" tab in Analytics, but it is greatly needed. 

2) Activity Log: There's no current way to see who did what and when. Being able to access a history of activity will help troubleshoot issues, have better-informed reports and see trends, and again, provide oversight of the many campus admins who have access.

- We currently have 135 administrators, and we are wanting to continue to grow the platform use, so naturally, admins will also increase. (Catalog needs to have ways to designate different access roles and the ability for main/root admins to restrict specific access or features per admin, subcatalog, and course. Subcatalog settings and even the bulk feature are good examples where root admins may need to impose restrictions. In the case of bulk purchases, agreements with organizations may be required before they would be allowed to make that purchase. Being able to restrict an admin from turning on that feature, or allow them to share a private bulk purchase link to an approved organization to a course while the public listing has the bulk feature turned off, would still allow for individual enrollments for the same course.) In the meantime, at least being able to view the full history of platform activity would help monitor university policy and best practice adherence, and quickly identify who/when/what/where to address the problem to ask the question, "why."

- Right now, we can only see "last activity" in Canvas, and that just means "accessed by a student." (Last activity could also be a masquerading admin, but there's no way to know the difference, or even what that activity was. With so many admins, and to ensure the student doesn't think their account has been hacked, the inability to know specifics is a safety and security risk.) To that end, it would also be helpful to specifically know student drop dates, as these affect access and refund policies, and course effectiveness assessments.

- Besides monitoring, we would be able to answer our most reasonable and frequently asked questions by upper management when they're trying to assess the use and viability of the platform: "How many courses were offered in a specific month/quarter/year?" or "How many times was a specific course offered?" We are not able to answer this basic question since any report pulled is only a snapshot of the point in time of the report date, and listings can be activated one moment, and deactivated the next (we consider an "active" course to be accessible via link or on the storefront, and enrollable).

In hopes that my questions, concerns, and requests are understood, I wanted to be sure to provide some context. To shorten/summarize my two main questions specifically regarding the Analytics improvement project, could you let us know 1) if the new filters will include the course listing fields, and 2) if there will be some access/activity details (ex: course active/inactive, student drop dates, etc.)? As a root admin, effective and detailed analytics tools are imperative in order to report to and properly advise upper management. Please reach out if you have any questions or may find it more valuable to discuss. I very much appreciate Instructure investing in truly meaningful and robust improvements, as well as actively requesting and considering feedback from those of us in the community. We are all quite excited to see Catalog evolve into this new and more flexible and effective tool.

[jkayew]

It was mentioned that the Reports tab would be going away, so I'm hoping/assuming that means consolidating that information into the Analytics Revenue tab. Does that mean the bulk and shopping cart feature purchases will be included under Revenue and administrators will be able to filter by type of purchase, how many seats were purchased in bulk, sent/accepted/revoked invitation status, as well as promo-codes used? When will refunds be visible in Catalog so that all enrollment/revenue data is accurate?

It would also be great if Analytics could pull data by administrator (at least at the root admin level), as some administrators have access to multiple subcatalogs and it's helpful to see at a glance all of the courses and enrollment numbers that they manage or assist with.

[jsowalsk]

In addition to @jkayew's points for the filter feature will this include dates? and also this is extremely important since this is currently a manual process: 

When will refunds be visible in Catalog so that all enrollment/revenue data is accurate?

[KataKeri]

Hi @jkayew and @jsowalsk ,

Thank you both so much for your questions and insights! Find my answers below.

1. Filter of Course Listing Field Data
   - Thank you for the feedback and the detailed use cases, adding course listing level filters will be considered for future iterations, but in the first iteration this won't be included - you will be able to create a course level import though, even with selecting multiple listings.
   - This can be set on a root catalog level, sub-catalog admins don’t have access to add these measurements. How about we’re give an other solution?  We are thinking about that the root catalog admin gives no measurement (selects blank from the dropdown), so the sub-catalog admins can’t add value on the listings. We think this is a great workaround, and I can shoot up a ticket to add this option.
   - All data is rather an overview than for straining information. If it does not carry value for you, you just need to select a listing as a filter.
   
   
2. Activity log
   - While you mentioned completely valid problems, these are separate features (and initiatives), and they are not specifically related to Analytics. Logging activities and improvements on restrictions are out of scope at the moment, and can be covered later, since others requested it too.
   - Regarding student drop dates: we need to add this as a live event to work, then it can be added as a filter. We will take a look around how we can make this happen.
   - Regarding on reports on "How many courses were offered in a specific month/quarter/year?" or "How many times was a specific course offered?" - We need to look into this, and I will raise it to the team, but most probably we won't be able to show retrospective data.
   
   Just an update so maybe it will help you with managing the 135 admins - With our lates beta release admins now are listed in alphabetical order.
   
   Summary for the two main questions:
   1) Will the new filters will include the course listing fields? - No, as of now.
   2) Will there will be some access/activity details (ex: course active/inactive, student drop dates, etc.)? - Active/inactive will be displayed, student drop rates/ teacher drop events will be displayed, but not dates of the drops, but if there will be an improvement of that we will keep you posted.
   
3. Reports tab and refunds
   Exactly, the Reports tab’s content will be a part of the Analytics Revenue tab.
   Bulk purchase numbers will be available via the API.
   Shopping cart purchases will be listed individually, since it’s a multiple course purchase by one purchaser.
   Sent/accepted/revoked invitation numbers will be available at the Purchases and Enrollments tab.
   
   Refunds: That is a bigger effort from our side since currently we support 13 payment providers, and we would need to implement refunds for every service. We know this is important and we will consider this for our roadmap for 2023.
   
   Seats purchased in bulk - This is available at Purchases and Enrollments tab, and we will consider how we can display this on the Revenue tab, so it will makes sense. 
   
   
4. Data by administrator:
   You will be able to multiselect catalogs and subcatalogs (admins only will be able to select those listings/subcatalogs where they are admins), so this way data by administrator can be pulled.
   

Regarding @jsowalsk question about dates:
Yes, multiple date filters will be visible on all of the different tabs - this table will be scrollable horizontally and I did not add all of the images. Here you can see (enrollment date, purchase date, registration date will be visible) :





Hope I did not left out any questions, but please let me know if I did and I will get back to you.
Thank you again for the replies!

[jsowalsk]

@KataKeri When would we be able to see these updates in beta so we can get a full picture about what you are mentioning? Also, when will subcatalog admins be placed in alphabetical order and will this be in beta? Will the same be done for listings for having subcatalogs be listed in alphabetical order and not by when they were last added?

[JakeStuckey]

Will user defined fields be shown in the enrollment report or will the registration report show what course the person signed up for? Currently, there's no way to see the user defined fields by course. For instance, we need to know where people who are taking the course are from which is one of our user defined fields, but the registration report doesn't show the course and the enrollment report doesn't show user defined fields. It doesn't make much sense because you can change the user defined fields based on catalog listings, but there is no way to to pull the data from those listings.

[KataKeri]

@JakeStuckey Thanks for the question - Yes, user defined fields will be a part of the registration reports, each field will have a separate column. You will be able to filter the registration report based on a course. 

@jsowalsk Alphabetical order of subcatalog admins will be released to beta next Monday.

As mentioned previously, we will have multiple beta releases for the different releases of Analytics, and I will keep this post updated about those. 

[jsowalsk]

@KataKeri How will the subcatalog admins be put in order by their first name/last name or by the subcatalog they are associated with? Will there be release notes? Will the second part of my feature request be fulfilled? https://community.canvaslms.com/t5/Idea-Conversations/Catalog-Put-subcatalog-admins-and-subcatalogs-...

[jsowalsk]

@KataKeri I see the change for subcatalog admins was updated in our beta instance - thank you for doing this. Will there be release notes for this? Also, in the future, will there be a way to filter by subcatalog? Since I don't understand what the below arrows do besides the first column for alphabetical order.

[jsowalsk]

@KataKeri I noticed the font was different in Beta than Production. Was this meant to happen? Will there be release notes on this?

[jsowalsk]

@KataKeri I havent seen anything yet regarding the alphabetical order and/or font style changes in the release notes, do you know when this will happen? https://community.canvaslms.com/t5/Canvas-Catalog-Releases/tkb-p/catalog-release

[KataKeri]

hi @jsowalsk , sorry for the late response, I was on Holiday.

Yes, the subcatalog admins will be put in order by their names. The second part of your feature request will be also fulfilled, the subcatalog drop-down menu will be displayed in alphabetical order too. Yes, there will be release notes about this, as always.

Yes, in the future, will there be a way to filter by subcatalog, we're on to fix the arrows, thanks for pointing it out.

Regarding the font changes - yes, it was intentional, and yes, it will be mentioned in the next release note too.

[jsowalsk]

@KataKeri ---

Yes, the subcatalog admins will be put in order by their names. The second part of your feature request will be also fulfilled, the subcatalog drop-down menu will be displayed in alphabetical order too. Yes, there will be release notes about this, as always. Thank you. When will these release notes be out? Also, to confirm this is at the listing level right? https://community.canvaslms.com/t5/Idea-Conversations/Catalog-Put-subcatalog-admins-and-subcatalogs-...

Yes, in the future, will there be a way to filter by subcatalog, we're on to fix the arrows, thanks for pointing it out. Thank you!

Regarding the font changes - yes, it was intentional, and yes, it will be mentioned in the next release note too. When will this be?

[KataKeri]

@jsowalsk Yes, that's the listing level. Release notes will be out next Monday. 

[jsowalsk]

@KataKeri Ok, what will be included in these notes? Thank you so much for your effort on this.

[KataKeri]

@jsowalsk The alphabetical order of the subcatalog admins and the dropdown menu, and also the update on fonts.

[jsowalsk]

@KataKeri Great, thanks. This upcoming Monday right?

[KataKeri]

@jsowalsk Yes!

[jsowalsk]

@KataKeri Thank you!

[jsowalsk]

@KataKeri Here are the release notes: https://community.canvaslms.com/t5/Canvas-Catalog-Releases/Catalog-Release-Notes-2022-09-05/ta-p/535...but I don't see anything reflecting the listing subcatalogs. When will this be done?

[jsowalsk]

@KataKeri Where can we tell the font type we have in Canvas versus Catalog? When will the subcatalogs be listed in alphabetical order like they will be in the subcatalog admin portal but for listings?

[KataKeri]

@jsowalsk hi, the font type is visibly different in Canvas then it Catalog, but I'll share two screenshots so you see the before/after, here you go:
beforeafterDo you mean alphabetical order here?

[jsowalsk]

HI @KataKeri,

I understand the fonts are different. What I am trying to identify is where I can find the Lato font identified in the Canvas LMS to have it match what it is in Catalog? Since I don't see it here: Font Type

The default font is updated to Lato. This update improves readability and further aligns Catalog with Canvas LMS.

Secondly, I see that the release notes got pushed out: https://community.canvaslms.com/t5/Canvas-Catalog-Releases/Catalog-Release-Notes-2022-09-05/ta-p/535...

Lastly, where I am asking for alphabetical order is on the listing under catalog:

This was apart of this feature request: https://community.canvaslms.com/t5/Idea-Conversations/Catalog-Put-subcatalog-admins-and-subcatalogs-...

[KataKeri]

Hi @jsowalsk Unfortunately I have no idea, since I don't work on Canvas. Best would be to ask your Canvas CSM where you can identify the font type.

Happy to see the changes are available to you.

Thanks for the further details on the last one, I misunderstood and thought you've meant the dropdown for Catalog listings in the account admin page. 

I'll take a note of this request and update you if we pick it up. 

[jsowalsk]

@KataKeri When do you think a decision will be made?

[KataKeri]

@jsowalsk we do our best to address this issue this year, we will keep you posted in the upcoming release notes. 

[jsowalsk]

 

 

@KataKeri Thank you. I noticed that there is no alphabetical order in the following areas:

Promotion:

Listing:

 Catalog:

Reports:

 It is reflected in Analytics:

Subcatalog admins:

Can these other areas be updated so the whole platform reflects alphabetical order for catalogs/sub catalogs?

 Jessica

[KataKeri]

@jsowalsk thank you, I'll take a note of these and raise it to fix these also. 

[jsowalsk]

@KataKeri Thank you. That way the platform is all reflected in the same way.  Does this need a feature request?

[KataKeri]

@jsowalsk No need for to submit a feature request, since I'll create a ticket about this. 

[jsowalsk]

@KataKeri Perfect, thank you!! Please keep us posted on the progress.

[jsowalsk]

@KataKeri I just noticed that for the analytics page that some of the subcatalogs are in order but then at the way bottom they are not -- do you know why this could be happening?

[JenniferJWhite]

@jsowalsk -- These are all grandchildren subcatalogs. The sorting mechanism doesn't seem to know how to handle alphabetizing the grandchildren?

[jsowalsk]

Per @JenniferJWhite comment, @KataKeri could this be looked into as well? Does this need a feature request?

Currently in our production instance, the listing isn't in any order for catalogs:

Same goes for the Catalogs page:

Same goes for the promotions page:

The reports page should be in alphabetical order as well:

The analytics page is in alphabetical order for the listings perfectly but for the subcatalogs it isnt - see Jennifer's note:

Subcatalog admins are listed perfectly with their subaccounts:

In Beta Catalog:

The outcome is identical for listings:

 Same outcome for catalogs:

I cannot get the promotions in beta since I am getting the following error - does the shopping cart need to be put back into Beta?

Reports are not in alphabetical order:

Analytics are in order except a couple at the bottom for subcatalogs and the listings are in alphabetical order:

Subaccount admins subcatalog is accurate:

Can the shopping cart be looked into as well to be added back to beta and NOT production?

[jsowalsk]

@KataKeri Ignore my comment about the shopping cart - it is now fixed.

[KataKeri]

hi @jsowalsk and @JenniferJWhite,

thank you both, I created a ticket about the issue of the alphabetical order of grandchildren subcatalogs and the team will look into it.

@jsowalsk thanks for collecting these, I'll add these to the task where we'll fix the alphabetical orders. 

[KataKeri]

@jsowalsk @JenniferJWhite my question is what is the need on the alphabetical order in case of grandchildren subcatalogs. I mean, in the picture they are in alphabetical order, but in a different section (down), so in this case you can differentiate them from subcatalogs. but if we place them in line, you won't be able to tell if that's a grandchildren subcatalog or not. what would be most beneficial for you? 

[jsowalsk]

Hi @KataKeri,

Thank you for looking into this for us to have the full Catalog be in alphabetical order like the listings currently are.

At the listing level, it would be nice to have the catalogs in alphabetical order including grandchildren in the mix so all catalogs are in order and easy to find.

Currently, it looks like this:

At the Catalog level, having all catalogs in alphabetical order as seen below is a nice to have this will include having grandchildren in alphabetical order/in the mix:

For promotions, it is the same request as catalogs:

For reports, it would be nice to have each of the catalogs including grandchildren listed in alphabetical order, currently, I think they are being added by when they were first created or they are in random order:

For Analytics, we would like the catalogs including grandchildren listed in alphabetical order, also known as thrown into the above section/into the mix:

For example, have TerrapinSTRONG listed below Terrapin Teachers.

What is exactly listed in the subcatalog admins page is what we are looking for with the grandchildren mixed in.

[KataKeri]

@jsowalsk The separate section for grandchildren subcatalogs are by design. If you'd like to mix them, that's a feature request - would you please submit it? No other schools requested this yet, and we'd like to measure the needs for it. For one school most likely we won't re-organize it.

Question: Wouldn't it cause a problem for you that you won't be able to differentiate grandchildren subcatalogs in this dropdown if we mix it?

Thank you for pointing out the needs on the alphabetical orders throughout the product. I collected these in a ticket and will raise it to the team. Target that to do this is still end of this year, I'll keep you updated. 

[jsowalsk]

Hi @KataKeri,

If you are saying that no other schools are currently requesting it then why should we go through the effort in creating a feature request? @JenniferJWhite can ask the Catalog Peers group at our next meeting in October if they are interested in having the grandchildren included in the list alphabetically and then if so we can create a feature request since this means that more people are interested in this feature. It wouldn't be a problem, since we would just compare it to the Catalog list that we have in Canvas or Catalog. 

We appreciate your effort in working on the alphabetical order throughout the platform. It has been a long time coming.

Jessica

[jsowalsk]

@KataKeri I just noticed the subcatalog admins page is the way we want all of the other pages to look for listing catalogs and subcatalogs. 

[jsowalsk]

HI @KataKeri,

A couple of questions regarding this, since we now see this feature in Beta Catalog.

When do you expect the All data to occur?

Will there be release notes on any of this coming soon?

I do not see a way to download any of this data to a CSV -- will this come soon?

I assume this data will update after every refresh of Catalog right? When will Reports go away?

[KataKeri]

Hi @jsowalsk 

When do you expect the All data to occur?
All data is the last phase (5th) of Analytics Improvements, we're still working on Phase 4. We are aiming for finishing Phase 5 until the end of this quarter (end of March).

Will there be release notes on any of this coming soon?

For the beta releases we won't do release notes, we update this article if something goes out to beta. We will do a release note when the whole Analytics Improvements is available on beta and then prod a week later.

I do not see a way to download any of this data to a CSV -- will this come soon?
This is part of Phase 4 which we are still currently working on, but will be released to beta in a couple of weeks. I'll update this post once it's out to beta.

I assume this data will update after every refresh of Catalog right? When will Reports go away?

Yes, the data will update after every beta refresh. We will deprecate the Reports tab after Improved Analytics will be available in production - maybe we will leave it there for a few weeks to gather feedback for the new analytics and ensure the old version at that time. 

[jsowalsk]

@KataKeri So to confirm, we should expect all of this in March in production? 

[KataKeri]

@jsowalsk End of March, yes, Q1 ends with March 31st - that is our target. 

[jsowalsk]

@KataKeri Ok, thank you.

[c-mc]

In the new Analytics, one change that would be really helpful would be to calculate dropped/refunded courses into the total revenue — otherwise, the Revenue tab would project misleading data and we wouldn't be able to use it reliably (or at all, we would rely exclusively on our payment platform reporting to calculate revenue). 

[c-mc]

In the new Analytics, one change that would be really helpful would be to be able to pull a report that has the enrollment data for a user AND the information that they provided when they registered their account. We collect important data in the registration (the only place where you can customize fields), such as if a learner needs accommodations, that is very difficult to cross reference when we pull registration reports for specific courses/listings. 

Right now, my understanding is that you can pull this information by listing but it only includes the data for users who have created new accounts when enrolling in that listing (not enrollees who have existing accounts). If this is already possible, please let me know. Thanks!

[KataKeri]

hi @c-mc,

Thank you for your insights, I'll add this to out use cases, and see how we can solve these to make it easier for you.