Course Readiness data seems inaccurate

[robin_galloway]

The majority of our spring semester classes begin tomorrow, so I thought today would be a good day to explore Course Readiness since we just recently got Intelligent Insights. However, the number of courses being displayed doesn't match the number of courses in the term. Insights shows 1143 courses with the criteria "published with at least one assignment published" and 1168 courses that do not meet this criteria. That's a combined 2311 courses, but there are only 1026 courses in our 2025 Spring Term. I triple checked the filters and criteria settings.

I also tried clicking on one of the courses in the data table listed as "Not Ready" which took me to a page in Canvas saying "Nothing here". Indeed, I found that course (ENG-106-101 2025/SP) listed in the Recently Deleted Courses report and confirmed with our registrar that it was cancelled Nov 21, 2024. I double checked the filters again to confirm that I did not have the "Deleted" Course Status selected. I did not, only the published with and without activity and unpublished statuses were selected. That course shouldn't have been in the list.

Unfortunately, my first attempt at using Intelligent Insights as we prepare for the new semester has been disappointing. I was hoping to message instructors with courses not yet ready, but I don't trust the data I'm seeing. How should we go about troubleshooting this? Posting here? Contacting support or our CSM? 

[carlycurran]

Hey @robin_galloway! I will absolutely check into this. Can you also work with your CSM to make sure this is documented as a support case?

[robin_galloway]

I sure will. Thank you for looking into it for us! We're excited to have Intelligent Insights and very optimistic about what we hope to do with it. 

[carlycurran]

Of course! I have the team looking at this now. We really want you to be confident using Course Readiness for your semester start. 👏

[morris_gart]

At our school we cross-list courses so teacher sections are contained in one course with multiple sections.  This creates orphan courses in our term that have neither students nor a teacher.  These get counted in the course readiness.  Not sure if that is the same for you.  I use Ask Your Data more than the course readiness due to this since there I can have the sql sort out courses with no enrollments.

[Jennifer_Mayes]

This is the same for our school too. I'd love to know more specific details about having the sql sort out courses with no enrollments.

[morris_gart]

In Ask Your Data, part of my prompt always includes "only courses with current active enrollments".  This will weed out those orphan courses from any data you are request. e.g. "give me a list of courses from the XXXX term that have less than 10 published assignments. List only courses with current active student enrollments"

[robin_galloway]

I appreciate the replies and tips for using "ask your data". We also have some cross-listed sections leaving behind the "orphan" sections that Morris described. However, Intelligent Insights is reporting 2311 courses in the term while the term details in Canvas show there are only 1026 courses in the 2025 Spring Term. 

Term Details

[robin_galloway]

I was playing with "Ask your data" tonight, and it tells me there are 4184 courses in the 2025 Spring Term, while Intelligent Insights Course Readiness reports 2311, and Term Details says 1026. So, which is it? Which data should I trust? 

[morris_gart]

I am going to assume that you only have one term with that particular name.  When I ask the same question I get identical information (1940 courses in the term, 1940 courses in ask your data) for my HS term.  I aslo get the same number of courses in the course readiness.  Term detail is likely accurate and the other two is a filtering issue where it is collecting more courses that just those in that particular term.  In Ask Your Data pose the question- "Give me a list of course names in the Spring 2025 Term (or whatever is your term name) - include the term_id and created_at"  See if that list has course that are not in your current term.

[robin_galloway]

Thank you for the suggestion, Morris! I did confirm that we only have one term with that name. Then I tried what you suggested, asking "Give me a list of course names in the Spring 2025 Term. Include the term_id and created_at". It returned 4186 courses, listing the same course name in duplicate 3 or 4 times. I downloaded the CSV and deduped that "CourseName" column of the CSV file. There were 1089 unique courses left– much closer to the 1029 courses listed in Term Details! Then I saw that the answer in Ask Your Data actually told me there were 4186 records in total representing 1088 unique courses. 

So, I refined my ask with "Omit deleted courses" and it returned "The updated data now includes a list of 1,029 unique course names for the Spring 2025 term". That's exactly the number of courses shown in "Term Details". 

So, thinking I just need to specifically tell it not to include deleted courses, I revised my previous query about the number of courses in a term. Instead of just asking "What is the total number of courses in the 2025 Spring Term?" I now asked "What is the total number of courses in the 2025 Spring Term? Do not include deleted courses."

Ugh– It responded "The query results indicate that there are no courses available for the 2025 Spring Term that are not marked as deleted. This means the total number of active courses for this term is zero." 

I restated the question as "How many courses are in 2025 Spring Term? Omit deleted courses." and it correctly reported 1029. Yes! Finally! 

[robin_galloway]

Thanks again for helping me identify that deleted courses were being returned in Ask Your Data. Now I know to refine my questions to omit deleted courses, and the result matches the number of courses shown in Term Details. I've begun using the phrase "only courses with current active enrollments" as was suggested. 

As for Course Readiness, a L2 support rep is looking into our instance to see why deleted courses are being included when the "Deleted" course status is not selected in Filters. (Case #11625801)

[robin_galloway]

While I'm still seeing huge discrepancies in the number of courses reported for a term between Term Details, Course Readiness, and Ask Your Data, I do want to share some success I'm having today using Ask Your Data. This is super useful and exciting. 

[carlycurran]

Hi @robin_galloway! Thanks for all the details on this. Our team found that your account did have an issue. We've fixed the bug associated with that, so:

• You should now see the correct number of courses in course readiness (though you might see orphaned courses for now)

Thanks for your patience on my response. And this is a great place to bring this kind of issue up. Really appreciate it and I hope you can use Course Readiness effectively now. 

As for orphaned courses (courses that have been cross-listed), I'd love to get feedback about how to best filter these out. 
Does it make sense to have a filter button to "remove all cross-listed courses" or "show only courses with enrollments"?
I'd love to hear from everyone in this thread. Does that nomenclature make sense? Do see any issues associated with filters like those? Thanks!  CC @morris_gart @Jennifer_Mayes 

[morris_gart]

I would like a button to exclude courses with no enrollments in the filter section.  This would remove all orphan courses in my instance.

[carlycurran]

Student enrollments only, right? I'm assuming a teacher could be enrolled in an orphan course? 

[morris_gart]

If you cross-list a course all enrollments are removed.  Teachers and Students.

[robin_galloway]

Yes, agree with Morris, and thanks for the question, Carly. I too would appreciate an option to exclude courses with no enrollments in the filter section of Course Readiness.

[turnermz]

We've also seen discrepencies in course numbers returned in these types of reports (albeit using Ask Your Data not Course Readiness) with what can be cross referenced in the UI. Also where term or sub-account has been specified in the query.

I'm intriged by the hoops @robin_galloway had to jump through to perfect the prompt to get the response to match the data which was easily confirmed against data available elsewhere. We've experienced similar issues.