Admin analytics outliers (and some discrepancies)

[DavidSchlater]

I am enjoying using Admin Analytics to find some of the oddities in our Spring 2023 term. The course with the most quizzes 100+, or discussions 100+, etc. However I've found that the stats are off in some cases, so I have these tickets in:

Syllabus word count incorrect - 09676185

Number of course discussions incorrect - 09673986

I'd encourage everyone to use this method to find your own outliers of any kind, then check them to see if the data is correct - if not  make a screencast and put in a ticket. I really like this tool and how it is working - my goal is to make it rock solid so we can all rely on what it tells us.

Thanks!

[carlycurran]

@DavidSchlater! Thank you so much for digging into the data and going through the support process. 

Also, I really appreciate the sentiment and your attitude here. Yes! We want this to be rock solid. 

We'll take a look at these tickets and get back to you.

[ldavenport4015]

I am digging into our Admin Analytics and am excited about the potential for some high-level reporting out of here!! Super kudos for giving us a platform to use without having to dust off our analytic and visualization skills!  I am also finding discrepancies between analytics reporting and our production environment. Are Support tickets the best way to bring attention to those? For example, I find that for our current semester there are 46 courses listed as "Deleted" course status, but they are active with students. Futher, the download of the data has duplicates of courses listed in the 46. 

[carlycurran · Accepted solution]

Thank you, @ldavenport4015 ! Yes, please submit a support ticket. I am wondering if those courses were reset. It seems they might still be in a deleted workflow state? 

[ldavenport4015]

Carly, we did submit a ticket and it was courses that had been reset and are showing up as deleted (as expected?). If it is helpful here is the Case #09704958. Would be great distinguish between deleted and re-set courses.

[Mikee]

This is what the reset does though - it deletes the course, and creates a new instance. I would think this is expected reporting. (check the course id in the url after a reset - it's a new shell each time)

[DavidSchlater]

Using the Analytics tools today to find current courses with 100+ announmnent and gently suggest to  instructors they review and delete announcements from past semesters they have been coping forward (that student can see) that are not relevant. There is a use-case for bringing forward old announcements to potentially re-use, but without easy bulk "hide from students" interface tools, keeping them does not seem practical.

[paul_fynn]

Hi David,

We changed our recommended rollover method for colleagues such that they don't do a whole course copy (in fact the select all is hidden by .js in the browser), and we prevent Institutional SIS generated assignments from being rolled over in a similar fashion.

Restricting rollover instructions to just the modules area means that only that which appears in or is connected to modules area gets copied. This also means that unused files are automatically decluttered as well. We also have certain module blocks that include the year in the title, for example those with academic regs, and these are deselected by the teacher as part of rollover.

I wouldn't restrict announcement rollover as some colleagues will reuse and reschedule existing announcements

I would be hesitant to completely block assignment rollover 

[DavidSchlater]

Interesting. We are coming from a different LMS, where if you did not do "select all" you would break the course in mysterious ways you only discovered later in the semester... Seems you are saying you make people select specific content so they have to select all Announmnent or pick just a few. I would guess most instructors would just check "files" and "pages" and that would take all those over even if they are not used (or do you try to discourage that too?). Is it true then that if you do an import and check just modules, assignments, quizzes, question banks, and discussions you end up with clean and functioning course where only linked pages and files are moved? Would rubrics in use move, even if you don't check the rubrics option? Is this documented by Canvas so we can understand the rules and dependencies? Thanks!

[paul_fynn]

Some general thoughts in response David. 

There are differences in the behaviour of the standard 'Import Course' and the ability to push content from another course using 'Copy To' from the source. The former will pull across all content associated with whatever you selected and transpose the course links, the latter will push across whatever you copied, but the links will still point back to the source course.

I need to do some more checking on Rubrics - I'm not currently clear whether a course import needs to be re-associated with the Rubrics. This is further complicated with Account/sub-account rubrics which may lose their association with the account level when/if copied with the assessment. 

We manage summative assessment provisioning through a SIS based custom tool, and use a separate custom tool to allow colleagues to selectively update the new assignment from an old version. You've prompted me to go back and investigate whether it is pulling the rubrics with it when it does that - thanks !

Taking all files over without discrimination creates a housekeeping 'bear' for later, and perpetuates copyright/accessibility issues. It also means that we can have student related content uploaded by teachers  that should be removed, or certainly not copied forwards. Note that files are not actually copied - they exist in a central location and the new course links back to the original location. Hence files imported from within Canvas don't count to quota. 

Two aspects of copyover (ref announcements but relevant to all content) are highly problematic for us and live courses - Canvas assumed that if you copied it you want to publish it.  However

• We are likely to want to edit before we publish it
• It publishes to everyone when we might want to restrict it to a specific section (in crosslists) or set of students
• We lose all the dates when it is copied over including due date, available from date and until date

[DavidSchlater]

I've added this idea to make the announcements tool more usable: https://community.canvaslms.com/t5/Canvas-Ideas/Hide-Announcements-from-Students-individually-in-bul... 

[Nancy_Webb_CCSF]

Thanks @DavidSchlater for starting this discussion.  We were trying to figure out if assignment participation numbers included graded discussions, so I picked one class to study..  The Interactions by Category report said there were 94 discussion participations, but in the class I found 104 posts (in 3 graded discussions), all made before March 20 so should all be included in the report.  Only one of the posts was from the teacher, so that means 103 student posts. Not sure why only 94 shows up in Admin Analytics. There were 1716 assignment participations, that will be harder to determine accuracy, it might include resubmissions for example and not sure if it includes graded discussions. I plan to download the gradebook and do some counting.

Maybe somebody at Instructure knows if Assignment participations include Discussions, @carlycurran?  Is that information somewhere in the documentation? Thank you!