Admin Reports: Outcome Results vs. Student Competency

[harker]

We here at SUU are using Outcomes at the institutional level. We are using these 'ELOs' as a means for university accreditation and etc.

We have been using these ELOs to track students General Education courses in our first phase.

What we have been finding is that the Outcome Results and Student Competency reports have some similarities, but the data tends to vary.

It would be helpful to know how these reports should be used.  There is no tutorial on how the data is gathered and what variables are used to generate them.

What we see is that there is more data and more courses in the student competency (SC) report than in the outcome results report (OR). Also, the SC report lists courses and ELOs not found in the OR report. In almost all cases, we get more outcomes scores for a given ELO in the SC report than in the OR report. There are a few ELOs for which it is the other way around.

1. What is the difference between the Outcome Results and Student Competency reports?
2. [Feature Request] It would be nice to have a way to see what account level directory that the outcome used is being pulled from.
   

   My thought is that the reports inconsistency in outcome scores could be the result of a couple of factors. To better illustrate:

   At the institutional level we have created 15 generic outcomes, along with these "ELOs" we have included a folder containing a breakdown of more defined outcomes. Faculty are asked to use at least one outcome (as assigned to College/Dept.) in their course. So, the logic is that they can search for the outcome and pull it down into their course. (Institutional level to Course) If the instructor pulls from the institutional level, it creates an issue for Chairs and Deans because they cannot see the outcome results in their reports.

   We have been told by Instructure that for the admins of these sub-accounts (Deans & Chairs) to have access to the data, that the outcome needs to be pulled down from Institution to College to Department to Course.

   This illustrates just two scenarios that are possible for the SUU ELOs to be pulled into the course.

   One thought here IS that the different locations (account directories where outcomes are found, where the same ELO can be found in) may cause the variance in the two- Outcome Results and Student Competency reports at the institutional level.

3. Can you confirm that this is the case?

   The other though is how the outcome is applied. Faculty using the ELO will include the outcome in different ways.

   The first is that the instructor will create an assignment that is worth no points in their course. This may or may not be in its own assignment group with a value of 0% of Total. Thus not affecting the final grade. In this assignment the rubric includes the outcomes listed as a criterion. Faculty, perform their ELO rating separate from everything else.

   The second is where the instructor does include the ELO in an assignment rubric but the outcome is not a scored criterion.

   The third is where the instructor does include the ELO in an assignment rubric with the outcome criterion points as part of the assignment score total.

4. Do the various scenarios described effect the report differences?
   

   We just need some clarification as to know which data report is best for our needs

So are there any ideas on this?

[Vanderford_n]

I would be interested in the answer to this question.  While I have not had the opportunity to run any reports in Canvas because my institution is currently in the process of adopting this LMS, I do know that creating Outcome reports is one of the factors of the move to this product.

I can address question #3.  In my department, we make the ELO part of the assignment rubric.  Then, when we need to report on a specific outcome, we can generate a report.

I would think the the type of report depends on the type of assessment that needs to be completed.  At my institution, we report on general education objectives.  However, there are seven objectives and we only need to assess one or two each year.  Further, we only need to report on students who are completing their general education curriculum (we go by credit hours in which a student would ideally complete all gen ed requirements by the end of the sophomore year).  In this assessment, one teach may only have to look at one student's performance.  So, I think that an individual report would suffice.

Again, I am new to Canvas, but my experience might help others to answer this question.

[Stef_retired]

harker​, I'd like to move your question over to the Canvas Admins​ group so that it can garner focus from that audience. Would that be all right with you?

Also, do you have a feature idea sandwiched in there between those questions?   In the new community, feature ideas​ go through a series of stages, and you can read about it here: How does the voting process work for feature ideas? Just like the old community, however, a feature idea needs to be submitted as its own discrete entity, so please use the How do I create a new feature idea?  guidance to submit it.

[harker]

Sure. Move it on up.

[jen]

Stefanie,

Would it be possible to get a Canvas response to this question? It really asks about basic system (reporting) functioning that Canvas should be able to answer. I appreciate that the Admin community knows a lot, but shouldn't Canvas know this too? Thanks!

[Stef_retired]

 @jen , I'm going to tag  @Renee_Carney ​ on this so she can help us with the response from Instructure.

[Renee_Carney]

Thank you stefaniesanders​ and  @jen ​

We have a phone call later this week with the school posting the questions.  We will share a thorough response soon.

[Renee_Carney]

harker​

Greetings!  On the topic of using outcomes reporting for accreditation - this is such an exciting topic to explore, and very necessary as standards have changed with the introduction of online and hybrid courses.  You are not alone in what you are seeking, but you are in the group seeking to find the best way!  This is something that was definitely on my radar when I was working for a higher education institution and it was on my to do list to figure out.  There were other eLearning administrators that I worked with at neighboring schools that were asking similar questions and seeking similar data for accreditation.

This is not your exact question, but you may want to participate in the conversation with phineas​ at Institutional Assessment for Accreditation Using Outcomes and Rubrics?​

Hopefully others exploring this topic will also join in.  I know it is fresh on the mind of  @CSoran  and others in Washington State.

[Renee_Carney]

harker​

There are two guides that will help answer question #1.

1. What is the difference between the Outcome Results and Student Competency reports?

What Reports are available for my account?​

1. Outcome Results shows the learning outcome results for all students. This report also includes question banks.
   
   Fields:  Student Name, Student ID, Student SIS ID, Assessment Title, Assessment ID, Assessment Type, Submission Date, Submission Score, Learning Outcome Name, Learning Outcome ID, Attempt, Outcome Score, Assessment Question, Assessment Question ID, Course Name, Course ID, Course SIS ID
   
   
2. StudentCompetency shows the outcomes results for all students. For detailed reports, create outcomes at the account or sub-account level and add those outcomes to course activities. Course level Outcomes are not included in the report.
   How do I create a Student Competency Report for an account?
   
   Fields:  Student Name, Student ID, Student SIS ID, Assessment Title, Assessment ID, Submission Date, Submission Score, Learning Outcome Name, Learning Outcome ID, Attempt, Outcome Score, Course Name, Course ID, Course SIS ID, Section ID, Section SIS ID, Assignment URL
   
   

The fields that differ between the two reports have been underlined above.

There is an important note on both of these reports:

Note: Outcome results are tied to level of outcome access. Account outcomes display on the account report, while the sub-account outcomes display on the sub-account report. The only time outcomes would be displayed on both reports is if they are stored in both accounts, such as copying an account outcome to the sub-account level. Outcomes created at the course-level display on the course-level and will not appear on account or sub-account level outcome reports.

[Renee_Carney]

2.  [Feature Request] It would be nice to have a way to see what account level directory that the outcome used is being pulled from.
My thought is that the reports inconsistency in outcome scores could be the result of a couple of factors. To better illustrate:At the institutional level we have created 15 generic outcomes, along with these "ELOs" we have included a folder containing a breakdown of more defined outcomes. Faculty are asked to use at least one outcome (as assigned to College/Dept.) in their course. So, the logic is that they can search for the outcome and pull it down into their course. (Institutional level to Course) If the instructor pulls from the institutional level, it creates an issue for Chairs and Deans because they cannot see the outcome results in their reports.We have been told by Instructure that for the admins of these sub-accounts (Deans & Chairs) to have access to the data, that the outcome needs to be pulled down from Institution to College to Department to Course.This illustrates just two scenarios that are possible for the SUU ELOs to be pulled into the course.One thought here IS that the different locations (account directories where outcomes are found, where the same ELO can be found in) may cause the variance in the two- Outcome Results and Student Competency reports at the institutional level.

Unfortunately, There is not a simple answer for this request.  It sounds like you have spoke with product team members about this already to see what works and what does not.  I would recommend really clarifying this idea with solid examples (and possibly some screenshots) and then share it with the Canvas Focus Group: Outcomes​ to further the conversation.  There is a related feature idea already shared there that I could see pairing with this idea very well in development - it starts the conversation of tiered outcomes. 

[Renee_Carney]

Your Final question I am going to combine/reword for my own clarity.  Please let me know if I am off base.

3.  Do the various scenarios described effect the report differences?

1. The instructor will create an assignment that is worth no points in their course. This may or may not be in its own assignment group with a value of 0% of Total. Thus not affecting the final grade. In this assignment the rubric includes the outcomes listed as a criterion. Faculty, perform their ELO rating separate from everything else.
2. The instructor does include the ELO in an assignment rubric but the outcome is not a scored criterion.
3. The instructor does include the ELO in an assignment rubric with the outcome criterion points as part of the assignment score total.

We'll need to do some exploration and testing to answer this fully.

[harker]

I'll ask the guys that asked me to ask you... such is the life of the middleman.