Releases Q&A: Improved Outcomes Management Feature Preview

[erinhmcmillan]

Looking to discuss this feature from the 2021-09-18 Release Notes? Post a reply and start a conversation!

Note: As of 2021-09-17, this feature has been delayed from being released in the production environment. An update will be posted when available.

POSTING GUIDELINES

• This topic is for feature discussion only—please share use cases, best practices, etc. regarding this feature
• Please thread replies as much as possible to keep posts organized

WHERE SHOULD I POST...?

• Idea enhancement feedback to product managers should be submitted in ideas.canvaslms.com (though linking to the idea here so others can find it is welcome)
• Bug reports should be submitted to Canvas Support—bugs will not be triaged in this thread

[dgrobani]

Hi,

In Canvas Feature Preview: Improved Outcomes Manageme... - Instructure Community, the API section says "In lieu of the REST API, this feature uses GraphQL mutations. Queries include learningOutcomeGroup, outcomeCalculationMethod, and outcomeProficiency."

I'm not clear what "this feature" refers to. I currently run several automated processes that manipulate outcomes and outcome groups via the Outcomes API and Outcome Groups API. Are these REST API endpoints on the way to being deprecated?

Thanks!

[erinhmcmillan]

Hi @dgrobani 

No deprecation notices have been issued. We are calling out that the Improved Outcomes Management feature uses GraphQL instead of REST API. We have several features that are using one or the other.

However, our product blog recently included a post from our engineering team, who has indicated that Canvas will move to using GraphQL in the future. See From the Engineering Deck: Instructure & GraphQL - Instructure Community

Thanks,

Erin

[dgrobani]

Thanks for the info!

[bbennett2]

I'm finally getting around to exploring this and I noticed that when we turned it on in beta, rubric scores were removed from the Outcome. I know that Rubrics and Outcomes are disentangling a little bit, but a major part of using Outcomes was the ability to give granular feedback via rubrics on a skill or idea attached to an assignment.

I've spent a lot of today playing with it and I can't figure out how to add a scorable Outcome back to an Assignment once the improved management tool is enabled. Am I missing something basic?

[Sopko]

Now that mastery calculation has been broken out into its own tab, do we still have the ability to set different calculation methods for different outcomes? For example, using the old interface I was able to set my grammar standards to n number of times as the method of calculation. I then had standards like identifying theme set to decaying average as the way to calculate mastery. In the new interface it appears I can only select one calculation method for all outcomes.

[nathanleesSSD]

I have the same questions/issues as bbennett2 and Sopko. 

Additional Question: For the Mastery tab, will we be able to set the mastery level? Also, will we be able to change the names of each level? 

[nelsonreg]

We have outcomes imported in different flavors as we could with the import CSV at various subaccounts and our top level subaccount. These outcomes have been aligned to assignments across our institution and our currently being assessed in an academic year for which we want an academic year data. What will this change mean to "already in progress" assessment of outcomes? Will the new changes also require re-entry of the program outcomes that already exist?

[BenFriedman]

I'm Ben Friedman, Product Manager for the Canvas LMS Outcomes Team. I will be keeping track of this thread and answering questions as they come up. Thanks to those of you who have posted comments and questions!

@bbennett2 , @Sopko , and @nathanleesSSD  Currently, the Improved Outcome Management feature is paired with the Account and Course Level Outcome Mastery Scales feature, and both “Feature Previews” will need to be enabled. Once the Account and Course Level Outcome Mastery Scales feature preview is enabled for your account, sub-account, or course, mastery scales and calculation methods are separated from individual outcomes and become consistent across that context. The Outcomes page then displays Mastery Scales and Calculation Methods as tabs (see screenshot below). Please note that we’re going to be doing some additional work to bring the capability of setting these at an individual Outcome level back over the next few months within Improved Outcome Management. Stay tuned if that is important to you! 

@bbennett2  In terms of adding an Outcome to an Assignment rubric, that can still happen with the new feature enabled. I love that you are giving granular feedback to students through rubrics - that is really taking advantage of the capabilities Canvas offers!

@nathanleesSSD  You can indeed customize Mastery Scales in the tab by setting the exact number of levels you would like, changing the names or “descriptions” and “points” for each, and even adjusting the color paired with each level. See more information here.

@nelsonreg  You can find the full impact of making a change here. Enabling the Account and Course Level Outcome Mastery Scales feature preview will affect current account data. If a rubric with aligned outcomes has been used to assess an assignment before Account and Course Level Outcome Mastery Scales are enabled, however, the assignment is not affected. Any assignments that have not been assessed by the rubric are affected if mastery scores are adjusted. Please connect with your CSM if you’re considering this so you can talk through the details.

[bbennett2]

Thanks @BenFriedman for the tip. I'll get in touch with our CSM today to have those turned back on in beta to keep testing.

[olguin]

Hi, our institution is just starting to test adding Common Learning Outcomes. Should I enable the 'Improved Outcomes Management' now while we are testing because eventually this will replace the older version? I am not sure what the timeline is for this since it is currently a feature preview. 

We are piloting a few courses with our new CLOs in Spring of 2022 with the idea of adding them to our instance in Fall of 2022. Any advice @BenFriedman ? And we do need to ask our CSM to turn this on for our test instance, correct? 

Thank you!

[BenFriedman]

@olguin You do need to work with your CSM to turn on this feature at this time. Part of the reason is that currently you also need to turn on the Account and Course Level Mastery Scales feature preview - the two are tied together. A few factors to consider:

- First, review whether you already have the Account and Course Level Mastery Scale feature preview enabled. There can be an impact on data (other than already assessed rubrics) if the rating scale changes so we do not generally recommend enabling this feature during a school year.

- We're currently in the planning stages for pulling the two features apart in order to support use cases where defining a mastery scale, calculation method, and description content on an individual Outcome basis is necessary. If any are needs in your context, then it would be good to wait until we finish this next set of work. 

[Chris_Hofer]

Yesterday our CSM team turned on the Improved Outcomes Management within our "Beta" environment so that we could start to do some testing on our own.  We already had already started building what we call "Career & Life Skills" into the "Outcomes" area of Canvas...prior to turning on the Improved Outcomes Management feature preview.  We have these stored at the Canvas account level.  Here are a couple things I've noticed so far...

• For our rating scale that we used for our C&LS (4, 3, 2, 1, 0), the descriptions didn't fully come over on the "Mastery" tab.  For example, the description for 4 is "Meets Expectations - demonstrates all required elements", but when Improved Outcomes Management was turned on for us, the description only showed "Meets Expectations " (including the space after the word "Expectations").  The same held true with the descriptions for 3, 2, and 1 ... where there was a dash between two words within the description.  So, it seems that Canvas stripped out everything from the dash to the end of the text, "- demonstrates all required elements".  I had to re-add the missing text.
• Related to the first bullet point, after I updated the descriptions of those ratings and clicked the "Save Mastery Scale" button, I was prompted with the message, "This will update all account and course level rubrics that are tied to the account level mastery scale and have not yet been assessed."  I clicked "Save".  Then, I navigated to my own sandbox course where I had several rubrics already built long before Improved Outcomes Management was enabled.  I went to my "Rubrics" screen and found a rubric that also included at least one Outcome that had already been added to it.  For example:
  
  
  
  When I clicked "Edit Rubric", the second row displaying the outcome changed rating scales on me and replaced the 5, 3, 0 scale with the 4, 3, 2, 1, 0 scale.
  
  
  
  This is slightly concerning to me.  I understand why this happened...because I chose to "Save" the changes I had made to the descriptions on the "Mastery" tab.  But, what if the instructor had wanted to make some kind of change to the original outcome row...and wanted to keep the 5, 3, 0 scale.  Those edits still need to be made via the "Outcomes" screen, but it wouldn't currently be able to use the 5, 3, 0 scale since the 4, 3, 2, 1, 0 scale is set as the default.  Or, let's say that the instructor instead needed to make an edit to the first row of the rubric (not the row with the outcome).  When editing the rubric, the 5, 3, 0 values still change in the outcome row even though the instructor really wants to make an edit to the row above it.  This would surprise and confuse me as instructor wondering what happened to the 5, 3, 0 values in the second row and why it was replaced with a completely different scale and descriptions...and potentially changes the overall total points of an assignment.  In this case, my first screenshot shows a total of 6 points for the rubric while the second screenshot shows a total of 5 points for the rubric.

[BenFriedman]

@Chris_Hofer Thanks for trying out this feature and posting about your experience. Right now, the "Improved Outcome Management" feature is paired with another feature -  "Account and Course Level Mastery Scales" and both need to be enabled with the help of a CSM in order for Improved Outcome Management to work. We are currently engaged in work that will bring back the ability to set a calculation method and criterion rating scale for individual outcomes within Improved Outcome Management. 

The changes to the rubrics you are identifying are related to Mastery Scales. The power of this feature is that is it efficiently creates a common mastery scale across Canvas so that they don't need to be set for individual outcomes and will be represented in rubrics and the Learning Mastery Gradebook. While this level of consistency is highly valued by many of our customers, it sounds like it may not be the best fit for your use case, where it sounds like more control over rubric rating scales and indicator descriptions are necessary. In this case, we'd recommend not using Improved Outcome Management until our work on calculation methods and criterion ratings for individual outcomes is complete.  

[kshaffer2]

I'd like to be able to add descriptions to the ratings criteria for Outcomes because we have many faculty that assess student work that rely on a printed Outcomes rubric.  Is this possible or in development?

[paul_witzel]

@kshaffer2 The teachers I work with have the same need. They want to be able to set up Outcomes with ratings descriptions, beyond a label, like "Exceeds Expectations". They would like the same editing capability that you experience when creating rubrics with non-Outcomes criteria. Uploading an Outcomes .csv may accomplish this, but is time-consuming and somewhat confusing.

I believe we had this flexibility before the Outcomes change in September.

[wdewysockie]

So glad to hear that the ability to set mastery scales and calculation methods at the individual Outcomes level will be restored. This is important to our institution as we have scenarios where setting these differently for individual Outcomes is very important in our school's Program Assessment.

[BenFriedman]

@kshaffer2 & @paul_witzel Currently, if you have Improved Outcome Management and the Account and Course Level Mastery Scale flags enabled, you are not able to edit criteria/descriptions in rubrics for individual outcomes since the mastery scales are consistent for all. As I mentioned above, we are currently in development to bring back this capability. It will then work as in the past - detailed descriptions can be added to individual outcomes and then brought into rubrics through alignments. 

[kshaffer2]

Fantastic.  Any idea what the timeline might be?

[BenFriedman]

Hi @kshaffer2, this is now available in Canvas. You can use Improved Outcome Management without the Account and Course Mastery Scale flag enabled. For the moment, you will still need to reach out to your CSM for support. 

[danaleeling]

I have 1618 outcomes in the admin root account that must remain read only for faculty at the institution. Our CSM team has enabled the new outcomes feature in beta. I note that there are permissions to grant instructors/teachers edit authority for rating scales and calculation methods. I am not seeing documentation on these permissions. I want faculty to be able to add their own outcomes in a course using their own rating scales and calculation methods that they prefer to use while not enabling editing or alteration of the outcomes pulled from the admin root account. Do these permissions grant faculty the ability to change the rating scales and calculation method used by the admin root account outcomes that they have pulled down into their courses?

In other words, what is the scope of those permissions? 

Conversely, if I disable those permissions, do faculty lose all ability to have their own rating scales and calculation methods even for outcomes they have created and added at the course level?

Thanks in advance!

[BenFriedman]

Hi @danaleeling - the permissions impacted by Improved Outcome Management are now listed in the feature overview. If permissions are enabled for the "Learning Outcomes - add / edit / delete," faculty will be able to add and edit their own outcomes within courses. They are not able to edit account-level outcomes regardless of whether this permission is enabled or disabled.