Hi, I'm Ben, the new Outcomes PM! Let's Chat!

[BenFriedman]

I’m excited to introduce myself to the Canvas Community! My name is Ben Friedman and I’m a new Product Manager at Instructure, working on Outcomes within the Canvas LMS. 

One of the reasons I’m so excited to work on the Outcomes team is how passionate I am about helping all types of students (and the stakeholders supporting them) understand what targets they are working towards and how they are making progress towards those targets. I believe this is essential to the process of students taking ownership over their own learning! As a former educator myself, I am also deeply committed to bringing products to in-person and virtual settings that make the learning experience better.

Over the past few weeks, I’ve been getting to know how people like you are using Outcomes through reading Community posts, talking with CSMs, and digging into feedback some of you have shared. I’m also in constant communication with Jody Sailor, the previous PM for the Outcomes team. I look forward to hearing from and working with you as we continue to improve Outcomes. Please check back here in the Community for exciting updates on what we’ll be releasing in the coming months. I’ll also be sharing research-based strategies for getting the most out of Outcomes, product tips and tricks, updates on my new pandemic related hobbies, and celebrity gossip (maybe?! - more on that later). 

If you are using Outcomes in Canvas and would like to share your experience, provide feedback, or discuss needed functionality, click this link and sign up for a time to have a brief chat with me. I look forward to speaking with you! Also, feel free to add a comment below. Thank you for your interest in Outcomes and dedication to your students! 

About Me

I’m lucky to have had varied experiences working in education and Edtech. I worked for 6 years as a Special Education teacher in New York City public schools just as new rigorous standards took hold across the country. Some of the proudest moments of my life were helping students who had struggled with reading or math have a breakthrough! I then entered the Cognitive Studies in Education program at Teachers College, Columbia University and got to dive deep into how people learn and what can facilitate learning. I love the science of learning! I even got to teach a few graduate level courses online and saw what value and difficulties an LMS can bring to teaching and learning in higher education. Since then, I’ve worked at the intersection of technology, product, and data - including on products that were built from the ground up with Outcomes use cases in mind like Competency-Based Education and Standards-Based Education. 

In my personal life, I’m the father of two amazing kids and husband to an ESL teacher (also amazing!), all of whom will be using Canvas next fall. During the pandemic I picked up gardening, teaching piano to my kids, and attempting to train our new rescue dog as hobbies - all of which are keeping me extremely humble on a daily basis. I’m originally from Wisconsin, and yes that means I love cheese, beer, and keeping up with the Green Bay Packers - including any Aaron Rodgers celebrity news ;-).

[dkpst5]

Welcome! I don't have a lot of feedback since few of our teachers are using the feature (we're relatively new to Canvas). But the top feedback I have seen is the desire just to mark an assignment as "counts toward this outcome" without using a rubric.

[Steven_S]

Outcomes would be a great way to track progress towards meeting college-wide objectives or course specific objectives.  However currently, an outcome must be created to match each individual rubric application.  For those overall objectives they may be met through multiple module level objectives and those often have assignments with differing rubric details and/or point values. It would be great to be able to link rubric achievements related to the specific assignment to meeting or exceeding expectations of an outcome.  That way the outcome would track the larger overall objectives that are not already tracked through rubrics. 

I would like to see this a dropbox of outcomes to link to any rubric criteria and then the ability to set each box of point levelsfor that rubric criteria individually to exceeds, meets, or does not meet expectations for that outcome.  In some cases it would be nice to be able to link a second outcome to the same rubric criteria with different points levels indicated meets/exceeds.  (For example, a rubric with one grammar criteria in a course that is not directly teaching grammar may be linked to multiple college-wide objectives individually related to grammar.)  The result in outcomes would be that the outcome reflects the achievement levels rather than the points that were applicable to various specific assignments.

This is essentially reversing the current system in which outcomes are created and then stacked to make rubrics.  Since different rubrics contribute together to the same overall objectives, the rubrics could be created to meet the needs of the assignment, and then each criteria linked to the overall outcome that it aligns to measuring achievement of.  The difference is the control of the rubric descriptions and points stays with the rubric.  Matching a criteria's description to the outcome accurately would be the responsibility of the instructor designing the course, instead of only allowing exact matches between rubric criteria and outcomes.

[jmurray6]

Hello Ben!

Probably the the most pressing need for our institution is for the outcomes in New Quizzes to pass back to the main learning mastery gradebook in the course and for those outcomes to also appear in the main outcomes reports pulled from hosted Canvas Data. Currently, for our faculty that have moved over to New Quizzes, they have to attach rubrics to the assignment settings for those quizzes (or create dummy assignments with rubrics) and manually click the boxes for those items to appear on our reports. If it's an auto graded quiz (and outcomes have been set on that quiz or by question in New Quizzes) Canvas will auto calculate and show those outcomes results for THAT quiz in the quiz analytics - but that information stays there and doesn't become part of the course learning mastery gradebook nor the campus wide reports we pull every semester.

A very big "want" (which would likely take an extensive overhaul of how outcomes work in Canvas) would be the ability to link outcomes on the backend.  Here's what I mean.  We have Course Outcomes (CLOs), Program Outcomes (PLOs) and Institutional/General Outcomes (GLOs). Our assessment process reports out on GLOs as the common point of comparison, but our instructors measure CLOs at the course level. There are also times (Program Review, Accreditation) where I need to report out on PLOs.  Currently we DO have links between CLOs > PLOs > GLOs but this is all done manually outside of Canvas.  What I would love to have the ability to do is for the faculty to access CLOs in their courses, and then as Assessment Admin be able to go into Canvas and say "These CLOs link to these PLOs and these GLOs" so that from the faculty perspective - they access on CLO, but I can aggregate reports on PLO or GLO from within Canvas reports and Canvas Data (so that when I select to report on GLO 1 for example, that pulls the records from all the CLOs associated with GLO 1).  Perhaps from within the Outcomes screen at the account/subaccount level, each outcome could have a few pull down menus (perhaps at least 2 or 3 for different levels of classification) to say which "higher level outcomes" that outcome would link to - these would become additional columns in the outcomes reports for that record so that those reports could be filtered/the data could be sliced by those "higher level outcomes" as well.

[Steven_S]

@jmurray6  That would also be useful at the instructor level.  In our school, instead of admins running a report in canvas, faculty submit their assessments of the appropriate GLOs for each course.  It would be ideal to flip to the learning mastery gradebook, and see the results there.  In our case the CLOs do not align directly with the GLOs.  That alignment exists as if the GLO was an additional course objective.  However, course objectives and GLOs, both, are too general to be used directly as the criteria with which we grade assignments.  Outcomes are currently designed to match those rubric criteria, but that means an outcome for every criteria of every rubric.  The CLOs and GLOs that we are really measuring progress in are measured repeatedly in multiple rubric criteria.

I have tried creating groups of objectives for this purpose within a course.  However those grouped objectives do not provide summaries in the learning mastery gradebook the way grouped assignments do.  Also, there is not a way to link 1 criteria to more than one objective group.

[jmurray6]

@Steven_S It sounds like our institutions have similar approaches just different ways of executing (I was somewhat simplifying in my original post). I'm at a community college where we wear many hats (for example, I'm faculty, assessment director, canvas administrator for academics, and I run the short-term study abroad program) and have a wide range of disciplines (general education, nursing/allied health, technical programs, etc). We also access every course in the institution (with a few exceptions) to try to get as complete a picture as possible.  It can be difficult enough to get all of our faculty to include outcomes in their overall grading as it is and I think having them submit assessments separately would be one more thing that I'd have to "chase down" every semester - so we have them "grade outcomes" as part of their grading process so that I can just pull the reports.

However, since outcomes do not stack - our faculty essentially are "grading" GLO items in their rubrics, so that we have that common point of comparison across disciplines. But they do this utilizing the CLOs.  Here's an example of the process from one of my courses (Social Problems).  Our process began with the links of outcomes. When we started assessing, department heads were tasked with linking outcomes.  One of my course outcomes is to "Define and identify social problems."  That course is part of 3 different departments (so those department heads considered which PLOs in their programs that CLO would associate with - they also linked their PLOs to the GLOs).  Through that process it was determined that "Define and Identify Social Problems [outcome code SOC2203 CLO 1] links to 6 PLO codes (across those departments) and those PLOs link to GLOs 1, 3 and 5 (which reference Critical Thinking, Interpersonal Relationships, Ethical Issues, and Issues of a Diverse Global Society).  So, when I start a social problems class, I get a document as faculty that tells me SOC2203 CLO 1 links to GLO 1, 3, and 5. 

I as faculty determine which assignments have elements that fulfill CLO 1, so on those assignments I add GLO 1, 3 and 5 to the rubric as "non graded items" (no point values in the rubric). When I grade those assignments I look back to my "linkage document" and assess the CLO skills related to that assignment and determine how well those skills have been mastered and which GLOs those skills are linked to - and use that to "click off" the "scores" on the GLOs.

That linkage document allows me as assessment admin to take the GLO values as the common point of comparison and report out the GLO aggregate values [assesses the courses], and the number of mastery attempts for each GLO [assesses student skill mastery] for each course, department, program and the entire institution.  It also allows me to drill back down and slice the data by PLO for specific programs if I need to.  However, reporting out and drilling down process has been setup by me somewhat manually (I've preprogrammed a series of spreadsheets and query relationships off of Canvas Data in Excel to do most of it for me -but that setup process is time consuming the first time a report needs to be run).

The faculty also don't really have the ability to see the CLO assessment in the learning mastery gradebook, because they're not technically assessing them directly. The way Canvas currently works with outcomes is that to get to the mastery gradebooks (or the reports) it has to be a rubric item and it has to be clicked off.  My faculty would revolt if I had them directly measuring every CLO, PLO and GLO especially since some courses (like my example) are in 3 different programs. That would equate to 4 CLO items, 18 PLO items and 6 GLO items to "click/assess" and I wouldn't want to do that either. But if we could get to a point where outcomes do stack, my faculty could just add the CLOs to the rubric and assess those as they grade - which would populate the learning mastery gradebook with those items AND allow me (on the backend of canvas) to slice that data through any aggregation (CLO, PLO, GLO) or level (Course, Program, Department, Institution) that I wanted simply by filtering values in the reports - instead of needing the linkage documents to do that. 

[Steven_S]

@jmurray6  We do have very similar systems at our community college.  A committee writes GLOs, some programs have additional PLOs, but in humanities and liberal arts, the GLOs meet our needs for now.  Rather than assess every course for every GLO, different courses and GLOs are assessed each term.  Assessment involves a special GLO rubric that is essentially a set of related objectives.  We choose an assignment to use at the beginning of the semester, fill out a starfish survey with an overall result for each student, and then turn in a few sample student submissions through a MS forms survey...

Alignment of assessments to institution wide objectives measured by the courses, is a requirement of the QM standards we apply to at least our online sections, but that does not mean grading using the GLO "rubric" it just means that what we do assess, includes measuring the same concepts as the GLOs.  In my courses, it meant adding more detailed descriptions to the structure and grammar criteria of writing assignments. But since there is no tool to pull the criteria from multiple assessments at different point values together to assess a GLO, we still use a separate assessment of the pre-selected assignment.  Grading twice is not the ideal way to accomplish this reporting, which is why we do not assess every objective in every term.

Instead, if outcomes allowed multiple rubric criteria with differing point values to be linked to them, then those criteria that are aligned with GLO(s) could be grouped via outcome.  Essentially I would put each objective of the GLO rubric into its own outcome, and then link the multiple aligned rubric criteria.  Then the learning mastery gradebook would display the results of the GLO without me doing a separate assessment, and those reports could be pulled the way you pull results.  (Except that our canvas admins would require supervisors to pull the results, and our supervisors would just add that to our end of term check out requirements - which makes seeing the results in the learning mastery gradebook helpful. 

To take that a step further, we assess each GLO as a whole, and so the ability to group outcomes together for a combined result (the way assignment groups create averages) would be particularly helpful.  For example, the individual outcomes are rated as exceeds/meets/needs-improvement, and then the lowest category identified in the outcome is selected for the entire GLO unless only one outcome needed improvement while all others exceeded expectations (for which "meets" is selected.)  It would be awesome to have groups of outcomes automate that type of summary in the learning mastery gradebook.

[hernandh]

Hi Ben, 

Welcome to the Canvas Community! Thank you for offering your help with Outcomes. We are working on a BIG project that includes Classic and NewQuizzes outcomes for assessment and accreditation purposes. We would love to get your ideas and thoughts around it. Would you be able to schedule a mini call with us to get your ideas about it? We may inform some potential/future enhancements.

Thanks, 

Humberto Hernandez

hernandh@dyc.edu

[BenFriedman]

Thanks for the welcome, all! I noticed a few people had a difficult time setting up a meeting through Calendly. If you'd still like to chat, I've updated the link in the original post. You can also click here: https://calendly.com/ben-friedman-1/outcomes-in-canvas-let-s-chat. 

@dkpst5 The request to align outcomes to all types of content, including without a rubric is something I've seen a few times. I'll start digging into what this would take. Thanks for sharing!

@jmurray6 & @Steven_S - Great to hear about your use cases around course, program, and general level outcomes. We're likely to do a discovery project in this area over the next couple months and I will be reaching out to see if you'd like to offer some input at that stage. 

@hernandh Looking forward to speaking with you and understanding what you are trying to accomplish for assessment and accreditation with outcomes. 

[gramos]

@BenFriedman 

Hi, Ben. I'm a community college faculty member who uses mastery grading in my courses. I just saw this post and wanted to share my feedback: 

• n mastery is currently limited to 5 items. There are some outcomes related to skills (example: "using claim-evidence-reasoning in your explanations" that I would want students to show mastery of more than 5 times since that outcomes is related to many assignments before being considered mastered.
• I like the ease of use of the new Improved Outcomes Management. I understand that those are course wide. However, it would be nice if there was some more flexibility such as
  • have more than one mastery scale that I could choose from (example: a two level scale and a three level scale) 
  • be able to edit mastery scale and/or calculation method on individual outcomes. (For *most* of my outcomes, I use n=2 mastery with a three level scale but for some I use n=5 mastery and for others I use a two level scale.)

Thanks!

--Gloria

[BenFriedman]

Thanks for your input @gramos!

[lettgo583]

Hi Ben, 

We are a nursing program within a university. We're looking at using Outcomes to help us in reporting to nursing accrediting agencies (AACN Essentials). There are specific outcomes prescribed by the agency that we need to show we meet. Have you worked with any other Nursing programs to set up outcomes? 

Thanks, 

Linda