Canvas Release: Anonymous Moderated Grading (2018-07-14)

[erinhallmark]

This document introduces Anonymous Moderated Grading as noted in the Canvas Production Release Notes (2018-07-14)

The current anonymous and moderated grading features have been updated to provide improved assignment flexibility with fully anonymized, multiple grader anonymity, and moderated options. Additionally, moderated grading has been simplified for a better user experience.

• Feature/Integration Setup
• Feature Options
• Existing Moderated Grading Functionality
• Existing Anonymous Grading Feature Option
• Account Management
• Permissions
• Anonymous Grading
• Moderated Grading
• Course Management
• Course Settings
• Course-Level Features
• Assignment Setup
• Anonymous Grading
• Anonymous Instructor Annotations
• Moderated Grading
• Anonymous Moderated Grading
• Gradebook Functionality
• Anonymous Assignment Details
• Moderated Assignment Details
• Grader Workflow
• Dashboard
• SpeedGrader
• Anonymous Assignments
• Moderated Assignments Grading Limit
• Moderator Workflow
• Dashboard
• Grade Summary Page
• Community Resources
• Future Enhancements
• Customer Feedback
• Feature Documentation
• Known Behaviors
• Audit Functionality
• Mobile Apps

Feature/Integration Setup

Feature Options

Anonymous Moderated Grading involves up to two feature options: Anonymous Grading and Moderated Grading. These features can be enabled individually or together to provide multiple options in the assignments page.

When one or both feature options are enabled, the assignments page displays several options according to each feature.

Existing Moderated Grading Functionality

Currently moderated grading is included in all assignments by default, but this update allows it to be enabled for the entire account or on a course-by-course basis.

Existing Anonymous Grading Feature Option

Accounts or courses that have set the current Anonymous Grading feature option to On will have the new Anonymous Grading feature option turned on by default. However, accounts or courses that have set the feature option to Allow must reset the Anonymous Grading feature option to Allow; by default it will be turned off.

Account Management

Permissions

The following permissions are available with each corresponding feature option:

Anonymous Grading

• Roles: Account, Course
• Permission: View audit trail
• Default: no user roles

Moderated Grading

• Roles: Account, Course
• Permission: Select final grade for moderation
• Default: enabled for TA- and Teacher-based roles

Course Management

Course Settings

If the required feature options have been set to Allow at the account level, individual feature options may need to be enabled at the course level. Graders and moderators may need to confirm the feature options are enabled in Course Settings. 

Course-Level Features

Assignment Setup

Anonymous Grading

In anonymous grading, graders cannot identify students associated with each submission, which helps remove grading bias.

When anonymous grading is enabled in a course, content creators can create assignments with the option to hide student names from graders.

Notes:

• This option can be edited and changed at any time by the final grader or an admin. Any edits made to the assignment will be tracked. Auditing tools will be added to help with accountability in a future release.
• Currently, enabling this option displays SpeedGrader submissions as read-only and does not allow annotations or comments in DocViewer.

Anonymous Instructor Annotations

Content creators can choose to anonymize instructor names in DocViewer-supported assignments. When the Anonymous Instructor Annotations option is enabled for an assignment, all annotations and comments made by instructors in SpeedGrader are anonymized. This option does not apply to annotations previously made to an assignment in SpeedGrader.

Note: Anonymous Instructor Annotations is not associated with the Anonymous Grading feature option and displays in all assignments. This option can be enabled by any content creator.

Moderated Grading

In moderated grading, anyone with assignment creation permissions specify an assignment submission to be reviewed by one or multiple graders, review the grades, and select which grade should be set as the final grade for each student. For each moderated submission, the user set as the moderator is the only user who can specify the final grade for each assignment. Any grader can be specified as the grader who can determine the final grade as long as they have the corresponding permission, but the moderator does not have to personally grade the assignment. Once the assignment is created, only the moderator and admins will be able to change the assigned final grader.

When moderated grading is enabled in a course, content creators can create assignments with moderated options:

• Grader number—the number of graders who should grade the submission. This number does not have a limit. However, the page displays a message if there are not enough course users with grading permissions to fulfill the desired number of graders.
  • A moderator does not have to specify which graders can grade submissions. If a course includes a surplus of graders higher than the desired number of graders, any grader can assess an assignment submission until the specified number of grades has been achieved. When an assignment has reached the specified number of graders, remaining graders will not be able to access the assignment.
  • The moderator is never included in the number of available graders since the moderator does not traditionally grade individual submissions but may provide a custom grade.
• Grader comment visibility—the option that allows graders to view each other’s comments. This option is selected by default.
• Final grader—the moderator, or the grader who determines the final grade. The Select Grader menu displays the names of course graders who have been given permission to determine the final grade. This permission is assigned to one or more roles at the account level.
• Final grader visibility—the option that allows the final grader to view the names of other graders. This option is selected by default.

Moderated assignments cannot be edited by any grader except the moderator. Any edits made to the assignment will be tracked. Auditing tools will be added to help with accountability in a future release.

Anonymous Moderated Grading

When anonymous grading and moderated grading are both enabled, content creators can create assignments that hide student names from graders and include moderated options. All options for both anonymous grading and moderated grading are displayed for the assignment.

When the moderated grading option for graders to view each other’s comments is enabled, the Anonymous Grading options list also includes a second checkbox option: Graders cannot view each other’s names. This option allows graders to view comments from other graders but not view the name associated with each comment.

Gradebook Functionality

Both anonymous and moderated assignments are shown in the Gradebook in a read-only state with gray columns. Additionally, both assignment types are automatically muted.

Note: The Anonymous label supersedes the Muted label. Assignments that are both anonymous and moderated display as anonymous, though the Grade Detail Tray displays the assignments as being muted.

Anonymous Assignment Details

• Grade cells do not display any content, including the Needs Grading icons.
• In the New Gradebook, columns display an Anonymous label in red text.
• In the New Gradebook, the Grade Detail Tray also displays options in a read-only state and shows the assignment as being muted.

Grades are not displayed in the Gradebook until the assignment is manually unmuted. Additionally, to remove anonymity from each submission, the assignment must be edited and changed so that graders can view student names.

Moderated Assignment Details

• Grade cells display dashes until grades have been posted.
• In the New Gradebook, assignment columns display a Muted label in black text.
• In the New Gradebook, the Grade Detail Tray also displays options in a read-only state and shows the assignment as being muted.

Grades cannot be edited in the Gradebook before grades are posted. However, after grades are posted by the moderator, grades can be edited in the Gradebook by any grader.

Grader Workflow

For grading, moderated and anonymous assignments are treated the same as any other assignment type. However, when an assignment is anonymous, some or all user name data is hidden.

Note: Anonymization and moderation options can be enabled or disabled at any time during grading, but audit logs will record the action and the user involved. Audit log visibility will be available in a future release.

Dashboard

In the Dashboard, all graders receive a To-Do notification when submissions have been received by students and are ready to be graded, regardless of assignment type.

SpeedGrader

All existing functionality is supported in SpeedGrader for both anonymous and moderated assignments.

The following restrictions apply to accessing assignments for each assignment type:

Anonymous Assignments

• When a grader accesses a SpeedGrader link, the first student displayed in the student list is always randomly generated for every assignment. Additionally, each assignment includes a random ID that cannot be traced back to any specific student.
• Unless an assignment is moderated, assignments can only be graded by one grader per current SpeedGrader functionality. If a submission has already been graded, other graders can still access the submission and change the grade.
• Comments in DocViewer display Grader instead of the grader’s name.
  • Note: Anonymity features do not hide any student identification that may be included directly in the submission.

Moderated Assignments Grading Limit

• For moderated assignments, when an assignment reaches its maximum number of graders, the grading notification will be removed from the Dashboard for graders who have not yet assessed the assignment. If a non-assessed grader opens the assignment directly, the SpeedGrader link will not display in the sidebar. If the non-assessed grader clicks the SpeedGrader link from the Gradebook, the assignment will redirect back to the Gradebook with a message that the maximum number of graders has been reached for that assignment.

Moderator Workflow

For moderating grades, moderated assignments mimic the existing moderated grading workflow but with simplified functionality.

Dashboard

In the Dashboard, the moderator receives a To-Do notification when at least one submission has been graded for a moderated assignment.

Grade Summary Page

The moderator can use the grade summary page to view all grades awarded by one grader and compare them to grades awarded by another grader. Grades are grouped by grader name.

Note: An assignment cannot be moderated until grades have been submitted.

If the assignment is anonymous, or if the moderator cannot view other grader names, the names in the grade summary page will be appropriately anonymous.

In the final grade column, the moderator can select a grade from the menu from any grader. The moderator can also open the menu and type a custom grade for the final grade.

When grades are final, the moderator can post the grades to the Gradebook.

Note: After grades are posted by the moderator, grades can be edited in the Gradebook by any grader. However, grades in the Grades Summary page cannot be changed.

Community Resources

Future Enhancements

This document outlines existing functionality included in the initial release of this feature. Anonymous moderated grading will continue to receive additional feature enhancements over future releases as indicated in Release Notes. Please follow the release notes for future functionality updates.

Customer Feedback

Community feedback for anonymous moderated grading is welcome per Canvas Community Feedback Guidelines, which outlines general feedback, broken functionality, and feature enhancements.

For anonymous and moderated grading, feature enhancements should include the anonymous&moderated tag. Before submitting a new idea, please review all existing anonymous moderated grading feature ideas.

Feature Documentation

All feature functionality will be updated in the Canvas Guides on July 14. Comments are welcome for functionality clarification. Lessons will also be updated with all future enhancements as indicated in release notes.

Known Behaviors

The following notable behaviors have been identified in anonymous moderated grading. Known behaviors are being triaged by the Canvas engineering team—feature ideas do not need to be submitted regarding their functionality.

Information in this document will be updated as made available by the Canvas product team:

Audit Functionality

• Full audit functionality that displays information associated with both anonymous or moderated assignments will be added in a future release. This functionality will ensure fair grading practices.

Mobile Apps

• Anonymous grading is currently not supported in the mobile apps. Both the Student and Teacher apps need to implement the new feature options into their code, which will be available in a future release. However, the Teacher app includes a local option to enable anonymous grading on a per-assignment basis.
• Moderated grading is not supported in mobile apps. No timeline for feature support is currently available.

[james_trueman]

Whilst I am really pleased that this AMG development has moved forward - I do have some concerns.

One is that there still appears to be a mismatch of understanding between what the workflow is (the specification) and the product that has been built - sad and surprising given how much effort has gone into consulting with Instructure on this (particularly from the UK HEI Sector needing this)?  For example, the opening line to Moderated Grading does not reflect the workflow that I'm aware of either locally, or from working with colleagues at other UK institutions 'In moderated grading, a moderator can specify an assignment submission to be reviewed by one or multiple graders, review the grades, and select which grade should be set as the final grade for each student.' This may just be semantics, but it conveys confusion.  Commonly there are two 'graders' involved, a 'marker' and a 'moderator' (moderated marking), or a 'first marker' and a 'second marker' (double marking), the 'moderator' does not usually make the decision described above, and the decision process to the final grade is also not reflected above.

I do not quite understand the thinking behind the controls and effect around when grades can be changed (i.e. not before posted but after, grades in the Gradebook can be changed but this does not affect the Grade Summary view)?

However, my biggest concern is that it seems an 'anonymity leak' still remains.  Below is a screenshot of the presented Gradebook view with AMG enabled: 

It shows that the student name column shows, well... student names. I understand that the Gradebook could hold a range of assignment outcomes, formative, moderated, double marked (not blind / anonymous), fully anonymous. Maybe not all on the same course / module, but you could easily have formative, non-anonymous and anonymous assessment in the same course site / module, so would have reason to want to see the student names in some, but not in others.

I am concerned that this design leaks anonymity where there are few students submitting, particularly one.  This is particularly because at the moment, Canvas only allows us to filter students in the Gradebook by Section, and this is actually a useful way to divide students into marking groups and filter down to them.  It is also a meaningful way to handle students with extensions, where we need to assign them a later Due Date and possibly (for various reasons) a different marker.  Under AMG (as links in the Gradebook go to random student papers in Speedgrader), it is a way of ensuring that markers can easily locate the individual students that need marking.

Below is an example Gradebook

By using the filter option in the top right I can filter down to the Extensions students (could be Extenuating Circumstances, a student resubmitting under Appeal, or any other reason, including extra-ordinary submissions that have to occur, or that only one student failed their first attempt and is submitting a second attempt paper/assessment).

And as a result, I can now see who that student is, whether or not it was an 'anonymous' assignment

Because at the moment we can only un-mute per assignment, I know that some colleagues are setting up new/duplicate assignments for events like extensions, so that they can better control the un-mute period.  Such workflows will also present the same issue, as one student in an 'Extension Assignment' column will clearly have their submission in the same row as their name, whether or not the Section filtering I've described is used.

Thoughts?

BW

[Tasha_Biesinger]

Some great improvements here!  We currently tell our faculty to stay far away from moderated grading because of all the 'gotcha's. We may start reconsidering when these changes role out.

Once you are able to incorporate anonymous DocViewer comments into this process, I know a lot of our faculty want all of the grader comments to appear to students (not just from the grader whose score is selected - or from neither if the instructor types in an average), for both completed rubrics to appear to students, and some faculty want the ability to post average scores of graders (without having to manually calculate). Is any of this being considered as future work?

Or...perhaps pull it all into a additional option:

When the moderator is selecting between grader 1 score, grader 2 score, or manual entry, add an additional option which is the "Average" option.  The average option will include all comments and rubrics from the graders. After posting scores, the instructor/moderator can remove any comment they choose, add their own rubric, and adjust scores to their heart's content.

A separate pain point: once a moderator chooses a grade, they can no longer see any comments/rubrics from the non-selected grader(s). This information should remain viewable to the moderator somewhere.

I'm happy to throw this stuff into feature requests, just want to plant a seed now while you all are really digging into this project just in case you can start building the hooks for the features now.

[erinhallmark]

Hi, Tasha,

This feature isn't fully completed and will include a few more phases. Our team is aware of comments not being able to be seen by non-selected graders, and that functionality will come a little later when the teams implement full auditing functionality.

All feature ideas need to be submitted in our Canvas Studio space as indicated in the document, otherwise our product team won't see them. We appreciate your feedback!

Thanks,

Erin

[natalie_norton1]

Hi - great to see this functionality coming!  I did some initial testing last week with no issue, but this week cannot see the moderated grading box correctly (testing in google chrome and safari - same view) 

[soberoi]

Hi Natalie, 

I'm glad that your initial testing went well with no issues associated with it! The issue that you present for the checkbox for Moderated Grading not appearing certainly seems odd. I have checked this myself and have had several other members on my team check it and we have been unable to reproduce it thus far. Are you still experiencing this issue? 

Best,

Sidh

[hesspe]

Hi erinhallmark   

Regarding:

...a moderator can specify an assignment submission to be reviewed by one or multiple graders, review the grades, and select which grade should be set as the final grade for each student. For each moderated submission, the user set as the moderator is the only user who can specify the final grade for each assignment.

Two things:

1) Re: "...a moderator can specify an assignment submission to be reviewed by one or multiple graders..."  

It looks one does not have to be  a Moderator (or someone with "Select final grade for moderation"permission)  to be able to select a moderator of an assignment from the list of those with grading privileges.  As a Course Designer creating a new assignment, I was able to  choose from the drop-down list the "Grader that determines the final grade."  

2) And aren't the "Grader who can specify the final grade" and the Moderator really one and the same?  If so, I'd find it clarifying if the label on the drop-down list read "Grader that determines the final grade  (the Moderator)."

I don't mean this as criticism; just checking my understanding.

[james_trueman]

Hi Peter, 

It is interesting wording, which seems to be predicated on the model that the 'professor, module/course leader, course convener etc.' is going to be the 'moderator' - e.g. that they will get other people to do the 'grading' and then check that by doing the 'moderation' (be the arbiter of whether the grading was good enough?).

Admittedly, that does happen (at least the 'leader' can sometimes be the moderator), but it's also frequently likely that when it comes to the marking load, the 'leader' is just one of the gang, who mucks in with everyone else.  It's even more common that each marker has a colleague acting as their 'moderator' - and highly probable that the 'leader' isn't doing any of either marking or moderation.  

I'm sure this could be styled a little more intuitively (and dare I say with better fidelity to practice?).

[g_bowie]

Hi all,

Just to echo other feedback, thank you for the developments in this area. Overall we feel they're great strides forward.

Of course during testing we have discovered some things that are probably worth highlighting - apologies if they've been covered already, and let me know if they need to go to an alternative forum / mechanism!

• Integration with Plagiarism Detection Framework - its not possible to create an Assignment with a 3rd Party Plagiarism tool selected. I think this has been previously noted, but its not listed in the current known issues. Is there an indicative timescale for this to be rectified?
• Old Gradebook - Assignment Groupings - If you use Assignment Groups, and only place a single assignment under that group, whilst the mark awarded to the student is hidden under the assignment, the score for the overall grouping is shown, making it relatively simple to identify the student.
• Old Gradebook - Export CSV - This shows the student names and IDs. We're not overly concerned about this, but thought the wider community may wish to know.
• Anonymous Instructor Annotations - When this is the only option selected on the Assignment creation page, it doesn't appear to hide or anonymous anything?
• Moderation / Double Blind Marking Workflows - If the grader who 'Determines Final Grade' is also one of the markers, then they get to see the Speedgrader Comments from the other markers.
• Analytics - Shows the mark awarded to individual students, regardless of the anonymous state. Again, not overly concerned about this, but the wider community may like to be aware.
• New Gradebook - Message Students Who - Clearly shows the student name and the marker awarded - another way to identify students. Once again, not overly concerned, but the wider community may like to be aware.

Finally, is there a way to get to the student specific submission in Anonymous Mode using the APIs - we've got some custom functionality where we'd like to drop someone to a specific submission? Any advice would be gratefully received.

[chriscas]

Hi g.bowie@herts.ac.uk‌,

• Integration with Plagiarism Detection Framework - its not possible to create an Assignment with a 3rd Party Plagiarism tool selected. I think this has been previously noted, but its not listed in the current known issues. Is there an indicative timescale for this to be rectified?

On my beta instance, I can do this without any visible issues right now.  Are you sure your plagiarism detection provider is setup in the Beta instance (I don't know if it automatically copies from production or not).

-Chris