Canvas Release: Anonymous Moderated Grading (2018-07-14)

erinhmcmillan
Instructure Alumni
Instructure Alumni
This document introduces Anonymous Moderated Grading as noted in the Canvas Production Release Notes (2018-07-14)

The current anonymous and moderated grading features have been updated to provide improved assignment flexibility with fully anonymized, multiple grader anonymity, and moderated options. Additionally, moderated grading has been simplified for a better user experience.

 

 

Feature/Integration Setup

Feature Options

Anonymous Moderated Grading involves up to two feature options: Anonymous Grading and Moderated Grading. These features can be enabled individually or together to provide multiple options in the assignments page.

 

When one or both feature options are enabled, the assignments page displays several options according to each feature.

 

Existing Moderated Grading Functionality

Currently moderated grading is included in all assignments by default, but this update allows it to be enabled for the entire account or on a course-by-course basis.

 

Existing Anonymous Grading Feature Option

Accounts or courses that have set the current Anonymous Grading feature option to On will have the new Anonymous Grading feature option turned on by default. However, accounts or courses that have set the feature option to Allow must reset the Anonymous Grading feature option to Allow; by default it will be turned off.

 

Account Management

Permissions

The following permissions are available with each corresponding feature option:

 

Anonymous Grading

  • Roles: Account, Course
  • Permission: View audit trail
  • Default: no user roles

 

Moderated Grading

  • Roles: Account, Course
  • Permission: Select final grade for moderation
  • Default: enabled for TA- and Teacher-based roles

 

Course Management

Course Settings

If the required feature options have been set to Allow at the account level, individual feature options may need to be enabled at the course level. Graders and moderators may need to confirm the feature options are enabled in Course Settings. 

 

Course-Level Features

Assignment Setup

Anonymous Grading

In anonymous grading, graders cannot identify students associated with each submission, which helps remove grading bias.

 

When anonymous grading is enabled in a course, content creators can create assignments with the option to hide student names from graders.

 

Notes:

  • This option can be edited and changed at any time by the final grader or an admin. Any edits made to the assignment will be tracked. Auditing tools will be added to help with accountability in a future release.
  • Currently, enabling this option displays SpeedGrader submissions as read-only and does not allow annotations or comments in DocViewer.

 

Anonymous grading assignment option

 

Anonymous Instructor Annotations

Content creators can choose to anonymize instructor names in DocViewer-supported assignments. When the Anonymous Instructor Annotations option is enabled for an assignment, all annotations and comments made by instructors in SpeedGrader are anonymized. This option does not apply to annotations previously made to an assignment in SpeedGrader.

 

Note: Anonymous Instructor Annotations is not associated with the Anonymous Grading feature option and displays in all assignments. This option can be enabled by any content creator.

 

Anonymous Instructor Annotations option

 

Moderated Grading

In moderated grading, anyone with assignment creation permissions specify an assignment submission to be reviewed by one or multiple graders, review the grades, and select which grade should be set as the final grade for each student. For each moderated submission, the user set as the moderator is the only user who can specify the final grade for each assignment. Any grader can be specified as the grader who can determine the final grade as long as they have the corresponding permission, but the moderator does not have to personally grade the assignment. Once the assignment is created, only the moderator and admins will be able to change the assigned final grader.

 

When moderated grading is enabled in a course, content creators can create assignments with moderated options:

  • Grader number—the number of graders who should grade the submission. This number does not have a limit. However, the page displays a message if there are not enough course users with grading permissions to fulfill the desired number of graders.
    • A moderator does not have to specify which graders can grade submissions. If a course includes a surplus of graders higher than the desired number of graders, any grader can assess an assignment submission until the specified number of grades has been achieved. When an assignment has reached the specified number of graders, remaining graders will not be able to access the assignment.
    • The moderator is never included in the number of available graders since the moderator does not traditionally grade individual submissions but may provide a custom grade.
  • Grader comment visibility—the option that allows graders to view each other’s comments. This option is selected by default.
  • Final grader—the moderator, or the grader who determines the final grade. The Select Grader menu displays the names of course graders who have been given permission to determine the final grade. This permission is assigned to one or more roles at the account level.
  • Final grader visibility—the option that allows the final grader to view the names of other graders. This option is selected by default.

 

Moderated assignments cannot be edited by any grader except the moderator. Any edits made to the assignment will be tracked. Auditing tools will be added to help with accountability in a future release.

 

Moderated grading assignment option

 

Anonymous Moderated Grading

When anonymous grading and moderated grading are both enabled, content creators can create assignments that hide student names from graders and include moderated options. All options for both anonymous grading and moderated grading are displayed for the assignment.

 

When the moderated grading option for graders to view each other’s comments is enabled, the Anonymous Grading options list also includes a second checkbox option: Graders cannot view each other’s names. This option allows graders to view comments from other graders but not view the name associated with each comment.

 

Moderated and anonymous grading assignment options

 

Gradebook Functionality

Both anonymous and moderated assignments are shown in the Gradebook in a read-only state with gray columns. Additionally, both assignment types are automatically muted.

 

Note: The Anonymous label supersedes the Muted label. Assignments that are both anonymous and moderated display as anonymous, though the Grade Detail Tray displays the assignments as being muted.

 

Moderated and Anonymous assignments in the Gradebook cannot be unmuted until grades are posted

 

Anonymous Assignment Details

  • Grade cells do not display any content, including the Needs Grading icons.
  • In the New Gradebook, columns display an Anonymous label in red text.
  • In the New Gradebook, the Grade Detail Tray also displays options in a read-only state and shows the assignment as being muted.

 

Grades are not displayed in the Gradebook until the assignment is manually unmuted. Additionally, to remove anonymity from each submission, the assignment must be edited and changed so that graders can view student names.

 

Moderated Assignment Details

  • Grade cells display dashes until grades have been posted.
  • In the New Gradebook, assignment columns display a Muted label in black text.
  • In the New Gradebook, the Grade Detail Tray also displays options in a read-only state and shows the assignment as being muted.

 

Grades cannot be edited in the Gradebook before grades are posted. However, after grades are posted by the moderator, grades can be edited in the Gradebook by any grader.

 

Grader Workflow

For grading, moderated and anonymous assignments are treated the same as any other assignment type. However, when an assignment is anonymous, some or all user name data is hidden.

 

Note: Anonymization and moderation options can be enabled or disabled at any time during grading, but audit logs will record the action and the user involved. Audit log visibility will be available in a future release.

 

Dashboard

In the Dashboard, all graders receive a To-Do notification when submissions have been received by students and are ready to be graded, regardless of assignment type.

 

To Do List for Graders

 

SpeedGrader

All existing functionality is supported in SpeedGrader for both anonymous and moderated assignments.

 

The following restrictions apply to accessing assignments for each assignment type:

 

Anonymous Assignments

  • When a grader accesses a SpeedGrader link, the first student displayed in the student list is always randomly generated for every assignment. Additionally, each assignment includes a random ID that cannot be traced back to any specific student.
  • Unless an assignment is moderated, assignments can only be graded by one grader per current SpeedGrader functionality. If a submission has already been graded, other graders can still access the submission and change the grade.
  • Comments in DocViewer display Grader instead of the grader’s name.
    • Note: Anonymity features do not hide any student identification that may be included directly in the submission.

 

Anonymous grading example in DocViewer comments

 

Moderated Assignments Grading Limit

  • For moderated assignments, when an assignment reaches its maximum number of graders, the grading notification will be removed from the Dashboard for graders who have not yet assessed the assignment. If a non-assessed grader opens the assignment directly, the SpeedGrader link will not display in the sidebar. If the non-assessed grader clicks the SpeedGrader link from the Gradebook, the assignment will redirect back to the Gradebook with a message that the maximum number of graders has been reached for that assignment.

 

Moderator Workflow

For moderating grades, moderated assignments mimic the existing moderated grading workflow but with simplified functionality.

 

Dashboard

In the Dashboard, the moderator receives a To-Do notification when at least one submission has been graded for a moderated assignment.

 

To Do List for Moderator

 

Grade Summary Page

The moderator can use the grade summary page to view all grades awarded by one grader and compare them to grades awarded by another grader. Grades are grouped by grader name.

 

Note: An assignment cannot be moderated until grades have been submitted.

 

Assignment with Moderate button

 

If the assignment is anonymous, or if the moderator cannot view other grader names, the names in the grade summary page will be appropriately anonymous.

 

Grade Summary page with two graders

 

In the final grade column, the moderator can select a grade from the menu from any grader. The moderator can also open the menu and type a custom grade for the final grade.

 

Grade Summary page selecting final grade

 

When grades are final, the moderator can post the grades to the Gradebook.

 

Note: After grades are posted by the moderator, grades can be edited in the Gradebook by any grader. However, grades in the Grades Summary page cannot be changed.

 

Grades posted to Gradebook

 

Community Resources

Future Enhancements

This document outlines existing functionality included in the initial release of this feature. Anonymous moderated grading will continue to receive additional feature enhancements over future releases as indicated in Release Notes. Please follow the release notes for future functionality updates.

 

Customer Feedback

Community feedback for anonymous moderated grading is welcome per Canvas Community Feedback Guidelines, which outlines general feedback, broken functionality, and feature enhancements.

 

For anonymous and moderated grading, feature enhancements should include the anonymous&moderated tag. Before submitting a new idea, please review all existing anonymous moderated grading feature ideas.

 

Feature Documentation

All feature functionality will be updated in the Canvas Guides on July 14. Comments are welcome for functionality clarification. Lessons will also be updated with all future enhancements as indicated in release notes.

 

Known Behaviors

The following notable behaviors have been identified in anonymous moderated grading. Known behaviors are being triaged by the Canvas engineering team—feature ideas do not need to be submitted regarding their functionality.

 

Information in this document will be updated as made available by the Canvas product team:

 

Audit Functionality

  • Full audit functionality that displays information associated with both anonymous or moderated assignments will be added in a future release. This functionality will ensure fair grading practices.

 

Mobile Apps

  • Anonymous grading is currently not supported in the mobile apps. Both the Student and Teacher apps need to implement the new feature options into their code, which will be available in a future release. However, the Teacher app includes a local option to enable anonymous grading on a per-assignment basis.
  • Moderated grading is not supported in mobile apps. No timeline for feature support is currently available.
Labels (1)