This document introduces Anonymous Moderated Grading as noted in the Canvas Production Release Notes (2018-07-14)
The current anonymous and moderated grading features have been updated to provide improved assignment flexibility with fully anonymized, multiple grader anonymity, and moderated options. Additionally, moderated grading has been simplified for a better user experience.
Anonymous Moderated Grading involves up to two feature options: Anonymous Grading and Moderated Grading. These features can be enabled individually or together to provide multiple options in the assignments page.
When one or both feature options are enabled, the assignments page displays several options according to each feature.
Currently moderated grading is included in all assignments by default, but this update allows it to be enabled for the entire account or on a course-by-course basis.
Accounts or courses that have set the current Anonymous Grading feature option to On will have the new Anonymous Grading feature option turned on by default. However, accounts or courses that have set the feature option to Allow must reset the Anonymous Grading feature option to Allow; by default it will be turned off.
The following permissions are available with each corresponding feature option:
If the required feature options have been set to Allow at the account level, individual feature options may need to be enabled at the course level. Graders and moderators may need to confirm the feature options are enabled in Course Settings.
In anonymous grading, graders cannot identify students associated with each submission, which helps remove grading bias.
When anonymous grading is enabled in a course, content creators can create assignments with the option to hide student names from graders.
Content creators can choose to anonymize instructor names in DocViewer-supported assignments. When the Anonymous Instructor Annotations option is enabled for an assignment, all annotations and comments made by instructors in SpeedGrader are anonymized. This option does not apply to annotations previously made to an assignment in SpeedGrader.
Note: Anonymous Instructor Annotations is not associated with the Anonymous Grading feature option and displays in all assignments. This option can be enabled by any content creator.
In moderated grading, anyone with assignment creation permissions specify an assignment submission to be reviewed by one or multiple graders, review the grades, and select which grade should be set as the final grade for each student. For each moderated submission, the user set as the moderator is the only user who can specify the final grade for each assignment. Any grader can be specified as the grader who can determine the final grade as long as they have the corresponding permission, but the moderator does not have to personally grade the assignment. Once the assignment is created, only the moderator and admins will be able to change the assigned final grader.
When moderated grading is enabled in a course, content creators can create assignments with moderated options:
Moderated assignments cannot be edited by any grader except the moderator. Any edits made to the assignment will be tracked. Auditing tools will be added to help with accountability in a future release.
When anonymous grading and moderated grading are both enabled, content creators can create assignments that hide student names from graders and include moderated options. All options for both anonymous grading and moderated grading are displayed for the assignment.
When the moderated grading option for graders to view each other’s comments is enabled, the Anonymous Grading options list also includes a second checkbox option: Graders cannot view each other’s names. This option allows graders to view comments from other graders but not view the name associated with each comment.
Both anonymous and moderated assignments are shown in the Gradebook in a read-only state with gray columns. Additionally, both assignment types are automatically muted.
Note: The Anonymous label supersedes the Muted label. Assignments that are both anonymous and moderated display as anonymous, though the Grade Detail Tray displays the assignments as being muted.
Grades are not displayed in the Gradebook until the assignment is manually unmuted. Additionally, to remove anonymity from each submission, the assignment must be edited and changed so that graders can view student names.
Grades cannot be edited in the Gradebook before grades are posted. However, after grades are posted by the moderator, grades can be edited in the Gradebook by any grader.
For grading, moderated and anonymous assignments are treated the same as any other assignment type. However, when an assignment is anonymous, some or all user name data is hidden.
Note: Anonymization and moderation options can be enabled or disabled at any time during grading, but audit logs will record the action and the user involved. Audit log visibility will be available in a future release.
In the Dashboard, all graders receive a To-Do notification when submissions have been received by students and are ready to be graded, regardless of assignment type.
All existing functionality is supported in SpeedGrader for both anonymous and moderated assignments.
The following restrictions apply to accessing assignments for each assignment type:
For moderating grades, moderated assignments mimic the existing moderated grading workflow but with simplified functionality.
In the Dashboard, the moderator receives a To-Do notification when at least one submission has been graded for a moderated assignment.
The moderator can use the grade summary page to view all grades awarded by one grader and compare them to grades awarded by another grader. Grades are grouped by grader name.
Note: An assignment cannot be moderated until grades have been submitted.
If the assignment is anonymous, or if the moderator cannot view other grader names, the names in the grade summary page will be appropriately anonymous.
In the final grade column, the moderator can select a grade from the menu from any grader. The moderator can also open the menu and type a custom grade for the final grade.
When grades are final, the moderator can post the grades to the Gradebook.
Note: After grades are posted by the moderator, grades can be edited in the Gradebook by any grader. However, grades in the Grades Summary page cannot be changed.
This document outlines existing functionality included in the initial release of this feature. Anonymous moderated grading will continue to receive additional feature enhancements over future releases as indicated in Release Notes. Please follow the release notes for future functionality updates.
Community feedback for anonymous moderated grading is welcome per Canvas Community Feedback Guidelines, which outlines general feedback, broken functionality, and feature enhancements.
For anonymous and moderated grading, feature enhancements should include the anonymous&moderated tag. Before submitting a new idea, please review all existing anonymous moderated grading feature ideas.
All feature functionality will be updated in the Canvas Guides on July 14. Comments are welcome for functionality clarification. Lessons will also be updated with all future enhancements as indicated in release notes.
The following notable behaviors have been identified in anonymous moderated grading. Known behaviors are being triaged by the Canvas engineering team—feature ideas do not need to be submitted regarding their functionality.
Information in this document will be updated as made available by the Canvas product team:
Whilst I am really pleased that this AMG development has moved forward - I do have some concerns.
One is that there still appears to be a mismatch of understanding between what the workflow is (the specification) and the product that has been built - sad and surprising given how much effort has gone into consulting with Instructure on this (particularly from the UK HEI Sector needing this)? For example, the opening line to Moderated Grading does not reflect the workflow that I'm aware of either locally, or from working with colleagues at other UK institutions 'In moderated grading, a moderator can specify an assignment submission to be reviewed by one or multiple graders, review the grades, and select which grade should be set as the final grade for each student.' This may just be semantics, but it conveys confusion. Commonly there are two 'graders' involved, a 'marker' and a 'moderator' (moderated marking), or a 'first marker' and a 'second marker' (double marking), the 'moderator' does not usually make the decision described above, and the decision process to the final grade is also not reflected above.
I do not quite understand the thinking behind the controls and effect around when grades can be changed (i.e. not before posted but after, grades in the Gradebook can be changed but this does not affect the Grade Summary view)?
However, my biggest concern is that it seems an 'anonymity leak' still remains. Below is a screenshot of the presented Gradebook view with AMG enabled:
It shows that the student name column shows, well... student names. I understand that the Gradebook could hold a range of assignment outcomes, formative, moderated, double marked (not blind / anonymous), fully anonymous. Maybe not all on the same course / module, but you could easily have formative, non-anonymous and anonymous assessment in the same course site / module, so would have reason to want to see the student names in some, but not in others.
I am concerned that this design leaks anonymity where there are few students submitting, particularly one. This is particularly because at the moment, Canvas only allows us to filter students in the Gradebook by Section, and this is actually a useful way to divide students into marking groups and filter down to them. It is also a meaningful way to handle students with extensions, where we need to assign them a later Due Date and possibly (for various reasons) a different marker. Under AMG (as links in the Gradebook go to random student papers in Speedgrader), it is a way of ensuring that markers can easily locate the individual students that need marking.
Below is an example Gradebook
By using the filter option in the top right I can filter down to the Extensions students (could be Extenuating Circumstances, a student resubmitting under Appeal, or any other reason, including extra-ordinary submissions that have to occur, or that only one student failed their first attempt and is submitting a second attempt paper/assessment).
And as a result, I can now see who that student is, whether or not it was an 'anonymous' assignment
Because at the moment we can only un-mute per assignment, I know that some colleagues are setting up new/duplicate assignments for events like extensions, so that they can better control the un-mute period. Such workflows will also present the same issue, as one student in an 'Extension Assignment' column will clearly have their submission in the same row as their name, whether or not the Section filtering I've described is used.
Some great improvements here! We currently tell our faculty to stay far away from moderated grading because of all the 'gotcha's. We may start reconsidering when these changes role out.
Once you are able to incorporate anonymous DocViewer comments into this process, I know a lot of our faculty want all of the grader comments to appear to students (not just from the grader whose score is selected - or from neither if the instructor types in an average), for both completed rubrics to appear to students, and some faculty want the ability to post average scores of graders (without having to manually calculate). Is any of this being considered as future work?
Or...perhaps pull it all into a additional option:
When the moderator is selecting between grader 1 score, grader 2 score, or manual entry, add an additional option which is the "Average" option. The average option will include all comments and rubrics from the graders. After posting scores, the instructor/moderator can remove any comment they choose, add their own rubric, and adjust scores to their heart's content.
A separate pain point: once a moderator chooses a grade, they can no longer see any comments/rubrics from the non-selected grader(s). This information should remain viewable to the moderator somewhere.
I'm happy to throw this stuff into feature requests, just want to plant a seed now while you all are really digging into this project just in case you can start building the hooks for the features now.
This feature isn't fully completed and will include a few more phases. Our team is aware of comments not being able to be seen by non-selected graders, and that functionality will come a little later when the teams implement full auditing functionality.
All feature ideas need to be submitted in our Canvas Studio space as indicated in the document, otherwise our product team won't see them. We appreciate your feedback!
Hi - great to see this functionality coming! I did some initial testing last week with no issue, but this week cannot see the moderated grading box correctly (testing in google chrome and safari - same view)
I'm glad that your initial testing went well with no issues associated with it! The issue that you present for the checkbox for Moderated Grading not appearing certainly seems odd. I have checked this myself and have had several other members on my team check it and we have been unable to reproduce it thus far. Are you still experiencing this issue?
...a moderator can specify an assignment submission to be reviewed by one or multiple graders, review the grades, and select which grade should be set as the final grade for each student. For each moderated submission, the user set as the moderator is the only user who can specify the final grade for each assignment.
1) Re: "...a moderator can specify an assignment submission to be reviewed by one or multiple graders..."
It looks one does not have to be a Moderator (or someone with "Select final grade for moderation"permission) to be able to select a moderator of an assignment from the list of those with grading privileges. As a Course Designer creating a new assignment, I was able to choose from the drop-down list the "Grader that determines the final grade."
2) And aren't the "Grader who can specify the final grade" and the Moderator really one and the same? If so, I'd find it clarifying if the label on the drop-down list read "Grader that determines the final grade (the Moderator)."
I don't mean this as criticism; just checking my understanding.
It is interesting wording, which seems to be predicated on the model that the 'professor, module/course leader, course convener etc.' is going to be the 'moderator' - e.g. that they will get other people to do the 'grading' and then check that by doing the 'moderation' (be the arbiter of whether the grading was good enough?).
Admittedly, that does happen (at least the 'leader' can sometimes be the moderator), but it's also frequently likely that when it comes to the marking load, the 'leader' is just one of the gang, who mucks in with everyone else. It's even more common that each marker has a colleague acting as their 'moderator' - and highly probable that the 'leader' isn't doing any of either marking or moderation.
I'm sure this could be styled a little more intuitively (and dare I say with better fidelity to practice?).
Just to echo other feedback, thank you for the developments in this area. Overall we feel they're great strides forward.
Of course during testing we have discovered some things that are probably worth highlighting - apologies if they've been covered already, and let me know if they need to go to an alternative forum / mechanism!
Finally, is there a way to get to the student specific submission in Anonymous Mode using the APIs - we've got some custom functionality where we'd like to drop someone to a specific submission? Any advice would be gratefully received.
- Integration with Plagiarism Detection Framework - its not possible to create an Assignment with a 3rd Party Plagiarism tool selected. I think this has been previously noted, but its not listed in the current known issues. Is there an indicative timescale for this to be rectified?
On my beta instance, I can do this without any visible issues right now. Are you sure your plagiarism detection provider is setup in the Beta instance (I don't know if it automatically copies from production or not).