Strategies with Multiple Markers

[MarkJeunnette]

Hi all,

I'm an instructor looking for some general advice and corrections to my understanding of options for large courses that use multiple graders to grade submissions.

Background. Assume...

• a 300-student course with an individual assignment
• students are split into tutorial sections of 30 students each
• assignment submission consists of one pdf file for each student
• single rubric defined for the assignment with 3 lines
• 3 graders to mark the assignment

Grading Option 1: split grading by submissions/sections

• each grader marks all of 100 students' submissions
  
  • Grader 1 grades submissions "Andrews" - "Hanson"
  • Grader 2 grades submissions "Henderson" - "Peters"
  • Grader 3 grades submissions "Pritchett" - "Watts"
• could also split along section lines (e.g., Grader 1 grades tutorial sections 1-10).
• Benefits
  • easier to organize
  • low risk of lost data from graders simultaneously marking the same submission
• Drawbacks
  • harder to maintain consistency in grading... one marker may be more generous than others, so fairness would require a post-review and potential adjustment of marks.

Grading Option 2: split grading by rubric line

• each grader grades ALL submissions for a particular line in the rubric
  • Grader 1 grades all submissions for rubric line #1
  • Grader 2 grades all submissions for rubric line #2
  • Grader 3 grades all submissions for rubric line #3
• Benefits
  • ensures grader consistency in grading, as we don't have multiple graders evaluating the same rubric
• Drawbacks
  • potential for data loss when two markers are marking the same submission
  • harder to track completion of grading, since SpeedGrader lists a submission as "graded" as soon as one line in the rubric receives a score.
  • could require double work if each rubric line requires reading the entire submission

I would like to utilize Grading Option 2, so am wondering if there are workarounds for each of the Drawbacks, or if utilizing something like Moderated Grading would work better.. Any thoughts or feedback would be helpful!