Assigning Intra-Group Peer Reviews

[James]

Quick Install

For those power users who are impatient, here are the quick install steps.

1. Install the Tampermonkey browser extension for Chrome, Firefox, or Safari
2. Install the Intra-Group Peer Reviews user script.
3. Navigate to a Peer Review page for an assignment and notice the Intra-Group Reviews section
4. Set the Group Set to use. This may not be necessary as it attempts to detect which one to use for you.
5. Click the Assign IntraGroup Reviews button.

If you run into problems, be sure to come back and read the instructions. In particular, if you're using a custom URL for your Canvas instance and not *.instructure.com, see the section on customization.

Introduction

It is sometimes useful to assign peer reviews to the other people in the group, rather than to people in different groups. I use them for students to provide appreciative and constructive feedback to their team members about their work in a team that week. Others may have other uses.

Basically, it's a pain to do this and since Canvas already has a list of who is in what group, it should be something that can be automated. That's what this script does.

Canvas Implementation

The current implementation within Canvas makes this very difficult. The peer reviews are assigned by name, not by group.

Common but Inefficient Workflow

The typical workflow would be to start at the top of the peer review page because that's where most of the work needs done.

I want to start with Bizarre Ability, but I don't know who is in her group. To find that out, I need to go to the People page and pull up the Accountability group set. I don't see Bizarre's name, what I see is this:

Except that I have 32 of those groups and have no idea which one Bizarre is in. I start expanding them ... one by one ... because there is no button to expand them all.

I finally find her -- she was in group 30 out of 32.

Now I switch back to the Peer Review page. I click on the button to Give Bizarre Ability another submission to assess and am presented with a dropdown list of all of the students in the class.

I scroll all the way down to S, looking for Selfish Hog, before I realize that the list of names on the People page is their first name followed by the their last name while on the drop down list, it's last name followed by first name and I should really be going to the H for Hog.

All right, I finally get Hog, Selfish and Sentence, Clicking added to Ability, Bizarre.

Now it's time to move on to Ruby Accolade, I mean Accolade, Ruby. Except I'm going to be smarter about it. Before moving on to Ruby, I'm going to go take care of Selfish Hog and Clicking Sentence while I know who is in their group.

Slightly Better Workflow

What I just described wasn't the most efficient way. What would be better would be to start on the People page with the first group.

Then I go to the Peer Review page and take of Radical, Stellar, Spiritual, and Unspoken. Then I repeat that process with the second group. Then the third, and so on, until all 32 groups have been assigned.

End Result

No matter which method I use, I still end up with assigning 355 peer reviews by hand. That takes way too long and makes me reconsider how important it is that I do intra-group peer reviews.

Before you argue that you would never have 32 groups, I had another groupset with only 12 groups of about 10 students each. That turned into 1141 peer reviews that needed assigned.

Of course, this wasn't my real class. My real class only has 30 students in it and had 103 peer reviews, but some people have larger classes and the problem of assigning intra-group peer reviews is even worse than I've described.

If there are n students and the average group size is k, then there will be about n*(k-1) assignments that need made.

Automatic Assignment

Now that we've seen how Canvas does things, let's look at what this user script does.

It adds a new block below Canvas' Automatically Assign Peer Reviews

You may need to pick the group set.

You click the button and you wait. There is a progress bar that shows while it's working so you have a sense of how much more there is to go. Each advance represents one student being assigned their peer reviews. There's a slight pause at the end while it catches up (probably inefficient programming on my part) and then the page reloads.

When the page reloads, you now have this.

Those 355 peer review assignments took my script about 30 seconds to complete. The 1141 peer reviews for the other group set took about 58 seconds. That was on a the test instance of Canvas over a DSL connection, your results may vary.

Video

I made a video so you could see it in action.

The Little Things ...

There are some things you may have not noticed, possibly because they weren't mentioned.

Automatically Picking the Group Set

If your class only has one group set in it, then it defaults to that group set rather than making you choose it. It also defaults to a particular group set if the assignment you're attaching the peer reviews to is a group assignment.

Proper? Sorting of Group Sets

The list of group sets is sorted alphabetically. I had, in order, Accountability, Communication, Project 1, Project 2, and Project 10.

Wait a minute! You know from other frustrations with Canvas that Project 10 is between Project 1 and Project 2. Well, alphabetically it is, but I threw in a little hack that will work most of the time except when it doesn't work.

The trick was to left-pad all of the numbers with 0's and then take the right-most 5 digits. 5 was an arbitrary number, but I figured that someone wouldn't have more than 100,000 group sets in their course. However, if someone named their group sets by zip+4 codes, which are 9 digits long, then it would not look good. Those people are free to modify the code to add extra 0's and use a longer string.

Interrupting and Restarting

The code only runs while you are on the page. That means that if you leave the page while it's assigning the peer reviews that it will not finish.

Luckily, all you need to do to finish the job is to come back, reload the page, and click the button again. It scans the list to see which peer reviews still need to be assigned so it picks up where it left off.

It only reloads the page if it assigns peer reviews. If you are completely done assigning and you try it again, it will show the progress bar at 100% and leave it there.

Reviews Assigned to the Entire Group

This program does not try to assign each a fixed number of peer reviews like you can do with Canvas' automatically assign. It just assigns peer reviews to everyone who is in your group. If you're on a team with 3 people, you get 2 peer reviews to complete. If you're on a team with 10 people, then you get 9 peer reviews.

Customization

This should work out of the box for people who are hosted by Instructure and have their Canvas instance ending in instructure.com. If your institution uses a custom domain like canvas.myschool.edu, then you will need to modify the // @match statement on line 5 to match your instance.

The only other customization is a Boolean flag on line 12 called reloadPageWhenFinished. It defaults to true, meaning to reload the page when finished. If you don't want the page to reload, then change that to false. I figured most people wanted to see the changes when they were made, but I made it a configuration option just in case. 

Technical Details

Most people probably don't care, but this is the first script I've written that tries to use a more modern version of JavaScript including Promises, the Fetch API, const and let instead of var, and arrow functions. It results in cleaner looking code and those things are supported on recent versions of the major browsers, but it probably won't work on older versions of the browsers.

Anyway, as my first script, I may not have everything right, especially with the Promises. But it works and is definitely easier than manually assigning the peer reviews, so I'm making it available to people.

Submissions are Still Required

This script does not attempt to get around the Canvas requirement that a submission be made before students can complete the peer reviews.

In my class, I have students provide feedback using peer reviews on an assignment different than the one they're turning in. That way I can assign a different rubric to their peer reviews than what I'm using for the project. As a separate assignment, there is no submission. It turns out that in Canvas, the peer review links that you get by going to the assignment page are tied to the submission for that assignment. For on paper or no submission assignments this doesn't happen until the instructor puts a grade into the grade book.

Even when there is a submission allowed, Canvas doesn't show anything until there is a submission made. Here is what Bizarre Ability sees from the assignment page before she has turned in the assignment.

There is no list of peer reviews.

However, if she goes to the To Do list that we tell students not to use, then she sees this:

There are two peer reviews that she needs to complete, but she has no idea who they are for.

When Bizarre goes click on the first one, she sees a screen and the only place she knows who she is responding to is in the breadcrumb.

This probably helps explains why one of less-tech-savvy students put feedback for all of the peer reviews into a single submission comment.

At least the item disappeared from the To Do list once it was completed.

What I've started doing in my class is faking a submission for every student and assigning them a default grade in the grade book. Of course I use 0 because I want them to know what will happen if they don't complete the assignment, but then you have to deal with "Why did I get a 0 for this?" because they didn't read the instructions on the assignment because they clicked on the To Do list instead of the assignment.

Once the grade is entered, then they see the submission details and the list of students who they need to peer review. In this case, Bizarre had already provided feedback to Clicking Sentence from the To Do list.

Update 2021-02-03

When I wrote the script, the browser limited the number of concurrent connections to 5 or 6 per host. That built-in limiting was enough to keep from hitting the Canvas throttling restrictions. Since then, things have gotten more efficient and the script began hitting those limits and then failing.  @mjennings has been after me for a while to fix this, but it hasn't been as a high priority as teaching my classes. I added throttling to another script a couple of days ago and decided to tackle another one while my mind was still on how to do it. The change could have taken less than 10 lines of code, but I took time to rework and update the script.

Canvas has an x-rate-limit-remaining value that starts at 700 and if it gets to 0, then it fails and basically stops responding. You can avoid reaching 0 by limiting the number of concurrent requests and staggering the requests so they don't all hit at the same time.

The script now uses the Bottleneck library to restrict the maximum number of concurrent requests to 10 and makes sure that there is at least a 20 ms delay between requests. In my beta instance, I tested a course with 369 students and 74 groups of 4-5 each. It took about 4 seconds to load all of the students and then about 45 seconds to assign 1472 peer reviews. The x-rate-limit-remaining never dropped below 569. This is actually faster than when I wrote the original script and it took 58 seconds to assign 1141 reviews (although using the test instance instead of beta).

There is the potential for speeding it up even more, but I decided to play nice with the API, siding on the error of taking more time but making sure it worked as opposed to doing it as quickly as possible. If you want to speed it up, then change the debug constant to true and it will log timings to the console in the browser's developer tools.

I also cleaned up some code through a linter and found a bug in detecting the next link for pagination.

References

There are a couple of threads in the Community that are related to this.

• https://community.canvaslms.com/ideas/4789-assign-peer-reviews-by-student-group is a feature request on Product Radar.
• Is there a way to automatically assign intra-group peer reviews? is a discussion where I saw other people had the same idea and asked some questions as I was writing this.

[scain]

Thanks  @James ‌, this is fantastic! I had been supporting a class that wanted to do group structured peer reviews like this and they ended up having to abandon it because of the time it required to set it up. This is perfect for their needs!

[isherman-youngb]

This is excellent--I was one of the folks who had requested this function a while back. This new script promises to save me a huge amount of time!

[Shar]

Thank-you once again  @James ‌! This canvancement is definitely worth a 3am write-up!! I've had to set up the intra-group peer reviews the long way for instructors by providing them a list of their student groups and reminding them to do it all manually :S  So I delegated to them, and they delegate to the TA, no one wants to sit there doing all that assigning. I did it once because they were slow getting on it and the students needed it-- I wonder if that was intentional. But yes very painful to do it manually and not make a mistake.

But now with this script... hmm do I teach them how to install it in their browsers or do I save that magic for myself so I look very wizardly. heh I'll just let them know there is some mercy and kindness in the universe and there is a way to do it very easily so no more shying away from the possibility.

Did I say thank-you? Thanks a bunch - that's 160 thank-yous for the batch of intra-group peer reviews I had to assign manually.

And just so I'm clear on the steps to tell instructors:

1. make the assignment, give default grade of 0 to everyone

2. run the script for peer review

3. change the grade after the reviews are complete

Cheers - Shar

[GideonWilliams]

This is astonishing and of massive help to so many departments in our school. Such a good idea too. Comprehensive explanation and video.

Many thanks for giving your support and time so freely to the community.

[James]

ishar-uw,

The three steps are certainly 1 way to accomplish the intra-group peer review. It depends on the settings and this is where flexibility makes it difficult to get a single set of instructions.

In my class, the peer review is different from the group assignment, so yes, those are the three steps I follow -- plus a huge note in the instructions that remind them that peer reviews are not supported in the mobile apps and they will need to use a browser and that their grade will be a 0 until the instructor gets the assignment graded.

Another way I've seen people work around it is to make the submission be a text box that requires the students to put something, anything, in and then they get to see the peer reviews.

If a teacher wanted the students to comment on the paper they had written and turned in as a group assignment, they may want them to submit before they can see the peer review.

There are other imperfect solutions as well. If you don't want the 0 to freak students out, you could tell it not to count the assignment towards the final grade or to make it worth 0 points and then change it later. I didn't take that approach because I felt it would encourage students argue that it was my fault they didn't do it since since it didn't count or was worth 0 points. 

This thread also goes to show what happens when you assume things. I wrote it in the higher ed group and then shared it with Canvas admins and K12. I thought about instructional designers, but figured it was something that teachers would be doing after the class was going and the instructional designers wouldn't be using it. It seems that many of the comments are coming from IDs. Sorry about that.

[James]

I thought of and tried another trick today. I muted the assignments. That way they don't get the 0 bringing their grade down but they can't argue that it wasn't worth anything and so they didn't do it. I typically don't mute, so I hadn't thought of that one.

Working through that process with the class brought up two other concerns -- now 12 weeks into a 16 week semester. The first was "why does Canvas say that my 10/10 isn't going to be counted towards the final grade?" My initial reaction was "what a wonderful spot to be in" but the student didn't find that helpful, so I went ahead and provided the full explanation. The second is "why do I have a 0 for this assignment [that I hadn't graded]?" Because you accidentally typed 0 and used the what-if grades in Canvas so you told it to pretend that assignment was a 0.

Neither of those is directly related to the issue of peer reviews, but it hopefully shows that there is a disconnect between where we think the students are and where they actually. And while that orientation that we make them complete (at our school) before they get into the class may benefit some, it's obviously not working as well as we thought it was.

[Shar]

I mute assignments all the time, I generally try to setup the course with the assignments already muted. And it's so easy to do with the Adjust All Assignment Dates on One Page

So you're saying the assignment can be muted and it will still show the student the names of the peer reviews they need to complete on the assignment page? The way we've used Peer Reviews with the instructors who wanted it was a 3 part assignment:

1. Student submits their draft. peer reviews other drafts using the assignment rubric. Graded for completion by Instructor/TA.
2. Student submits a text-box write up of the reviews they completed and what they learned from looking at others. Graded by Instructor/TA.
3. Student submits the final version incorporating the feedback from the reviews. Graded for content by Instructor/TA with rubric.

If just the mute will still show who they have to review, then that's grrrreat! I'll test and get back to the thread once I have screenshots to show to my instructors as well.

I do like the 0 though because then the students cannot get away with not turning in a draft. Hmmm or maybe a default of 1. I guess it will depend on the impact to the instructor's overall grading. We have some instructors who use straight points already portioned into weights and others who (wisely) use grading categories.

And we do strongly believe our students use the To-Do list so we've got Peer Review instructions that reference it showing up in the to-list and on the assignment page to see who specifically.

Cheers - Shar

P.S. Why is Bizarre Ability a girl?

[James]

I went in and tested to make sure. Accountability Paper is muted in the gradebook and the top row is Bizarre Ability.

On the assignments page (the list of assignments), it does not seem to affect what is displayed, whether muted or not. I don't see peer reviews assigned there at all.

On the To Do list, no, it does not show the name of the student -- whether muted or not.

On the assignment page, it does the names of the assignments, even when muted, provided there is a submission for them.

That last comment is because it is muted. Which means students may not be able to see the feedback that others have provided. To confirm, I went to Clicking Sentence and the feedback from Bizarre Ability was not visible while the assignment was muted, but became visible as soon as I unmuted it. That may be good if you want them to wait until everyone has had the chance to provide feedback, but it might be bad if you want the receipt of feedback to encourage them to provide it others.

There's never a simple solution is there?

You're putting way more thought into this than I did. I just wanted a way to assign intra-group peer reviews but thought I should mention that there still has to be a submission for them to show up to the students on the assignment page.

For Bizarre Ability, I'm merely using the response she provided at the beginning of the semester when I asked what pronoun set (open ended question) the student would like to be addressed by that semester. Or maybe it was a he but I want to mask his real identity so I changed it to a she for purposes of the video. Inquiring minds will never know.

[pimmel2]

 @James ‌ Thank you so much for all the wonderful things you have created for the Canvas Community!  

I have a couple of questions / feature requests....and NO is a perfectly acceptable answer!

1.  Would it be possible to make the intra group reviews be anonymous?

2.  For one of my group projects I have the students fill out a rubric for each student.  The questions in the rubric consist of 5 "ratings" questions ie, "on a scale of 1-10, how responsive was this student to communications from group members", and a couple of short answer questions.  I then collect all of the responses, crunch numbers, then return to students anonymously. 

As you can imagine, this is a fairly time intensive project for me...especially in a class of 100+ . I am just wondering if this script could handle something like this.

Thanks again for all of your hard work...it is appreciated!

Pat Immel

[James]

 @pimmel2 

I'm not sure I follow.

The checkbox to select Anonymity is when you set up the peer reviews, not when you assign the peer reviews. That is a totally separate process from what my script does.

I don't use the anonymous feature, so I had to look up the page in the guides to see how they worked: How do I use peer review assignments in a course? 

Unfortunately, that didn't help me understand your question. The anonymity goes both ways, you can't see who you're reviewing or and the person who got the review won't know who provided it. But if you want them to provide feedback to each other, then they would need to know who they were reviewing.

I am going to make a guess about what I think you're saying, but this is definitely prefaced by my personal experience and what I've done in the past, so it may not be anywhere close to what you're talking about. I apologize ahead of time for being obtuse.

There have been times where I have asked each student to tell me about the other students in their group in an effort to determine the group dynamics, who contributed, who slacked, and so forth, so that I could assess group participation. I had them write a paper with a paragraph about each student's contributions, including their own, and give a score for each student. I contributed my own score for how well they completed this assignment. Each student's final result was the average of what the students scored and my score. I then provided that information back to the student as a summary result.

Is that anywhere close to what you're asking for?

I was never able to get Canvas to do that with rubrics, but I haven't used peer reviews much until this semester (I did a few times a couple of years ago, but they were a pain to do with peer reviews so I gave it up). If you have anonymous peer reviews, then the student would not know who they were providing the feedback to unless there was an assignment turned in that had the student's name visible -- and just that student's name without the names of the other people in the group.

I suppose that if you had an assignment with a Text Entry submission that asked for their name, then you could enable anonymous peer reviews and assign them to each member of the group. You could then fetch the rubric results through the API and compile a composite score. But the students would still be able to see the individual rubrics that were filled out -- unless perhaps you muted the assignment so that they couldn't see the feedback.

If all you were after were the rubrics so that you could compile an anonymous composite result, that you could make an assignment with peer reviews but not anonymous, mute it, assign intra-group peer reviews. Let the students complete the feedback. Then you do your processing to come up with a composite score that you want to give them anonymously. You then delete the assignment so that it can never be unmuted. This last paragraph is untested -- I do not know what happens to peer review rubrics when an assignment is muted.

[Shar]

Hi  @pimmel2 ,

It sounds like you're after group ratings rather than group peer reviews. We've had a few successes using a Google Form with the group member names already populated and some selective branching and using Canvas quiz with Fill-in-the-blank to input the group member name and rating.

The spreadsheet version of the google form can do the math calculations for you. We actually did an iframe-embed of the google form into a Canvas survey worth a few nominal points for completing the survey. I do not know if the instructor provided the feedback back to the students, but I do believe the rating scores were compiled (averaged?) to put into a Canvas gradebook assignment.

With the Canvas quiz, you can see the quiz statistics in the responses... I'm actually not sure what the instructor does with the canvas quiz group rating-- I don't know if the score and feedback gets back to the student or if the person providing the feedback just gets a score for rating.

At any rate, is that what you're looking for? A way to have students rate each other in the group and then you collect the responses to give back to the group members?

Cheers - Shar

[skhagen]

 @James , could you elaborate on the phrase, "What I've started doing in my class is faking a submission for every student"?

How do you fake a submission? (And is there a workaround for an instructor to upload a file on a student's behalf?)

Thanks for this script!

[James]

 @skhagen , I go into the gradebook and set the default grade to 0. That is sufficient to get the Submission block that contains the list of peer reviews to complete to show up. Whether you need to do that depends on how your peer reviews are set up, but it's necessary for mine.

There is a feature request on Product Radar asking for the ability to https://community.canvaslms.com/ideas/1124-submit-an-assignment-on-behalf-of-a-student" modifiedtitl.... I don't want to say there's no way to do it. Certain roles (typically admin roles) can masquerade or act as another user to submit some types of assignments, but in general, it's not easily accomplished.

[Boekenoogen]

This looks very interesting. We are looking at doing more peer review. 

[kbink]

I've two things to add, after of course another THANK YOU!  So awesome.

First, you can now assign peer reviews without a submission!  Not sure when then changed that but we discovered it a few months ago.

Second, I couldn't get tampermonkey scripts to work in the past.  I was working with the developer of the chrome quiz printing extension and we discovered that it is our vanity url (canvas.umn.edu) that was not working.  All I had to do was go to my page in canvas and change from the canvas url to the umn.instructure.com url and it worked.  So I decided to give that a try with this tool, and voila!  the tampermonkey script worked.  I'm not sure if anyone else has run into this problem so I thought I'd share.

Thanks again for taking what was an hour's work and making it 5 seconds.  More so this makes it possible for me to do group member evaluations in some of our large classes with several hundred students too!  Awesome work!

Kalli

[James]

 @kbink 

You can modify the // @include or // @match line of the script to refer to the vanity URL (in this script, it's match, but some of my scripts use include). This way, you don't have to switch URLs to get it to work, it's just part of Canvas. This is briefly documented in the "customization" section above.

It's nice to know that you don't have to have the submission first. I'll have to do some testing before I change the instructions to make sure I understand how that works.

[mjennings]

 @James 

Is there a tweak that could be made that would make it "INTER" group instead of "INTRA" group, so that it will automatically assign based on people not in their group (ie reviewing something posted by someone not in their group).

Backstory - 

Instructor wants to have a group project and each group would post a short video to a discussion board. Then each group member would be assigned other videos (not their own to review). 

In writing I am seeing a potential flaw as only one group member would be posting for the entire group so this would have to account for which person has actually posted so they could review. We could have each team member post the a copy of their video but not sure if that would help.

Any thoughts on the feasibility of this? 

[James]

 @mjennings ,

I used to do this in my statistics course many years ago (before writing this script). It wasn't as straight forward as it seems, but it wasn't so much the group thing you're concerned about unless Canvas has changed the way things work since then. I'm working from about 4.5 year old memories here so I may have something wrong. I did go back to Spring 2015 to double check some things and that doesn't mean it hasn't changed since then.

The problem was that even though it was submitted by a group, the review was of an individual and only that individual could see the reviews.

For example, let's say that we have Team 1: Alice, Bob, and Carol; Team 2: Dave, Ed, and Fiona; and Team 3: George, Hilary, and Iona. In each case, let's assume it's the first person alphabetically that turned in the group assignment. The person who turns it in is irrelevant.

I cannot tell Dave to review Team 1's submission. Furthermore, when I'm assigning the peer reviews, I have no way from that screen of knowing who is in what group, so I had to have a list of group members somewhere else.

I had to tell Dave to review Alice or Bob or Carol as an individual. If I assign Dave to review Alice, then only Alice can see the review from Dave; Bob and Carol cannot. That means that I have to mix it up and assign Dave and George to review Alice, Ed and Hilary to review Bob, and Fiona and Iona to review Carol. It doesn't have to be that exact match up, but I wanted each person to have some feedback in case the three of them couldn't get together to go through things. I would hate for all of the feedback to go to Bob and then have him be a non-existent member of the group and others be left reviewless.

Back then, all of this was done in an Excel spreadsheet making sure that I balanced the names (both on the giving and receiving end) and kept people out of their own groups. Mine was complicated even more because I had two sections combined, so I made section 2 review section 1 and vice-versa. That way when it came time to give the presentations, they hadn't already read about it.

It wouldn't be as much of a tweak as a rewrite, but one could use it as a starting point.

I know I try to come through for you when I can, but I do not expect I can help this time, certainly not in enough time to help your faculty. Our school just decided 2 hours ago to extend spring break a week (it was set to end this week) and go with online instruction because of concern about the cororavirus. Since all of my classes are face-to-face, my spare time just dropped to nil. I haven't reached full panic mode yet, but I'm sure it will come quickly.

[mjennings]

 @James 

Thanks for the thoughtful reply. I ABSOLUTELY understand as my university just extended our spring break (starts next week and is now two weeks long) and then is coming back as online for at least another week after that. I, in no way was expecting a solution, especially within the timeline my instructor needs it. It was more just a question of feasibility  or just in case there was a quick change of code that my make it work.

Focus on your classes and taking care of that wonderful family of yours... and wash your hands. 

[mjennings]

@James 

Are there any inherent limitations based on the number of users? I have a very large course with 449 students split into 90 groups (I know that's a lot). Every time I am running this it seems to "stall" out.

Thoughts?

[James]

@mjennings 

Matthew, there are two things that I can think of that might be affecting this. I haven't looked at it in a long time.

There is a limit on the time that a script can run inside a browser. I don't think that's it. It should be able to handle that load within the allowed time.

The second thing I can think of is that I used to rely on the browser to limit the number of connections that are being made to put in implicit throttling. Now that HTTP/2 is getting used, Canvas doesn't seem to have that limitation and I'm hitting the API rate limiting. When that happens, Canvas stops accepting requests for a while.

To test the second theory, try opening up the developer tools in Chrome (F12) right before you run the script and switch to the network tab. With that open, try running the script and see if it starts generating errors (I think the requests show up in red).

[mjennings]

@James 

I am definitely getting some errors. But I am not sure how to read them. Attached is a screen shot. If there is a better way to share this data, please let me know.

[James]

@mjennings 

The 403 errors are forbidden and what will happen if you exceed the API rate limit threshold. You can click on the "users" at the top (preferably the one in red) and look for the response headers. It should contain an x-rate-limit-remaining (or something like that) that is 0 or so close to 0 that there was a request that pushed it over the top.

The solution is to limit how quickly the requests are made. I've used the Bottleneck library to do that in some of my more recent projects (maybe in the one that fetches the rubric results for an entire assignment), but I don't know that I've published any of them.

You may have noticed I'm not in the Community as much as I used to be. I'm swamped with school work (teaching online doesn't give me as much free time as when I was teaching face to face) and the new Community and me do not get along. I do not have an ETA on this, nor even an ETS (estimated time to start).

Out of curiosity, what happens when you run the script multiple times. Does it finish the ones the that it didn't pick up the first time? (you may want to try that in the Beta or Test instance)

[mjennings]

I completely understand the swamped and adjusting to the new community. I have been a bit lax here in the community as well over the last six months.

When I run this multiple times, nothing happens and it just sits looking like it is still loading. I have left it for over an hour and nothing changed. The additional students are never assigned peers to evaluate.

When I click on the first user error I got an x-rate-limit remaining of -79.72145190569995.

I appreciate any help.

[raw]

How does one change the @match?

When I try on this page  "chrome-extension://dhdgffkkebhmkfjojejmpbldmpobfkfo/ask.html?aid=8e74f718-627a-4939-83d3-229b543aad0c" it won't let me make any changes. 

Nevermind I figured it out!  

in case anyone is wondering after you install it.  Click on the extension from the chrome toolbar, then click on the installed scripts tab and it will show you all your installed scripts.  There is an edit icon on that page.   

Once you edit it click on file and save.

[adessler]

Howdy!  This looks really great, but I can't get it to run on my page.  I hate being the guy who can't get software to run, but here goes ... I installed tampermonkey and the peer review script, then I edited the header to show:

but when I get to the peer-review assignment page, the intra-group button does not show up and if I go to the tampermonkey menu, it says "no script is running".  I'm sure I'm doing something dumb ... any suggestions?

[adessler]

nevermind — I just saw that I had tamu.com in the @ match statement.  changed it to tamu.edu & it's working.

[James]

@mjennings 

Matt, I updated the script to throttle. Sorry it took so long.

Once Tampermonkey updates, it should handle your needs without hitting the rate limits. I couldn't remember if you made any customizations to the old code, but it's a substantial enough change that it may not remember your customizations.

More information is in the Update 2021-02-03 section near the bottom of the blog post.

[mjennings]

@James 

You are the absolute best!! Thanks for the update. This will really help me out. I don't think I had any customizations to this one, so I should be good to go.

Thanks again!

[rvoronov]

Hello, the plug-in helps a lot to assign the intra-group peer reviews. Thanks!  However, after spending all the time on setting up rubrics and installing this plug-in, I've run into a brick wall with the anonymity issue (which I believe has been raised here before, and is also the first bullet point here:  https://community.canvaslms.com/t5/Question-Forum/Is-there-a-way-to-automatically-assign-intra-group... ).  I DO want the students to be able to see whom they are reviewing within their group (obviously they need to know which group member they are reviewing); but, I do NOT want them to see the names of the people who scored them.  For example, that would create problems with future retaliation, if the students who graded each other poorly in one group happen to meet again in a different group later in the semester.

It seems like Shar suggested above using some combination of Google Forms and Canvas quizzes, but that's just another project (e.g., how do I send a bunch of these forms to each group member and make them unique to their particular group composition, while at the same time somehow protecting these forms with a unique identifier to prevent abuse...).

I don't understand why there is not a solution to this seemingly common and simple scenario, where one way anonymity is required for intra-group peer reviews, instead of two way.

[James]

@rvoronov 

What type of assignment are you having the students do a peer review on? The document they submitted as a group or a separate, non-group assignment?

I haven't tried this, but it could work:

Since a submission is required for the students to do a peer review, you could have the students submit a text-field assignment that has their name in it. Then make the peer reviews anonymous, Since the students can see the document they're providing the peer review for, they would be able to see the name of the person they're reviewing as part of the document but the reviewers wouldn't know who was doing the reviewing.

With a few exceptions, I think that you shouldn't be a coward and hide behind anonymous feedback. People shouldn't say anything behind someone's back that they wouldn't say to their face. I felt this way before our institution went through accountability training and the importance of feedback, but it's been reinforced since then. Anonymous feedback is too easily dismissed through the filtering we do when provided feedback.

[rvoronov]

@James , thank you for the suggestion.  It seems to be working!  One counter-intuitive thing I had to do to make it work was to make it an individual assignment, instead of a group assignment (in case of the latter multiple peer reviews would receive the same name label of the student within the group who submitted the assignment last, because canvas accepts only one submission per group in this scenario; and I guess it takes the latest one, given that there unlimited attempts).

Now the issue I am having is how to export the results of the peer reviews from Canvas to something like an excel file.  For example, I would like to average the peer review scores that each student received from their groupmates, in order to assign an individual grade for each group member.  But I really don't see any type of export options for the peer reviews.  So, I am wondering whether you have any suggestions for how to automate the peer-review-based grading? Thanks

Regarding your philosophical point that one shouldn't be a coward, I'll just say that the way course evaluations are done at the end of the semester is exactly the same one-way anonymous mechanism as what I am trying to do here.  And the reason why they are done this way is so that the instructor can't go back and retaliate against the students after reading the reviews.  This makes sense and is probably the safer thing to do from the legal point of view also.  Imagine an extreme scenario where students get into a fight over a peer review, and you would be held responsible for enabling it.  It just not a good idea.

[James]

It seems there are a couple of things I've done over the years to get peer review feedback out. I haven't used peer reviews in a quite a while, though.

One thing I did in fall 2018 was to save the speed grader information as a JSON file. Then I went through and parsed it. The speed grader information is not available through the API, it's loaded when you open up SpeedGrader. I'm doing this in Chrome, but other browsers have similar capability.

1. Open SpeedGrader.
2. Press F12 to open the Browser's Developer Tools. Switch to the Network tab and select XHR instead of All.
3. Reload SpeedGrader (Ctrl+R in the devtools will do it). SpeedGrader opens in a new window and the data was already fetched before you could get to it, so we have to reload the page.
4. Look for network traffic with a name that starts with speed_grader.json?assignment_id ...
   
5. Select that name and then click on Preview. What you are seeing is the JSON that drives everything that SpeedGrader users. Since SpeedGrader can display the rubrics and feedback for the peer review, it's in this file.
6. To do anything useful with it, you need to copy it and then paste it somewhere. You can click on Response and then select all (Ctrl/Cmd+A) followed by copy (Ctrl/Cmd+C) or you can right click on the name, select Copy, and then choose Copy response.
   

After you save it in a file, preferably something with a .json extension, you can use code to parse the file.

Here are some things to look for.

• The submission comments that peers left are under submissions[i].submission_comments[j] where i is for each student who received feedback and j is for each student who left student i feedback.
• The rubric entries and comments that peers left are under context.students[i].rubric_assessments[j].data

There are tools (including online ones) to convert JSON to CSV so that you can bring it into Excel, but I wouldn't try them. This file is one object with lots of parts and the tools that convert to CSV work best with arrays of objects rather than objects with objects that have arrays. It might work if you took each student and copied just the rubric_accessment object for each one, but then you would need to to that for each student.

Writing code is really going to be the most efficient way extract this.

We also have anonymous student feedback at the end of the term, but there wasn't universal agreement on it when we implemented out system. The compromise was that deans could take no disciplinary action based solely on the anonymous feedback provided. My take is that faculty who retaliate against students should be held responsible for their actions. Perhaps teaching isn't the right profession for them? I understand that some people want anonymous feedback and Canvas  supports it. When I have students review other's work (we're doing that in a discussion this week), I tell my students something like this: "Ultimately, it is your work and you are responsible for what you submit. You need to decide whether the person providing the feedback has a legitimate point. Just because someone says something does not make it true." I don't give examples, although both sides of our political spectrum provide plenty, but we have already talked about Daniel Kahneman's Thinking Fast and Slow and how people who blurt out answers quickly may not be right.

[TimJensenUSC]

hey @James,

Thanks for everything you've contributed to the Canvas Community.

I have a developer question— Outside of Tamper Mokney, is there a way to 'install' your Canvancements (and my own that I want to build in JS) into Canvas? I have Admin access.

Thanks,

Tim

[James]

Tim ( @TimJensenUSC )

You're welcome.

Many of my scripts will work without modification if added to the custom global JavaScript through the theme editor. I don't test them there, but others have reported success. I try to program a check for the page it runs on inside the code itself rather than relying on the metadata for the userscript. It may seem redundant as a script, but it's there in case someone wants to run it for everyone.

[TimJensenUSC]

Thanks @James.

Yeah I can see how Tamper Monkey comes in— you don't want to be uploading a global script into the Theme every time you want to run/test your code.

I've installed Tamper Monkey, and taken a look at one of your scripts and now understand what you mean by the metadata feature in Tamper Monkey - its a great browser tool isn't it!

I'm working on my first userscript for Canvas called 'Canvas Module Context - Basic Page Context for Modules in Canvas'. I must say I was so amazed to discover this isn't already in the UI for the Module view- it's an essential in a higher ed context in my opinion.

[Hi212]

Can anyone tell me if this script works on Macs (Big Sur 11.6)? I followed steps 1-5 in order, and tried loading Tamper Monkey and the Intra-Group Peer Reviews user script onto Firefox v92.0.1 (and on Opera v79). I got confirmations that both loaded, and I restarted the browsers, but the Intra-Group reviews block isn't showing up. Thanks much!

[kbink]

@Hi212   If you have a vanity address for your Canvas, you may need to login to your instructure.xxx.com address instead and the button will show up then when you navigate to the assignment.

[James]

@kbink 

Thanks for helping out. That catches some people and with the inability to see what domain people are using, it's harder to diagnose. I hope it's something as simple as that.

People can also edit the heading in the script to refer to your custom domain. That's what I would recommend so that people don't have to keep switching URLs for Canvas.

[arnanas]

Thanks James for this really useful tool. I have a custom Canvas domain so I did have to change the script heading for it to work. Just wanted to share where that particular line is, in case it is useful to others. 

In the code at the very top you should see lines like these. Edit the one highlighted in red to match your custom Canvas domain. 

// ==UserScript==
// @Name IntraGroup Peer Reviews
// @description Assign intra-group peer reviews
// @namespace https://github.com/jamesjonesmath/canvancement
// @match https://your-canvas-domain-here/courses/*/assignments/*/peer_reviews
// @require https://cdn.jsdelivr.net/npm/bottleneck@2/light.min.js
// @version 2
// @grant none
// ==/UserScript==
/* global Bottleneck */
 

[gramos]

Hi, @James .

I just read through all this as I think about implementing peer reviews next semester. Since a submission is required for the students to do a peer review, I was thinking that a custom grading scheme would work here...something like:

"Submit Peer Reviews" = 0 pts <-- this would be the default grade

"Peer Reviews Complete"  = how ever many points you want 

even a third option if you want to allow them to make-up points they missed: "Peer Reviews Incomplete" = between 0 and max pts (for missing reviews, poor quality reviews etc.)

It doesn't change the issue/concern with zero points, but students will see "Submit Peer Reviews" in their Grades page and realize that they have to do it. 

Thoughts about this approach?

[James]

@gramos 

I'm not sure that I follow what you're saying, perhaps because I haven't eaten anything yet today. I think it's the word "since" as in "since a submission is required ..." is throwing me. What you describe doesn't address the problem of requiring a submission.

Are you using this custom scheme for the primary assignment or for a second assignment that is just for doing the peer reviews? If it's for the primary assignment (the one that needs peer reviewed), then I'm not sure that it makes much sense to tell a student they need to submit peer reviews when they haven't been assigned yet. It also doesn't tell them that they need to submit the assignment to get on the list.

If you're saying that's the grading scheme for the second assignment where you award the points for doing the peer review, then it makes sense. I have absolutely no problem putting in 0's for scores and find it serves as a gentle reminder (or wake-up call) to students who missed it. I have the automatic 0 turned on in my class for missing assignments and (for some assignments) give them additional time after the due date.

What I do for that second assignment is to put instructions that explain what they need to do and how to get there, but it might help to make the grade be instructional as well.

[gramos]

I was referring to the "fake" submission you had for the students.

I thought that the fake submission mentioned below was the second assignment (peer review) assignment, but I may have misunderstood. (It wouldn't be the first time 🤣)  I thought giving them a default grade of zero was necessary so that students could see the submission details and the students who they need to peer review.

(and I should clarify: I haven't done the peer reviews yet and was exploring how to do it so I can implement it next semester. For what I plan to do, I'm not sure I'll need that second assignment.)

---

@James wrote:

Even when there is a submission allowed, Canvas doesn't show anything until there is a submission made....

What I've started doing in my class is faking a submission for every student and assigning them a default grade in the grade book. Of course I use 0 because I want them to know what will happen if they don't complete the assignment, but then you have to deal with "Why did I get a 0 for this?" because they didn't read the instructions on the assignment because they clicked on the To Do list instead of the assignment.

Once the grade is entered, then they see the submission details and the list of students who they need to peer review. In this case, Bizarre had already provided feedback to Clicking Sentence from the To Do list.

[James]

@gramos 

I haven't used this in several years, but I think my default grade of 0 to get past the submission requirement to do a peer review is on the assignment that you the students to peer review. That's what I'm calling the primary assignment not a secondary assignment. I could have that completely backwards though. If you take the route explained at the end, then pre-filling the secondary assignment with 0's would encourage them to do the peer review, but it wouldn't help in the assigning of the peer review in the first place. That can only happen if you set the default grade to 0 for the first assignment.

There doesn't have to be a secondary assignment at all, depending on how you're wanting to use peer reviews. If you want the people in the group to review how each other performed in the group, then you only need one assignment and you can set the default grade to 0 and use a letter grade with a custom scale of "Submit peer reviews" for a 0. Student reviews do not become the grade, so you are free to use that same assignment as the score you give for completing the peer reviews. You just need to let the students know that the grade they receive is the grade for the peer review, not the grade the other students have determined for them. Your letter grade scheme could help with that, but it should also be in the instructions.

I would use intra-group peer reviews a little differently. I set up two assignments because I want two rubrics. One rubric is for the students to provide feedback to each other and the other is for me to provide feedback on their peer reviews. The items that I want them to mark each other on (primary assignment) are certainly not the things I'm looking for in a good review. But I want to make it easy for me to grade and provide feedback on how well they provided feedback to others so I have a second assignment with a different rubric. If needed, I could make either assignment worth 0 points or mark it as "do not count towards final." One could make the grade on the primary assignment the average of the peer reviews and then not count it as a grade so the student has one number to represent how well they did as opposed to having to look through the rubrics to determine it. Other people may be willing to let the students determine another student's grade and count the grade towards the final, but I'm not willing to go down that path.

I haven't used peer reviews (intra-group or of any kind) myself in several years (even pre-COVID).

[gramos]

Thanks, @James ! 

[MAbalos]

Thank you @James!!! 224 peer reviews added in seconds! 

Found this resource -

Using Canvas Peer Review for Group Peer Evaluations https://cpb-us-w2.wpmucdn.com/u.osu.edu/dist/b/4855/files/2017/12/Using-Canvas-Peer-Review-for-Group...

[aaron_burger]

This is a much needed feature on Canvas, thank you James. I think that from a teaching and learning standpoint this makes a lot of sense; the peer review process should be something that they can do in groups!

I don't think that I understand why Canvas hasn't incorporated something similar.

[mjennings]

@James Greetings! It has been a while, but I am running into an issue with the Intra-Group Peer Review User Script. It looks like Canvas has added pagation to the Peer Review page that I do not remember being there before. Now when I run the Assign Intra-Group Peer Reviews only the first page is getting student assigned and subsequent pages are not.

Have you seen this? Is it just me? If you have time, is there a way to fix this?

Thanks for all your help!

[James]

@mjennings 

Yeah, someone put a pull-request on my GitHub Canvancement site about a month or so ago for this. I looked at it and felt there was a better way that didn't rely on pagination at all and worked on it, but didn't get it finished. I've been too busy with my classes to find time to work on things. I'm just glad no one has written that the latest upgrade to Inst-UI that they warned us about actually broke things. Or, if they did write, they forgot to tag me on it, so I don't know about it.

I think the temporary solution is to run the script for every page of students, which is kind of a pain with large classes. The solution offered was to automatically move to the next page on the reload. That still meant clicking on each page but you didn't have to mess with the pagination as well.

The problem was that I wrote the original script to use the names in the document to save time since Canvas had already loaded everyone. Now they load everyone in one spot (for the drop-down list) but not for the assignment part and that was what my script was using to decide who to assign them to.

[mjennings]

@James 

Thanks for verify that I am not crazy. A few additional problems I found is that if students in the group are on a different page then they are not being assigned. This makes sense given that it is only using the names on the page as you described. 

As for going one page at a time, the button was greyed out after I ran it on the first page and would not let me run it on the second page, or even on the original page a second time. 

Ultimately if it is only pulling the students from the page and they may not alway be grouped that way, even going page by page is not feasible for the high enrollment courses. 

Fortunately the course I was dealing with today only had two pages with about 6 students on the second page. I was able to manually get that sorted out. 

Thanks for all you do and I hope things calm down in your courses.