[ARCHIVED] Peer review or group member evaluation tools

[skhagen]

I'm looking for peer review and/or group member evaluation (i.e. students rate each other's performance on group work) tools that others are using.

I'm aware of Peerceptiv and PeerGrade. What else is out there?

Thanks!

[cdavis1921]

I have used PeerGrade and liked many of its functionalities. But, one thing that they don't have available is peer review of individual group members. In a support chat, they indicated that the idea is on their development radar, but there is no estimated date of implementation.

A colleague of mine introduced me to CATME as a tool that can be used to assemble teams and provide a platform for self and peer evaluation of group members. 

CATME Information Website

CATME is "currently" free, however, they have been trying to convert over to a paid model based on the number of unique logins for a given institution. The embedded video below still states that the service is free for academia. The platform allows you to create teams first be having students complete on online survey that asks them questions regarding availability, degree of dedication to the course, leadership styles, etc. You can edit the survey to add or delete questions depending on what types of information you would need to know when building a group. 

CATME Video Introduction

[michellez]

I'm following this thread - I too am interested in this question.  Thank you  @cdavis1921 ‌ for the suggestions above. Do you have any idea about the proposed pricing structure if CatMe does in fact become a paid program?

[cdavis1921]

I did receive the following information via email in March (3/9/17). Since it's been more than  six months, I don't know if it is all still accurate. 

• The unique student price of $2.00/yr. is for unlimited student use across all their instructors.
• Instructors pay nothing for unlimited use of CATME.
• Data storage indefinitely is free.
• New CATME features like written peer-to-peer comments are available, no charge.
• We will bill at your choice; departments, schools, colleges, universities or instructors.

[michellez]

Thank you for your prompt reply!

[a_craik]

Hi Sara

We have been using WebPA for a while, which is free and open source. You can read more about it here: WebPA . We are yet to try the Canvas LTI integration, but I believe there may be some hitch in the practicality of it to do with setting up groups before students have clicked on the tool. I'll update if/when we do investigate the LTI, and otherwise keen to hear what others are using that integrates with Canvas!

Cheers 

Adam

[skhagen]

Hi  @a_craik , Do you happen to know whether WebPA data is stored in the UK or could be stored in the US? We have restrictions limiting us to US-based data storage. Thanks!

[a_craik]

Hi  @skhagen ‌

It's open source software that you'd need to install and host yourself (see here) - maybe not as simple to adopt as cloud hosting, but you do get to have the data on your own server!

Cheers

Adam

[KeithWilley]

This is a late reply and may not now be relevant.

SPARKPLUS   has been around since the late 1990s.    The tool has been updated and enhanced many times since then. 

SPARKPLUS is centred on self and peers’ assessment of and feedback on team work and skills and student's contribution to group projects.   SPARKPLUS brings assurance of learning and assessment validity to group work activities, in that students are graded according to their demonstrated contribution to the group project task and activities.  The monitoring tool and moderation facilities allow instructors to make informed decisions if and where to intervene in the case of dysfunctional teams, free riding and non-contributing students. 

SPARKPLUS allows the instructor to use preselected criteria or to develop their own.  This flexibility allows for both the team skills and knowledge/technical skills that contribute to group work projects to be evaluated and assessed with written feedback. In addition, you're able to weight each category of criteria appropriately to generate the Relative Performance Factor RPF.  

If you go to the sparkplus webpage https://sparkplus.com.au/  you can see more details.
 
The Using SPARKPLUS tab provides an easy-to-read overview of using SPARKPLUS to facilitate self and peer assessment of group activities.  The information includes a discussion of the assessment factors, the results and feedback available to instructors and students including radar diagrams, choosing appropriate criteria and feedback prompts, when to use the different RPF formulae and dealing with any student objections to their feedback and factors.
The video tab contains 16 short videos (generally 3 to 5 minutes in length) that cover a range of topics.

[skhagen]

In case anyone is interested in the conclusion to this exciting saga, we're now using FeedbackFruits (LTI 1.3) for group member evaluation.

[lebuhn]

Thanks.

[lettgo583]

@skhagen How do you like Feedback Fruits? Is it working well for your instructors? We're looking at it too ... since Canvas hasn't yet addressed the challenges with discussions. Would love to know what you like and don't like about FF! Thanks! 

[skhagen]

@lettgo583 I like it a lot! In the Spring semester we had nearly 1500 unique students using it in 18 different engineering courses, mostly for the group member evaluation tool. Also some peer review, which we're expanding into next year. Most people find the UI to be very intuitive, and the 24/5 help is great, so I only spend an hour or two per month troubleshooting. There's an initial learning curve, but we have folks who've used it for 2-3 semesters now. It shifts some instructor time from administrative figuring out who rated whom and averaging things to being able to quickly identify which groups or students might be struggling. And I think it will be useful for our ABET documentation of the teamwork outcome, especially in design courses where students work on semester-long (or longer) projects.

Overall student response has been positive. They like the ability to give and receive anonymous feedback. Students have been generally good about providing constructive criticism. (Students can flag harmful comments--to my knowledge this hasn't happened yet. And we've provided a resource about how to report and how to escalate if needed.)

We're pairing the rollout with faculty development opportunities for thinking about the structure of group work and thoughtful design of group assignments. Just like with any tool it's important to have a quality rubric/rating scale and clear instructions for students. We also consult with instructors about how to design assignments using FeedbackFruits.

The LTI integration is great. Canvas groups get synced into FeedbackFruits, and grades get pushed back to the Canvas gradebook with very few issues, and that's stress-testing it with some courses over 200 students, over 50 groups, and large teams of collaborating instructors.

[lettgo583]

All this is good to know! Thank you!!