Today I Learned ...

[tdw]

Do you have a cool use case for the Rubric Creator?  A gem of wisdom you want to share with other Early Adopters?  Feedback or troubleshooting that just can't wait for survey?  Then you've come to the right place!

[ngarimella1]

Many faculty members I've worked with had well-developed assignment instructions. The challenge, however, was helping them take the next step: creating a rubric. While they understood the value of a good rubric, they often worried about two things: capturing all the criteria accurately and writing clear, descriptive language for each performance level. 

[HeatherSaigo]

I tried the tool yesterday and I am pleasantly surprised with the rubric it generated from my assignment description. It was a fairly thorough assignment description, and the rubric was nicely worded, easy to understand as a grader, and customizable so the number of criteria fit the scale of the assignment. I'm looking forward to seeing what it does with less robust assignment starting material. 🙂 First impression is GREAT!

[YasiAfsharnive]

@HeatherSaigo I'm so happy to hear that! Thank you for sharing. 

@ngarimella1 Good to know. Hopefully Rubric Creator can help them build those clear, descriptive rubrics! 

[HeatherSaigo]

Hello! I had an excellent result when I used the rubric generator on a thorough, well-presented assignment with a title and a description. When I trimmed down the same assignment but kept the rubric parameters the same, the tool would not let me save the rubric. It still was able to generate a decent rubric with the same number of criteria and points, but fewer details (because I didn't provide them). I can preview the rubric, but I can't save it. Is this a "me" problem or is anyone else having trouble? Thank you!

[HeatherSaigo]

Oh, update. Looks like the feature options have been turned off in my instance somehow. Is there an update on whether we can run it in test? Thanks!

[HeatherSaigo]

I tried it in our test instance, and I'm getting the same error.

 

[tdw]

Hey Heather, thanks for the report and I was able to reproduce it.  The issue is ...

The rubric requires a name!

We should definitely be giving you a better error message indicating exactly what the problem is.

[HeatherSaigo]

Thank you! Yes, specific and actionable feedback is where it's at. 😄

[HeatherSaigo]

Also, it doesn't run in test but it does run in beta. I think it wasn't available when our test instance last reset?

[kapjones]

When I turned off AI Rubrics in Feature Options, it automatically also turned off Enhanced Rubrics.  (I turned AI off in production once I noticed it in my Beta instance - but quickly realized our Enhanced Rubric option was turned off as well)

[YasiAfsharnive]

@kapjones Hm that's interesting. Do you mind DM'ing me your Canvas URL? Turning off one feature flag should not turn off the other! 

[TrentMcMahon]

Posting my feedback here in case others echo these suggestions or want to expand on them. I will also fill out the survey. Hope this helps!

Pros

• Flexibility of choosing the rating display, order, and scoring.
• Generally, the AI-generated criteria does align well with the assignment. We especially liked that the language itself was refined and straightforward.
• Being able to build onto existing criteria using the "Regenerate" button or modify a single criterion. It's a nice feature when you need to fine-tune the output.

Improvements/Suggestions

• Global/Account Settings. Our institution would be more likely to adopt if admin or the design team could be involved with setting up on the back-end. We use blueprints, so individual instructors would almost never be using this tool.
  • We have very standardized rubrics at our institution. The ability to set our own rating names for all rubrics would be beneficial. I noticed even the tool itself changes the rating scale. Exceptional/Satisfactory/Inadequate one time, Excellent/Satisfactory/Needs Improvement the next. We believe that students should expect a standard rating system throughout the course, and this tool is not delivering this. Was able to use Regenerate to make the change but having to do this for every rubric does not save us time.
  • We follow specific point distributions depending on degree level. The ability to set a standard for distributions for all rubrics would be beneficial.
• User Experience
  • Need to be able to see the assignment details while setting up the rubric, to more easily determine whether the criteria it generated aligned with the assignment.
  • Would be nice for rubric name to auto-populate with assignment name.
  • Would be nice for the tool to remember previous settings to help streamline rubric building within the same course.
  • Points per Criterion is clunky, honestly a bit useless. Rarely is a rubric going to have the same point distribution per criterion. This becomes a manual process to update and does not save time. Points per Assignment may be better--could the tool be trained to consider the criteria it is creating and assignment points to each row accordingly?
• Additional Features
  • Need rubric support for graded discussions or this tool is not viable.
  • I could not get the AI to change the point totals of the assignment, either overall or to an individual row. This seems limiting, and like something that it could be trained to do.
  • Prompt history. We would like to be able to see and tweak prompts previously provided and regenerate. No real way to do that without completely regenerating.
• Functionality Issues
  • Changing "Rating Display" or "Rating Order" and clicking Regenerate does not actually change these things.
  • Not sure if this is a feature of Enhanced Rubrics or this tool, but you cannot delete a rubric from within the Rubrics menu item, only the assignment itself? Seems like an issue.
  • Not sure if this is a feature of Enhanced Rubrics or this tool, but "Rating Scale" and "Points" being 2 different numbers could cause some student confusion. Why is both needed?

Overall Analysis

While this tool is a step in the right direction, our design team is able to much more quickly, easily, and reliably use our in-house rubric generation tool to create rubrics for assignments. Perhaps if some of the items above were addressed, it could become a viable solution for us.

[tdw]

Trent, thank you for this thorough feedback, this is exactly what we're looking for, and bonus points for filling out the survey.  A peek behind the curtain; we use the same demographic and sentiment questions in every EAP so we can get more-or-less an apples to apples comparison of the loveability of all features.