Rubrics have needed a lot of love and attention for a really long time. I've been using Canvas for 12 years and I've written several Canvancements that enhanced the functionality of rubrics, so I feel like I know something about them.
- My rubric importer allows you to copy/paste a rubric from a spreadsheet and create a rubric in Canvas.
- I added the ability to reorder criteria in an existing rubric with drag and drop. It wasn't accessible, there were no handles, but it worked.
- I added the a "Max Points" button at the top of the rubric to automatically award the maximum points possible.
- I wrote QuizWiz, which included several SpeedGrader enhancements. The most popular one was the ability to save the rubric results, advance to the next student, and then re-open the rubric for that next student. It really cut down the number of clicks.
- I wrote a script that would sort the rubrics when you go to add it to an assignment and put the current context at the top of the list.
- I wrote, but never told people about it, CSS tweaks that would fix SpeedGrader so that the rubrics weren't so big (I have small titles), hide the long descriptions (I don't need them when filling out a rubric), and hide the possible points for each criterion (it just wastes space).
I was a little excited that Canvas was finally providing much needed attention to rubrics. I was hoping that Canvas would fix things and I could retire my Canvancements. Thankfully, they've been fairly stable as it takes a while for Canvas to change things -- as an example, the rubric importer was last updated 7 years ago.
I've stayed out of the rubric redesign until now. People would ask me to make something work with the rubrics and I would say that I don't want to code for something that is in flux and could quickly change. I've got a full-time job and write Canvancements as a labor of love, but they have to take a back-seat to my real job.
With the July 20, 2024, release, Canvas finally deemed that the Rubrics Enhancements feature preview was ready for people to use. It is now on by default in your beta and test instance. Many of you will want to go in and enable the new exciting feature for your users without fully understanding what it means.
My first experience with the rubric enhancements came on July 25 when a user wrote on my rubric importer page that their entire assessment system for 110 courses and 30k students was broken because my rubric importer button was gone. It turned out that their college had enabled the Rubrics Enhancements feature and things broke. It was still there for me as my college had not enabled the feature. I went into the beta instance and sure enough, the button wasn't showing. It turned out that the button was being created, but they were hiding the right-side panel. I quickly came up with a one-line code that would re-enable the right-side menu and have been working on adding the functionality back. As I write this, the part to add the button back is fixed to work with Rubric Enhancements, but I'm tracking down some other bugs that came along with a code cleanup. The user decided to turn off the rubrics feature for the time being so they could use the rubric importer.
It turns out that all of the rubric improvements I mentioned, except for the one to sort the rubrics when attaching one to an assignment, are broken. The only one that has been implemented natively is the ability to reorder criteria.
If you rely on my rubric Canvancements, I would wait a while before enabling the Canvas rubric enhancements preview feature.
At some point, Canvas will force this rubric enhancements on people and my enhancements will stop working. Canvas is moving towards not including jQuery or jQuery UI, so other customizations will run into issues. The ability for end-users to improve Canvas is getting harder and harder. If you can find something that doesn't rely on modifying the JavaScript or CSS, you should probably start looking into those solutions.
Since that initial message on the 24th, I dug deeper into the Rubrics Enhancements from the role of an admin and teacher. I have not looked at it from the perspective of a student. This is a review of phase 1 and hopefully Canvas will be able to improve it a lot before they actually enforce it on us.
The first thing I noticed is that only the admin can enable the rubrics enhancement feature. It is enabled by default in the beta and test environments, rather than copying the setting from production. In the past, admins have had to enable features, but then instructors can enable it at their course level if they like. The rubrics enhancement feature is different. It can only be enabled or disabled at the account level. There is no override at the course level. Instructors cannot decide whether or not they want to use it, it can only be done account-wide.
When I went to use a rubric from within SpeedGrader, I noticed that they are now using the Instructure UI (InstUI) library. This was their great solution to accessibility and consistency that led to the deprecation of the Canvas Style Guide that we used to rely on. I call it a right-side flyout, but technically, it's probably a DrawerLayout or perhaps a Tray component. It slides out from the right side of your screen, covering up information. Because it's done with ReactJS, they have non-human-friendly names for classes, making it hard to target or modify. That doesn't bother people who are happy with the default functionality, but it makes it harder for developers to tweak.
My biggest problem there is that it is slow to fill out a rubric.
I've seen this before and suspect it is related to the the event listeners and where they add them. An event listener is triggered when something happens. For example, if you click somewhere, then the click event listener is triggered. When I add them in my code, I add them to the objects getting clicked so they don't get triggered needlessly. When Canvas adds them, they put them on the whole document, so any change anywhere on the page fires the event listener. I could be typing a submission comment in SpeedGrader and it cannot keep up with me. I'm not that fast of a typist to start with, so if I notice it, it must be really bad.
I'm seeing a similar delay when using the rubric. You click a rating, it updates the score at the top, and then you have to scroll to the next criteria. More on the scrolling later. I have a rubric I use for testing purposes that has 20 criteria and 5 ratings for each.
Without the enhancements, I could fit six criteria on my screen. With the traditional view of the enhanced version, I can only fit two at a time. I now have to scroll nine times while completing the rubric while it used to be that I only had to scroll twice.
It's worse than just additional scrolling. When you click, it adjusts the instructor's score at the top of the page, which is nice. But there is a noticeable lag in doing that. Again, I suspect it's because of where they attached their event listeners as it has to check for anything that could happen anywhere on the page every time I click rather than checking just for the clicks within the rubric.
That delay also makes scrolling difficult. I tried to click on the two criteria that were on the screen and then scroll (using the mouse wheel) down to the next two, but it was jerky and unreliable.
To put some numbers to it, I took that rubric with 20 criteria and went through and clicked the maximum points for each criterion (I did not use my Max Points button with the old version). Without the enhancements, it took me 13 seconds to scroll through and click each one. With the enhancements, it took me 28 seconds to scroll and click each one. Except that it turned out that I missed clicking on 3 of the criteria because of the jerky nature of the scrolling. I wouldn't have known that I missed three if I hadn't gone through and checked. Gone are the colored rubrics, there is just a black border around selected ratings it's hard tell that something is selected.
The reason that I only get two criteria on the screen at one time is because they decided to fix the height of the rows in the traditional view. Because I have optimized my rubrics for the way rubrics have been forever (until now), they had short titles, no descriptions, They took up minimal space in SpeedGrader so that I didn't have to do a lot of scrolling to complete them. I almost never provide feedback on individual criteria, but leave comments that the students may actually see.
In the traditional view of the rubric enhancements, the comment box is given considerable space.
If I enable free-form comments, I get a little bit of space back and can now fit three criteria per page. However, there may be up to three scrollbars while I'm typing a comment and the scrollbars may start doing a rapid blinking action.
I could probably go through and write a CSS tweak to hide those comments and to un-fix the height of the rows if I had to, but that's not something that most people could do.
The flyouts are fixed width. In SpeedGrader, you could adjust the size of that right panel and still can. But because the rubric is no longer part of that right panel but its own flyout tray, the width is fixed. It's about half of the screen width on a 1920x1080 monitor, but if you have a narrower screen, you won't be able to see what you're grading because it will be covered up by the rubric tray. All of that is in the traditional view; you can sometimes select a vertical or horizontal view. These are very narrow, but the heights are not fixed, so typing a lot just makes the comment box taller without adding a scrollbar.
If you have more than five ratings an any criteria, you cannot use the traditional view. Instead, you must use the horizontal or vertical views. As I mentioned, these are narrower and allow you to see the content your grading on a narrow screen.
However, I find these very unintuitive and downright dangerous.
Here is the horizontal view for a rubric that has five choices.
You get to select 4, 3, 2, 1, or 0. Because these are numbers, many are going to be confused thinking they are point values. You may think this is okay because when you mouse over it, Canvas will show you the text. You know, some kind of accessibility / usability thing that they value and is a core component of who they are? No! The number is the label, so from an accessibility perspective, it is labeled. However, from a usability perspective, you won't know what the 4 represents until you click it. Then you can see what the text and ratings are.
Vertical mode is slightly better in that you get the rating titles along with the number. There is still no point value given, but the text is helpful. Once you select an item, it will show the description of the rating and the point values.
You must still click on the number to select the rating. You cannot click on the label, which is where intuition would tell you to click. The 4 is an artificial construct Canvas is forcing us into. We don't want 4, we want "superior".
The problem here is that I may only get one criterion per screen, depending on how many ratings there are.
I've settled on a holistic rubric of awesome, good, okay, fair, poor, none. The titles are short and I have no long descriptions as it allows me some subjectivity. For my discussions, I have two rows, one for the initial post and one for the follow-up. It was a combination of what I want, but also making it fit the restrictions on Canvas's non-enhanced rubrics. It worked well. No scrolling and I could see everything at once. When you count those ratings, there are six of them.
When I view my rubric in SpeedGrader using the rubric enhancements, I get the vertical view. I have more than 5 ratings, I cannot use the traditional view. I can select horizontal view (it's not greyed out or disabled), but Canvas ignores my request. I cannot view it in a horizontal mode. Presumably because it has more than 5 ratings.
My old process (because of QuizWiz and its SpeedGrader enhancements) would be: (1) Open SpeedGrader, (2) click on View Rubric, (3) click on first rating, (4) click on second rating, (5) click advance and re-open [my Canvancement], and then repeat steps 2 through 4 for each student. Three clicks per student after the initial opening. Now, it's (1) Open SpeedGrader, (2) click on View Rubric, (3) click on the first rating, (4) scroll down to second rating, (5) click on the second rating, (6) click Submit Assessment, (7) move to the top to advance to the next student, and then repeat steps 2 through 7 for each student. An enhancement should not make things harder or require more clicks, scrolling, or mouse movement. Yet that's exactly what this rubrics enhancement does.
Although it would be an extra click, I would consider hiding all of the unselected ratings from the vertical view. You would deselect a rating (that's the extra click) before you could pick another, but it would save so much time scrolling through things.
The new interface is scrollbar heavy. In the traditional view, ratings with long descriptions gave me two vertical scroll bars. There was a vertical scrollbar for my rubric (since only two fit on the page at one time), and a vertical scrollbar for the page itself (the content behind the flyout rubric tray). There was also a horizontal scrollbar for some reason because the width of the content in the tray was slightly larger than the width of the tray.
There are some enhancements that I consider improvements.
You can now save a draft version of a rubric so that you can come back later and finish it. I sometimes like to grade the initial posts after they're made but hold off on the follow-up discussion until the discussion is over.
The submit assessment button is always visible at the bottom so that you don't have to scroll through the rubric to get to it. This was a feature my wife asked me to add just this week to the un-enhanced rubrics. She said not that my maximum points allowed her to quickly fill out the rubric, it would be nice to not have to scroll to the bottom to submit it. Canvas added that.
The main rubrics page now has more information about each rubric, including where they are used.
There is additional functionality. There is now the ability to archive or duplicate rubrics. You can also tell it to show the rubric ratings in order from lowest to highest instead of highest to lowest. More on that in a moment.
With the unenhanced rubrics, the clicking on a title from the main page would take you to a separate page that would show the rubric. Now clicking on a title will cause the rubric to flyout from the right so you get the preview right there. When you hovered over a title on the old interface, a pencil (if you could edit the rubric) or X (if you could delete the rubric) appeared. Now, you need to open the actions menu, an extra click, to edit or delete a rubric. That actions menu also allows you to archive or duplicate a menu.
The creation / editing of a rubric interface reminds me of the New Quizzes interface.
This is where you can select the rating order for those who want lowest to highest rather than highest to lowest. That said, selecting lowest to highest doesn't change the creation process. You still need to enter your ratings in order from highest to lowest. It does change the order when you go to view or use the rubric.
The new editing interface allows you to drag and drop the ratings to reorder them. The display column (4, 3, 2, 1, 0) is an artificial construct and cannot be changed. That threw me at first when I moved ratings around but the display stayed the same. It serves as a row identifier so that you know which button to click on from the horizontal or vertical views in SpeedGrader. The point values do not change when you drag the ratings as that would put them out of order -- only the rating name and description move.
I know some people had been asking for negative point values on the rubric ratings. This enhancement does not add those. You cannot type a negative sign, it won't let you.
The other thing it won't let you type for a rating value is a decimal point. The old rubrics supported decimals as rating values. The new enhanced version does not. The underlying rubric object does, it's the interface that does not allow it. You cannot create a rating that is worth, for example, 2.5 points. For some people, that wouldn't be a problem, but for me that is huge. The only rubrics I have that only have whole number values are the ones I use for testing my rubric importer for other people to use. Every rubric I actually use has decimals for most values.
For some people, the rubric enhancements will be an actual enhancement. For others, there are too many deal breakers.
I went into this today to let my Canvas Admin (I have admin rights, but I don't make the decisions) know what to expect. He kept asking me if there was anything from InstructureCon that he should let faculty know about and the two things I kept saying were Discussions and Rubrics. But I didn't have any further information on Rubrics until I explored it. After looking at them, my email to him started off with:
Short version: DO NOT turn on Rubric Enhancements or tell people how great they are or perhaps mention them at all. But you should know why they're not enabled or why doing so would be a bad thing in case people ask about them.
For those who know me, the fact that I even added a short version is unusual. I don't believe in TL;DR, but I didn't want him to miss the major point.
My wife is no longer the Canvas Admin at our school but I asked her if she wanted me to include her on the email message. She suggested making a blog post in the Community about it, which is why I wrote this.
I had great hopes for rubrics as there is a lot of improvement that could be done. Those hopes were dashed.
Canvas likes to roll things out a little at a time, making incremental changes. They got bitten by New Quizzes, which has been in development for a long time and gone through several name changes and still has issues that keep me from using them. That suggests, along with the "phase 1" name, that they will improve rubrics more. However, "phase 1" does not suggest they will improve what they have given us now, but make other changes down the line. They don't want to redesign and reimplement the same thing multiple times.
More recently, especially with the Discussions Redesign, they have decided that everyone should enjoy the improved features regardless of whether it is an improvement or not. The lack of the ability to control Rubric Enhancements at the course level may be indicative that this is headed in a path similar to Discussions Redesign.
The rubric enhancements might be great for you and your institution and if so, enable them. For the rest of you, I would suggest waiting until they fix some of the problems.
Personally, implementing rubric enhancements at my school would be a major setback. It's going to take more clicks, scrolls, mouse movements, and time to grade things. That is not my definition of enhancement.
I did not look at the rubrics from the student perspective, but if it's too hard for faculty to use, then there won't be anything for students to look at.
My comments here are my own. I didn't go looking for what other people had written about the issues with the rubrics enhancements, so sorry if I repeated a lot of what has been said. This is not intended to be a comprehensive list of pros and cons. I just wanted to share what I found so people can make their own decisions.
