Hi @bgesteland,

Unfortunately, I don't believe this is possible at this time.  There is a somewhat related idea at [Rubrics] Hidden Rubrics - Instructure Community  which might apply to peer reviews if it's chosen for development in the future, but looking at the idea theme status, I think it would be 9 months minimum if the idea theme is chosen during the next round of voting, so don't expect anything too immediate.

Hope this helps a bit, even though it may not be the answer you wanted.

-Chris