Hello @rwollner
I don't think this is possible, unfortunately. However, you could experiment with the permissions available for a custom course level role. HERE are the permissions available at that level.
The permissions to allow a user to access Speedgrader/submissions are some-what tied to the grades since Speedgrader displays the assessment grade box that feeds to the gradebook.
Another option I have seen some institutions do is setup a custom role based on the Observer role. From there they pair the observer with the student they need to monitor. However, the observer role is still able to see the grades for the student they are observing when they are posted by the instructor.
-Colton