In SpeedGrader, click on the student's name in the top right corner. You should see a scrolling list of all your students. If you know whose assignment you need, you can click their name. That way you do not have to scroll through all assignments, including those with feedback, to find those you need.
I also suffer from having to click through a whole class's worth of assignments to find one or two that need comments or regrading. If all your students turn in their quizzes at the same time, you can go through them all in one sitting, and stop when you see you are back to the beginning. If they take their quizzes at different times, then yes, you get stuck clicking through all of them several times. I use SpeedGrader to check the quiz grades after the class period when 90% of my students take them, and then again after the last time to submit the quiz has passed.