It is not clear how to undo the assignment sort filters. Once you have picked one, it looks like you can only pick a different one. To undo the filter, you have to make a selection in the Student Name display options. You should be able to either uncheck an option or select a None option, behavior that exists in other display menus.
I agree that there needs to be an easier way to clear selected filters. It looks like, for right now, you are expected to select the top option in each filter menu which states All Assignment Groups, All Sections, etc.
The other thing I would note in relation to this, is that I would expect that if I uncheck the Filter group in the View menu, that it would remove that filter criteria from the gradebook view at the same time that it removes the filter dropdown from being displayed.