I'm afraid this is a challenge for all of us once we have been using Canvas for more than a short period of time, @maychu. What I do to get around this is to rely on the Syllabus page with the Course Summary table enabled. This has all of the events that appear on the calendar (assignment due dates, manually created events). When I want to look at a course's calendar (e.g., to create manual events) I use the View calendar link on the course homepage. This opens the Calendar tool but with only the current course's events selected.
I was going to suggest that you create an idea to allow hiding course calendars, but I think the response would be that we already can - each of those calendars can be selected to have its events appear or not appear on the main user calendar. This does not solve your problem, however, especially for all those courses that don't use the calendar anyway.
This likely isn't a satisfying reply, but I hope our colleagues here in the Community share how they deal with the issue you shared.