Hi @MirandaDilbert,
Unfortunately, I don't think there is a way to limit access to observers on the Canvas side of things. There are 3 basic options administrators can set for visibility of external tools: public (everyone), members (everyone enrolled in the course, including observers), and admins (just administrators and teacher role). The admins setting would remove it from observers, but also from students, which is not what you want. There may be a setting on the Zoom administrative side to configure what it does with roles, but I'm not the Zoom admin for my campus so I can't say for sure what options may be available there.
-Chris