I have tried to invite people to my course as Observers, but they aren't getting the invite email at all.
If they have course invitations turned off in notifications they won't get an email. If they login, the option to accept the course should be there. A Canvas administrator in your institution can tell you what notifications have been sent to a user.
Retrieving data ...