cancel
Showing results for 
Search instead for 
Did you mean: 

Email Notification Delays

Email Notification Delays

Engineering expects email notification delays to continually decrease over the next 4 weeks. As an alternative, notifications can be received in the Canvas mobile apps.

 

The notification delivery platform (Notification Service) is periodically experiencing queuing delays up to a few hours with the recent rise of Canvas usage. The primary bottleneck of delivery throughput is Amazon SES (Simple Email Service). SES send capacity for the IAD region is currently at about 800 deliveries / second. This send capacity is no longer sufficient to meet COVID usage patterns in Canvas.

While scaling up other components of the notification service to respond to increased load, a request to increase SES send throughput was also implemented. Unfortunately, increasing email send throughput is not as straightforward as other services. New AWS machines and IP addresses are provisioned as part of this process, but there is a mandatory "warming" period during which a very slow ramp-up of the new hardware takes place. As the new machines come online, a slowly (as in, over a period of weeks) increasing number of emails are delivered through the new machines. This allows us to ramp up our send throughput while also minimizing the potential that email firewall systems will see those new machines as being email spammers.

It is possible for AWS to disable the "warming" process all together and allow us to send at full capacity, but doing this greatly increases the likelihood of long-term issues for a short term benefit. While waiting for the warm-up process to complete, our team is also looking into regionalizing the message delivery infrastructure to split up the work more efficiently and safely over the various geographic regions. Currently, all global Canvas messaging (except YUL) is routed through our IAD datacenter. This will be a longer-term initiative, but will hopefully mitigate scaling issues going forward.

For better reliability, students and teachers can choose to enable push notifications within the Canvas Student app and Canvas Teacher app, respectively. Parents can set alerts within the Canvas Parent app. All apps are available in the iTunes store and the Play store.

Labels (1)
Comments

Thank you for posting. We hare having issues of specific teachers whose notifications through the Canvas inbox are not going through to students. Could this be a linked issue?

Hi, Emily,

Notifications through the Inbox are separate and are sent through Canvas application servers and databases. So they're not related to the same delivery behavior.

However, it could be a linked issue if students are setting their notification preferences to receive emails. Instructors can't manage notifications for students directly.

Are students using the Canvas mobile apps? They're more reliable in receiving notifications and I'd recommend helping students know how to use them and log in. 

Erin