Thoughts about the Canvas Outage

[laurakgibbs]

CROSSPOSTED. Updates will appear at Teaching with Canvas blog.  

~ ~ ~ 

I wanted to write a post to express my admiration for the way Canvas handled the AWS-related outage that took the Canvas system down for a chunk of the day on Tuesday, February 28. If you look at their Status page, you will see how diligent they were in updating us at regular intervals, acknowledging our need to know: there were 18 detailed, helpful updates over the course of the outage. There's even an RSS feed for the page, which of course makes me happy (I am the Empress of RSS).

So, THANK YOU to all the people at Canvas who managed that event yesterday: it looks like you did a great job of coping with the problem behind the scenes, and I really appreciated the clear, steady communication throughout!

Below are two observations that came to mind:

1. Twitter is powerful. The communications from our own IT were not so good, but I really benefited from my Twitter network. I only found out about the incredibly useful Canvas Status page because friends at Twitter shared that URL when I tweeted about the outage. Instead of the generic "OU IT technicians are working with the vendor" message, it would have been great if our IT had shared the Canvas Status page with us in the alert message; that's clearly where we needed to go for information.

2. It's risky to put all your eggs in one basket. I sometimes get pushback from people who tell me I should be using the LMS for everything; my response is simply that I choose the right tools for my needs, and the LMS does not meet my needs, so my class operates as a series of blogs, wikis, and websites outside the LMS (my own blogs and wikis, and my students' blogs and websites), plus our class Twitter. As a result of this distributed system, it's usually pretty easy to ride out any outage that comes along. If one area of the class is unavailable, we can always get by with the other areas.

So, because I do not rely exclusively on Canvas messaging (I have all my students' emails in a spreadsheet), I was able to contact the students about the outage, and because I do my class announcements in a blog, I was able to update the blog throughout the outage, and my students could get updates there too while they continued doing their regular work for the class.

And of course Growth Mindset Cat has some advice about variety: Play with many different toys. Variety is how you grow. 🙂

[dwillmore]

The issue is that 76% of the internet's data is held by four companies.  Of those companies, Amazon holds 40% of the data.  When one of those companies take a hit it brings severe pain to many.  In a very real way, we are all in just a few buckets no matter how much we diversify our products.

I am surprised that a single region of AWS took out sites completely as I would have thought they would have cross region replication.   Of course, I have never worked on anything as large as their datacenter and do not know the technical/cost issues.

It probably depends on your student base and your class as you determine the best way to provide information to your students.  In our general space, providing the simplest access is very important. Managing multiple login credentials for multiple sites would become a hurdle to learning.   This does not matter if there is no need for comments or assessment allowing a read-only access with no login need, I suppose.

With the very good uptime we have with Canvas,  I favor keeping data contained and accessible in one container.   Yes, this presents a risk of shutting down instruction completely, but again, this is a rare occurrence.  By splitting my data between several different vendors, I have multiplied the risk of some of my instruction being shut down, and in many cases, those vendors have uptime that is worse than Canvas or rely on AWS just like Canvas and are just as vulnerable to what happened yesterday.

I guess it is more of a philosophy/religion topic when it comes down to it.

By data here, I am really talking about the presentation of data to others.  By all means, maintain actual data on multiple systems for security.

[laurakgibbs]

 @dwillmore  yesterday was indeed VERY weird. When I saw the list of services that were down because of AWS, it was very eye-opening.

Another issue to think about is longevity. I'm in this for the long haul (i've been teaching fully online since 2002), and over those years I've seen services and tools come and go, including LMSes (I've used WebCT, Blackboard, D2L and now Canvas, just at one school in that time period). Just selfishly speaking, in the same way that I feel better able to weather outages by having my courses diversified at any one time (while also offering my students the best tools as I see it for each task), I also feel better able to manage changes that are coming down the line as different tools age out of the system and I have to shift and accommodate.

When I first got started I my students were using EZBoard and Netscape Composer for web publishing (we used EZBoard as a kind of proto-blogging tool!)... and through various permutations (including a long stint with Ning), I've ended up with Blogger and Google Sites. Since Canvas does not really even try to offer tools for student web publishing, I'm always working outside the LMS. Student web publishing is the single most important element of my classes and always has been, and I've seen a steady progression in the quality of the tools available for that. The LMS world, by comparison, feels pretty stuck in time.

When I think about the way my students manage their digital lives, I don't think an additional log-on is the issue. The real issue as I see it is digital literacy: making sure they have set up accounts properly so that they have a way to recover a lost/forgotten password, and also making sure that they are backing up locally what they have in the cloud. For example, as I said above, most of my students use Blogger for their blog, and Google Sites for their website (although I leave that up to them; if they prefer other blogging and web publishing platforms, anything ad-free works for me), and I make sure they know how to back up their work locally. No student of mine has ever actually lost their class work at one of these cloud-hosted services, but I think I am helping their general web literacy by making sure they understand why it's a good idea to back up locally. Just in case.

[dwillmore]

Laura,

Those are all good points, and I did not even think about moving from one LMS to another.  Our moves from WebCT to Moodle to Canvas were all painless getting the data over to the new LMS, but it took work to get the formatting of that data down. We disagree on several points here, but as I said, it depends on what one is trying to accomplish, what the need is, and what you want to present to the student.

Our moves from WebCT to Moodle to Canvas were all painless getting the data over to the new LMS, but it took work to get the data properly formatted after getting it to the new LMS.

My big goals in the role of an administrator are to ensure ease of access, little downtime, security/Ferpa compliance, and a centralized tool for reporting.  Canvas offers this structure, and their LTI compliance allows, even more, freedom.    These are not the priorities of everyone, and I can see that these priorities could even be an impediment to accomplishing someone else's mission.

[hallmans]

I have to say, this outage was inconvenient but far from a disaster. A number of our faculty are so used to the uptime of Canvas that they keep their course materials/PowerPoint's in the files section and just log on instead of bringing jump drives to class or logging into the classroom PC locally. When our campus hosted other LMS systems on local servers there seemed to be at least two outages a year.

One interesting observation was that faculty were mad that Canvas was down, but quieted down when we explained that Amazon servers were down.

[laurakgibbs]

Exactly hallmans ! That's why I was so surprised at lack of communication from our local IT / Canvas team: knowing that it was an AWS problem and also knowing about the Canvas Status page made all the difference for me! This is our soft rollout year, so it's not like any of our faculty would have known about the Status page (this is the first outage we've had to my knowledge), so that was an opportunity missed for letting people know that Canvas provides solid communication during downtime. I am glad to know about the Canvas Status page for future reference, and that is where I'll check first from now on if/when there is some kind of service problem.

[hallmans]

One of the great things about Canvas is their communications channels. Radio silence from our last LMS during outages was infuriating. We have an internal helpdesk and sent out the message below. We heard from several faculty how much they appreciated the updates and the heads up to bring their materials on alternate media to the classrooms. Once things were back to normal, we sent out an all clear message. We also posted on our Intranet site. 

From: IT HelpDesk

Sent: Tuesday, February 28, 2017 1:42 PM

To: Kenan-Flagler Students, Kenan-Flagler Faculty and Staff , PhD Students

 Subject: Amazon Web Services issues affecting Canvas, Panopto and other services:

Who:     All students, faculty and staff

What:  Amazon Web Services issues affecting Canvas, Panopto and other services

When:  Tuesday, Feb. 28^th, 2017.

Amazon Web Services is experiencing operational issues.  UNC Kenan-Flagler Business School students, faculty and staff may experience issues with some of our services including Panopto and Canvas due to an outage that originates from an Amazon Web Service issue affecting the Eastern coast of the United States. 

Amazon is actively working to identify and resolve these issues. 

Thank you,

IT Helpdesk ▪ Information Technology ▪ UNC’s Kenan-Flagler Business School

Campus Box XXXXX ▪ McColl Bldg ▪ Chapel Hill, NC XXXXXX

[laurakgibbs]

That's great! I contacted my students right away (as I mentioned in my post) because I knew that there would probably not be anything sent to students from our IT... but I was surprised nothing went to faculty. Thank goodness for Twitter buddies who came to my rescue! 🙂