We are using canvas quizzes for an in-class exam. Class enrollment is about 120 students. We are in a new classroom with robust wifi. Nonetheless, roughly 20% of students experience load times of 10-15 minutes at the beginning of the exam.
Canvas functions normally during the exam.
At the end of the exam, some students experience extended submission load times (not the same students who experience the initial load problems).
Any answers as to why canvas hangs at loading and submission? We know that this is not a wifi issue. Is there a data bottleneck somewhere?
Was this a one time incidence or is it repeatable? If it happens every time, I would look into the wifi. If it happened once and you want to prevent it from happening again, I would look at the wifi. If it happened once and hasn't happened again, then I would chalk it up to other problems.
Who told you that it was a "robust wifi"?
Do you know the band 2.4 GHz or 5 GHz? Is it 802.11g, 802.11n,or 802.11ac? How fast is the wired connection back to the network or is it all done over wifi where one access point connects to another? How many network SSIDs are visible when you try to connect?
I don't expect you to know answers to all those. The point is that there are a lot of things that could go wrong with wifi. Brown University's network was impressive when I visited there a decade ago, so they probably have things together, but there's a lot of things out of their control as well.
The more connections that you make to a single access point, the slower the connections will be. You didn't specify how many access points, but 120 is a lot for a single access point. Most likely, there are also people not in your classroom that are using the same access point or wireless network. Also, there are potentially other connections within the room as students have smartphones and other devices that connect.
If you have multiple SSIDs that you don't belong, then there are other routers or access points conflicting for the signal and reducing your "robust" wifi. For example, someone may have their cell phone in hotspot mode or there may be a competing wifi from another classroom nearby. You need to make sure that they aren't using the same channel.
It's not that you can't have 120 connections to a single access point, but when you all start downloading the quiz at the same time it puts a load, especially if it's all at once like it would be at the beginning of the quiz. During the quiz, there isn't much traffic going on, so it is fine. At the end, everyone is submitting data and it may slow down again. All that assumes that you're the only group using that wifi -- if someone else is using the wifi to download a huge file, it could slow it down enough that when there's the additional load in Canvas it crawled.
It might help to stagger the quiz. Have some start the quiz, wait a couple of minutes, have another group start, etc. That way they're not all hitting at the same time.
You might also have someone with a network sniffer come in and look for problems.
That's no guarantee that it's a wifi problem. It might have been a network slowdown somewhere else that corrected itself. We have those on our firewall machines periodically. Canvas periodically experiences issues and slowdowns as well.
But using "robust" to describe "wifi" is a recipe for disaster -- it gives a false sense of confidence, so that's where I would start. Some day when you're not taking a test, have people all go somewhere that involves big downloads -- like run a Speed Test and see what happens to the robust wifi. If it holds up, then it might have been something else. If it doesn't, it may not be as robust as you think.
One thing you might do with those students who are taking 10-15 minutes is have them stop their browser from loading than then attempt to reload the quiz after a few minutes. That's the staggering effect I mentioned. If you're waiting 15 minutes, there's probably something wrong that often corrects itself when you reattempt the download.
We just upgraded wifi across campus last summer. The room has 7 Aruba 220 APs; 5ghz; 802.11ac. There may be a few other people on those access points, but not many. There's no other nearby classrooms. We stress tested our wifi with the network services team before the class. Also, with the same cohort of 120 students, we were collecting webcam feeds over wifi from each of their computers during a different exam. No issues. There have been several other online exams in the room. Some of these exams have experienced similar (those less extreme) submission lag issues in canvas.
How fast is the wired connection back to the network or is it all done over wifi where one access point connects to another?" I do not know for certain, can find out.
Basically, the wifi handles everything we throw at it except starting and submitting quizzes in Canvas.
Hi @theodore_kalait ...
Out of curiosity, what web browser are the students using to take the exam? Might this have something to do with it? This might be a pretty basic question, but maybe one to look at? Typically, Canvas works best using Mozilla Firefox or Google Chrome vs. Internet Explorer. Though I'm not 100% certain on this, maybe there are some issues that IE is having with the exam that could be resolved using a different browser?
Awesome setup. I was impressed with Brown's IT department a decade ago and you've just boosted that even more.
I'm still not ruling out wifi problems. You can't really, ever, unfortunately. Unless you're purely 802.11ac (don't fall back to g or n) then some devices might connect to the older network. But to do that, you would have to have fairly new machines. My HP laptop that's a couple of years old doesn't support 5Ghz or 802.11ac, so it would have to use the n to connect. So that suggests a couple of different things to look at.
Is there anything in common about the machines the students who experienced the delays were having? There were two groups - some at the beginning and some at the end, but if they all had machines running on the 2.4 GHz network, that might be part of the cause. Are these the student computers or the universities? Were the other pages in Canvas loading slowly or was it just the quiz. If the computers are the students, a mischievous student could have been running something that was sucking down bandwidth in an attempt to crash the network and get out of taking the exams (the university might -- really no guarantees -- have network logs that would show unusual activity during that time period).
But there is one thing that Canvas has that the webcam test you mentioned may not have. Did you collect webcam feeds locally to the University or did they go out over the Internet? If the webcam stress test was local, then the issue with the Canvas tests might have been one out of your control. That's why I asked if it was a one time thing or it happens multiple times.
Canvas is in a better position to know whether there were issues connecting to them. There are slowdowns that happen periodically (every now and then they are reported in vague terms like "some institutions may have experienced a slowdown between 12:43 MST and 12:48 MST on blah blah blah...". I would never call Canvas fast, and although we have a decent connection outside of the college, Canvas always seems to crawl for us.
Luckily for us, Canvas has just decided that they will release our Canvas Data daily in the form of flat files. That includes all page requests and so someone might be able to look at those files and determine a pattern among the students who experienced the delay that might shed light on what happened. Then you're not at the mercy of Canvas to do all the investigation. You'll need to contact your Canvas Admin (unless you are that person) to have someone look at that data.
However, if your people contact their people (have your Canvas support contact Canvas), they might be able to look across the board during the time frame in question and see if there were network problems. They have the ability to look broader than you could with the Canvas Data.
Another issue to look at is the complexity of the exam. I just created a quiz that has 9 questions with 150 possible choices for each one for a total of 1350 questions. When I edit it, it takes a while and my browser often freezes while a script processes. It's probably not that bad while taking the quiz, though, as they're only sent 9 questions. If your quiz had lots of images or other content that people had to download, that could have contributed to the amount. If it was just textual questions, it should have loaded quickly.
There are other issues that could come into play, but what you describe is consistent with network problems somewhere in the pipeline. Wifi is the easiest to start with and many aren't as well designed as yours. It could be Amazon (who hosts Canvas) having problems or a routing issue somewhere along the line. It could have been something those students had in common. It could be a problem with Canvas.
As annoying and frustrating as it is, if something only happens once, it's generally chalked up to an "issue out of our control" (whoever "our" is) because it's so hard to track down. On our network, when we have a slow down, I can test up until it leaves the college and so I can tell if the problem is internal or not. If it's something outside the college, there's nothing I can do about it and that's why I always have a backup prepared for days when we're using technology.
Unfortunately, I'm not going to be able to do much more than offer suggestions as we're all community volunteers without access to the network or the data.
If this has only happened once, it might be a fluke. What you might do if you have some class time to kill (I know, I know), is make a copy of the quiz in Canvas and then have the students access the quiz in the same way they did during the exam. If it works this time, then it was likely something external (or you just didn't hit the perfect firestorm). But if it messes up again, then you have a stronger case that something is wrong and needs fixed. If the quiz is really complex, then there might have been an issue with the submissions. In that case, there would be a bottleneck on Canvas' side (or Amazon's cloud who hosts it).
You may still want to file a ticket with Canvas with the date/time of the situation and have them look for issues on their end.
Designing for the anticipated usage is an important aspect of any network design. So, where is the bottleneck? Could it be the wireless connection, the connection to the service provider, or somewhere in between?
I suggest asking your IT Dept. to do some load testing. Also, some IT Dept's get fancy and rate limit wireless to a percentage of the overall available bandwidth. This surely would create a bottleneck.
We have had a couple courses this semester (440 students, 630 students) that tried to deploy a quiz ( 40 questions, various question types, one at a time). They were trying to mimic the simultaneous deployment you get in classes by enforcing a narrow window in which the students can take the quiz, and the quiz takes up most of the available time (2 hour quiz, available 7:30-9:30am)
The result of this was a mess. Working with Canvas, we determined that there were database locks happening on the quiz object.
We figure that we can resolve this by spreading out the start time of the quiz, allowing more time to start & complete the quiz, thus reducing contention on the particualr database object while the quiz is being generated.
Other than Canvas fixing their quiz system, any other tips for deploying quizzes to very large courses?
That is extremely helpful and seems to match what @theodore_kalait was experiencing at Brown. He'll be relieved to know it probably wasn't his wireless after all.
If you want to keep keep roughly the same testing time frame, one thing that you might try is to stagger the start times as you mentioned, but let the entire quiz be downloaded at once instead of one question at a time. This is something the teacher could do without any intervention and programming genius from IT. Divide the class up into groups of roughly 50 people and then make announcements at 30 second intervals: "All students whose last name begins with A-C may now begin." Announce that the time you start the quiz will be recorded and if you start early, you will be penalized. Be sure to put a time limit on the quiz to force the A-C group to finish at a different time than the W-Z group and make it slightly less than the full two hours so that everyone has an equal time to take it within the 2 hour time period. I would even randomize the order of the names so students felt like they had an equal chance of going first.
I have noticed severe slow-down when I was editing a quiz that had a small number of questions 9 but with lots of items in question groups (150 for each question) so there was a large number of available questions on the quiz (1350). My 35 students were taking it whenever they wanted to over a five day period, so we didn't experience the lockup issues.
But based on that experience, I have a couple of questions that might spark an path towards something useful.
Were question groups used on the quiz? If so, were the questions part of the quiz or were they linked to a question bank? Have you tried the other way?
When I had my extreme slowdown, all 1350 questions were part of the quiz itself rather than being linked from a bank. That meant that Canvas had to load all of them whenever it generated a quiz.
There are benefits to having questions as part of the quiz itself and that's the way I've decided to go with all of mine, but it might be faster to have them as part a the bank.
Warning! A logistics nightmare may will most likely ensue from the following approach. Consider it an extreme measure for when the instructor isn't willing to budge on other things like one question at a time.
Is it possible to create multiple versions of the quiz and then use differentiated assignments to spread out the load so it seems like different quizzes instead of the same one? The problem here is automation. The students could be divided into smaller groups using Create an Assignment Override from the Assignments API. Don't use the Quiz Assignment Overrides, it just returns a list of overrides but doesn't allow you to create them.
If there were multiple sections all taking an exam at the same time, you might be able to differentiate by section, but if all of those students were in the same section, it would have to be broken up by individual students. You take each copy of the quiz and assign it to just some of the students. Even if you assign each one to 50 students, you're still talking 13 quizzes.
The other issue is copying the quiz multiple times. You could try just using differentiated assignments without copying the quizzes, but since Canvas has determined it was a lock with the quiz objects, that it probably wouldn't be fixed by just using different assignments.
Depending on where the quizzes come from, there are various ways to get them into the same course. I believe one recommended way is to copy it from the course back into self. Set up the quiz once with the dates and assigned to one of your assignment override groups (you can't have a quiz assigned to no one). Then go to course Settings and Import Content into this Course. The problem with this route is that it will provide the same name for the second assignment that the original one had. That may be fine from an automated approach, but if you're a teacher using the web interface, it because difficult to tell other than looking at the Quiz ID that shows up when you mouse over the quiz just which one is the original and which is the newest. Then do that 11 more times.
Then, when the quiz is all said and done, there are now 13 entries in the gradebook for the quiz. You would want a 14th to hold the actual grade. Then you could either programmatically copy the grade from the quiz they took or you could download the spreadsheet, create a formula that took the max() of all 13 quizzes and put into the single grade, and then upload the spreadsheet back into Canvas. The individual quizzes would need to be in a assignment group that didn't count since anyone getting 0 for 12 out of 13 assignments isn't going to be happy with their grade.
Another way to coalesce those grades would be to go through and excuse the grades for the 12 versions the student didn't take. Don't blank excuse any missing value without checking the other quizzes as the student who didn't take the quiz at all would then be excused. This might be the best way, especially if if the gradebook isn't using assignment groups.
If it turns out that question groups were used on the quiz and that neither banking nor including the questions on the quiz itself help, that you could combine the two approaches and make different versions of the quiz that removed the question groups and had a single question on them. That way each quiz is quicker to prepare and access.
Anything involving making different versions of a quiz is going to be a nightmare. Hopefully Canvas will fix the database lock issue.
Oh, and I don't know that "50" is the magic number. I was just throwing out a number that seemed smaller than anyone has complained about.
I like the Group approach noted by @James and might modify it slightly by adding a password which would be included in each group announcement. However I would increase the time between the announcements by perhaps a full minute or more assuming some delay on the student initiating the quiz. Perhaps evaluate such an approach with a low stakes quiz? And of course verify with your favorite Canvas expert.
ps. merging sections into a common course sure pays dividends however in this instance it sure created a headache. And as James noted, we have a Plan D (a fully separate copy of the quiz for each group) and may be motivated enough to place each course section into a separate course. But that certainly would be a last resort. Or perhaps going back to pencils and paper fill-in-the bubble would be the last resort. ;o)