Degraded AWS performance is currently impacting some Canvas users in the North American region. Check Canvas Status for updates.

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
bbisbee
Community Participant

Mass load Student Avatars

Jump to solution

Hello,

I'm working on developing a process in PowerShell to mass load all of our student id photos as our student's avatar. 

I've based it off of the python script that is available in the unsupported github

While running into our test/beta sites, I've noticed after about +/- 2100 students, I stop getting the image tokens needed to make the call to set that image as the users avatar. I've noticed that the accounts that aren't getting the token back, no thumbnail has been generated for the image that was uploaded. For all of the accounts that have been successfully set, a thumbnail has been generated. 

I'm not sure if since these are not production sites if there is perhaps a delay in those processes happening?

Does any one have any ideas?

Thanks,

Brian

1 Solution

Accepted Solutions
maguire
Community Champion

There is a more aggressive rate-limiting of API operations against the beta and test instance. Did you look at the error response that you were getting?

When I was loading thousands of quizzes into a course a test instance, I introduced an exponential back off when I got errors in uploading new quizzes and reduced the rate of new attempts. This seemed to work and I could inject many thousands of quiz questions for a course. [This was when migrating content from an earlier LMS. Some of my colleagues had generated an enormous number of quiz/exam questions over 17 years.]

 

View solution in original post

Tags (1)
11 Replies
maguire
Community Champion

There is a more aggressive rate-limiting of API operations against the beta and test instance. Did you look at the error response that you were getting?

When I was loading thousands of quizzes into a course a test instance, I introduced an exponential back off when I got errors in uploading new quizzes and reduced the rate of new attempts. This seemed to work and I could inject many thousands of quiz questions for a course. [This was when migrating content from an earlier LMS. Some of my colleagues had generated an enormous number of quiz/exam questions over 17 years.]

 

View solution in original post

Tags (1)
bbisbee
Community Participant

It definitely got better after switching it into Production, I was able to get all 60K of them loaded in. Thanks

reynlds
Community Participant

@bbisbee Can you provide some details around this? I've been looking at the bulk load of avatars for a bit now, and have been told that unless they are posted on a publicly accessible website (uh, no) there is no way to achieve this. I think the Instructure folks think I was asking how to dynamically load avatars as they are requested. However, I'd like to simply upload them somehow directly into the application. While I won't be using Powershell (Python is my "go to") I'd like to see your process/code if possible.

bbisbee
Community Participant

@reynlds 

Our process works in multiple stages. Every morning after our SIS import runs, I download a complete list of all our canvas users and import them into a SQL Table. I then match those users and identify which ones are students, parents, teachers, etc. After I do that, I grab all the student id’s and scan a file share that has all our student id photos. If they have a photo, I update the table with the full path, and an md5 hash (to compare when a new photo is taken).

At the end of each week, our process runs gets of a list of students who have new id photos or do not have an avatar already in canvas and sends it. I took this approach rather than just reuploading them each week because Instructure seems to have a rather aggressive rate limit and would cause the process to die out. It took about 19 hours to get the main batch up. I also took this approach so that we do not keep uploading the same photo each week causing duplicates.  Another thing to note is if the user is over their quota, you will not be able to upload a user avatar for them. From there it runs the API queries to create the upload form and then upload the avatar and then to assign it to the user. I log the unique ids and tokens to a database that I use to compare when a user changes their avatar or when a new one needs to be uploaded.

I know very little about python, but this is the script that I based my powershell code on:

https://github.com/unsupported/canvas/blob/b3acbb55267feb69e14078d83655f594e992c94f/api/bulk_assign_...

 

As far as the powershell side of things, I store all the canvas ids in a variable and then put it in a parallel foreach loop to go through each id and make the changes. If you are curious about more of the specifics on that let me know. I hope this helps.

MikeBrinkman
Community Participant

Thanks for sharing that link, @bbisbee ! I was trying to get this to work using bulk_upload_avatars.py, but no luck. Then I tried bulk_assign_avatars.py, which ironically did work, since the page it is hosted on specifically says 'Also note that this script does not upload your images to Canvas in any way, it simply links each user to an image you're already hosting.' However, the images I used were located on my computer, not links to files on a web server. Very glad this is working!

bbisbee
Community Participant
Awesome, glad you got it working!
reynlds
Community Participant

I was able to this this working as well (in my Test environment). I'm estimating that it's going to take 3.5 days to upload the 300K avatars to Canvas. Fun on a bun!!!

robotcars
Community Champion

@reynlds 

Can you try spinning up 2-6 workers and trimming that down? You'd have to incorporate rate limiting, but I'd definately shoot for not trying to do those 1 at a time.

Also, if you task ends, do you have some kind of 'memory' to store what records have been completed... this way if it ends, you can resume?

reynlds
Community Participant

Extra workers is a good idea...I'll investigate that. The process that I'm using logs all activity, and if it encounters any type of issue, it logs that (separately) and proceeds past it.