Mass load Student Avatars

[bbisbee]

Hello,

I'm working on developing a process in PowerShell to mass load all of our student id photos as our student's avatar. 

I've based it off of the python script that is available in the unsupported github. 

While running into our test/beta sites, I've noticed after about +/- 2100 students, I stop getting the image tokens needed to make the call to set that image as the users avatar. I've noticed that the accounts that aren't getting the token back, no thumbnail has been generated for the image that was uploaded. For all of the accounts that have been successfully set, a thumbnail has been generated. 

I'm not sure if since these are not production sites if there is perhaps a delay in those processes happening?

Does any one have any ideas?

Thanks,

Brian

[maguire · Accepted solution]

There is a more aggressive rate-limiting of API operations against the beta and test instance. Did you look at the error response that you were getting?

When I was loading thousands of quizzes into a course a test instance, I introduced an exponential back off when I got errors in uploading new quizzes and reduced the rate of new attempts. This seemed to work and I could inject many thousands of quiz questions for a course. [This was when migrating content from an earlier LMS. Some of my colleagues had generated an enormous number of quiz/exam questions over 17 years.]

[bbisbee]

It definitely got better after switching it into Production, I was able to get all 60K of them loaded in. Thanks

[reynlds]

@bbisbee Can you provide some details around this? I've been looking at the bulk load of avatars for a bit now, and have been told that unless they are posted on a publicly accessible website (uh, no) there is no way to achieve this. I think the Instructure folks think I was asking how to dynamically load avatars as they are requested. However, I'd like to simply upload them somehow directly into the application. While I won't be using Powershell (Python is my "go to") I'd like to see your process/code if possible.

[reynlds]

I was able to this this working as well (in my Test environment). I'm estimating that it's going to take 3.5 days to upload the 300K avatars to Canvas. Fun on a bun!!!

[robotcars]

@reynlds 

Can you try spinning up 2-6 workers and trimming that down? You'd have to incorporate rate limiting, but I'd definately shoot for not trying to do those 1 at a time.

Also, if you task ends, do you have some kind of 'memory' to store what records have been completed... this way if it ends, you can resume?

[reynlds]

Extra workers is a good idea...I'll investigate that. The process that I'm using logs all activity, and if it encounters any type of issue, it logs that (separately) and proceeds past it.

[bbisbee]

@reynlds 

Our process works in multiple stages. Every morning after our SIS import runs, I download a complete list of all our canvas users and import them into a SQL Table. I then match those users and identify which ones are students, parents, teachers, etc. After I do that, I grab all the student id’s and scan a file share that has all our student id photos. If they have a photo, I update the table with the full path, and an md5 hash (to compare when a new photo is taken).

At the end of each week, our process runs gets of a list of students who have new id photos or do not have an avatar already in canvas and sends it. I took this approach rather than just reuploading them each week because Instructure seems to have a rather aggressive rate limit and would cause the process to die out. It took about 19 hours to get the main batch up. I also took this approach so that we do not keep uploading the same photo each week causing duplicates.  Another thing to note is if the user is over their quota, you will not be able to upload a user avatar for them. From there it runs the API queries to create the upload form and then upload the avatar and then to assign it to the user. I log the unique ids and tokens to a database that I use to compare when a user changes their avatar or when a new one needs to be uploaded.

I know very little about python, but this is the script that I based my powershell code on:

https://github.com/unsupported/canvas/blob/b3acbb55267feb69e14078d83655f594e992c94f/api/bulk_assign_...

As far as the powershell side of things, I store all the canvas ids in a variable and then put it in a parallel foreach loop to go through each id and make the changes. If you are curious about more of the specifics on that let me know. I hope this helps.

[MikeBrinkman]

Thanks for sharing that link, @bbisbee ! I was trying to get this to work using bulk_upload_avatars.py, but no luck. Then I tried bulk_assign_avatars.py, which ironically did work, since the page it is hosted on specifically says 'Also note that this script does not upload your images to Canvas in any way, it simply links each user to an image you're already hosting.' However, the images I used were located on my computer, not links to files on a web server. Very glad this is working!

[bbisbee]

Awesome, glad you got it working!

[morris_gart]

@MikeBrinkman did you have to change much in the script to get it to work?  I have been trying to use the bulk_assign as well but it keep returning errors.  I feel like I am missing something with my limited knowledge.

[reynlds]

Just wanted to update any interested parties that I initiated this a couple weeks ago with just under 300k user avatars to upload and set within Canvas. The process took just under 2.5 days with a single worker thread. About 1/3 of the avatars were unset for various reasons (missing image, etc.). However, everything else went off without a hitch.

Also, I know Instructure says there is currently no mechanism for "locking" avatars on user accounts, but I've been testing this using headless Firefox/Chrome and python/selenium. I estimate that for the almost 200k pictures that were set it would take 5 days or so to perform this using this automation.

[bbisbee]

That was about my experience as well. It took about 19 hours to do about 68K students. I wish there were a better way. As far as locking the avatars, I believe there is an option in the settings that will let you disable the ability for the users to change them, but I have not played with that yet because we still let Staff manage their own. 

[reynlds]

You can either turn the avatars on or off...but to lock them you have to individually "review" the avatar. I searched for a while and then checked with my CSM who confirmed that the option to lock does not appear in the admin GUI. 

[fisher1]

It appears that unchecking the box for "Users can edit their profile" locks the ability for all users to change their avatar...at least from what I tested.

[reynlds]

@fisher1 I was more interested in locking only the avatar, not necessarily the whole user profile.

[fisher1]

@reynlds When I disable "Users can edit their profile", I'm not sure what else it is disabling besides the ability to change the avatar. I tested it out and the user can still click "Edit Settings" and change their name (if enabled), communication channels (if enabled), language, time zone, password, access tokens, and feature options. The only thing I see that changes when I disable "Users can edit their profile" is the little pencil button to edit their avatar disappears. Maybe Instructure meant to title that setting "Users can edit their profile picture"?

[reynlds]

if that's the case, it wouldn't surprise me. Instructure is notorious for calling a field one thing in the GUI, something else in the API and yet something else in canvas data. Their timestamps are also inconsistent with some fields showing my local time zones and others on UTC. I don't usually work in the GUI, so I defer to more graphical admins when it comes to those things. The last few years we've written a pretty extensive framework for administration, etc., that only uses the API and SIS processes.