Bulk Avatar assign/upload

[morris_gart]

I am stuck and have decided to turn here to find assistance.  I am not a developer but have tinkered around enough with API calls, python, postman, curl, and I am just about to lose my mind.  AI was useless and just went in circles.  I am just trying to upload about 250 student profile pictures as their Canvas Avatar.  I have tried the bulk_assign_avatar and bulk_upload_avatar from Github but I always get errors that I am not sure are because of the age of the script or the age of the programmer (me).  Any help would make this tired educator very grateful.

[jerry_nguyen]

@morris_gart 

What errors are you getting?

[morris_gart]

Using bulk_assign_avatars I am getting this returned

File "/Users/gartmorris/Downloads/bulk_assign_avatars.py", line 72, in <module>

    upload_file_res = requests.post(upload_url, data=upload_params, files=files, allow_redirects=False)

[fisher1]

I also want to add that restarting VSCode and re-opening the script file seems to help it run again after getting hung up on errors...

[fisher1]

@morris_gart I was also getting the same error when I attempted to edit the script and use it. I'm not a programmer and still learning how the Canvas API works, but after much research and trial and error, I finally got it to work. 

It seemed that the "upload_url" value was returning "None", so that is why it gave me the same error that you received. I haven't figured out yet how to get the upload_url value fixed without changing the script, but I did figure out how to get a temporary upload_url using the Canvas API documentation. On this page I used the example curl command in the terminal (on my Mac) and replaced it with my information:

curl 'https://<canvas>/api/v1/users/self/files' \
     -F 'name=profile_pic.jpg' \
     -F 'size=302185' \
     -F 'content_type=image/jpeg' \
     -F 'parent_folder_path=my_files/section1' \
     -H "Authorization: Bearer <token>"

After entering that into the terminal, I was able to get a response that contained an upload_url. I copied and pasted that upload_url into the github script where it sets the value of the upload_url. 

So I changed this:

upload_url = data.get('upload_url')

to this:

upload_url = "my-upload-url-I-copied-from-the-terminal-response"

It ended up working after that! Until I ran into a student id that didn't exist in Canvas anymore that was in the spreadsheet. And it also seems if I get a Key Error for 'location' that simply getting a new upload_url fixes it. But now after a hundred rows or so, it's stalling on the "Done prepping Canvas for upload, now sending the data..." response. So back to the drawing board! But at least I'm closer!

Maybe a programming genius like @James would be able to help us troubleshoot if he has the time?

[James]

@fisher1 

Strangely enough, I uploaded my first file to Canvas using the API just a few weeks ago. I was getting tired of manually uploading my trig lecture notes after I changed them and recompiled the LaTeX, so I wrote a script that would scan the folders on a local disk and compare them to the files inside Canvas, updating the files in Canvas when they local file was newer.

It took some trial and error to get it working properly. The thing that I thought would give me trouble, but ended up not, was the step 3: confirm the upload's success. I told my request to follow redirects and that worked.

I was only uploading a few files at a time (40 at most), definitely not what you would call a bulk avatar upload.

I notice in the source code that they have allow_redirects=False. I don't do any Canvas work in Python (I don't do much in Python), but based on what I found that with Node JS, that could be set to true and then skip step 3.

Here's my step 2. I took the upload_url returned from step 1. I then created a form and copied all of the upload_params over. Then I added a file parameter to my form as the last thing. However, it was still part of the form.

Here are some questions I would probably know the answer to if I used Python.

• Python is synchronous and requests are made in serial format? I've been using promises in JavaScript and have to use a lot of then statements to wait for the promise to finish before moving on.
• Is it the same file that fails every time or is it just after a bunch of them? That's tied to the synchronous nature. If too many calls are made asynchronously, like with JS promises, then you could hit the rate limiting. But if you can only make one call at a time, that shouldn't be an issue.
• The upload_file_res request has files=files in it. Does it manually add a form entry for file? I had to add the file parameter to the form itself. That's probably irrelevant since you said that's not the part having issues, but you also mentioned not using the upload URL that the request returned.
• Is there something different about the student that fails? An extra space in there somewhere? I normally add trim() to my code even though I'm the person creating the file, just to make sure. Maybe that student has a profile picture already?
• Is there something different about the file that fails than the others? For example, does it have capital letters or special characters in it?

You might try some debugging. For example, print out the inform_parameters object before you make the request and see if it's got something unusual in it before the request that gets hung up. I would also print the result and look for something unusual there.

You say it's hanging up on the "Done prepping for Canvas", but you don't really know that because there is no print statement in step 2. It might be step 3 or 4 that is failing. Make sure that the data object (end of step 1) has upload_params and upload_url in it. Then print the results of upload_file_res at the end of step 2 to make sure it's giving you what you think it should.

In step 3, the script is unilaterally making a get post call. First, it may not be necessary, you should check the http status from step 2. More importantly, the documentation says to make a get, not a post. That said, the example request shows a post with a content-length of 0. You're not specifying a content-length of 0, but that might be something that Python does for you. Still, the documentation reads like it should be a get. That's why my following redirects worked.

In step 4, why are you masquerading as the user when fetching the avatar information when you're specifying the user ID? I would also add a check to make sure that the type is attachment as all three types have a display name. You might also consider printing the display_name for each record to make sure it's not failing to find one. That may not be necessary, but it might be trying to get a token that doesn't exist. This is unlikely to be the problem -- you just asked me to troubleshoot a programming language I rarely use and a script I have never used and it's almost 2 am and I should have been in bed a couple of hours ago since I have to teach this morning so my mind is asking things it might not if I had more time.

In general, the script doesn't do any error checking. It could be a failed network request that is causing the problem. It just plows ahead like everything works without checking that it does. That could lead to a hang, which is why I'm suggesting adding debugging by printing what is sent and what is returned by the requests. I also have ran into trouble in my own scripts where I think something is right but I end up having a null inside a request (ex: /api/v1/courses/null/users) -- and of course, it fails.

[fisher1]

@James Thank you so much for your reply and wise advice as always!

After some testing it appears that I didn't need to switch out the upload_url value after all and that it was throwing the error because there was a student id in my spreadsheet that didn't match any student id in Canvas because they had left the school and no longer existed as active.

So with some help, I was able to use this code to include error handling (as you suggested) and skip the rows of students no longer at our school.

import csv
import requests
import os
import mimetypes

# Configuration
working_path = 'workingpath/'
csv_filename = 'data.csv'
images_path = 'imagefolderpath/'
domain = 'mydomain.instructure.com'
access_token = "ACCESS_TOKEN_HERE"

# Headers and mimetypes
header = {'Authorization': f'Bearer {access_token}'}
valid_mimetypes = ('image/jpeg', 'image/png', 'image/gif')

# Process CSV
with open(f"{working_path}/{csv_filename}") as csv_file:
    read_csv = csv.DictReader(csv_file)
    for row in read_csv:
        # Check if image file exists
        image_path = f"{working_path}{images_path}{row['image_filename']}"
        if not os.path.isfile(image_path):
            print(f"{image_path} does not exist, skipping to next record")
            continue

        mime_type = mimetypes.guess_type(image_path)[0]
        if mime_type not in valid_mimetypes:
            print(f"Invalid mimetype for file {row['image_filename']}, skipping to next record")
            continue

        # Inform Canvas about the file upload
        inform_api_url = f"https://{domain}/api/v1/users/self/files"
        inform_parameters = {
            'name': row['image_filename'],
            'content_type': mime_type,
            'size': os.path.getsize(image_path),
            'parent_folder_path': 'profile pictures',
            'as_user_id': f"sis_user_id:{row['user_id']}"
        }

        # Error handling for user not found
        try:
            res = requests.post(inform_api_url, headers=header, data=inform_parameters)
            res.raise_for_status()  # Raises an HTTPError if the response is not successful
            data = res.json()

            # Upload file
            files = {'file': open(image_path, 'rb')}
            upload_params = data.get('upload_params')
            upload_url = data.get('upload_url')
            upload_file_res = requests.post(upload_url, data=upload_params, files=files, allow_redirects=False)
            upload_file_res.raise_for_status()  # Check for upload success

            # Confirm upload and set avatar
            confirmation_url = upload_file_res.headers['location']
            confirmation = requests.post(confirmation_url, headers=header)
            confirmation.raise_for_status()  # Check confirmation success

            # Find uploaded image and set as avatar
            avatar_options = requests.get(f"https://{domain}/api/v1/users/sis_user_id:{row['user_id']}/avatars", headers=header)
            avatar_options.raise_for_status()  # Check for avatar options success

            for ao in avatar_options.json():
                if ao.get('display_name') == row['image_filename']:
                    token = ao.get('token')
                    params = {'user[avatar][token]': token}
                    set_avatar_user = requests.put(f"https://{domain}/api/v1/users/sis_user_id:{row['user_id']}", headers=header, params=params)
                    if set_avatar_user.status_code == 200:
                        print(f'Profile image set for user - {row["user_id"]}')
                    else:
                        print(f'Failed to set profile image for user - {row["user_id"]}')
                    break
            else:
                print(f"Uploaded image not found in avatar options for user - {row['user_id']}")

        except requests.exceptions.HTTPError as e:
            print(f"Error processing user {row['user_id']}: {e}")