Error 500 when uploading a file via API

[jack_c_kiefer]

I swear to the LMS gods that the following Python requests code was working just yesterday:

import requests

# Step 1: Tell Canvas you're going to upload a file

filename = 'test.svg'
folderPath = '/myFolder/test'

size = os.path.getsize(filename)
data = { 'name' : filename,
         'size' : str(size),
         'content_type' : 'image/svg+xml',
         'parent_folder_path' : folderPath }

response = requests.post(filesURL, json=data, headers=headers)
response.raise_for_status()
response = response.json()


# Step 2: Upload the file

files = list(response['upload_params'].items())    # Get all upload params
file_content = open('svg/'+filename, 'rb').read()  # Read in file data                                       
files.append((u'file', file_content))              # Add file data to payload
response = requests.post(response['upload_url'], files=files)
response.raise_for_status()
id = str(response.json()['id'])‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

However, in step 1, when I get the first response, it looks nothing like it used to or what's in the documentation:

{
    'file_param': 'file', 
    'progress': None, 
    'upload_url': 'https://inst-fs-iad-prod.inscloudgate.net/files?token=<crazy long token>', 
    'upload_params': 
    {
          'filename': 'c02s04n02a.svg', 
          'content_type': 'image/svg+xml'
    }
}
‍‍‍‍‍‍‍‍‍‍

The code produces the following error:

500 Server Error: Internal Server Error for url: https://inst-fs-iad-prod.inscloudgate.net/files?token=<blah etc.>‍

Am I missing something? Has a change of file repository occurred from AWS?

Would appreciate some help!

Thanks

[jstauffer]

I'm having the same issue. Below is a transcript of the requests/responses:

02/07/19 12:41:06 - ExecutePostRequestWithResponse [ StringContent : System.Object ]
[ request.RequestUri ] = https://<site>/api/v1/users/self/files
[ request.Content ] = {"name":"0771056.jpg","size":71912,"content_type":"image/jpeg","parent_folder_path":"profile pictures","on_duplicate":"overwrite","as_user_id":"1146"}
[ response.StatusCode ] = OK
[ response.ReasonPhrase ] = OK
[ response.Content ] = {"file_param":"file","progress":null,"upload_url":"https://inst-fs-iad-beta.inscloudgate.net/files?token=<token>","upload_params":{"filename":"0771056.jpg","content_type":"image/jpeg"}}

02/07/19 12:41:11 - ExecutePostRequestWithResponse [ MultipartFormDataContent ]
[ request.RequestUri ] = https://inst-fs-iad-beta.inscloudgate.net/files?token=<token>
[ request.Content ] = --49d2b870-1d97-4f67-994d-75bfb6dcb5f5
Content-Type: text/plain; charset=utf-8
Content-Disposition: form-data; name=filename

0771056.jpg
--49d2b870-1d97-4f67-994d-75bfb6dcb5f5
Content-Type: text/plain; charset=utf-8
Content-Disposition: form-data; name=content_type

image/jpeg
--49d2b870-1d97-4f67-994d-75bfb6dcb5f5
Content-Disposition: form-data; name=file

<stream data>

[ response.StatusCode ] = InternalServerError
[ response.ReasonPhrase ] = Internal Server Error
[ response.Content ] = {"error":"No file uploaded","sentryId":"708c005f324343c1993d5b2693feaf48"}

[jstauffer]

Note that when we were uploading to amazon it was working fine.

[James]

There was another person asking about 502 errors. We've had 3 SIS imports fail in the last 22 hours. Canvas says they're having trouble with uploads and that was fix was implemented at 12:56 MST (19:56 UTC) on February 7. I say that because they claimed the problem was fixed on February 6 by 4:12 MST. Hopefully this one truly resolves the issue.

Canvas Status - Some users are unable to upload files into Canvas 

Canvas Status - A small amount of Canvas users are experiencing slower load times than normal 

Here's the main Canvas Status Page. I check there first whenever some background task that was working fails for some reason.

[jstauffer]

We were having this issue well before the above incidents and the fixes didn't help us at all. We're still seeing what I posted above.

[jack_c_kiefer]

LIkewise; still seeing the same error.

[jstauffer]

We have a case open w/ them so hopefully we can figure out something. It's possible to upload the image manually by logging into canvas and I'm assuming they are sending the file to the same offsite location so something is definitely weird. If we resolve this I'll update my original post w/ the solution.

[jack_c_kiefer]

That would be very much appreciated; thank you!

[jstauffer]

I figured this out finally. It appears that posting to inscloudgate requires that you include the "filename" as part of the form data of the actual file (even though this is already specified as separate form data per Canvas uploading files documentation).

So instead of:

--cb243f9c-3bd8-48be-a88e-19c7579e0d68
Content-Type: text/plain; charset=utf-8
Content-Disposition: form-data; name=filename

12345.jpg
--cb243f9c-3bd8-48be-a88e-19c7579e0d68
Content-Type: text/plain; charset=utf-8
Content-Disposition: form-data; name=content_type

image/jpeg
--cb243f9c-3bd8-48be-a88e-19c7579e0d68
Content-Disposition: form-data; name=file
<image memory stream>
--cb243f9c-3bd8-48be-a88e-19c7579e0d68--

It should be:

--cb243f9c-3bd8-48be-a88e-19c7579e0d68
Content-Type: text/plain; charset=utf-8
Content-Disposition: form-data; name=filename

12345.jpg
--cb243f9c-3bd8-48be-a88e-19c7579e0d68
Content-Type: text/plain; charset=utf-8
Content-Disposition: form-data; name=content_type

image/jpeg
--cb243f9c-3bd8-48be-a88e-19c7579e0d68
Content-Disposition: form-data; name=file; filename=12345.jpg; filename*=utf-8''12345.jpg
<image memory stream>
--cb243f9c-3bd8-48be-a88e-19c7579e0d68--

Of course I'm not manually creating the above streamed data but rather using .NET's HttpClient and you just need to specify the filename for the image memory stream, the filename*=utf-8''12345.jpg is done automatically by the HttpClient.

[James]

You waited 13 days between responses and I know that Canvas was having problems again on the 20th when you responded. My wife was trying to upload a file through the web interface and couldn't, but nothing ever showed up on the Instructure Status Page about broken file uploads. The coincidence of the timing makes me wonder if it was working between Feb 7 and Feb 20 and just broke again or it has been failing this entire time and you just got around to responding?

I don't know one way or the other, I'm just pulling information from separate locations together trying find a common thread and the this is the third person I've seen say something about having upload problems of one kind or another on the 20th.

[jstauffer]

Sorry about the delay in response. We have automated processes, written in .NET, that run throughout the day which attempt to upload images for users (as well as other things). On failures we receive a report that indicates what the problem is however the we also get false errors on these reports which are due to attempting to verify that the image was successfully uploaded (a check we included after step 3 when uploading a file) and Canvas responding that it was unsuccessful or timing out, although the images were actually uploaded fine. The person that reviews these error logs didn't report the issue until last month but going back over the logs we can see that they have been failing since late last year, more specifically when we were switched from using amazon storage to inscloudgate storage. And even after the "fix" we have been having the issues every day.

As a note, when I manually upload an image by logging in and going to my files the files upload fine. Also when using curl (pausing application execution step 1's post to canvas to get the upload url) it uploads the file fine.

[James]

That definitely helps clarify the issue, thanks. Right now, I can't access Canvas at all -- getting server back end issues or the site cannot be reached. At least that is documented on their status page. Even if I could access it, I probably won't be able to help with this one. The only issues I've personally encountered with inscloudgate was getting messages about blocked third-party cookies. I was just trying to tell if it was related to the other file upload issues going on and it doesn't sound like it. Hopefully someone else has ideas.

[nicholas_matteo · Accepted solution]

I had the same errors, and found that a "filename" parameter which Requests adds to multipart/form-data parts was the problem.

To prevent Requests from adding this, the files argument for requests.post can be provided with tuples of values, giving None in the first position, so we have a list like [('Name', (None, 'value')), ('Name2', (None, 'value2'))].

So in the code sample in the original post, it might help changing line 21 to

files = [(key, (None, val)) for key, val in response['upload_params'].items()] # Get all upload params
‍

I figured this out from this Stackoverflow answer, so thanks to Martijn Pieters!

[jack_c_kiefer]

Ah, thank you! This has solved the issue for me! :smileygrin:

What a strange format! But, this definitely seems like the best solution as it is somewhat resilient to future changes in the API.

You have saved us!