read csv file

[julielkz]

Hi 

I can get a URL for a error report. the url likes this

https://domain/files/userid/errors2018-10-13+04%3A34%3A03+UTC20181013-17095-16eso8l.csv

How can I read this csv in python. Can I use request too?

thanks

li

[robotcars]

These error reports? Error Reports - Canvas LMS REST API Documentation 

What process is generating this file, or where is the error report coming from?

[julielkz]

Hi Robert,

i use this https://domain/api/v1/accounts/account_id/sis_imports 

from there I can find 'url' for downloading the report , for example, the url can be

https://domain/files/userid/download?11330000005243390_processing_warnings_and_errors2018-10-13+04%3...

I want to open this csv, and read data from it. 

I use python. Hope this helps

thanks

li

[robotcars]

Once you have the file in a local directory.

Try CSV File Reading and Writing — Python 3.7.1rc2 documentation 

import csv
with open('names.csv', newline='') as csvfile:
    reader = csv.DictReader(csvfile)
    for row in reader:
        print(row['first_name'], row['last_name'])
‍‍‍‍‍

Some expanded examples

https://realpython.com/python-csv/

CSV - Python Programming Tutorials 

[julielkz]

Is there a way to read the file without downloading it? Currently I use csv.reader, but it only returns the log in page. 

I wonder if I can use the same token in the same script to request the data, like I did for sis import api call above?

thanks

li

[robotcars]

Without downloading it seems unlikely, at best you'd probably have to read it from memory (after it's downloaded), but I have no idea how that would be done or if it's even possible.

You can use the same token, just make sure you continue passing it in the headers of any request you make.

Not sure where dgrobani‌'s reply went but he and  @millerjm ‌ are far better at python than I am...

[dgrobani]

julielkz, I'm not certain, but I don't think you can read the file without first downloading it.

You should be able to do that like this, including  your token as carroll-ccsd‌ indicated:

auth_header = {'Authorization': 'Bearer {}'.format(token)}
content = requests.get(file_url, headers=auth_header).content
with open(output_file_name, 'wb') as file:
    file.write(content)‍‍‍‍

[julielkz]

Hi all,

The reason that I do not want to download the files manually because the report is generated daily(the filename and URL vary).

my script will find the URL of error report from sis_import API, and find user UIDs in that error report(the report is csv file downloaded from the URL).

If it is impossible to read the report file (csv) without downloading it, do you guys know whether these data can be pulled from another API? or maybe other method?

thanks again

li

[robotcars]

Are you getting the file from the SIS Import Status?

SIS Imports - Canvas LMS REST API Documentation 

  "errors_attachment": {
    "id": 59781524,
    "uuid": "poiuytrewzxcvbnm",
    "folder_id": null,
    "display_name": "sis_errors_attachment_6540920.csv",
    "filename": "26000000123456_processing_warnings_and_errors2018-10-15+04%3A46%3A19+UTC20181015-2864-1l3asdf.csv",
    "workflow_state": "processed",
    "content-type": "csv",
    "url": "*.instructure.com/files/59781524/download?download_frd=1&verifier=asdfghjkl",
    "size": 3794,
    "created_at": "2018-10-15T04:46:20Z",
    "updated_at": "2018-10-15T04:46:20Z",
    "unlock_at": null,
    "locked": false,
    "hidden": false,
    "lock_at": null,
    "hidden_for_user": false,
    "thumbnail_url": null,
    "modified_at": "2018-10-15T04:46:20Z",
    "mime_class": "file",
    "media_entry_id": null,
    "locked_for_user": false
  },‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

You could rename the file after downloading to the display name, or just errors.csv, and your script would see the same file. Or you could download the file, use the display name and match by

regular expression: sis_errors_attachment_(\d+).csv

concatenation: 'sis_errors_attachment_' + importid +'.csv'

Or try these...

SIS Import Errors - Canvas LMS REST API Documentation 

This one, takes the import id and returns a JSON response

/v1/accounts/{account_id}/sis_imports/{id}/errors :

{
  "sis_import_errors": [
    {
      "sis_import_id": 456789,
      "file": "daily-employees.csv",
      "message": "Could not save the user with user_id: 'E123456'. Unknown reason: Something broke with this user. Contact Support with ErrorReport id: 260000088012651",
      "row": 3610
    },
  ]
}‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

This one does not take an import id, last import is returned

/api/v1/accounts/{account_id}/sis_import_errors

Do you know about the Live API?

You can see the different endpoints and responses and choose accordingly.

https://youredu.instructure.com/doc/api/live#!/users.json

[julielkz]

Daniel

I will try your approach now.

li

[mnahmias]

I was able to read the report CSVs without saving them to a file. Here's what I did:

import csv
import io

import requests


# Assuming you already got the download link from somewhere else
download_link = 'asdf'

# The download link saves the CSV into memory
response = requests.get(url=download_link)
# Raise errors if any occurred
response.raise_for_status()

# Use StringIO to process the CSV contents since we're not working with a file
# Cast the DictReader as a list so you can iterate over it more than once
data = list(csv.DictReader(io.StringIO(response.text)))‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

[dgrobani]

Nice!