Is it possible to automate the execution and downloading of reports via curl commants?
Canvas LMB, given the technical nature of this question I'm going to share it with the Canvas Developers group. They are the ones who do the back-end programming with Canvas and other systems and should be able to answer your question.
Take a look at the API documentation here:
Account Reports - Canvas LMS REST API Documentation
You should be able to form your curl commands based on the API definitions.
Generating a report will require at least three steps:
If there is a specific report you are trying to automate, reply and I'll look at the curl commands with you.
** subject to delay due to InstructureCon : )
I can see how to start the report
I can monitor the status of the report
I am unable to download the report. Do you know that curl command?
I would like to be able to download the report on to a Linux server
Look at the definition of the "Report" object:
One of the fields is defined as:
//The url to the report download. "file_url": "https://example.com/some/path"
When you ask for the status of the report is returns the report object.
When the report comes back as "complete", use the file_url to download the csv file.
To complement what Garth said, file_urls are not API calls, so do not include your authorization header when you download them.
If you're running from cURL from the command line, you'll want to use curl -o to specify the output name. wget might be better suited. If you're downloading it from within a program, many languages provide a function that will download a file for you and those are probably easier than using cURL. It's basically just an HTTP GET request, so you don't need anything fancy (like cURL) to get it.
When I use curl to call the file_urls link to download the file, I am immediately redirected to the login page. The same thing happens when I make the request using postman or drop the link directly in a browser. I guess that makes sense because I run and monitor the report via the api using an api access token to authenticate, and downloading the file is a different authentication process since it is not done via the api. We are using CAS on the front end for single sign on. How can I overcome this login requirement and use curl to retrieve the file generated by the report?
I'll do some testing on my end and come up with specific steps you can take.
Here are the steps I took to start a report, query the status of the report, then download the report:
1. Start the report by calling making this API call Account Reports - Canvas LMS REST API Documentation - Start a reportIn my scenario for ":report" I passed the value of "lti_report_csv"The response I got was the following:
2. Now make this API call until the progress is 100: Account Reports - Canvas LMS REST API Documentation - Status of a report
You may need to make this call multiple times, if you're automating this you might want to consider throttling the frequency of the calls.
The result I received looks like this:
3. Download the file using the "url" value, highlighted above in bold red. I put this link in a browser (where I was not logged into Canvas) and the file downloaded without needing to authenticate.
I apologize for calling out the wrong download link variable in my previous post.
Let me know if this gets you what you need.
Thanks for the research Garth! Those are actually the exact steps I am following. However, when I drop the url in a browser, I am immediately redirected to the login page if I do not currently have an authenticated session. Just to be clear, I am being redirected to our CAS single sign on page and not the canvas logon page. I am wondering, since we use CAS for authentication, is there some canvas configuration that is using cookies or some other mechanism to check for an authenticated session when trying to access any canvas page? In any event, I am not able to access the url from a browser, an http get, or using curl without being prompted to authenticate, so I can do everything I need to do programmatically except download the report. I kind of feel like CAS might be causing (or at least be part of) the issue here but I am not sure what to do about it.
I understand what you are saying, and I am able to download my report with zero authentication.
Please confirm which "url" variable you are using to download the file.
In my previous post, there is a json object, with the following variable:
Notice that this is not the "file_url".
Also notice that this "url" has a parameter "verifier".
If I use this url in a browser that is not logged into Canvas, with a session that has not been authenticated, I am able to download the file without logging into Canvas or passing a bearer token, i.e. I am able to download the file with zero authentication involved.
Please confirm that the json for your result matches the json that I have posted, and that you are using the "url" variable, not the "file_url".
I double checked and I am using the "url" parameter. Your post did highlight something that I had not noticed before. The url that is returned to you contains a section that looks like "download_frd=1&verifier=". The link that is being returned to me looks like "download_frd=1\u0026verifier=". When I replace the "\u0026" with "&", I am able to drop the link in a browser that does not have an authenticated session and successfully download the file. However, when I try the same link through postman, I get a long error message that begins "SignatureDoesNotMatchThe request signature we calculated does not match the signature you provided. Check your key and signing method". When I try the same link through curl, it still downloads a file with a redirect link to the login page.
Postman is a tool allowing you to execute the API calls and pass custom headers and parameters and inspect the response, Postman allows you to execute different HTTP verbs (i.e. GET, PUT, POST, etc.)
In your previous post you said the URL worked when you put it in a browser:
"I am able to drop the link in a browser that does not have an authenticated session and successfully download the file"
Why are you trying to use Postman to download a file? It isn't necessary.
When you click on a file link on a web page, it is not using Postman to download that file to your local machine.
Same concept, the "url" value is simply a link to the file.
Are you executing the API manually?
Or are you wrapping the API calls in your own code?
If you are writing your own code to execute the API calls, what language are you using?
So far I have just used postman to manually execute the API calls. I was using postman to understand all of the pieces and have my calls lined up before I develop the final product. I will be wrapping the API calls in an oracle pl/sql package that will be executed daily by control-m. Oracle pl/sql is what our group uses to develop so there is not really any wiggle room here. The goal is to have an automated process that runs the report, monitors the report progress, downloads the file, and then combines the results with some data from our student system so our reporting group can access the info from cognos locally. The download bit is the part I am struggling to automate and that is why I have been attempting to download the file via a GET in postman or through a curl script. I think I would rather use curl if possible because we have run into some API reply size limitations when making API calls in oracle.
Allen C Kelly Implementing the API in pl/sql seems like a harder way to go, but it boils down to what tools you are most comfortable with.
I was kind of expecting you to say PHP, or Ruby, or .NET, or Java, or something along those lines : )
Take a look at this article:
13 Developing Web Applications with PL/SQL
To quote the article: "Typically, developers have used Java or Perl to perform these operations"
Search the article for this text: "Retrieving the Contents of an HTTP URL from PL/SQL"
I also found a StackOverflow post on this subject:
database - Procedure to download file from a given url in Oracle 11g and save it into the blob type column - Stack Overf…
Have you looked into Oracle config options regarding reply/download size limitations?
I'm not sure what version of Oracle you are working with, but maybe these articles will lead you to the information you need.
You should create a blog post with your solution after you get it working, I would be interested to read it.
Retrieving data ...