Quicker ways to Download Canvas Data

Jump to solution
venkat5121
Community Participant

We have created a json script to download Canvas Data through API. When we first started in 2017 it downloaded all 72 files then in less than 5min. now it takes anywhere between 30min-80min. After that we are loading these files individually to Oracle database which takes whole Night.

I am trying to see if there is a better design I should opt to to and solve this in atleast 3-4hr window.

I tried direct scripts to MySQL in this community and Canvas Data Viewer tool they have their own limitations. 

Any expert opinions or suggestions most welcome.

1 Solution
robotcars
Community Champion

Since CD is constantly growing from the start of your instance, the download and load time is always going to increase, until there is delta data option... but we've come up with some faster solutions, check these out.

Build a Canvas Data Warehouse on AWS in 30 minutes!

low cost solution with AWS CloudFormation Stack

or my setup

Managing Canvas Data with Embulk YAML configs for importing Canvas Data with Embulk, for any RDBMS and more

Canvas Data: CLI Instructure's command line tool to connect and download files from Canvas Data

If you're using Oracle and choose Embulk,  @afarnsworth ‌ was working implementing the OCI drivers vs the JDBC defaults, for quicker import.

View solution in original post