We have created a json script to download Canvas Data through API. When we first started in 2017 it downloaded all 72 files then in less than 5min. now it takes anywhere between 30min-80min. After that we are loading these files individually to Oracle database which takes whole Night.
I am trying to see if there is a better design I should opt to to and solve this in atleast 3-4hr window.
I tried direct scripts to MySQL in this community and Canvas Data Viewer tool they have their own limitations.
Any expert opinions or suggestions most welcome.
Solved! Go to Solution.
Since CD is constantly growing from the start of your instance, the download and load time is always going to increase, until there is delta data option... but we've come up with some faster solutions, check these out.
low cost solution with AWS CloudFormation Stack
or my setup
Managing Canvas Data with Embulk YAML configs for importing Canvas Data with Embulk, for any RDBMS and more
Canvas Data: CLI Instructure's command line tool to connect and download files from Canvas Data
If you're using Oracle and choose Embulk, @afarnsworth was working implementing the OCI drivers vs the JDBC defaults, for quicker import.