Overview
A small Command Line Interface (CLI) tool for syncing data from the Canvas Data API directly to any of the tool's supported database types .
NOTE: This is currently an example application, but is supported, and welcome to contributions. Please report any bugs or issues you find!
Benefits of this tool compared to manually downloading files:
It pulls all Canvas Data flat files, and schema, into a specified database type.
- Is dynamic to handle schema version changes without breaking your database.
- Can be automated to run daily to keep your database constantly up to date.
- Has the ability to skip historical refreshes (backload of all pageviews in the request table).
Install
All of this needs to be done through your terminal (OSX) or command prompt (Windows).
The direct Github page can be found here.
Prerequisites
This tool should work on Linux, OSX, and Windows. You may need to have full admin access to modify, and download, certain tasks listed during the configuration steps below.
Clone Canvas Data Loader From Github
git clone https://github.com/instructure/canvas-data-loader.git
Configure
The Canvas Data CLI requires a configuration file with certain fields set. Canvas Data Loader uses a small toml file as the configuration file. To generate this configuration:
1. Run
cp ./config/default.toml ./config/local.toml
2. Navigate to local.toml
Navigate to the directory that the Canvas Data Loader was cloned to on your computer. Under the config folder, open the local.toml file with the text editor of your choice.
3. Edit 'Save Location'
Within the file, edit the saveLocation and unpackLocation to point to where you want to save the Canvas Data output files.
Example #1: saveLocation: '/Users/PandaUser/Desktop/dataFiles'
Example #2: saveLocation:'/Users/PandaUser/Documents/Canvas_Data_Ex/dataFiles'
4. Generate API Credentials
Click on the Canvas Data API Guide for reference on generating API credentials. Once you have this you must do one of the following:
A. Hard Coding Credentials (easier, but less secure)
Open your local.toml file from step 2
Replace the api_key and api_secret with the secret and key you generated from your Canvas Data instance surrounded by double quotes.
5. Provide URL to database
Provide the authenticated url in the local.toml file to your database that you wish to import into.
Supported databases at this time are:
- Postgres - Help on finding URL.
- Mysql - Format of URL should be mysql://<username>:<password>@<host>:<port>/<db_name>
6. Specify the database type
Within the local.toml file, specify the database type that you are using.
Supported databases at this time are:
- Psql
- Mysql
End result should appear like this:
# This determines the log level
save_location = "/tmp/cdl-save"
rocksdb_location = "/tmp/cdl-rocksdb"
skip_historical_imports = true
[canvasdataauth]
api_key = "12345"
api_secret = "6789"
[database]
url = "postgres://pandauser@localhost/panda-db"
# Valid Values are Psql, Mysql
db_type = "Psql"
7. Save
Save and close the local.toml file.
8. Install Rust
The Canvas Data Loader is built using the Rust programming language. Rust is necessary to be installed in order to use the Canvas Data Loader. You can install Rust by visiting their webpage here.
9. Add Rust to the PATH environment variable
Within your terminal/command prompt, Rust will finish installation with steps to add to your PATH environment variable.
For Example on Mac, you will see:
Rust is installed now. Great!
To get started you need Cargo's bin directory ($HOME/.cargo/bin) in your PATH
environment variable. Next time you log in this will be done automatically.
To configure your current shell run "source $HOME/.cargo/env"
You will need to copy source $HOME/.cargo/env to the next line and press Enter. Ensure that you are running this command within the root canvas-data-loader directory.
The exact command will differ for Windows. The concept will be same for adding to your PATH environment variable.
10. Build a Release Version
Copy cargo build --release into your terminal and press Enter.
11. Run the Importing Process
Navigate to the Canvas Data Loader Directory within your terminal/command prompt.
Example: cd ./canvas-data-loader
Once in the correct directory, run RUST_LOG=info ./target/release/cdl-runner
[OPTIONAL] Automate Downloading Into Database
It is beneficial to automate the running of the Canvas Data Loader to ensure that you are always using the most recent data set.
MAC
Information on crontab automation variables
Setup a crontab to run the importer every hour:
crontab -e
- Enter on it's own line, replacing the path to your importer:
0 * * * * cd <my_cdl_location> && RUST_LOG=info ./target/release/cdl-runner > /var/log/cdl-log 2>&1
WINDOWS
Create a scheduled task to run RUST_LOG=info ./target/release/cdl-runner.
Information on creating scheduled tasks
I'm having some trouble with the last step and am getting this error when running cdl-runner.