Official Canvas Document

• • Overview
  • Install
    • Prerequisites
    • Clone Canvas Data Loader From Github
    • Configure
      • 1. Run
      • 2. Navigate to local.toml
      • 3. Edit 'Save Location'
      • 4. Generate API Credentials
      • 5. Provide URL to database
      • 6. Specify the database type
      • 7. Save
      • 8. Install Rust
      • 9. Add Rust to the PATH environment variable
      • 10. Build a Release Version
      • 11. Run the Importing Process
    • [OPTIONAL] Automate Downloading Into Database
      • MAC
      • WINDOWS

Overview

A small Command Line Interface (CLI) tool for syncing data from the Canvas Data API directly to any of the tool's supported database types .

Benefits of this tool compared to manually downloading files:

• It pulls all Canvas Data flat files, and schema, into a specified database type.

• Is dynamic to handle schema version changes without breaking your database.
• Can be automated to run daily to keep your database constantly up to date.
• Has the ability to skip historical refreshes (backload of all pageviews in the request table).

Install

All of this needs to be done through your terminal (OSX) or command prompt (Windows).

The direct Github page can be found here.

Prerequisites

This tool should work on Linux, OSX, and Windows. You may need to have full admin access to modify, and download, certain tasks listed during the configuration steps below.

Clone Canvas Data Loader From Github

git clone https://github.com/instructure/canvas-data-loader.git 

Direction to install git.

Configure

The Canvas Data CLI requires a configuration file with certain fields set. Canvas Data Loader uses a small toml file as the configuration file. To generate this configuration:

1. Run

cp ./config/default.toml ./config/local.toml

2. Navigate to local.toml

Navigate to the directory that the Canvas Data Loader was cloned to on your computer. Under the config folder, open the local.toml file with the text editor of your choice.

3. Edit 'Save Location'

Within the file, edit the saveLocation and unpackLocation to point to where you want to save the Canvas Data output files.

Example #1: saveLocation: '/Users/PandaUser/Desktop/dataFiles'

 

Example #2: saveLocation:'/Users/PandaUser/Documents/Canvas_Data_Ex/dataFiles'

 

4. Generate API Credentials

Click on the Canvas Data API Guide for reference on generating API credentials. Once you have this you must do one of the following:

A. Hard Coding Credentials (easier, but less secure)

1. Open your local.toml file from step 2

2. Replace the api_key and api_secret with the secret and key you generated from your Canvas Data instance surrounded by double quotes.

5. Provide URL to database

Provide the authenticated url in the local.toml file to your database that you wish to import into.

Supported databases at this time are:

• Postgres -  Help on finding URL.
• Mysql - Format of URL should be mysql://<username>:<password>@<host>:<port>/<db_name>

6. Specify the database type

Within the local.toml file, specify the database type that you are using.

Supported databases at this time are:

• Psql
• Mysql

End result should appear like this:

# This determines the log level
save_location = "/tmp/cdl-save"
rocksdb_location = "/tmp/cdl-rocksdb"
skip_historical_imports = true

[canvasdataauth]
api_key = "12345"
api_secret = "6789"

[database]
url = "postgres://pandauser@localhost/panda-db"
# Valid Values are Psql, Mysql
db_type = "Psql"

7. Save

Save and close the local.toml file.

8. Install Rust

The Canvas Data Loader is built using the Rust programming language. Rust is necessary to be installed in order to use the Canvas Data Loader. You can install Rust by visiting their webpage here.

9. Add Rust to the PATH environment variable

Within your terminal/command prompt, Rust will finish installation with steps to add to your PATH environment variable. 

For Example on Mac, you will see:

Rust is installed now. Great!

To get started you need Cargo's bin directory ($HOME/.cargo/bin) in your PATH
environment variable. Next time you log in this will be done automatically.

To configure your current shell run "source $HOME/.cargo/env"

You will need to copy source $HOME/.cargo/env to the next line and press Enter. Ensure that you are running this command within the root canvas-data-loader directory.

10. Build a Release Version

Copy cargo build --release  into your terminal and press Enter.

11. Run the Importing Process

 Navigate to the Canvas Data Loader Directory within your terminal/command prompt. 

 Example: cd ./canvas-data-loader

• How to navigate to a directory on MAC
• How to navigate to a directory on WINDOWS

Once in the correct directory, run RUST_LOG=info ./target/release/cdl-runner

[OPTIONAL] Automate Downloading Into Database

It is beneficial to automate the running of the Canvas Data Loader to ensure that you are always using the most recent data set. 

MAC

Information on crontab automation variables

Setup a crontab to run the importer every hour:

• crontab -e
• Enter on it's own line, replacing the path to your importer: 0 * * * * cd <my_cdl_location> && RUST_LOG=info ./target/release/cdl-runner > /var/log/cdl-log 2>&1

WINDOWS

Create a scheduled task to run RUST_LOG=info ./target/release/cdl-runner.

Information on creating scheduled tasks