How to Use the Canvas Data Loader

Document created by Sydney McCann

on Jul 31, 2017•Last modified by Sydney McCann

on Oct 2, 2017

Version 9Show Document

Like • Show 3 Likes3
Comment • 12
View in full screen mode

Official Canvas Document

Overview

Install
Prerequisites

Clone Canvas Data Loader From Github

Configure
1. Run

2. Navigate to local.toml

3. Edit 'Save Location'

4. Generate API Credentials

5. Provide URL to database

6. Specify the database type

7. Save

8. Install Rust

9. Add Rust to the PATH environment variable

10. Build a Release Version

11. Run the Importing Process

[OPTIONAL] Automate Downloading Into Database
MAC

WINDOWS

Overview

A small Command Line Interface (CLI) tool for syncing data from the Canvas Data API directly to any of the tool's supported database types .

NOTE: This is currently an example application, but is supported, and welcome to contributions. Please report any bugs or issues you find!

Benefits of this tool compared to manually downloading files:

It pulls all Canvas Data flat files, and schema, into a specified database type.
Is dynamic to handle schema version changes without breaking your database.
Can be automated to run daily to keep your database constantly up to date.
Has the ability to skip historical refreshes (backload of all pageviews in the request table).

Install

All of this needs to be done through your terminal (OSX) or command prompt (Windows).

The direct Github page can be found here.

Prerequisites

This tool should work on Linux, OSX, and Windows. You may need to have full admin access to modify, and download, certain tasks listed during the configuration steps below.

Clone Canvas Data Loader From Github

git clone https://github.com/instructure/canvas-data-loader.git

Direction to install git.

Configure

The Canvas Data CLI requires a configuration file with certain fields set. Canvas Data Loader uses a small toml file as the configuration file. To generate this configuration:

1. Run

cp ./config/default.toml ./config/local.toml

2. Navigate to local.toml

Navigate to the directory that the Canvas Data Loader was cloned to on your computer. Under the config folder, open the local.toml file with the text editor of your choice.

3. Edit 'Save Location'

Within the file, edit the saveLocation and unpackLocation to point to where you want to save the Canvas Data output files.

Example #1: saveLocation: '/Users/PandaUser/Desktop/dataFiles'

Example #2: saveLocation:'/Users/PandaUser/Documents/Canvas_Data_Ex/dataFiles'

4. Generate API Credentials

Click on the Canvas Data API Guide for reference on generating API credentials. Once you have this you must do one of the following:

A. Hard Coding Credentials (easier, but less secure)

Open your local.toml file from step 2
Replace the api_key and api_secret with the secret and key you generated from your Canvas Data instance surrounded by double quotes.

5. Provide URL to database

Provide the authenticated url in the local.toml file to your database that you wish to import into.

Supported databases at this time are:

Postgres - Help on finding URL.
Mysql - Format of URL should be mysql://<username>:<password>@<host>:<port>/<db_name>

6. Specify the database type

Within the local.toml file, specify the database type that you are using.

Supported databases at this time are:

Psql
Mysql

End result should appear like this:

# This determines the log level
save_location = "/tmp/cdl-save"
rocksdb_location = "/tmp/cdl-rocksdb"
skip_historical_imports = true

[canvasdataauth]
api_key = "12345"
api_secret = "6789"

[database]
url = "postgres://pandauser@localhost/panda-db"
# Valid Values are Psql, Mysql
db_type = "Psql"

7. Save

Save and close the local.toml file.

8. Install Rust

The Canvas Data Loader is built using the Rust programming language. Rust is necessary to be installed in order to use the Canvas Data Loader. You can install Rust by visiting their webpage here.

9. Add Rust to the PATH environment variable

Within your terminal/command prompt, Rust will finish installation with steps to add to your PATH environment variable.

For Example on Mac, you will see:

Rust is installed now. Great!

To get started you need Cargo's bin directory ($HOME/.cargo/bin) in your PATH
environment variable. Next time you log in this will be done automatically.

To configure your current shell run "source $HOME/.cargo/env"

You will need to copy source $HOME/.cargo/env to the next line and press Enter. Ensure that you are running this command within the root canvas-data-loader directory.

The exact command will differ for Windows. The concept will be same for adding to your PATH environment variable.

10. Build a Release Version

Copy cargo build --release into your terminal and press Enter.

11. Run the Importing Process

Navigate to the Canvas Data Loader Directory within your terminal/command prompt.

Example: cd ./canvas-data-loader

Once in the correct directory, run RUST_LOG=info ./target/release/cdl-runner

[OPTIONAL] Automate Downloading Into Database

It is beneficial to automate the running of the Canvas Data Loader to ensure that you are always using the most recent data set.

MAC

Information on crontab automation variables

Setup a crontab to run the importer every hour:

crontab -e
Enter on it's own line, replacing the path to your importer: 0 * * * * cd <my_cdl_location> && RUST_LOG=info ./target/release/cdl-runner > /var/log/cdl-log 2>&1

WINDOWS

Create a scheduled task to run RUST_LOG=info ./target/release/cdl-runner.

Information on creating scheduled tasks

2 people found this helpful

Attachments

Outcomes

Nil Shah Aug 17, 2017 7:09 PM

I'm having some trouble with the last step and am getting this error when running cdl-runner.

Actions

Bob Edmison @ Nil Shah on Nov 17, 2017 8:25 AM

I'm getting a similar error:

INFO:cdl_runner: Setting up API Client...

thread 'main' panicked at 'Failed to get List of Dumps: Error(Hypererror(Io(Error { repr: Custom(Custom { kind: Other, error: ErrorImpl { code: Message("invalid type: map, expected a sequence"), line: 1, column: 1 } }) })), State { next_error: None, backtrace: None })', src/libcore/result.rs:906:4

note: Run with `RUST_BACKTRACE=1` for a backtrace.

Actions

Jeff Longland @ Bob Edmison on Dec 14, 2017 3:12 PM

I ran into this error when I had the incorrect API keys in conf/local.toml.

Actions

Stuart Smith Aug 24, 2017 5:44 PM

This sounds as though it is an improvement on the canvasDataCli tool. Is there a plan to extend support to other databases?

Actions

Sydney McCann

@ Stuart Smith on Nov 17, 2017 10:53 AM

Hi Stuart,

We are planning to add support for SQL server. If we can manage it, our long-term goal is to try and add support for Oracle as well.

Actions

Stuart Smith @ Sydney McCann on Nov 19, 2017 3:23 PM

Hi Sydney,

Thanks for the update. In my experience Oracle is the only database platform which can handle a dimensionally-modelled star schema well. It supports bitmap indexes for fast fact-dimension joins, implicit star schema transformation, partitioning and compression to handle data such as requests data. Oracle support for materialised views is also the best available, with fast refresh capability and with the latest release you can even get real-time data from a materialised view.

The ability of the Data Loader to handle upload direct to the database and to skip full historical requests dumps sound very useful. However, I'm not clear how schema changes are handled, clearly if a new table is included or an old one dropped or a column added or dropped, the database will need to be changed accordingly.

Regards,

Stuart.

Actions

Benson Chen @ Sydney McCann on Feb 21, 2018 3:31 PM

Is there a timeline for when Oracle will be added to the supported list?

Actions

Brett Dalton

Nov 15, 2017 4:50 PM

Sydney McCann is there any facility in this tool to skip the requests table given its size grows extremely quickly?

Actions

Sydney McCann

@ Brett Dalton on Nov 16, 2017 11:25 AM

Actually, the Canvas Data Loader only keeps one day's worth of requests. It does not accumulate each day's worth of requests for this exact reason.

Actions

LiLing Yan Mar 1, 2018 5:15 PM

When I ran the CLI loader, it take 40 mins to download, but ~8 hours to unpack those files because some files are extremely big. Is there way we can make the process faster? 8 hours is not acceptable for business. Appreciate your thoughts and creative ideas.

Actions

Sunder Nagarajan @ LiLing Yan on Mar 9, 2018 2:23 PM

I have been running a powershell script to unpack and load the files on a Windows machine and it does not take very long - but to your point our files are not too big as we are newbies on Canvas.

Actions

Sunder Nagarajan Mar 9, 2018 2:19 PM

I have been trying to compile the cdl-runner on a windows machine and get the following error:

liblibrocksdb_sys-8964c77fbce86bb9.rlib(compaction_job.o) : error LNK2001: unresolved external symbol "public: void __cdecl rocksdb::port::WindowsThread::join(void)" (?join@WindowsThread@port@rocksdb@@QEAAXXZ)
          liblibrocksdb_sys-8964c77fbce86bb9.rlib(backupable_db.o) : error LNK2001: unresolved external symbol "public: void __cdecl rocksdb::port::WindowsThread::join(void)" (?join@WindowsThread@port@rocksdb@@QEAAXXZ)
          liblibrocksdb_sys-8964c77fbce86bb9.rlib(env_win.o) : error LNK2001: unresolved external symbol "public: void __cdecl rocksdb::port::WindowsThread::join(void)" (?join@WindowsThread@port@rocksdb@@QEAAXXZ)
          liblibrocksdb_sys-8964c77fbce86bb9.rlib(delete_scheduler.o) : error LNK2001: unresolved external symbol "public: void __cdecl rocksdb::port::WindowsThread::join(void)" (?join@WindowsThread@port@rocksdb@@QEAAXXZ)
          liblibrocksdb_sys-8964c77fbce86bb9.rlib(threadpool_imp.o) : error LNK2001: unresolved external symbol "public: void __cdecl rocksdb::port::WindowsThread::join(void)" (?join@WindowsThread@port@rocksdb@@QEAAXXZ)
          liblibrocksdb_sys-8964c77fbce86bb9.rlib(threadpool_imp.o) : error LNK2019: unresolved external symbol "public: bool __cdecl rocksdb::port::WindowsThread::detach(void)" (?detach@WindowsThread@port@rocksdb@@QEAA_NXZ) referenced in function "public: void __cdecl rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned __int64)" (?BGThread@Impl@ThreadPoolImpl@rocksdb@@QEAAX_K@Z)
          C:\Users\canvas-data-loader\target\release\deps\cdl_runner-ca016ca32e7d556d.exe : fatal error LNK1120: 6 unresolved externals

No other errors were encountered during the compile.

Actions

12 Comments

Nil Shah Aug 17, 2017 7:09 PM
I'm having some trouble with the last step and am getting this error when running cdl-runner.
Like • Show 0 Likes0
Actions
- Bob Edmison @ Nil Shah on Nov 17, 2017 8:25 AM
  I'm getting a similar error:
  INFO:cdl_runner: Setting up API Client...
  
  thread 'main' panicked at 'Failed to get List of Dumps: Error(Hypererror(Io(Error { repr: Custom(Custom { kind: Other, error: ErrorImpl { code: Message("invalid type: map, expected a sequence"), line: 1, column: 1 } }) })), State { next_error: None, backtrace: None })', src/libcore/result.rs:906:4
  
  note: Run with `RUST_BACKTRACE=1` for a backtrace.
  Like • Show 0 Likes0
  Actions
  - Jeff Longland @ Bob Edmison on Dec 14, 2017 3:12 PM
    I ran into this error when I had the incorrect API keys in conf/local.toml.
    Like • Show 0 Likes0
    Actions
Stuart Smith Aug 24, 2017 5:44 PM
This sounds as though it is an improvement on the canvasDataCli tool. Is there a plan to extend support to other databases?
Like • Show 0 Likes0
Actions
- Sydney McCann @ Stuart Smith on Nov 17, 2017 10:53 AM
  Hi Stuart,
  
  We are planning to add support for SQL server. If we can manage it, our long-term goal is to try and add support for Oracle as well.
  Like • Show 1 Like1
  Actions
  - Stuart Smith @ Sydney McCann on Nov 19, 2017 3:23 PM
    Hi Sydney,
    Thanks for the update. In my experience Oracle is the only database platform which can handle a dimensionally-modelled star schema well. It supports bitmap indexes for fast fact-dimension joins, implicit star schema transformation, partitioning and compression to handle data such as requests data. Oracle support for materialised views is also the best available, with fast refresh capability and with the latest release you can even get real-time data from a materialised view.
    The ability of the Data Loader to handle upload direct to the database and to skip full historical requests dumps sound very useful. However, I'm not clear how schema changes are handled, clearly if a new table is included or an old one dropped or a column added or dropped, the database will need to be changed accordingly.
    Regards,
    Stuart.
    Like • Show 0 Likes0
    Actions
  - Benson Chen @ Sydney McCann on Feb 21, 2018 3:31 PM
    Is there a timeline for when Oracle will be added to the supported list?
    Like • Show 0 Likes0
    Actions
Brett Dalton Nov 15, 2017 4:50 PM
Sydney McCann is there any facility in this tool to skip the requests table given its size grows extremely quickly?
Like • Show 0 Likes0
Actions
- Sydney McCann @ Brett Dalton on Nov 16, 2017 11:25 AM
  Actually, the Canvas Data Loader only keeps one day's worth of requests. It does not accumulate each day's worth of requests for this exact reason.
  Like • Show 0 Likes0
  Actions
LiLing Yan Mar 1, 2018 5:15 PM
When I ran the CLI loader, it take 40 mins to download, but ~8 hours to unpack those files because some files are extremely big. Is there way we can make the process faster? 8 hours is not acceptable for business. Appreciate your thoughts and creative ideas.
Like • Show 0 Likes0
Actions
- Sunder Nagarajan @ LiLing Yan on Mar 9, 2018 2:23 PM
  I have been running a powershell script to unpack and load the files on a Windows machine and it does not take very long - but to your point our files are not too big as we are newbies on Canvas.
  Like • Show 0 Likes0
  Actions
Sunder Nagarajan Mar 9, 2018 2:19 PM
I have been trying to compile the cdl-runner on a windows machine and get the following error:
liblibrocksdb_sys-8964c77fbce86bb9.rlib(compaction_job.o) : error LNK2001: unresolved external symbol "public: void __cdecl rocksdb::port::WindowsThread::join(void)" (?join@WindowsThread@port@rocksdb@@QEAAXXZ)
          liblibrocksdb_sys-8964c77fbce86bb9.rlib(backupable_db.o) : error LNK2001: unresolved external symbol "public: void __cdecl rocksdb::port::WindowsThread::join(void)" (?join@WindowsThread@port@rocksdb@@QEAAXXZ)
          liblibrocksdb_sys-8964c77fbce86bb9.rlib(env_win.o) : error LNK2001: unresolved external symbol "public: void __cdecl rocksdb::port::WindowsThread::join(void)" (?join@WindowsThread@port@rocksdb@@QEAAXXZ)
          liblibrocksdb_sys-8964c77fbce86bb9.rlib(delete_scheduler.o) : error LNK2001: unresolved external symbol "public: void __cdecl rocksdb::port::WindowsThread::join(void)" (?join@WindowsThread@port@rocksdb@@QEAAXXZ)
          liblibrocksdb_sys-8964c77fbce86bb9.rlib(threadpool_imp.o) : error LNK2001: unresolved external symbol "public: void __cdecl rocksdb::port::WindowsThread::join(void)" (?join@WindowsThread@port@rocksdb@@QEAAXXZ)
          liblibrocksdb_sys-8964c77fbce86bb9.rlib(threadpool_imp.o) : error LNK2019: unresolved external symbol "public: bool __cdecl rocksdb::port::WindowsThread::detach(void)" (?detach@WindowsThread@port@rocksdb@@QEAA_NXZ) referenced in function "public: void __cdecl rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned __int64)" (?BGThread@Impl@ThreadPoolImpl@rocksdb@@QEAAX_K@Z)
          C:\Users\canvas-data-loader\target\release\deps\cdl_runner-ca016ca32e7d556d.exe : fatal error LNK1120: 6 unresolved externals

No other errors were encountered during the compile.
Like • Show 0 Likes0
Actions

How to Use the Canvas Data Loader

Overview

Install

Prerequisites

Clone Canvas Data Loader From Github

Configure

1. Run

2. Navigate to local.toml

3. Edit 'Save Location'

4. Generate API Credentials

5. Provide URL to database

6. Specify the database type

7. Save

8. Install Rust

9. Add Rust to the PATH environment variable

10. Build a Release Version

11. Run the Importing Process

[OPTIONAL] Automate Downloading Into Database

MAC

WINDOWS

Attachments

Outcomes

Related Content

Recommended Content

Incoming Links