Skip navigation
All Places > Canvas Developers > Blog > Author: Robert Carroll

Canvas Developers

6 Posts authored by: Robert Carroll

Embulk is an open-source bulk data loader that helps data transfer between various databases, storages, file formats, and cloud services. embulk.org github contributors

 

Simply put, Embulk makes importing gzipped CSV files into any RDBMS* and managing the data and workflow necessary for Canvas Data using command line tools easy, really easy, specifically solving issues we experience working with Canvas Data without fancier tools.

 

with support for

Linux, OSX, Windows https://github.com/embulk/embulk#quick-start

MySQL, MS SQL Server, Oracle, PostgreSQL, RedShift https://github.com/embulk/embulk-output-jdbc

 

* Embulk goes beyond SQL, List of Embulk Plugins by Category

 

and features useful for Canvas Data

  • Decode gzipped files
  • The ability to intelligently guess the format and data types of CSV files
  • Parallel execution of tasks, multi-threading per CPU core, and a task for each batch file
  • Input CSV Plugin as default Input for Embulk
  • Filter data with Filter Plugins, https://plugins.embulk.org/#filter
  • Output Data to SQL
    • Insert, Insert Direct, Replace, Merge, Truncate and Truncate Insert
    • Timestamp formatting
    • TimeZone conversion from UTC for date time columns
    • before_load and after_load, config options to run queries before (truncate) and after import (indexes)
    • and more

 

Embulk uses YAML config files for each task, for Canvas Data this means each input source (table files) and it's output destination (db table) is 1 file. This includes differences between staging, test and production destinations. I imagine your workflow and setup will be different than mine and many others. You may only need a few tables, or only have one database, or you could simply use Embulk to manage, manipulate, filter and possibly join CSV files to examine with Tableau if that's your thing. For this reason, I have only shared each set of config files for MySQL, MSSQL, Oracle, and PostgreSQL. I have not worked with RedShift.

 

Our old workflow, requires that we attempt to maintain the newest data from Canvas Data for reporting, attendance, API services and automation, and LTIs. One of our biggest issues is the size of the daily batch without deltas and the growing use of Canvas within our schools and how long importing everything can take, how slow and unnecessary it is to hold 6 years worth of data for this semester, tried different things in SQL and bash to limit the data quickly for the current school year in production, never implement. LTI queries for attendance and submissions are really slow. Then some days the downloaded files are 0 bytes, we must have lost internet, or there was duplicates and the table didn't load, and it takes until 2pm to get everything loaded. Sometimes there's new columns in the table and I forgot to read the release notes and we've truncated the table before importing, and it takes hours to import. And so on.

 

Some of these are human, some of these are manageable.

 

Our new workflow uses Embulk

  1. Download with Canvas Data CLI, some of that documented here
  2. Import all CD tables using CSV in SQL out to staging environment with Replace mode, this creates temporary tables for the import, if it fails, the previous version is still intact. After successful import, Embulk will drop the old table and run the after_load queries, I use this for enumerable constraints and indexes. I left a lot of examples in the configs.

    The Requests table config uses Insert mode to append the new rows.
  3. I use staging for Tableau reporting. For production, I only need to load the tables necessary for our LTIs and API services. Some of these configs are straight copies of the staging imports, except they point to production. Some of the configs create new tables using SQL in SQL out and importing filtered or composite tables from query results using https://github.com/embulk/embulk-input-jdbc

    heres' an example https://github.com/ccsd/canvas-data-embulk-configs/wiki/SQL-in-SQL-out

 

Using RHEL7, 6 CPUs with 12 cores, and 16GB Ram, Embulk imports 7.9GB of CSVs into >1TB of SQL (no requests) in less than 4.5 hours, depending on which indexes you keep in the configs.

 

GitHub - ccsd/canvas-data-embulk-configs: YAML configs for importing Canvas Data with Embulk

 

CHANGELOG

Sep 19, 2018 > Sep 21, 2018 12:28 PM

Jun 6, 2018

May 31, 2018

 

README

If you're a system administrator of a Canvas LMS instance with a deep organization of sub accounts you have inevitably found yourself in one of Dante's 9 Circles of Hell, repetitively clicking Sub Accounts at each and every level as you drill down to where you need to go. Here at CCSD we have over 8,000 sub accounts, and yet this only affects 5 people. So we will share this to help anyone else who is trapped.

 

The JavaScript and CSS files here add a directory style recursive HTML menu to the Canvas global navigation admin tray...

 

In Pictures

CollapsedExpandedSearch
Admin Tray CollapsedAdmin Tray ExpandedAdmin Tray Search

Configuration & Setup

Zero to little configuration is needed for admintray-subaccmenu.js, so I won't even bother explaining subacctray.cfg

 

Option 1

  1. Host the file admintray-subaccmenu.js on a secure server (https) or somewhere like Amazon S3
  2. Copy/append the contents of admintray-subaccmenu.inc.js to your Canvas Theme JavaScript file, point the URL to the hosted file.

Option 2 (Quick Start)

  1. Just copy the example snippet below that uses this repo's version and RawGit   to your Canvas Theme JavaScript file. However, please note that the RawGit URL points to the repo source and any changes to it may affect your usage later. I recommend hosting your own for stability, but this can get you started.

 

Once loaded the script will initialize an API call (to /api/v1/accounts/self/sub_accounts) and loop through x/100, collecting every sub account in your instance, compile a recursive HTML unordered list and store it your browser's localStorage.

Depending on the number of your sub accounts and internet speed, this will take a moment, be patient for the first load, open the tray and wait for it to show up. This takes us about 45-60 seconds on production. You will see a loading indicator in the Admin Tray while it compiles.

 

Features & Use

 

Instance Independent

There are some users (like Instructure Canvas Remote Admins) who may login to multiple Canvas Instances/institutions, so each menu will be stored in localStorage based on the uniqueness of x.instructure.com, including x.beta.instructure.com, x.test.instructure.com, and xyz.instructure.com.

For most users and institutions this will go mostly unnoticed unless you have different sub accounts between your Production, Test and Beta environments. I made this update to help those users it will affect. I also personally hate copying, pasting or replicating files just to change 1 line. This update allows 1 file to be hosted on a CDN like Amazon S3 and simply change the var show_results_parent value in admintray-subaccmenu.inc.js or leave it blank.

Therefore an already minified file has been provided in the repo.

 

Alphabetical Sorting

Yup! As far as I know, the Canvas API does not allow sorting API calls alphabetically during pagination. The entire stack of sub accounts is sorted prior to building the menu.

Localized Search

Using JavaScript/jQuery to search within the stored HTML, preventing further API calls. You can search the entire sub account menu by name.

Search Result - Prefix Parent Account (Skip-to-Depth Display)

I don't know what else to call it, so here is an explanation.

Suppose your directory structure looks something like the following, where (#) is the depth of the account.

  • High Schools (1)
    • George Washington HS (2)
      • Course Shells (3)
      • SIS Courses (3)
        • English (4)
        • Math (4)
        • Science (4)
  • Middle Schools (1)
    • Betsy Ross MS (2)
      • Course Shells (3)
      • SIS Courses (3)
        • English (4)
        • Math (4)
        • Science (4)

 

When we search for Science and get multiple sub accounts of 'Science', we can't identify which one we want to choose.

  1. Science
  2. Science

So if we map the results depth of 4 to it's parent depth at 2 (instead of 3, SIS Courses) we can get a result like:

  1. George Washington HS > Science
  2. Betsy Ross MS > Science

 

And both are links to their respective account.

 

Examples

// one skip
show_results_parent = { 4:2 }
// expected result:
// George Washington HS > Science

// or multiple skips, must be unique
show_results_parent = { 4:2, 3:2 }
// expected results:
// (4:2) George Washington HS > Science
// (3:2) George Washington HS > SIS Courses

Note: If you don't define the skip, it will not display a parent

Reload

The ↻ button at the bottom right of the menu will clear the menu from localStorage and recompile it for the current Canvas instance. Use this when you or other admins have added or removed sub accounts.

 

User Impact

As mentioned above, CCSD has 5 system admins, with 40,000 Employees and over 300,000 Students. Be kind to your users, use the snippet below to reduce the impact of your admin-only tools. This is included in the repo as admintray-subaccmenu.inc.js

if (ccsd.util.hasAnyRole('admin','root_admin')) {
// used for search results, result:parent, see documentation for more details
var show_results_parent = {}
// async load the sub account menu script
$.ajax({
url: 'https://cdn.rawgit.com/robert-carroll/ccsd-canvas/master/admintray-subaccmenu/admintray-subaccmenu.min.js',
dataType: 'script',
cache: true,
data: {skipd: JSON.stringify(show_results_parent)}
})
}

 

User Script

At the suggestion and some coaching by James Jones, I have created a User Script version. This is useful for those admins that can't or don't want to install this script in their Canvas Theme/Global JavaScript. This script is identical to the global JavaScript file, except that the CSS will be added to the DOM by the script and has the various userscript requirements at the top for Tampermonkey.

  1. You will need to install and enable the Tampermonkey browser extension
  2. Install the UserScript version of admintray-subaccmenu.user.js

 

Known Issues

If you are not a root admin of your Canvas Instance, you will need to set (1) sub account in the root setting. At this time you cannot add multiple sub accounts. I am planning to fix this.

 

Code repo here...

ccsd-canvas/admintray-subaccmenu at master · robert-carroll/ccsd-canvas · GitHub

 

 

Credits for Blog Post Banner Image

Public domain

Chart of Hell
File:Sandro Botticelli - La Carte de l'Enfer.jpg - Wikimedia Commons
 

Botticelli : de Laurent le Magnifique à Savonarole : catalogue de l'exposition à Paris, Musée du Luxembourg, du 1er octobre 2003 au 22 février 2004 et à Florence, Palazzo Strozzi, du 10 mars au 11 juillet 2004. Milan : Skira editore, Paris : Musée du Luxembourg, 2003. ISBN 9788884915641

 

{{PD-US}}
This media file is in the public domain in the United States. This applies to U.S. works where the copyright has expired, often because its first publication occurred prior to January 1, 1923. See this page for further explanation.

Developer Tools for the Canvas User > Tutorial #3

 

For this tutorial we are going to combine what we learned in tutorial's 1 and 2. The goal will be to make an AJAX request with JavaScript to the Canvas API, and then when the request is successful update the DOM with an updated value. The Document Object Model or DOM, is the webpage as your browser sees it. I'll get you through this tutorial without any more details, but if you're curious here are some resources:

W3 Schools - JavaScript DOM Methods

MDN - Document Object Model (DOM) - Web APIs

 

As part of this rapidly escalating tutorial I am no longer going to use pictures. Anything you need should be referenced from the previous tutorials.

 

  1. Grab the snippet from the last tutorial and add the following code (lines 10-14) to the end.
    var payload = {
         "nickname":"My W○RKR∞M ⸚ Test"     // what we are changing the nickname to
    }

    $.ajax({
         url : '/api/v1/users/self/course_nicknames/1214595',
         type: 'PUT',
         data: payload
    })
    // when the request is resolved
    .done(function(response, status) {
         console.log(response)
    })
    • The jQuery done() method allows us to do something when the AJAX request is complete. For now, copy or type this code into your Console tab, and press Enter
    • The console.log() method allows us to display messages to the console. This is generally used for debugging or displaying other messages while developing. The user will never see these messages unless they have the Console tab open.
    • The response should look just like the one you saw in the Preview pane of the Network tab from Tutorial #2
  2. Now, you will update the name of the Course Card...
    Note: We could have done this with the payload value, but this is a better example of making an AJAX request and doing something when the request is finished and successful.
  3. First we have to find a Selector we can use to identify the element we want to change. 
    Right click on the Course Card where you see the course nickname (the text with color), and Inspect Element
    Hopefully, you will land on a element that looks something like this
    <span style="color:#FDC010;" data-reactid=".1.$1266609.0.2.0.0.0">W○RKR∞M⸚CARROLL</span>
    Note the style="" attribute with value color:#FDC010;
    We will use this as our selector for this tutorial.[1][2]

  4. Let's update our code and use jQuery to update the element after the AJAX request completes, see line 12
    We will use the response.nickname value
    var payload = {
         "nickname":"My W○RKR∞M ⸚ Test"     // what we are changing the nickname to
    }

    $.ajax({
         url : '/api/v1/users/self/course_nicknames/1214595',
         type: 'PUT',
         data: payload
    }).done(function(response, status) {
         $('span[style*="color:#FDC010;"]').html(response.nickname)
    })
    Press Enter, after updating the code in the Console tab
  5. You should see the nickname of your course card update
    If not, try refreshing the page and reset or adjust some of the values you are passing with the code.

 

Congrats, AGAIN! 

This is the basic workflow of a Canvas end-user modification. These are some of the fundamental practices used in making interactive web pages all across the web. You can expand these skills to make your own custom Canvas modifications. Remember to review the Canvas API Documentation and thoroughly plan out the objective and goals your script is trying to achieve.

 

[1] There a plenty of resources on the web for CSS and HTML selectors for interacting with DOM elements via JavaScript, check with Google.

[2] This is because Canvas uses React - A JavaScript library for building user interfaces, which allows elements to load or update after the Primary DOM loads. This is noticeable via the attribute data-reactid="". Usually unique elements will have an id="" attribute if the element is unique and a class="" attribute if there are similar elements. This creates problems when jQuery tries to interact with these elements, because they are not part of the page when jQuery loads. We would prefer to use predictable Selectors whenever possible. This is a more advanced topic and may be covered in a later tutorial.

 

 Rapid Escalation

Repeat Tutorial 3 and update the color of the Course Card, after sending a new color to the Canvas API.

Developer Tools for the Canvas User > Tutorial #2

 

Before getting started with APIs, read through paragraphs 1-5 of the linked thread by Stuart Ryan

Canvas APIs: Getting started, the practical ins and outs, gotchas, tips, and tricks

Stuart confirmed there is a working beta environment on the Canvas Free For Teachers accounts at https://canvas.beta.instructure.com


To save on lengthy screenshots, I've combined some panels, please let me know if this causes confusion.

 

  1. Now that you have the console open, click on the Network tab, and the ⃠ Clear button on the left
    This will clear all the requests that happened during page load
    Drag out some space with the border between the ViewPort (on the left) and the Developer Tools
  2. Now click on the edit button at the top right of any Course Card, change the nickname and the color, click Apply
    You should see two new requests with the course number
  3. Click on the request prefixed with course_ and then the Network Headers tab
    Take a quick peak at the following:
    Request URL - this is the URL (or endpoint) of the request we want to send to the API to change the color, it contains your user and course id's
    MethodPUT - HTTP, and all the Request Methods - HTTP | MDN 
    Status Code200 OK, and all the Response Status Codes - HTTP | MDN 
    Content Typehttps://www.google.com/search?q=JSON
  4. Now click on the Network Preview tab
    This is JSON, I hope you clicked that last link.
    I'll walk you through with Colors first, it's a simple place to start.
    // this is a simple JSON object, with the hex value of the color you chose
    // you'll usually see JSON like you see in the console, with all the white space removed
    {hexcode: "#F0592B"}

    // but it's often useful to nest it with tabs while you're working and learning
    {
         hexcode: "#F0592B" // this is the data that was sent to Canvas to save the color
    }

    // it has one (1) value, simple right?

    Canvas Developer Tools | Change Color

  5. That's 5 pieces of information we need to change the color with the API
    URL - API endpoint/api/v1/users/:id/colors/:asset_string
    MethodPUT - replaces all current representations of the target resource with the request payload.
    ID - your user idself - use self when you can't remember your user_id [1]
    Asset Stringcourse_# - get the course number from the course URL
    Parameters{hexcode: "#F0592B"} // the payload you will PUT to the URL
  6. Let's keep it simple and use jQuery to make this API request
    Type or copy the 10 lines below into the Console tab, and edit the course number in the URL
    After you get this snippet into the console, hit Enter
    var payload = { hexcode: '#FDC010' } // yellow

    // http://api.jquery.com/jquery.ajax/
    $.ajax({
        url  : '/api/v1/users/self/colors/course_1214595',
         type: 'PUT',
         data: payload
    })

    // #8BC34A - a green hex code

    {readyState: 1, setRequestHeader: ƒ, getAllResponseHeaders: ƒ, getResponseHeader: ƒ, overrideMimeType: ƒ, …}

  7. You should immediately see a response that looks something like the one above, we'll cover it in tutorial #3, click on the Network tab
    Review the Network Headers and the most recent request for course_#
    Hopefully you'll see Status Code of 200 OK
    If not, compare your code to the snippet above // comments can be ignored, or take a peak at tutorial 3
  8. Click on the Network Preview tab
    You should see a JSON object with the hexcode you sent with the request in the Console tab
  9. Refresh the page
    Unlike tutorial 1, this time you sent the data to the Canvas API where it was saved. When you refreshed the page, you got another copy of the Dashboard with the new color.
  10. Try It Again!
    Go back to the Console tab. To re-execute the last command/code you ran in the console, press the up arrow on your keyboard
    Copy the green hex code from the snippet, and replace the payload value, press Enter
    Pick a different Hex Code with W3 - HTML Color Picker

 

Congratulations!

You've just used javascriptjqueryajax, and the Canvas api to Update custom color - Users - Canvas LMS REST API Documentation 

Review the documentation anytime you want to interact with Canvas programmatically. The documentation is a great starting point to any project, then planning how you will use it in your script or program.

 

 Escalating

 

  1. Let's continue to the Course Nickname.
    Click on the Network request without the course_ prefix, in the screenshot below it's 1214595
    Review the Headers and the Preview tabs as you did above
    Canvas Developer Tools | Set Course Nickname
  2. Some more JSON, this time with a few more parameters, let's take a look
    // you see a string like this
    {"course_id":1214595,"name":"W○RKR∞M⸚CARROLL","nickname":"My W○RKR∞M ⸚"}

    // we can indent and format, making it easier to read
    {
         "course_id":1214595,          // the id of the course we want to rename
         "name":"W○RKR∞M⸚CARROLL",     // the current nickname of the course
         "nickname":"My W○RKR∞M ⸚"     // what we are changing the nickname to
    }
  3. Three values were returned in response to setting the Course Nickname using the Dashboard Card. However, we only need to send 1 value to change the nickname.
    Take a look at Set course nickname - Users - Canvas LMS REST API Documentation
    Note the Endpoint and the Method, it looks like this

     PUT /api/v1/users/self/course_nicknames/:course_id

     

  4. Let's update the snippet of code we used in the first section of Tutorial #2

    We need to update the payload with the nickname, and change the URL

    // update the payload with the values we need to set
    // indent them to be easier to read
    var payload = {
         "nickname":"My W○RKR∞M ⸚ Test"     // what we are changing the nickname to
    }

    // http://api.jquery.com/jquery.ajax/
    $.ajax({
         url : '/api/v1/users/self/course_nicknames/1214595',
         type: 'PUT',
         data: payload
    })
  5. Copy and paste your code into the Console tab (unless you edited it there) and press Enter
  6. You should immediately see a response that looks something like the one above, we'll cover it in tutorial #3, click on the Network tab
    Review the Network Headers and the most recent request for course_#, hopefully you'll see Status Code of 200 OK
    If not, compare your code to the snippet above // comments can be ignored, or take a peak at tutorial 3
  7. Click on the Network Preview tab
    You should see a JSON object with the hexcode you sent with the request in the Console tab
  8. Refresh the page
    You should see your course change names
  9. Try It Again!
    Edit the snippet, change the nickname and press Enter

 

[1] Profile - Users - Canvas LMS REST API Documentation

      or in the browser console, try ENV.current_user_id

 

 Rapid Escalation

Check out Developer Tools #3 - Update the DOM with an AJAX Response  and we'll update the Dashboard without Refreshing the page

Developer Tools for the Canvas User > Tutorial #1

 

  1. On your Canvas Dashboard, Right Click on a Course Card and choose Inspect
    If you miss and the highlighted element is not exactly as shown, hover over the elements until you have the whole Course Card
    Look for the <div> elements with a class of ic-DashboardCard
  2. Choose Delete Element
    Canvas Browser Tools - Delete Element
    It's Gone!
    Canvas Browser Tools - Element Deleted
    Not Really.
    And you didn't break anything either. You deleted the element from the browser's local copy of the page, nothing on the server has changed.
    To get a new copy from Canvas, continue to step 3
  3. Refresh the page

 

All Done! After the refreshed page downloads, your Canvas Dashboard is back to normal.

 

There's no real purpose for doing this other than to walk you through finding elements to edit on the page, or to make this tutorial. I'm enrolled in a few more than the three courses seen here. For the average Canvas user this may be helpful for writing tutorials and other documentation, or maybe removing and modifying personally identifiable information to take and transfer screenshots safely.

 

 Escalating

 

  1. To modify an element on the page, right click on it or find it in the Elements tab, and choose Copy Copy Selector
    Canvas Developer Tools | Change HTML with JavaScript
  2. Next, click on the Console tab and type in // and paste the selector, press Enter (that is 2 forward slash's)
    This is a JavaScript comment (code that isn't executed) with an identifier we can use to change the content of the <h1> that displays the Dashboard page title
    You should also see undefined on the line below, this is because nothing was executed and nothing was returned
  3. Using either of the 1 line jQuery* snippets below, you can change the title of the Dashboard page or remove the last Dashboard Card from the page
    // change the page title/header
    $('#dashboard_header_container > div > h1').html('My Dashboard')
     // remove the last dashboard card
    $('.ic-DashboardCard:last').remove()

    Each line should edit the page immediately and the console will respond with an updated DOM object

    Canvas Developer Tools | Change and Delete
  4. Refresh the page to clear your changes

 

* The usage of jQuery here is for demonstration purposes. Canvas currently provides jQuery, however it may be deprecated in the future. The following snippet shows how to change the page Title with vanilla JavaScript.

// change the page title/header
document.querySelector("#dashboard_header_container > div > h1").textContent = "My Dashboard";

 

 

 

 Rapid Escalation

If you want to make changes to your Canvas Dashboard more permanently, check out this thread and Github repository.

Sorting Dashboard Course Cards 

canvancement/dashboard at master · jamesjonesmath/canvancement · GitHub 

 

For a deeper look at modifying Canvas pages with Javascript, look at Hiding Content from Certain Roles

While collaborating recently, I’ve found myself referring to the Javascript Console, or is it Developer Tools, maybe Web Inspector; what about adding Browser before, do I provide a link? Sometimes, I don’t know whether to provide a link to an article that discusses the particulars of opening, inspecting or getting to the tab I want to reference; or if I’m insulting someone. So, I've decided to create a series of short tutorials that will hopefully walk new developers through some basic tooling around on Canvas with a Browser and JavaScript. I will also try to use these as references in future posts and discussions. I hope they are helpful, feedback is welcome. Enjoy!

 

Before we get started, let me point out that I won’t be getting lengthy on how to open the tools in each browser since everyone can do that with Google, and the easiest way is to right click the web page and select Inspect Element or Inspect, if you're using Chrome.

 

The following is a brief, but detailed, and rapidly escalating tutorial of things any Canvas User can do with Developer Tools. Along the way, I will share some community resources that will help guide you with additional information.

 

tutoriallevelfocus (tab in developer tools)
#1 - Delete and modify elements, and Refreshfreshmanelements
#2 - Update a course nickname with the APIsophomorenetwork, console
#3 - Updating the DOM and Handling Responsesjuniorconsole, elements
#4 - JavaScript XHR for Canvasseniorconsole, network
#5 - Mutation Observers & Reactmutant

 

// indexing a few related resources

guidelevelfocus
API Testing: Postman by Garth Egbertadvancedapi requests, json

 

 While searching around the community for this series, I stumbled upon this comment by James Jones.

  hiding-content-from-certain-roles#comment-86823 

The problem with JavaScript resources is that, for the basic stuff, you're not trying to learn JavaScript, you're trying to accomplish a specific task. There's no book or website that I want to sit through the whole thing to learn how to program in JavaScript. I don't want to start with "Hello world." I don't have time to start with "Hello world." I want to solve the problem that needs solved and get back to life.

 

My primary resources are the encyclopedia (Mozilla Developer Network documentation) and the commentary on it (Stack Overflow), using Google to get me to the content I need. I often start with a solution from Stack Overflow and then go look up the right way to do it using MDN.

I'll also add that W3 and W3Schools are great resources for what we're doing.

 

If you have questions or need assistance, please ask in a comment, it will help others.

If you work through these tutorials and find any issues (or Canvas breaking changes) please let me know and I will get it fixed right away.

Please let me know if you would like to contribute, I would be happy to add new tutorials and information. Contribute new tutorials with API endpoints that work for user/student roles, no advanced permissions.