Canvas APIs: Getting started, the practical ins and outs, gotchas, tips, and tricks

Document created by Stuart Ryan Champion on Mar 26, 2018Last modified by Stuart Ryan Champion on Apr 17, 2018
Version 23Show Document
  • View in full screen mode

Overview:

Welcome one and all, this document is designed to assist everyone from those that have never heard the API acronym before, to seasoned programming veterans who may be looking for tips and tricks that are specific to the Canvas API. I have worked to structure this document to get more complex as it goes on, so you should be able to find your comfortable starting point easily. Most of all, if something doesn't make sense, please leave a comment, if you have some ideas for inclusion, or even if you would like to challenge something that I have proposed, I encourage you to leave a comment!

 

API Foundations - Let's go through the basics

So, you may be asking 'What is this API acronym'? API stands for Application Programmer Interface, and in this section, I will cover off some basics about what APIs are. When you access a website (such as Canvas), you are accessing an application. The application has a web interface that is written for humans to interpret information and interact with that information. These interactions may be adding data such as discussion posts, pages and module items or they might be changing data, such as updating student or staff information or enrolments in courses. This list is not extensive, and there are many other functions available. However, these are just some examples I will reference for highlighting the differences between the web interface of Canvas and the APIs.

 

The key difference between the Canvas web interface and the APIs is that the Canvas web interface is designed for a human to interpret and interact with the application, it presents information in a visually appealing way, using things such as iconography and visually logical structures to users such as staff, students, and administrators. The APIs differ in that they are designed for consumption by a computer. You can think of the APIs as 'just another website', but a website that computers can access to interact with Canvas and the information contained therein. For a computer to most effectively communicate with another computer, the visual elements and structures of a web interface such as Canvas' web interface, only get in the way. Therefore, the APIs present functions and data in formats that other computers and programs can call and consume in a standardised way.

 

Canvas Web Interface ExampleAPI Interface Example
Image displaying the Canvas Web Interface with Two Test CoursesCanvas API Interface displaying only the first of the Test Courses on my user account

Important points:

  • Displays a variety of information including the courses the logged in user is enrolled in
  • Sources and displays information from multiple courses such as what is 'Coming Up'
  • Provides visual guides and references to guide you through the Canvas application
  • A user can easily bounce to different areas of Canvas by clicking buttons such as their inbox, calendar, Courses, Commons and so on

Important Points:

  • The image above shows only a representation of the first item on my dashboard the 'Test Stu' course. To obtain this information I used the API for 'Courses by User' instructing Canvas to provide me with a list of courses based on my user ID.
  • From the image above, you can see this is verbose, and only represents a small portion of what the application pulls together for users in the Canvas web interface
  • The information presented includes information generally not exposed to users such as internal system IDs, Account IDs, Term IDs, enrolment IDs and so on
  • There are no visuals included. However, you may see that there is a logical defined structure to the information. The structure displayed is known as JSON and is the structure Canvas utilises to tell a computer (via scripts and programs) to interpret the data that Canvas sends back.

 

What can the APIs Do?

The APIs can achieve most things that are possible within the Canvas web interface. It is important to note that the APIs are for Canvas itself. Therefore tools installed such as LTIs will not be accessible through the Canvas APIs and are out of the scope of this document. The concepts herein, however, could be applied to any LTI vendor's APIs if they publish an API specification similar to the Canvas LMS API documentation

 

The functions listed in the Canvas LMS API documentation are extensive, and some of the more common applications of the APIs include:

  • Integrating with other systems at your institution to provision user accounts (for systems not supported by direct SIS integration)
  • Enabling LTI tools greater levels of access to information to interpret more about a user or context
  • By savvy admins who wish to automate elements of their work, setting up new courses, copying content and more
  • This list is by no means exhaustive and represents just some of the ways you can use the APIs, though, in all honesty, your imagination is the limit

 

Can anyone use the APIs?

Indeed they can! Canvas publishes the full API specification, and any user can interact with the APIs. The key to note here is that a user will only be able to access as much information in the APIs as they are permissible to within the Canvas Web Interface. Therefore, if a student tries to query information about their account, they will have the authorisation to get the details back. However, if they attempt to run a query for another users' account, they would get an access denied error. Equally, I will reiterate here that a user must have an authenticated account or access token for your institution to access the APIs, without which, the API calls will fail.

 

A great way to illustrate this is the Canvas Mobile App for iOS and Android. The mobile apps use the same APIs that are available to institutions and users, to interact with Canvas and present information to users on the run based on their user account. The mobile apps take the information that Canvas presents in the API Interface Example above, interpret that information, and then re-present it to the user with visual elements much like the Canvas Web interface.

 

What should I know before getting started if this is my first rodeo?

I am glad you asked!  There are some incredibly important things you should know before you get in and start playing.

  1. It is incredibly easy to change things in your environment with the API, and this means it is also incredibly easy to accidentally do something wrong or unintended when you are learning. For this reason, start out using your Beta environment (i.e. yourinstitution.beta.instructure.com ). The Beta environment refreshes weekly, therefore, if you break something on a large scale, you only have to wait for the next week to get a clean slate.
  2. Consider leveraging your Test environment also (i.e. yourinstitution.test.instructure.com ). After you have played around and have a script or integration working in Beta, I recommend testing it against your Test environment. You may think of it along the lines of User Acceptance Testing of your script or integration before migrating it to your production environment. When developing something that modifies data at scale, I highly recommend this progressive process.
  3. Don't think you have to start from scratch! With the Canvas APIs being completely open, and with such a great community here, it is highly likely what you are trying to achieve has been done by someone else at some point. The Canvas Developers group is a great place to look through discussions on the API, examples that people have posted, and issues they have faced as well.
  4. Don't try to do too much, too fast. Start small, get your authentication working with a simple script (say, pull the user details for a single hardcoded ID), once you have that, build on it further, and so on. I would also recommend, save versions of your script every so often (you may wish to use Github.com). That way, if you accidentally break something, you can go back to a known working version.
  5. Be mindful that you will need programming skills. If this is your first time programming, you may need to step back and look at some basic programming courses first. I would probably suggest something like Python (despite it not being my language of choice, give me PERL any day).
  6. Lastly, and most importantly, NEVER EVER EVER EVER (EVER) run a script you have downloaded from the internet against your production environment without first reading the script through fully and completely. You need to ensure you understand what every line or block of code is attempting to do before you run the script. Often scripts are provided best-effort, some may have errors, some might have testing code left in them, some may not quite meet your exact needs, and hence they should be taken as a starting point. You should also be mindful of the potential for malicious scripts, though unlikely, it is still possible. I stress that it is good practice and policy to only ever run something in your environments that you fully understand yourself. 


When are Canvas APIs most appropriate?

What are the available alternatives?

There are several alternatives you may consider before you look at the APIs. I won't cover these in great detail, as there is a wealth of information already available, hence, I suggest you check out these links for more information.

 

Which alternative should I choose?

So, now you know that Canvas has multiple ways to get data in and out of the system, you may be wondering which one you should choose!

  • Prebuilt SIS integrations – These are great for automatically hooking into supported systems with minimal effort. For institutions that are looking for a holistic, out of box solution, pairing Canvas with one of the supported SISs can lower the time and ongoing administrative investment over both the short term and long term.
  • Canvas Data – The Canvas Data platform provides a read-only view into some normalised data about your Canvas Instance. It is great for understanding ‘what HAS happened’ (with a delay of up to 36 hours). Primarily it is designed for reporting. Some institutions choose to leverage Canvas data to trigger when they should do something with the APIs (though, this is far less common). Another important note for Canvas Data is that 'deleted' data will not show in the Canvas Data exports, you will need the API to see any deleted records.
  • APIs – The APIs provide Real-time, access to both active and ‘deleted’ data. The APIs are great for ‘what IS happening’, 'what HAS happened' or ‘I want to change something’. You may wonder why you would use the APIs to ask the 'what HAS happened question' as Canvas Data provides this as well. Canvas data can answer this question rapidly en masse for large sets of data (hence, great for reporting), the APIs can answer this question for small subsets of data, usually far more quickly than setting up a Canvas Data environment. Also, you would pick the APIs should you wish immediately up-to-date (i.e. real-time) information.

 

Design concepts and considerations:

Oauth vs Developer Keys vs User Access Tokens

One of the first things you must decide is how you connect to the API (i.e. with which type of authentication method). The method you choose will largely depend on your needs.

  • User Access Tokens (generated from within an individual user's settings page) are useful for tinkering, or a single-user run script. They will only provide the equivalent permissions of the owning-user.
  • Developer Keys (generated at the account level, from within the account settings page) provide complete 'root' access to the system for an integration or application. The assumption when using these keys is that you implicitly trust the integration/application, and that any governance of access or permissible functions is the responsibility of the system/application/script using the developer key.
  • OAuth. OAuth is a little different as it enables user logins for an application, then the application can interact with the API as that user. You can read more about this at OAuth2 - Canvas LMS REST API Documentation. The best way to describe this would again be with the Canvas mobile apps. The apps use a shared key and secret (developer key) to permit a user to authenticate to Canvas and log in, without the app itself ever storing the password, Canvas generates a session key and provides this to the mobile app for that user. For any user-facing apps or development, you must use OAuth to protect your user's credentials.

 

A little side note on developer keys. I would re-stress that a developer key gives full administrative access to your system, and should, therefore, garner the same respect that you would a full administrator account. You should regularly review the developer keys that are active on your system, and ensure that you remove keys that are no longer required as soon as possible.

 

Understanding API rate-limits: a practical guide!

Canvas outlines the API rate-limiting policy in the API Rate Limiting document. However, when I first read this, it took me some time to get my head around the best way to approach designing my solutions to ensure I wouldn't run into any issues. I recommend having a read of that first, then come back here for some of my additional tips.
 

There were a few of things that we came up with as possibilities to ensure we did not run into any issues:

  1. You can avoid the rate-limit entirely by using sequential processing only - by doing things in a sequence, such as a loop, waiting until one call had finished to continue, you avoid the rate-limiting entirely. The 'bucket' (as described in the official Canvas guide) is to stop a flood of requests from a single 'application' or 'user' so to speak. Hence, sequential is not an issue. With regards to user tokens, these operate in the same way. The documentation suggests a user token as, in theory, if you had 1000 students who needed to do something via the API concurrently, they each have their own API rate-limit 'bucket' and therefore would not get throttled. The best example I could give is the way the mobile apps work. Rather than a single API key for a mobile app for an account, when a user logs in, Canvas returns a user session token which is used for the API. That way when each student uses their mobile device, it is parallelising requests from all students on their individual session tokens. 

  2. Spread the load across multiple API keys - The second option you can opt for is using an API token per request type (as one example) - you could (if you had a significant need to do parallel processing) use a different API key per API request type. For example, if you need to get course details, you use one API token, if you need to set course details you use a second API token. There are other implementations of this (such as having a pool of keys and using round-robin to select a key for each new call), but I would anticipate this is the most common.
  3. Do minimal parallelisation - with only a small number of parallel threads (less than 5) the likelihood of hitting the API limit lessens, more on that below.
  4. Parallelise and gracefully handle the rate-limit - if you are planning to run parallel queries, you can (and I dare say should, regardless of which solution you select) gracefully handle the rate-limit, set a 'sleep' time and then try again when the rate-limit has gone away. Gracefully handling the API limits ensures you implement your solution in a programmatic, logical, and supported way.

 

The option we have used for our major integrations are three and four. Personally, if you are doing anything other than sequential processing, I think it is good practice to gracefully check for, and handle the rate-limit error (i.e. option four). In our instance, we decided to run five threads in parallel, with the rest of the operations in those threads running sequentially. That gave us a good hybrid of performance, along with ensuring we would not hit any API limits as the most we have running at once is five API calls on a single token. 

 

Pagination concepts

The next item you should be aware of is the need for pagination when you are using the API. Pagination is not overly difficult but is often encountered as one of the first stumbling blocks when getting into the Canvas APIs. 

 

The Canvas API Pagination mechanism is documented in the Pagination - Canvas LMS REST API Documentation. So, what is pagination exactly? When you have large amounts of data that you are pulling out with the API, it would be incredibly system intensive to pull all that data out in a single call. One example includes using the API to list all courses in an account. You could conceivably have hundreds, if not thousands (if not possibly tens of thousands) of courses returned, depending on the size of your institution. If you made a single call and that retrieved all results, it could result in significant delay in even getting the results, and this is where pagination comes in!

 

Canvas (by default), only returns the first ten items by default in an API call and will also add an item in the 'header' of the return message indicating there are more 'pages'. While you can increase the per_page_limit, be aware that (as per the documentation) the per_page_limit maximum value is undocumented, so be aware, your mileage may vary if you choose to tinker with this. What you need to be most aware of with pagination is that no matter what you are doing with the API, no matter how simple, one-off, or just playing, you will need to handle pagination.

 

Once you implement pagination, things work a little differently. Using our previous example, when you call the API to list all courses in an account, Canvas will return the first ten and also tell you there are more pages. Therefore, your script/integration must recognise that, and then call the Canvas API asking for the next ten, and the next ten, and so on. For those starting out with programming, this is a great way to get into playing around with looping in your chosen language. 

 

When you don't handle pagination, you will find you may struggle as you seem to get small result sets, you may think there is no real rhyme or reason to the result sets you get back from the API, and you may think data is missing. Therefore, if you run into any of these, I highly recommend ensuring you have handled pagination in some way, shape, or form, and that it is working correctly.

 

Developer Keys and User Access Tokens in Beta and Test Environments

All righty, if you only read one section of this document (surely you want to read them all right?!?), please make it this one! I can not stress the importance of this concept, especially for those starting out with the APIs as things can go very very wrong, very VERY fast.

 

One of the great things Instructure provides for hosted clients is several copies of their production environment including Beta and Test. There is a slight downside of these the environments' regular cloning to be aware of. You will quickly find that if you want to follow best practice and only use a key in Beta or Test linked to that environment, and to do so you have to create a new key every one or three weeks respectively.

 

I am realistic, I know this is best practice, but after a time it becomes unsustainable. Before long I guarantee you will create your keys in a production environment and just let them clone down through the systems. However, you should be completely aware of the ramifications of doing so. 

 

The best practice option ensures that when you think you are playing around with a Test environment if you only have a key that works in the test environment, there is no risk of it connecting to your production environment, it would simply give access denied. When you have your 'test' keys sitting on the production environment, you run the risk of accidentally thinking you are working with Test (or Beta), when in fact you are modifying production data.

 

I have yet to find a best practice way around this. I feel a new feature idea coming along to support this. Nonetheless, please be careful when you have keys that work across your environments. When you are starting out, I highly recommend working in Beta and creating a new key each week, trust me, it is safest in the long run!

 

Remaining Concepts

There are a few more concepts you should know about in getting started in the APIs, for these I will refer you to the existing documentation already available, though if there are any that people would like me to expand on, please let me know:

  • SISIDs - Canvas allows you to store unique, institution/school provided identifiers for things like courses, terms, students and so on. These are known as SISIDs (Student Information System IDs). Utilising these can mean integrating with the API is significantly easier (and requires less logic). Check out Object IDs, SIS IDs, and special IDs - Canvas LMS REST API Documentation  for more information.
  • Masquerading - When using a developer key, you can have a program masquerade as another user and have an API call appear that the specified user made it. See Masquerading - Canvas LMS REST API Documentation for more information.


Planning, as you have to crawl before you walk:

Tips for getting started and some useful resources

When you want to get started, the best thing I can suggest is start small, don't try to build Rome in a day, build your foundational column and work on that until you get your head around it. Equally, don't think you are alone, there are some great resources out there to help you get started:

 

Picking something to tackle

Figuring out your first task to tackle might seem simple, you have an immediate need, and you want to dive into that. I would propose something a little different. If you are starting out completely fresh, consider doing something you already know about, to simplify the learning curve. For example, you may need to automatically enrol some users with the API. Before jumping into this, I would propose taking the time to put together a script in your favourite language that does something more simple. By starting with functions and features you know well, the learning curve of how the API works will be easier to grasp. Some possible starting places you may want to consider include:

 

Tools to help you on your way

  • Canvas Live API - This one will blow your mind (well, OK it blew my mind, so hopefully it does for you too)! When getting started with the API, you should visit yourinstution.instructure.com/doc/api/live. Please note, you will need a user access token or a developer key to utilise the Live API. The Live API is a great way to tinker and learn about the data, the data structures, and how to send and receive responses to and from the API. Remember, this is exactly the same as making changes on your live systems, so when you are playing around you may want to use yourinstitution.beta.instructure.com/doc/api/live for safety as you learn.
    screenshot of the Canvas API with arrows to the access token
    Remember to put your access token in and press 'Save Token' to ensure the API has an authentication key to work with. Otherwise, all you will get back is access denied.

    Screenshot of the Canvas Live API with Courses functions displayed
    With the Canvas Live API, you can see some great detail on what each field in an API response contains, for every one of the API functions.

    Screenshot of the Canvas Live API with Courses functions for sending data displayed
    The great thing is you can also prepare a call in a visual interface that helps understand the process as you build a query and get a response back. The Canvas Live API is a really great way to troubleshoot when things aren't working as expected.

  • Postman - Check out this awesome blog post Garth Egbert put together on what Postman is and how you can use it API Testing: Postman.
  • I will add more here as I come across them, or please feel free to make recommendations also!

 

Useful Video Resources:

John Raible from UCF talks about getting started with the Canvas APIs.

 

Matthew Emond introduces canvasapi (Formerly known as PyCanvas) - A Python Wrapper for the Canvas API

You can check out the source code for canvasapi at the official Github page --> GitHub - ucfopen/canvasapi: Python API wrapper for Instructure's Canvas LMS  


API Calls Made Simple - Curtis Rose

Thanks so much for the recommendation of this video Audra Agnelly!

 

Optimising – The evolution from tinkering to a future-proofed and production ready script:

Please note, this section is to be completed and will be done within the next few weeks. I wanted to get the bulk of this information out for people to start with, and to get early feedback. Hence the optimising and troubleshooting sections are a work in progress. Feel free to follow the document for updates as I make changes.


Covering your bases – adding robustness to your scripts and integrations:

  • Logging
  • Timeouts and error handling
  • Rate limit handling
  • Rate limit optimisation

 

Troubleshooting – what you can do when things don’t go to plan:

  • Check what you are attempting with the Canvas Live API
  • Check your function calls
  • Search the Canvas Community for any related information
  • Contact support (if you are an Institution on the SaaS hosted Canvas product)
    • What you should include
    • What you can ask for
    • What you can expect

 

Appendix A: Resource List

Most of these I have laced throughout this document, and are included here also as a complete list of resources:

11 people found this helpful

Attachments

    Outcomes