Batch insert calendar events programmatically

[philip4]

Hi!

I am trying to figure out a way to insert 12000+ calendar events to users in our Canvas account. Doing it through the API renders quite a few requests... To minimise the requests, I'm wondering if there's a way to batch insert all these calendar events?

[raj_veeravalli]

Philip,

Have you investigated threading or multiprocessing options in your programming language of choice?

I am easily able to run 30 simultaneous requests and I can go to 50 if I want, which roughly equates to around 4-7 minutes execution time for 12000 requests.

If you program in Python there are some great example from  @bbennett2 ‌ in the developer blog

https://community.canvaslms.com/groups/canvas-developers/blog 

[philip4]

Hi Raj!

Thanks for your input.

Yes I've tried running concurrent requests. However, the throttling of the API puts a penalty when firing multiple simultaneous requests (Throttling - Canvas LMS REST API Documentation😞

Parallel requests are subject to an additional pre-flight penalty to prevent a large number of incoming requests being able to bring the system down before their cost is counted against their quota.

I've prepared my client to retry the request if it fails due to exceeding the rate limit. So I get it to work with parallel requests, however it's quite error prone to do 12000 individual requests, albeit concurrent.

I believe batching the data into fewer requests would be the better alternative, given that this will be a reoccurring action, for synchronising users' schedule on a weekly basis.

[raj_veeravalli]

Hi Philip,

You're on the money!

I do batch requests and I'm also unable to surmount API limits and hit errors. Here's an example from one of my scripts

relevant_courses = np.array_split(relevant_courses, 10)
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
    for relevant_course_list in relevant_courses:
        kw = {
            'courses_list': relevant_course_list,
        }
        executor.submit(publish_reports, **kw)

Here, relevant_courses is a list containing a bunch of sis_course_id which I have split using numpy.

I can run 10 threads in parallel, each thread running a function that processes a list of courses. This can significantly reduce the running time, but has to respect the API limits. It works best if the workloads are all similar, so they can run simultaneously and finish running around the same time.

I can think of reducing the running time further, if you can reduce the API lookup operations by referring to another database; perhaps, a CSV file from a provisioning report, canvas data or perhaps something else such as SIS and use the compute resources (which are "free") in lieu of canvas's network resources. This is not always possible and the gains may be marginal.

If you are using a library, I'd set the loglevel to DEBUG or equivalent to ensure duplicate queries are not made and tweak the queries if necessary.

Other than these, I don't have any ideas. Maybe someone else has found a better way to do this.

[James]

 @philip4 

There is not a way to insert multiple events with a single API call. The intent of the throttling is to keep you from overloading the system for others. You can make multiple requests, but the limit on how many you can make at one time is low.

Instead of batching them, there is another way to do it that has been pretty reliable for me.

Here's what I've discovered about pushing the limits. I am using Node JS and the Bottleneck library. There may be something a similar capability for your programming language. Most of my requests are GET, which probably have a lower cost that POST, so the numbers I give will change but the principles will the same.

The x-rate-limit-remaining starts at 700. When it gets to 0, things start to fail / stop.

There is a penalty of 50 for each simultaneous request. If you make 14 concurrent requests that had 0 cost, you would hit the pre-flight penalty. Since the calls do have cost, it will be less than in a sustained environment -- you might be able to get by with 14 one time, but the 15th would likely cause issues.

In the Bottleneck JS library, this is controlled by maxConcurrent and it is normally not the limiting factor for me. I can allow 40 or maybe 50 concurrent calls depending on the call -- but it doesn't ever hit that.

The limiting issue is making them all at the same time and with that pre-flight penalty, you can't get nearly that much.

The trick is to stagger the requests so that they don't all start at the same time.

Bottleneck has a minTime parameter, which specifies the number of milliseconds to wait between requests. The first one is made right away, but after that, there is a delay between requests to space them out.

I just got through revising my Obtaining and using Access Report data for an entire course script, which is entirely GET requests. I ended up with a minTime = 30 and maxConcurrent = 25. 

The other thing I do with GETs is to limit the per_page to reduce the cost of the page, but that is not an issue with POST or PUT requests. Instead of using per_page=100 (the maximum allowed), I use per_page=50 or per_page=20 for expensive calls.

I could have pushed it more, but I was trying to make sure that it didn't time out with large classes so I sided with caution. With the current setup, my x-rate-limit-remaining is normally above 500.

It turns out that it is endpoint specific. There are three endpoints I use.

• I fetch the sections for a course with a per_page limit of 50. The per_page limit of 50 is probably higher than it needs to be. At least at my institution, we never have more than a few sections in a page. This call isn't particularly expensive anyway. It's throttled, but it's not the bottleneck.
• I fetch the list of users with their enrollments with a per_page limit of 20. This is an expensive call when the system is busy, which is why I went down to 20 per page. When I was doing 100 per page, I was sometimes seeing the x-rate-limit-remaining drop to below 200 with the second request (before using Bottleneck JS). Since this is an expensive call that takes a while to process, and limiting the number of simultaneous requests comes into play.
• I also fetch the access report for every enrollment in the course. This is a non-API call, but it uses the x-rate-limit-remaining header as well. This is a fairly quick request, even with per_page=100, so I let it fetch 100 at a time. I could make a lot of these requests simultaneously, except for that pre-flight penalty. Even if I could run 50 of those at a time, and I probably could, I couldn't get up to 50 because of the pre-flight penalty. The limiting factor on this call is the minimum time between requests.

I start fetching my access reports as soon as I have the first 20 students. Then it gets the rest of the students before it finishes fetching the rest of the access reports.

I could set up two limiters, one for the enrollments, and one for the access reports. In other programs, I've waited for one type of request to finish before starting the second and change the parameters between types. This one has a front-end that people see in the browser, so I didn't want them to wait even longer but I still may make the change.

Anyway, for the getting of the enrollments, since it takes a long time to run, I want to cut back on the number that are running at the same time, so the minTime=30 and maxConcurrent=25 is really for this one. I ran into a few issues with minTime=30 and maxConcurrent=40. With a minTime=30, then you can only make 1000/30 = 33.33 requests per second, so setting maxConcurrent=40 was never getting hit when the requests are short, but these requests were taking more than 1 s to complete, so I was hitting the maxConcurrent limit and occasionally getting limiting errors.

For the access report calls, these were quick and not very expensive. I could go minTime=20 and maxConcurrent=50 and probably get by with it in a sustained setting -- provided my x-rate-limit-remaining had not already been drained by the enrollments. In my other program, I wait for one type to completely finish before starting the next time of fetch, so my x-rate-limit-remaining is almost back to 700 for each new type of request.

I had originally monitored the x-rate-limit-remaining throughout the process with my other program that downloads data nightly, but the Bottleneck JS library doesn't recognize changes made to existing calls in the queue, so the would timeout before the change I made took place. I just slowed the whole system down enough that it exhaust the x-rate-limit-remaining in the first place. That program averages about 10-11 calls per seconds, but it never times out since I slowed it down that much. It now takes 12 minutes to run instead of 9. Last night's run was only 3 minutes and 17 seconds, but we're between semesters and summer classes are fewer than fall or spring, so it's not a good indicator.

If you need to make 12000 requests, at 10 per second, you're only talking 1200 seconds or 20 minutes. Even at 5 calls per second, it's only 40 minutes. That's nothing to a server running a backend process.

You don't have to get these done instantly. Pushing the limits can make the system slower for others and getting events added to a calendar is probably less important than the user experience. You can make more than one request at a time, but you'll need to space them out to avoid that pre-flight penalty. If the calls take a while to complete or are really expensive, you may need to worry about maxConcurrent or you can just space them out more to get the same effect.

[raj_veeravalli]

James,

Thank you for your detailed explanation. It is clear that I've completely

misunderstood batching.

How about Canvas mutations via GraphQL endpoint? Is this a possible option

for those that want to batch requests?

I have written some simple code to configure multiple course settings via a

single mutation (POST). Happy to share what I have when I'm back at my desk

tomorrow.

Sent from my phone. Please excuse typos and abbreviations.

On Mon, 1 Jun. 2020, 10:01 pm james@richland.edu, <instructure@jiveon.com>

[James]

 @raj_veeravalli ,

Here's how I'm defining the word batching. Whether it's the correct definition or not, I don't know. Sometimes people use the same word but mean different things so I want to make sure I'm clear about what I'm talking about.

Batching can be 

1. sending multiple requests at the same time in the same API call. Canvas often uses the word "bulk", "multiple", or "batch" in their API definitions for this.
2. sending multiple concurrent API requests at the same time, waiting for them to finish, sending a bit more, etc. 

I normally think of definition 1 as batching and that's what I meant when I said the API does not support batching, but I will grant that some people would consider 2 to be batching.

Definition 2 normally runs up against the API x-rate-limit-remaining header issue. For this technique, I would use some kind of array slicing to grab a bunch to send, send them, wait for them to finish, and then slice off another part of the array. With JavaScript, this could be done with a Promise.all() construct.

Definition 1 is designed to handle multiple requests at the same time, but Canvas often throws them into a background process for PUT and POST operations and then you have to query the progress status if you want to see if it is complete. You are sometimes limited by the length of a query string (for a PUT operation) or stated restrictions like 500 for update courses.

When Canvas supports definition 1 and operating on multiple items, you still may have to use method 2. In the example of the batch update of courses, if you had 700 courses to update, you would still have to make multiple requests to not exceed the stated limit of 500 (or the actual limit due to query string length of around 300).

The throttling libraries like Bottleneck JS are variations of definition 2. Instead of you having to do the slicing yourself, they take care of it for you. There is a queue and you don't normally have to wait for the queue to completely empty before you send the next batch. It keeps the maximum number going at all times. This is better in the long run, but it means that the x-rate-limit-remaining doesn't have a chance to replenish like it would if you were sending, waiting, and then sending again. If you can space out the requests, all the better.

I haven't done anything GraphQL other than through the web interface, so the only issue I've ran into is requesting too much data that it times out. Pagination is still an issue, but you have to tell it when to do it, otherwise it tries everything and comes back with an error and you GET no data.

GraphQL mutations are mostly overlays on top of existing functionality. There are a few new things that were developed just for GraphQL, but in most cases, it was exposing existing data structures. Unless the underlying code supported multiple objects (batch definition 1) then it is unlikely to support it with GraphQL. Those were the ones that threw it into a background process and that doesn't get added just because you tack a GraphQL interface on top of things. You would have to add the background process as additional code.

As far as definition 2 of batching goes, then you may be able to use the GraphQL. This assumes that you can even use GraphQL in the first place (not all of the REST API has been converted over). This post about course post policies from December 2019 discusses sending 75 updates a time with GraphQL. I haven't attempted this myself, though, and I would note that this was one of those GraphQL-first things, so it might have been designed to handle multiple requests, rather than being a retrofit of existing functionality. I just haven't tested it and try no to speak authoritatively about things I don't know about (educated conjecture is what I'm doing).

If you were to send multiple mutations as a single requests, you're still likely to run up against the issue of the x-rate-limit-remaining. Your x-request-cost is going to be high, so you cannot send as many of them at the same time. For the person sending 75 mutations in a single request and it taking 1-2 seconds, you probably won't be able to send 25 of those requests at the same time, even if you stagger them 50 ms apart.