Trying to figure out module_item creation behavior (parallel requests)

[mike_young]

Hi all!

Long-time lurker, first-time poster. Many thanks to everyone who has shared their tricks for negotiating the Canvas REST API, especially @James with parallel requests and pagination strategies! Huge help. Big kudos. 😃

As it says in the subject, I'm trying to figure out some module_item creation behavior. Here's the story:

1. I am doing stacks of parallel cURL requests in PHP using curl_multi handles.
2. Specifically, I am building multiple module items at once. I am building my POST requests correctly, and everything works successfully, including positioning, EXCEPT... for completion_requirements.
3. The module item objects in the requests look like this in JSON format (real #s scrubbed, obv):
   
   {
   "module_item": {
   "title": "Whatever",
   "position": #,
   "indent": 0,
   "type": "Assignment",
   "module_id": #,
   "content_id": #,
   "completion_requirement": {
   "type": "min_score",
   "min_score": 80
   },
   "published": true,
   }
   }
   
   
4. The module item objects I get back from Canvas look like this (PHP print_r output):
   
   

   [id] => #
   [title] => Whatever
   [position] => #
   [indent] => 0
   [quiz_lti] =>
   [type] => Assignment
   [module_id] => #
   [html_url] => correct URL
   [content_id] => #
   [url] => correct URL
   [completion_requirement] => stdClass Object
   (
   [type] => min_score
   [min_score] => 80
   )

   [published] => 1
   [unpublishable] => 1

5. Nothing weird I can see except that "unpublishable" property, which I can't quite track down in the Canvas source code API behavior. The problem is that even though completion_requirement is returned correctly for every module item I create, it inconsistently ends up missing in the actual course when I check through the UI and through the API (by grabbing the new module item info).
6. As implied by the mysterious italics, this behavior is inconsistent. I've thought about patterns based on where the call is in the batch, and there is no seeming correlation between whether it's the last call of the batch, the first call, etc.

I suspect the issue is related to the parallel cURL calls because I know the Canvas REST API doesn't like them. Maybe @James or another kind parallel request guru with time on their hands might be able to shed some light?

Thank you all very much! 😃
Mike

[James]

@mike_young 

It's not that Canvas doesn't like some parallel requests, they're okay as long as you don't hit the API too hard and put some delay between them. One way to test the theory that it's the parallel nature that's messing things up (I suspect it's something else), is to make them one at time and see if it's still inconsistent applied. 

The reason I don't suspect the parallel nature is because the requests will fail if your x-rate-limit-remaining value goes to 0. Since you're getting the response back, then the request itself didn't fail. Now if some of your requests are coming back and others aren't, then it definitely could be the parallel request causing the issue.

One place that the parallel nature will bite you is if you're trying to set the position. You can really get things in a seemingly random order if you try setting position if you send a bunch of requests at one time and they finish out of order. For those, I always go to serial requests rather than parallel requests.

Let's look at what you might do if you really are hitting the throttling issue.

I use Node JS instead of PHP/cURL since asynchronous calls are supported out of the box. It also has a Bottleneck library I use to control the throttling. There are two aspects to consider: how many requests can you send at a time and how long do you delay between requests. With PHP/cURL, you can control the first but I don't see a native way to control the second. In my experience, setting the second is more important than setting the first.

Let's say you have 20 module items to set requirements for. Sending them all at once will almost certainly cause a throttling issue. But if you allow up to 20 at a time but put 100 ms between them, you will probably be okay. That's because the server isn't hit with all of them at once. In fact, with 100 ms between each one, you're limiting it to 10 requests per second. That's not the same thing as allowing 10 simultaneous requests.

The 100 ms delay could probably go lower. I use 20 ms for some of my parallel requests. It really depends on the requests, which is why I debug and dump the x-rate-limit-remaining after each request finishes to make sure it doesn't go to low. I try to keep it above 500 (it starts at 700).

The number of simultaneous requests is more problematic. It's that initial request that could burn up all 700 of the x-rate-limit-remaining in one shot. How low it should be? When browsers used to limit the number of connections, it was 5 or 6 connections and I didn't have throttling issues. It's when they opened that up that I started running into issues. That might have been with HTTP/2 connections allowing pipelining and though there were just five connections to the server, it was sending multiple requests through each connection.

PHP supports the curl_multi_setopt options that may help. I'm going from the descriptions, not looking up old code here.

• CURLMOPT_MAXCONNECTS sets the maximum number of simultaneously open connections. The default is four times the number of handles added with curl_multi_add_handle. I would consider keeping this low, say 5 or less.
• CURLMOPT_MAX_TOTAL_CONNECTIONS has a very similar description except the part about the default.
• CURLMOPT_PIPELINING allows you to send multiple requests through the same connection. This may run into a problem if enabled since you could send multiple requests through the same connection, therefore bypassing the effort to limit the number of connections.

There are other options that could be tweaked. Basically, I would like to keep the number of requests at any time to 5 or fewer to avoid that initial hit. If I could spread out the requests, I would do that instead. To accomplish that, turn off pipelining and set the maximum number of connections to 5.

There are a couple of alternatives to multicurl that might work. I'm not recommending either of these, just listing them as alternatives.

• rolling-curl was last updated 14 years ago. It doesn't appear to put space between the requests, but it does limit the number of requests at a time. In their example, they're hitting separate sites, so they don't have to worry about throttling. However, it's a single file.
• Guzzle is a whole framework. I tried Guzzle a couple of times, but eventually gave up as I prefer doing things the hard way. I don't see where it puts delays between requests.

The best solution would be if you could stagger when things are added to the multicurl request. Start it off with 1, then use usleep() to add a slight delay before adding the next request. The rolling-curl code may give some insight into that. Starting with line 304 in RollingCurl.php is the code to add a new request when one has finished.

If it's not a parallel issue, then we need to dig further into which ones are failing.