[PHP - Curl] Handling pagination with acces_token

[abelardolg]

Hi there!

https://canvas.instructure.com/doc/api/file.pagination.html

I would like to retrieve +100 students per account and per page.

According to Canvas API documentation, "The one exception is that if an access_token parameter is sent for authentication,"

I always send the acces_token for authentication; if I would remove it, then no api call is done.

How to receive a Link header in order to iterate through pages with the access_token parameter? I have checked the response from curl but it doesn't contain this piece of information

I used Postman but, again, this program only retrieves 100 results, not 500,000 (yes, I am exaggerating but only for testing purposes).

Best regards.

[DecoyLex]

Hi @abelardolg,

Because the access token is an active session for your (presumably highly privileged) account, Canvas treats the token like a password. This means that it won't send it back to you in plain text. Canvas's pagination code duplicates all the query parameters on the request and adds the pagination information (in most cases setting a page number).

Assuming you're using the built-in cURL functions in PHP's standard library, instead of sending the access token in the URL as a query parameter, you can also send it in the headers. Something similar to the following should work:

$curl = curl_init();

curl_setopt($curl, CURLOPT_HTTPHEADER, [
    'Authorization: Bearer YOUR_CANVAS_ACCESS_TOKEN'
]);

curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl, CURLOPT_HEADER, 1);

...

$response = curl_exec($curl);

$header_size = curl_getinfo($ch, CURLINFO_HEADER_SIZE);
$header = substr($response, 0, $header_size);
$body = substr($response, $header_size);

Then you can extract the Link header from the $header variable and further parse that to get the next page.

I hope this helps!

[abelardolg]

Hi Decoylex!

I tried your strategy but...it didn't work!

I am trying to reach out this url:

CURLOPT_URL => "https://XXXX.instructure.com/api/v1/accounts/" . $account . "/users?enrollment_type=student&per_page=" . $perPage,

but the response gives me records, not header information. The same when I print $header info.

I always put the canvas_acces_token value inside the header.

When I try to display the header_size value, an empty information is printed out.

curl_setopt_array($curl, array(
CURLOPT_URL => "https://XXXXX.instructure.com/api/v1/accounts/" . $account . "/users?enrollment_type=student&per_page=" . $perPage,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => '',
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 0,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => 'GET',
CURLOPT_HTTPHEADER => array(
'Authorization: Bearer ' . $this->MY_CANVAS_ACCESS_TOKEN,
'Content-Type: application/json',
'Cookie: _csrf_token=XXX; _legacy_normandy_session=YYYY;
canvas_session=ZZZZ
),
));

I removed the cookie data but it didn't work too.

Sorry, but unfortunately I can't share the information that is displayed because of sensitive.

[abelardolg]

I used the Postman and it shows the Link variable in the header.

What am I doing wrong?

[abelardolg]

Hi there!

Why doesn't it exist a reference to the "page" query param at this webpage related to pagination?: 
https://canvas.instructure.com/doc/api/file.pagination.html

Could you update this page with this query param, please?

With this variable I could request as many pages as I want to get. 

I could achieve my primary goal. Now my secondary goal is to get 1000 per page; so, it's possible?

Thanks and best regards.

[JamesSekcienski]

@abelardolg 

The API documentation only details features that Canvas/Instructure support for use of the API.  Based on the documentation, they are only officially supporting for users to get to the next page by using the "next" link in the Link Header.  This is most likely to encourage users to take an approach that minimizes creating a strain on the system.

While there is a page query parameter, depending on the API endpoint it could be numeric based or bookmark based.  The numeric based is predictable, but it could change to bookmark based if the developers choose to make that change.  Since this isn't specifically documented, you would need to monitor for these changes yourself since it isn't something they would need to announce as a change.

As for the per_page amount, in most cases, it seems that 100 is the maximum supported amount for a single request.  This is again to ensure that requests don't cause an unnecessary strain on the system.  Since the documentation says the maximum allowable amount is unspecified, this could change, but based on conversations in the Canvas Developers group, it seems fairly stable as the maximum amount.  Nonetheless, this would also need to be monitored since there is no official maximum stated.  

I'm not sure what you are trying to achieve, but you may also want to consider utilizing Canvas Data 2, if the data doesn't have to be live and is ok to be partially delayed.  You could also consider using the provisioning reports from Canvas to get some initial data that is live and then use that as a starting point for a script to automate additional actions that are needed.  If you created a new post about what your goal is and what you have tried so far, some members of the Canvas Developers group may be able to make additional suggestions or point you towards something someone has already shared.

[abelardolg]

Then, I discard the "page" strategy and I should return back to "Link" header.

At this point, I don't get this header. I followed the steps indicated above but I don't get this header.