Can return enrollments- how to handle pagination?

[salmonsm]

This Node script is working well to return enrollments- but I am struggling with pagination.

const request = require('request');

let url = 'https://mysite.instructure.com:443/api/v1/courses/123456/enrollments?type[]=StudentEnrollment&state[... rel="next"';

request(url, function (err, response, body) {
if(err){
console.log('error:', error);
} else {
let results = JSON.parse(body)

for (var i in results) {
console.log(results[i]);
}
}

});

the only way i know how to cycle through the pages is manually with page number in the URL, example: page=2, page=3, etc.

How can this be automated?

How can I design the script so it knows it's at the end? I assume an if-then structure that checks the state of rel. As long as it's "next," it keeps going, but what about when it's "last?"

Any help would be most appreciated. It's a rather urgent need. Thank you.

[ColinMurtaugh]

Hi Michael --

The easiest and safest thing to do is to rely on the Link header that comes back in the response. The Link header will contain opaque URLs for the current, first, last, previous, and next pages of results. You can just check for the presence of a "next" reference, and request the URL that you're given (if you're on the last page, there will be no "next" reference).  There's some info in the Canvas API docs, with an example of what the header value looks like.

I'm not a Node developer, but you probably want to use a library to parse the link header -- maybe this one? Actually, check out this thread here in the community, and especially this comment which includes a Node example!

Good luck!

--Colin

[anthony_harris]

I'm also not a Node developer, but perhaps the logic in this Python code will help.  The same sort of abilities exist in JavaScript/Node.js I'm sure, I just don't know the syntax for them.

This is a while loop that runs through and makes the calls one by one to the API URL in url, passing in the token via the headers defined in headers.  Then it grabs the returned "Link" header, splits it up, looks through all the links passed to see if there's a "next", and if so, sets the next url to be that.  Then your code would do stuff, and at the end of the loop you see if you set nexturl, and if you did, set url to that and loop back up.  Otherwise set the exit flag and drop out of the loop.

# Now the meat of the process.  We're going to loop here to get each of the 10-record pages
# the Canvas API will give us in response to our request.
keepgoing = True
while keepgoing:
    #  Make the call to the API URL and pass in our custom header.
    r = requests.get(url, headers=headers)

    # Make sure the call worked, and if not, we want to throw an error.
    if r.status_code != 200:
        print("ERROR: Status code returned={} for {} -- Exiting.\n".format(r.status_code, courseurl))
        sys.exit()

    # Since there are a lot of these, we'll need to paginate.  That means getting the link
    # header to the next page out of the LINK header.  If there isn't one, then this is the
    # last page.
    linkheader = r.headers['Link']
    # The header has an array of links, separated by commas...
    linklist = linkheader.split(",")
    # Loop through.  We're looking for the "next" link, if there is one...
    nexturl = None
    for linkitem in linklist:
        # Each link item is really two things, the actual URL, and a rel= that tells us what the link is to...
        onelink = linkitem.split(";")
        # If we have a next...
        # Otherwise, if no next, empty nexturl so we'll know.
        if "next" in onelink[1]:
            # Get it, strip the first character (a "<") and the last letter (a ">").
            nexturl = onelink[0]
            nexturl = nexturl[1:]
            nexturl = nexturl[:-1]
    # End of for loop

**** do stuff with the data on the current page here ****

    # Now, if the nexturl is blank, set the keepgoing flag to false so we'll drop out of the loop.
    # Otherwise, set url to nexturl and we'll get the next page worth.
    if nexturl:
        url = nexturl
    else:
        keepgoing = False

    # End of while loop