Am I mistaken or is pagination in the "List user page views" API broken?

[david_taylor]

I have been playing around with the "List user page views" API endpoint and pagination appears to be broken...or I'm an idiot.  Either possibility is equally likely.

Normally, when you go and get a page of data you will get a "next" in the header as explained here:
Pagination - Canvas LMS REST API Documentation 

I've written lots of API calls and this always works fine.  HOWEVER, when I try to bring back page views it *ALWAYS* shows that there is a "next" page even when there is not.  Obviously this is a problem because my code keeps going back for more data.

I would love some feedback.  I'm hoping I'm just missing something, but I'm afraid that this API endpoint just does not work like all of the others I've used. 

[pklove]

It does work slightly differently.  The paging links have page=bookmark:sadfsdfd rather than the usual page=number.

There are some other endpoints like this.  I first came across this in this post: https://community.canvaslms.com/message/61158-re-students-login-logs-download?commentID=61158#commen... 

I haven't actually tried endpoints like this, so a few questions.  What happens when you get to the end of the data?  Do you get an empty return, or just a repeat of the last lot of data you got?  Does the bookmark change?  If not you could check to see if the next bookmark matches the current one.

[david_taylor]

It just keeps returning the same data.  In fact, I tried to bring back page=2 of data and it brings back the same data as page=1.  But as you said, I guess it doesn't use the same page=number format.

I'm not familiar with what you mean when you say "page=bookmark:sadfsdfd".  Where is this sort of pagination documented?  I've only used the "page=number".

[pklove]

I haven't seen page=bookmark:sadfsdfd documented, its just what appears in the links.

[pklove]

BTW, page=number isn't documented either.  The link urls are opaque, so the intention is that they are used as returned.  So, although you can usually make up your own one with page=number, there is nothing to say you can do this.

[david_taylor]

Hmmm.  See, I'm parsing out the header manually.  I'm not using any libraries.  I flat out do not see the URLs in the header text.  How do you actually see/use "opaque" URLs like this?

[James]

If this is your literal Link header, then it does appear that something is getting stripped out.

"Link": "; rel=\"current\",; rel=\"next\",; rel=\"first\"",

My Link headers look something like this:

Link: <https://richland.instructure.com/api/v1/courses?page=1&per_page=10>; rel="current",<https://richland.instructure.com/api/v1/courses?page=2&per_page=10>; rel="next",<https://richland.instructure.com/api/v1/courses?page=1&per_page=10>; rel="first",<https://richland.instructure.com/api/v1/courses?page=5&per_page=10>; rel="last"

I hope this isn't just another Jive stripping out things when you post them in the community but it's really there.

[david_taylor]

Ha!  Thanks for questioning me on this.  I was writing the contents of the header to the browser screen and that was not displaying.  I dumped it to a text file and it *IS* coming back. I've just always looked for "next" to be there and I go and grab the next page if it is.  In this case, "next" always seems to be there but it doesn't work correctly.

Anyway, this is what is coming back in the header:

rel="next",<https://zzzzzzzzzz.instructure.com/api/v1/users/XXXXXXXXX/page_views?end_time=2018-09-04&start_time=...>

But this looks REALLY strange to me.  Why are there TWO end_time and start_time parameters in this "next" link?  This is really confusing.  My original URL that I sent looked like this:

https://zzzzzzzzzzzz.instructure.com/api/v1/users/XXXXXXX/page_views?start_time=2018-07-01&end_time=... 

[pklove]

Note that that isn't the next link.  The next link is the one before the ;rel="next".

The date repetition does look odd, but it doesn't seem to affect the pagination working.  I've tried with three different users and the next link disappears at the end.

[James]

I've seen that duplication myself and consider it a bug in Canvas code, but Peter is right about it shouldn't be affecting things. He's also right that you want the one before the "rel". It's a comma separated list of links and the semicolon separates the link from the relationship.

You can split on the comma and iterate through array. This is what I do with my PHP code, but it's not absolutely necessary to split it. Here's a fairly lazy check for a "next" link in PHP. It assumes that the link headers are in $link. If it succeeds, then $matches[1] will contain the URL.

preg_match( '/<(.*?)>; rel="next"/', $link, $matches )‍

There's a lengthy discussion on the Canvas Developer's group about Handling Pagination that has contributions from several people and code examples in multiple languages.

[david_taylor]

I see--thanks for the clarification. Since I have not actually used those URLs coming back in the header before (I always use the page=#, incrementing the number myself) I'll need to play around with it a bit.  But you guys have put me on the right track.  Many thanks, pklove‌ and  @James ‌.

[jperkins]

david_taylor   @James  I've seen the duplicate parameters on the when I resend the parameters on the returned next href. So instead I hard code the parameters into the initial request_url instead of in the request body. Below is an example of a request in Ruby.

request_url = "#{canvas_url}#{api_endpoint}" #include parameters for API call in url
count = 0
more_data = true
while more_data 
    request = Typhoeus::Request.new(
        request_url,    #we need a variable here because we need the api url to change
        method: :get,
        headers: { authorization: "Bearer #{canvas_token}" }
        )

    request.on_complete do |response|
        #get next link
            links = LinkHeader.parse(response.headers['link']).links
            next_link = links.find { |link| link['rel'] == 'next' } 
            request_url = next_link.href if next_link 
            if next_link && "#{response.body}" != "[]"
                more_data = true
            else
                more_data = false
            end
        #ends next link code

        if response.code == 200
            data = JSON.parse(response.body)
            #do stuff here
        else
            puts "Something went wrong! Response code was #{response.code}"
        end
    end

    request.run
end
puts "Script done running"‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Hope that helps!

[pklove]

That's  not the case here.  Making the call with the request parameters as a query string does result in them being duplicated.

BTW, as the page views endpoint call is a GET, I think you would expect the request parameters to be passed as a query string and not in the body anyway.

[pklove]

I just tried List user page views and on the last page the next page dropped off like usual and a last link appeared (which equalled the current link, so all good there).

So it worked like usual for the user I tried.

[david_taylor]

I think that is getting stripped out. I never actually see anything in link other than something like this:

  "Link": "; rel=\"current\",; rel=\"next\",; rel=\"first\"",

So I always just look to see if there is a "next" and if there is I go back for the next page. 

[pklove]

Ah, well I'd get that fixed first.  What language are you using? 

[david_taylor]

I'm using PHP, but that was actually copied/pasted from the live API tool that Instructure provides. But I never see the URLs in the header text that comes back to my PHP scripts. 

[pklove]

Does this help with getting the headers: https://community.canvaslms.com/message/115743-re-list-active-courses-in-account-not-paginating?comm... 

[david_taylor]

I just wanted to thank everyone again for the help.  Once I started dumping the header text to a text file instead of trying to print to the browser screen for debugging I was able to see the links being passed back and everything began to make much more sense.  I was able to get things working this morning.  You guys really saved me a ton of time by pointing me in the right direction!

[agschmid]

David - Maybe give Insomnia a whirl...it's very useful for testing out calls before writing a script: https://insomnia.rest/