prevent making thousands of calls to the API from web page? Is there a better way?

[jsimon3]

I want to display the analytics for observees to the observers. On the observers page >

```

/profile/observees/

 ```  in order to do so the api calls that I would need to call are:

```

//api/v1/users/self/ --to get the user id

/api/v1/users/:user_id/observees/ - -using the initial api call to substitute for user id to get an array of the ids

//api/v1/users/:user_id/page_views -- pass in each id history to display to parents (repeat api call for each student)

```

I am using the fetch api to get the information:

```

const apiTest = () => {

  const url1 = "/api/v1/";

  const options = {

    'credentials': 'same-origin',

    'headers': {

      'accept': 'application/json'

    },

    'timeout': 5000

  };

  fetch("/api/v1/users/self/", options)

  beep boop

        })

        ).then(res => {

          blah blah

        }).etcetera);

}

apiTest();

```

and then innerHTML to display information fetched.

Is there a better approach to this?

I feel like I am going to breeze through our API cap or something.

[robotcars]

I don't think you'd be in danger of hitting the cap limits.

Can you use the ENV* variables to get the user id, or better yet, use self on endpoint #2

/api/v1/users/self/observees/

* console.log(ENV)

> ENV.current_user {id:}

> ENV.current_user_id

Tagging Canvas Developers‌ for additional visibility.

[jsimon3]

ok, I can eliminate the first call completely using the self on the second, thanks.

but am I approaching this in the right way?

if a parent has 6 kids that is 7 api calls...

and OMG never new this ENV variable existed

where is documentation of this witchcraft?

[robotcars]

OMG never new this ENV variable existed

where is documentation of this witchcraft?

I don't think there's documentation, not even really sure it's there for us/developers, I think we just found it and started using it?

The rate limiting question is hard to evaluate without more details, and honestly I've never worried about it, and I've done some pretty terrible things against production, but not as crazy as  @James  and his concurrent data pulls. He definitely has more insight here than I do.

One thing to remember is that each user is using their own quota.

For more info read,

API Rate Limiting

Dynamic Throttling Implementation using rate limit header

Is there any limits on the number concurrent calls to canvas REST API?

If you are writing something for use within Canvas Themes those requests are limited as the user. Never use your own token in a theme hack. If you are using an LTI or something that uses a developer token, have the user issue the token, not your own.

if a parent has 6 kids that is 7 api calls...

More than that :smileygrin:

// when I first read that, I saw it as a student with 6 courses, leaving this bit for detail.

The user page_views endpoint is by user, not by course. If you want to split the data by course, you'll have to code it.

{
  "url": ".../courses/1234/files/6789/download?wrap=1",
  "context_type": "Course",
  ...
  "links": {
    "user": 4567,
    "context": 1234,
    ...
  },
  ...
}‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

// 6 kids is 6 requests + pagination

By default you will get 10 rows from the end point, with pagination you can get 100 rows at a time, collecting additional page views with additional requests.

Pagination - Canvas LMS REST API Documentation 

Handling Pagination 

but am I approaching this in the right way?

With the API you always have to start with what's available, but after that, the options are in your imagination and desired product.

Scope

How many users are you expecting, how many of your observers log in each week/month/term?

What are you trying to show the observers, what do they want to see?

Is it a theme hack or an LTI?

Do your observers use the Canvas Parent app, or do they use the desktop/browser?

User Experience

What does the user see when they get to your feature?

Will they get just a long list of page views? Can you load the most recent 100, and build in something like an infinite scroll that collects/shows more if they browse that far?

The following get considerably more complicated, because you have to do enough collection to gather the data and split by course or range...

Can you give them a date picker and let them set a range? Then break the results up by course?

Can the observer click into each course instead of seeing hundreds of page views per course on 1 page...

maybe they don't want to see every course all at once, but are checking into activity of just 1?

Since the endpoint has a date range param, maybe you could default them to the current week, and then give them a month filter or the ability to scroll back to see previous weeks?

[jsimon3]

Mr. Carroll, just plain thank you. Your terrific insight and additional references that now have taken refuge in a new bookmark in my browser.

Is it a theme hack or an LTI?

Do your observers use the Canvas Parent app, or do they use the desktop/browser?

This is a theme hack I guess you could say.

the observers use both but it seems they tend to utilize the browser a bit more

User Experience

What does the user see when they get to your feature?

Will they get just a long list of page views? Can you load the most recent 100, and build in something like an infinite scroll that collects/shows more if they browse that far?

-Honestly I would just lock it down to the past 20 things that the student has done.

   this was just to give the parents a since of when or if they students where doing their work

-Is there something I am missing that I could pass in to share with parents that would give them a since to know when their kids are or if they are doing things(not just in one class but any classes)?

-If you could so happen to infinitely scroll through your wisdom to point me in the right direction for this little feature I would be ecstatically pleased.

[robotcars · Accepted solution]

You seem enthusiastic about trying to solve this so I'm going to lay out what I can and just state, I'm always open to collaborate, especially with like-minded K12 folks.

Years ago when I started looking to provide tools for our district, which is not small, the amount of endpoints and code and limited staff (me) necessary to bring in the data we wanted to use was just overwhelming. Then Canvas Data came out it solved much of our reporting and data needs, and we focused on using the API to send data to Canvas and manage background tasks. One of our purposes for CD was an LTI for teachers to discern between submissions/activity and just course access, which is page views or Requests, so I get the need for showing access vs course progress stats.

Then, Live Events was released and we no longer have a delay in data, and can use Canvas Data or Live Events, or combine them based on purpose. We currently use LE to provide a 1 page attendance screen of last submission by attendance week, number of submissions for that attendance week, and most recent activity, with a count of students current online for the course. This gives teachers a much faster way to do SIS attendance than bouncing around Canvas to collect that info on each student, and we're eventually going to use this data to pass a boolean course attendance to the SIS.

Both of those solutions require some technical architecture and deployment to ingest the data, and a secure web server to host the LTI. If you're interested in that, we can go down that rabbit hole.

However, an LTI for your purposes could be much simpler start using a web server and server side LTI as  @James  suggests. The Awesome CanvasLMS has tons of open source resources, including a single page Node LTI examples from pklove‌ that might be a decent starting point. With that, you could use a developer token to collect page views when the user accesses the LTI and provide the information you want to the parent, or other endpoints only available through admin privileges.

Additionally, I tried to look into what API and GraphQL endpoints are available for observers but was directed to look at the permissions by role in Canvas, this revealed some really absurd options.

Analytics - Page Views is not available, cannot be enabled for an Observer. Parent should have access to this. Alternatively, a parent should have a view of /users/:kid profile page by default, which would provide page view list.

Really absurd, permissions page has the option to allow an observer to:

Add/Edit/Delete:

• Course Pages
• Learning Outcomes
• Course Files
• Course Calendar events
• Course Content*

But for some odd reason, we can't let them Manage the Course State, and Add/Edit or Delete course sections.

We can give them the option to Import Learning Outcomes, though. /sarcasm :thumbsdown

Most of this, I'd hope isn't actually possible through the views/routes, but it shows some not-thought-out-to-well purpose of the Permissions page. Some testing will be done and a conversation with our CSM.

edit: nope!

In test instance, I allowed observers to set course content. I deleted assignments, modules, and edited the syllabus... but can't set page_view rights to their kids.

[jsimon3]

Sorry about the late reply handling other issues that popped up yesterday.

I also have a huge team to deal with all of the shenanigans that may arise(Me, myself & and a former student of mine working as an intern). We have written a Library in C# and hosting on a NuGet server to handle most back end things and reports. I would love to work with you to see how you have approached things in the past and moving forward, I just started in September and are, as they say, "building the plane as we fly it". I would like to share our libraries and everything I done so far as well. We are also utilizing doxygen to document as we go. 

ATM I don't have good latch of how to process the information and pass the info to the parents.

[James]

It's more likely that you'll bore the person to death while you wait for the results to load. Determining anything from page_views in real-time is slow. The page_views uses bookmarks for pagination, not the page= notation, so you cannot take advantage of concurrent requests. It took my computer 2.4 seconds to fetch 100 results and then 1.7 seconds to fetch the next link supplied in the headers. The next one after that was 4.9 seconds. Normal API calls are much, much quicker.

However, there is an even bigger issue. The page views is limited to the admin users and a user cannot even fetch page_view information about themselves unless they're an admin. 

GET /api/v1/users/self/page_views

{"status":"unauthorized","errors":[{"message":"user not authorized to perform that action"}]}

That means that you're not going to be able to do it completely within the browser since including an admin account token would be a huge security no-no. One could write a server-side script that contains an admin token so you can download the page view information or go to an LTI.

[jsimon3]

However, there is an even bigger issue. The page views is limited to the admin users and a user cannot even fetch page_view information about themselves unless they're an admin. 

Thank you for crushing my dreams sir. But analytics is locked to self so I thought that page views would be my next best bet.

Problem (K-12 BTW) : I would love to show parents their students activity logs or something of that nature to when the last time they interacted in classes.

As I dig through the API I am becoming more discouraged.

Would you with all of your Canvas wisdom be able to point me in the right direction kind sir? 

[robotcars]

With 320k students, 95k Canvas Enrollments, and 44k unique students... the API is discouraging when trying to do large collections and analysis. Posting ideas, replying to your comment above.

[robotcars]

The boring aspect just seems like a matter of UI and displaying the information in a useful way. I was thinking some JavaScript filter,map,reduce on page views would give a decent report of courses visited, assets accessed in a bite size form of 'did my kid work this week'.

I had tested, ironically, using a school admin who has observees, so missed that detail, but I don't often API.

I really wish there was a better role/privilege level description of endpoints...

...observers should be able to view page view endpoint of their kids. That seems obvious.

Always helpful input.

[James]

Wow, I was reading this whole thread wrong. I thought Jewell wanted to show the students what their parents were up to.That changes everything -- or perhaps nothing.

With an LTI that has admin access, you can get the analytics for any student. There's enough there with the existing analytics API, and the new analytics stuff is done with graphQL, that you don't need the page views. The ability to view analytics is a course-level permission, although it's locked for the observer level for me and I don't use permissions enough to know how to get around that right now (I'm also taking a short break from a major project that needs done tomorrow, so I don't have a lot of time to look into it).

If the analytics route is available, one wouldn't have to make it an LTI, however ... Canvas would prefer that you use an LTI instead of hacking the page with JavaScript. I routinely ignore that advice.

[jsimon3]

If the analytics route is available, one wouldn't have to make it an LTI, however ... Canvas would prefer that you use an LTI instead of hacking the page with JavaScript. I routinely ignore that advice.

Please tell me more if you would so happen to find time...

[robotcars]

Not sure we're all on the same page, here's my thinking/tests...

$.getJSON('/api/v1/users/self/observees/', function(r) { console.log(r) })
// > (2) results in test user
$.getJSON('/api/v1/users/123/page_views', function(r) { console.log(r) })
// > 401 returned
$.getJSON('/api/v1/courses/987/analytics/activity', function(r) { console.log(r) })
// > 401 returned
$.getJSON('/api/v1/courses/987/analytics/users/123/activity', function(r) { console.log(r) })
// > 401 returned

//401 {status: "unauthorized", errors: [{message: "user not authorized to perform that action"}]}
// errors: [{message: "user not authorized to perform that action"}]
// status: "unauthorized"‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Cannot enable page views and analytics for Observer role.

[jsimon3]

That is correct I thought that  @James ‌ might have had some trickery up his sleeve. I was intrigued. 

[James]

One may be able to create a custom role based off a different user type and then go through and change the permissions to match that of observer but give then analytics permissions. I wouldn't advise that, though as there are probably some other issues lurking that would be bad.

One could download the analytics for each student once a day and archive it locally in a database. That's what I do. Extending that, you could then pull from that database using the server side implementation and display the analytics with a note that it was last updated at a particular time.

One could use the API to fetch the analytics, but I wouldn't put this on a frequently displayed page inside Canvas. Since you need to be the user or an admin to view the analytics, it's possible that the back-end script might be making too many requests if you have a large school. It's probably not a terms of service violation since you're just displaying a widget, but it also has the slim potential to hit the limit threshold if you make a lot of those calls in a very short time. You may also want to implement a caching feature so that it only updates the information once an hour. It depends on what Jewell is after.

One could implement the Canvas Live Events that Robert likes and then have the information quickly available in a local database.

We don't really use observers much (college level) so I'm not familiar with their capabilities. Is there nothing there that lets observers see what the observees are doing, just what assignments they have coming up?

[jsimon3]

As I am looking into the methods I believe that the live function sounds the most reasonable carroll-ccsd‌ mentioned it earlier. Thank you both for your time and thoughtful responses.