422 Error when making POST Request to Favorite Courses API

[lbeveridge]

Hello,

I am working some custom Javascript that needs to be able to add and remove favorite courses for users through the Canvas API. I can get the list of a user's favorites without any issues, but I am getting a 422 "unprocessable entity" response when I try to do a POST request to add a favorite course for a user.

First, I tried making the POST request without any authorization, since the GET request for favorite courses does not require it, however I got the 422 response. Then, I used Chrome's dev tools to look into the request that Canvas sends when a user clicks on the start icon in the "All Courses" menu to favorite a course. I found that the payload Canvas sends includes an "authenticity_token" which is called the "_csrf_token" in Canvas's cookies. This token is regenerated each time it is used, but I tried manually adding it to my code to see if that would solve the error. Here is the code I am currently using:

data = {}
fetch('<our canvas url>/api/v1/users/self/favorites/courses/<course number>', {
  method: 'POST',
  headers: {'Content-Length': 0, authenticity_token: '<csrf_token copied from Canvas's cookies>'},
  body: JSON.stringify(data),
})
.then(response => response.json())
.then(data => {
  console.log('Success:', data);
})
.catch((error) => {
  console.error('Error:', error);
});

With this code, I still unfortunately get the 422 response. The user I am testing this with is enrolled in the course that I am trying to favorite, and the course is not already favorited. I am thinking I must be doing something wrong with this request!

I would greatly appreciate assistance with this if anyone has ideas as to why this error is occurring!

[James]

@lbeveridge 

You don't send the CSRF token as an authenticity header. You send it as part of the data that becomes the body rather than as a header. Move authenticity_token from the header object to the data object.

Since you're sending JSON, you may need to include "content-type: application/json" in the headers. You generally do not need to include content-length. I tend to throw in an "accept: application/json" header as well.

I didn't check the rest of the code for any problems, just the thing that jumps out at me. I've used the CSRF token before for API calls, so I just searched my GitHub repository to see how I handled it. I was looking at a PUT instead of a POST, but that shouldn't affect where the authenticity_token goes.

[lbeveridge]

Thank you for the help here James! I appreciate it!

I went ahead and tried adding those fields in, and moving the CSRF token into the body, but I'm still getting the same error unfortunately. Here is the code I have now:

data = {'authenticity_token': '<CSRF Token>'}
fetch('<Our Canvas Domain>/api/v1/users/self/favorites/courses/656', {
  method: 'POST',
  headers: {'content-type': 'application/json', 'accept': 'application/json'},
  body: JSON.stringify(data),
})
.then(response => response.json())
.then(data => {
  console.log('Success:', data);
})
.catch((error) => {
  console.error('Error:', error);
});

If you have any other ideas about what mistake I may be making, I would be grateful for the help. Thank you!

[robotcars · Accepted solution]

@lbeveridge 

I've never tried to pass the token in the form data, just the headers. I did some practice with this awhile back for XHR testing and applied it to your fetch. I was able to make this work. Since the token is refreshed, I use a function to strip the token from the cookie and decode it, the function making it reusable for subsequent requests. You can see this by making any AJAX request on the page and calling CSRFtoken().

const CSRFtoken = function() {
  return decodeURIComponent((document.cookie.match('(^|;) *_csrf_token=([^;]*)') || '')[2])
}

fetch('/api/v1/users/self/favorites/courses/1234567', {
    method: 'POST',
    headers: {
      'content-type': 'application/json',
      'accept': 'application/json',
      'X-CSRF-Token': CSRFtoken()
    },
    //body: JSON.stringify(data),
  })
  .then(response => response.json())
  .then(data => {
    console.log('Success:', data);
  })
  .catch((error) => {
    console.error('Error:', error);
  });

[James]

Nice job Robert ( @robotcars ). Reminds me of the good old days when there was no response when I started typing and there was by the time I was done and hit post.  You beat me by almost 30 minutes, which means I spent way too much time on it.

@lbeveridge - see what Robert said. We said the same thing, he just did it much more succinctly.

[robotcars]

Those were good days. They will come back. I thought you were gonna beat me to it. Your detailed explanations are always on point!

[James]

@robotcars 

Speaking of old times ... something bothered me about the regexp when I first saw it. It works, but it took a while to figure out why it worked. This is me mostly rambling ... avoiding grading finals.

My first question was whether the || '' was misplaced? I originally though you were trying to return '' for an undefined value, but then it hit me that you were trying to avoid the error of when the match comes back null.

document.cookie.match('(^|;) *_csrf_token=([^;]*)') || '')[2]

The regexp.match() returns null, the same as regexp.exec(), when the string is not found. If _csrf_token is not in the cookies, then the match returns null and you assign it the empty string '', which is a string and not an array. I originally thought that accessing [2] on a string would return an error, but it turns out it returns the character in that 0-indexed position. The empty string has 0 length, so accessing ''[2] returns undefined and decodeURIComponent(undefined)='undefined' (it literally returns the string 'undefined').

How bad is it that I originally ended that sentence with a semicolon?

Since I had always used substr() to get a portion of a string, I didn't know the other method worked, so I'm glad I dug into it.

Since match returns an array, my internal linter (under heavy influence from the external linters) would suggest that I'm returning different types and proffer that I should use || [] instead of || '', but the end result is the same. Rather than getting undefined because the length of the string is 0, you're getting undefined because the length of the array is 0.

The external linter I'm using keeps telling me to reduce the complexity of my code from 98 to 15 or less. It really doesn't like deeply-nested if statements within loops.

Now that I know your code is good and works, the first thing I saw was that you could use a non-capturing group ?: as in (?:^|;) to use [1] at the end instead of [2].

I played around with it some more. I investigated word boundaries \b, but it turned out that won't work since it picks up on things like !_csrf_token.

What you're doing is simpler than what I came around to. I used regexp early on, especially with the link headers for pagination, but then switched to splitting the cookies into an array and comparing one by one.

The JavaScript Info website says the easiest way to get a cookie is through a regular expression. Then they show how to creating a regular expression from the name instead, which is essentially what you've done. They do use (?:^|; ) -- note there's a space after the semicolon, so it looks like the space is required after a semicolon to be a valid cookie. I like your approach of making it optional instead. They also split up the thing into two lines. One that does the match and then one that checks to see if there was a match before decoding it or returning undefined (not the string) if it isn't there. That is a cleaner way than using the || '' or || [] and decoding an undefined value. Of course, given the system we're operating in, we should always have the _csrf_token header available to us so we take shortcuts that wouldn't work other places.

You know that I don't like doing things the established way and I know you like writing one-liners 🙂 Here's my most recent attempt [at least 3 years ago] at getting the csrf token cookie. I used a generic getCookie() rather than writing it specifically for the csrf token -- even though the only time I ever called it was for _csrf_token.

  function getCookie(name) {
    return document.cookie.split(';').reduce((a, c) => {
      const d = c.trim().split('=', 2);
      return d[0] === name ? decodeURIComponent(d[1]) : a;
    }, '');
  }

It accomplishes the same thing as the getCookie() function from JavaScript Info, but it returns an empty string instead of 'undefined' when it cannot find the cookie. 'undefined' is truthy while '' is not, so if you wanted to check to see if a cookie was present before adding it unconditionally, then you might run into issues. If I wanted to do that, I should probably start with null or undefined instead of '' in my reduce function. Like you, I added the results unconditionally, so I didn't want undefined.

The file that had that code in it was the one I was looking at when I made my original response to this thread. I did put it into the data.authenticity_token since that's what Canvas did and so when I looked at the original code, I knew that I had done something like that before. It was so similar to what I had that I skipped past the part about getting the token and assumed it was correct. That code also included bottleneck so I made sure I didn't hit the API too heavily (it turns out there are too many missing labels that needed fixed to send them all at once).

[MattHanes]

---

@James wrote:

How bad is it that I originally ended that sentence with a semicolon?

---

I'm just here to say that I read this and quite literally laughed out loud and snorted my diet root beer up my nose.

[James]

@MattHanes 

As long as there's no permanent damage. I'm glad it was a diet rootbeer, there are certain brands of cola that you shouldn't admit to snorting up your nose.

[robotcars]

That line came from some tinkering around the concept of what happens if they dump jQuery. The only expectation may have been to visually see 'undefined' in testing. Your post may have more thought put into it than I made in the decision back then.

[James]

@robotcars 

I still think the jQuery removal is coming at some point. I just don't know which will happen first -- removal of jQuery or switching the entire codebase over to GraphQL.

[lbeveridge]

@robotcars & @James Thank you both for your help with this, it is working now!

My issue seems to have been a combination of me calling the CSRF token 'authenticity_token' instead of 'X-CSRF-Token' and the way I was manually copying in the CSRF token. I had been copy pasting the CSRF token out of the cookies each time it updated without also selecting "show URL decoded" for the cookie value. The un-decoded value causes the request to fail, whereas if I paste in the decoded value, it works. Either way, having the function to automatically read in the CSRF token, as well as changing the value in the header to 'X-CSRF-Token' solves it!

Thank you much for the assistance with this!

[robotcars]

Great! Glad it works.

Copy/Pasting the token at any point might be fruitless as any other AJAX request in Canvas will update the token, and by the time you make the request it may be out of date. Canvas has a lot of unread and ping requests that will test your timing.

[James]

@lbeveridge 

Not necessarily relevant to the issue, but are you trying to add this to the custom global JavaScript that runs inside Canvas or are you trying to do this from outside Canvas?

An example of the first kind would be to go through and make something a favorite every time people log into Canvas (a course that cannot be un-favorited). An example of the second kind would be if you want to go through and add a favorite to everyone and then be done with it and they can remove it if they want?

Both can be done with JavaScript. The first kind needs the CSRF token and runs within the browser, but may not get those people coming in through the mobile apps. The second kinds needs to be ran externally, say using a NodeJS script but needs to use the authorization header and won't have access to a CSRF token. If you're trying to take the CSRF token from within Canvas and then use it externally, you're going to have issues (401 error invalid access token).

The reason I ask is because if you want to do it outside of the browser, you can do it for all users using masquerading and an access token belonging to someone with the ability to masquerade. If I was a student and someone kept on favoriting a course for me each time I logged in and I kept on unfavoriting it, I wouldn't be happy as space on the dashboard is limited.

Okay, now to the issue. I'm including the whole investigative process so other people with the issue can come along later and have a resource. What I think the issue is at the bottom, but no one has ever accused me of being a TL;DR kind of person.

There are a couple of things that come to mind.

The first was is the person who is running the script (the person you're logged in as within Canvas or the person owning the token) actually enrolled in course 656? That is, can you go to the Courses page for that user and click the favorites to add or remove it? Not all courses are can be made favorites. However, I don't think this is it. Canvas returns a 200 OK when I try to add a favorite for a person who isn't enrolled in a course (maybe they will be at some time in the future).

The second is whether the course ID is correct. If I try this with a bogus ID that doesn't belong to us, I get a 404 Not Found error, not a 422 code.

So the next thing I do is to open up the developer tools in Chrome and look at the network traffic that Canvas sends to see what's missing. The only two things it's sending in the payload is the _method and the authenticity token. I notice you don't have _method in there.

So, I right click on the request and copy it as a fetch so I can replay it without the _method. Nope, that still works.

Then I look at the headers and something jumps out at me. They do include the CSRF token in the header, but they don't call it authenticity_token and then don't call it authorization. It's called x-csrf-token. So I take out that header and it still works.

That makes me wonder what you are including in the authenticity_token. It should just be the value of the CSRF.

If I remove the x-csrf-token header and garble the what I'm sending as an authenticity_token, I'm able to get the 422 error.

Garbling the authenticity_token so that it doesn't match the x-csrf-token gives a 500 code.

Changing the x-csrf-token while using the right authenticity_token works.

It seems that the x-csrf-token header is optional. If it is included, you do not need to include the authenticity_token in the data, but if you do include the authenticity_token in the data, then the x-csrf-token is overridden by the authenticity_token.

That seems that the easiest way to do it would be to include the x-csrf-token in the header and not mess with a body at all.

But all that then comes down to what you are using as the token. You didn't share that (good reason for not doing so), but then you have to wonder, after tracking everything else down, whether you're sending the right thing or not.

You should send just the value of the csrf-token and not the _crsf_token= portion. Depending on where you send it will determine whether you should decode it or not first.

If I look at my set-cookie header that is returned as part of the response or I use document.cookie to obtain the _csrf_token value, I get a uriEncoded value that contains things like %2F, %2B, and %3D.

Those cannot be passed in either the x-csrf-token or authenticity_token (if you are using JSON). You have to pass the decoded form.

If I let run my cookie _csrf_token through decodeURIComponent(), then I can send it as either x-csrf-token or authenticity_token. Note that decodeURI will not fix it, it needs to be decodeURIComponent.

If you are using the content-type of application/x-www-form-urlencoded; charset=UTF-8, then you must include the encoded form. If you try to send the decoded form as the authenticity_token, it will give a 422 error.

In short, use a header of x-csrf-token and make sure you pass the _csrf_token cookie value through decodeURIComponent() before doing so.

[patfm]

@lbeveridge   

I am curious if you have succeeded with your javascript for adding favorites to a user.  We are trying to find all published courses for the current term and favorite the published courses for instructors that are not yet favorited.  We have an automated process for moving courses into Canvas for the next term and when that happens any published courses that have not yet been favorited in the current term get removed from the dashboard.

Thanks.

[James]

@patfm 

I've had a rough last 35 days, so my brain is likely mush, but this sounds like a strange thing to do.

My experience has been that when new courses you are enrolled in become available, they are automatically added to my favorites. When courses conclude, there is no need to remove them from the favorites after the course closes. There's no need for faculty or students to go through and do anything. If you get more than 20 favorites, then it may not show all of them. 

• How do I customize my Courses list as an instructor?
• How do I customize my Courses list as a student?

Generally, you should not have to be setting favorites for published courses. If you are, then maybe something else is off like you're having too many classes favorited at one time? Maybe someone decided that favoriting was something that people should do -- or worse decided to do it for them. Now you have issue mentioned at the bottom of the first link that favorited courses are not automatically removed as a favorite.

In other words, I'm not favoriting or unfavoriting much at all, I'm letting Canvas handle displaying current courses without favoriting them. If I have manually favorited a lot of courses, then I may not be able to see my new courses.

Besides the technical issues, favorites are personal. Just like setting the color of a course. Setting them for someone else could be problematic, especially if they have already removed something from a favorite and you go back and set it.

To clear up the technical mess, you could unfavorite all of the old classes that never should have been favorited in the first place and then stop marking courses as favorites for people. Just let Canvas show the current list of courses and whatever people have favorited. If you have non-academic courses -- say a club -- then don't automatically unfavorite those.

Moving on past the potential issues with messing with people's favorites ...

If you want to make this on behalf of someone else, then you have some extra work to do. You should not do it through the browser. You should do it once a term, rather than every time someone logs into Canvas. Otherwise, you keep on resetting people's favorites.

That means that you should use the AP, which doesn't use the CRSF token at all, you need an access token with enough permissions to masquerade as someone. There is a Favorites API that allows you to do this. You would only do it through the browser and use the CRSF token if you were adding global JavaScript of a userscript.

When the Favorites API,  you can get a list of favorite courses for the current user. The endpoint is 

GET /api/v1/users/self/favorites/courses

In most API endpoints, the path looks like /users/:user_id/ instead of /users/self/, but this is a case where you cannot specify a user ID in the path. If you try GET /api/v1/users/123/favorites/courses, it doesn't like you (404 not found code, meaning the route doesn't exist).

You can only get favorites for yourself (the person with the access token). Within a browser, that would be the current user. Outside the browser, you need to masquerade.

Use the exact path above, but then masquerade as the user by adding ?as_user_id=123 to the end (123 gets replaced by the Canvas user ID). Use ?as_user_id=sis_user_id:456 (if their SIS user ID is 456).

The favorites API also has add course to favorite and remove course from favorites to accomplish what you're trying to do.