Empty AJAX response when URL works fine in browser

[dramus]

I'm trying to retrieve Canvas data for a project where I work, which self-hosts Canvas for its faculty and students. I'm using Fetch API (but have tried with other ajax methods too). However, anytime I try to GET something I just get an empty response back. Body is null and statusText is empty. 

The thing that's weird is if I access the URL directly in my browser, I see the JSON just fine. But when I Fetch with the exact same URL I just see an empty response. For example, in the browser https://canvas.uchicago.edu/api/v1/users/self shows me my ID, name, sortable name, etc. Hitting that with a GET fetch gives just an empty response.

Status of the call is 200 and I'm not getting any CORS errors or anything like that, so as far as I can tell it's going through OK.

Any thoughts?

Thanks everybody!!

var token = // my temp token here
var url = 'https://canvas.uchicago.edu/api/v1/users/self/';

var headers = new Headers();
headers.append('X-Requested-With','XMLHttpRequest');
headers.append('Authorization',`Bearer ${token}`);

var options = {
  method: 'GET',
  headers:headers,
  credentials: 'same-origin',
  mode:'no-cors'
}

fetch(url, options).then(function(response) {
  console.log(response);
})‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

[themidiman]

I was able to repeat your issue with my hosted version, so I think that rules out the fact you are using a self-hosted version of Canvas.

One thing that I'm not entirely familiar with is the if-then deferred promise code you've supplied, or the limitations of the Fetch API vs XMLHttpRequest. What I do know is that if you're doing AJAX queries on API endpoints within the app, that you may be encroaching on JSON Hijacking. Canvas inserts a bit of prepended text to the JSON returned within the confines of a web browser JavaScript engine, so using JSON.parse() will cause it to choke on that prepended text. This is a security measure to be sure that JSON will never execute any code.

However I was able to work up another AJAX code snippet that does return actual data to the console. It assumes you're already in an authenticated state so you would have to add the authorization information to the header.

var url = 'https://canvas.uchicago.edu/api/v1/users/self/';

function ajax_get(url, callback) {
    var xmlhttp = new XMLHttpRequest();
    xmlhttp.onreadystatechange = function() {
        if (xmlhttp.readyState == 4 && xmlhttp.status == 200) {
            try {
                var data = xmlhttp.responseText;
                    
            } catch(err) {
                console.log(err.message + " in " + xmlhttp.responseText);
                return;
            }
            callback(data);
        }
    };
 
    xmlhttp.open("GET", url, true);
    xmlhttp.send();
}

ajax_get(url, function(data) {
    console.log(data);
});‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Hope this helps.

[James]

 @dramus  ,

I was able to make your code work (with the appropriate 2 changes at the top) and make it not work as well.

If you are on a page in Canvas already, so that the domain for the page where you're viewing the browser's console is your Canvas instance, it works.

If you open the developer tools from a non-Canvas page, then I get the results you're describing.

To clarify, I originally opened up the the developer tools in Chrome from a blank tab, so it thought google.com was the domain. When I tried your code there, it did not return any results, the status was 0 (not 200) and the ok was false instead of true.

Then I logged into Canvas, opened the developer tools, and re-executed the same code and I got a valid response.

Now, you may be saying ... but, but, but ... I'm sending an authorization header so it should work without being on a page inside Canvas.  That's what I thought originally. What tipped me off was the error messages I was getting when I played around with your options.

First, if I'm on the Canvas page with the developer tools, I do not need credentials or mode to be specified in the options. I do need the authorization header, but this is explained in the Using Fetch - Web APIs | MDN instructions where it says:

By default, fetch won't send or receive any cookies from the server, resulting in unauthenticated requests if the site relies on maintaining a user session (to send cookies, the credentials init option must be set).

The takeaway there is that if you are running this script from within Canvas (like a user script), then you don't need to add the authorization headers, but you need to use credentials: 'include' in the options rather than credentials: 'same-origin'. You do not need to use the mode line when you do this.

If you are running this from within Canvas, then all you need to accomplish everything you have is this:

var options = {
  method: 'GET',
  credentials: 'include'
};

fetch(url, options).then(function(response) {
  console.log(response);
});‍‍‍‍‍‍‍‍

Now, why wasn't it working from outside Canvas when you were including the authorization header? Let me go back to opening the Developer Tools from a blank tab. I am still logged into Canvas, just not on a Canvas page when I opened the developer tools at this point.

If I take out the credentials property completely, then I get a 404 not found for the API call. When I remove the mode: 'no-cors', then I get this (I removed the URL so Jive doesn't hyperlink it)

Failed to load <webpage URL>: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'https://www.google.com' is therefore not allowed access. The response had HTTP status code 422. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

Of course, you already had that in there. What I noticed was that the Authorization header was not showing up in the Request headers, so I did some research into whether that might be an issue. I still don't know, but the Headers.has() said it had it, it just wasn't showing up in the Request Headers, but Chrome was also saying that "Provisional headers" weren't being shown.

At this point, I switched over to Firefox and still did not get the Authorization header in the request headers. But if I take out the request header completely, it comes back with a 401 (Unauthorized) error now instead of a 404 (Not Found) message, so it must be that the header is getting through.

It really appears to be related to the cross-origin restrictions that are messing you up. And it might have been as simple as you weren't on a page in Canvas when running it and you might have your solution and not need any more information.

On the other hand, if you plan on running this script from within a browser from other site, then you should probably reconsider as the token will be exposed. If you're running it from a server or a script with Node JS, then it shouldn't have the cross-origin restrictions since it wouldn't be in a browser.

Hopefully that's enough to get you headed in the right direction.

[dramus]

Interesting... thanks for looking at this JEFHQ12951 and  @James . After researching and trying more things based on your responses, I'm pretty sure my troubles all come down to the backend servers I'm trying to reach not allowing cross origin access, like you mentioned James.

Jeffrey, I was able to get actual data back like you showed when I used a chrome extension that forces Access-Control-Allow-Origin, but other than using that I'm stuck with the CORS error (422). Additionally, if I add an authorization header to include my access token, that triggers preflight, which again runs into the CORS thing. Currently I've only been able to get this to work by including the access token in the URL, which of course I don't want to do, or by being logged into Canvas in another tab. 

James, I do want to ultimately run this from a browser from another site outside Canvas. The end goal is a web app that assists teachers in some tasks, and I want to be able to run ajax calls to do things like load in the class roster. From what I've read on Canvas's documentation, as long as I use OAuth and put the token in an authorization header, I should be okay. Am I missing something? Is there a better way to go about that?

Thanks. 

[James]

 @dramus ,

Maybe someone else can comment, but I've not been able to successfully get around CORS (I think that's by design) except by using the custom jqXHR functions that GreaseMonkey implemented that bypassed the browser's version. That said, I've never tried making Canvas REST API calls from within a browser using anything other than from a page within Canvas. From Canvas' perspective, accessing it from a browser that isn't logged into Canvas probably isn't secure, but if it's a program that's going to be used by multiple users, you should probably be doing oAuth rather than putting an access token in -- the user gives permission for your app to access Canvas and then your app works out the oAuth negotation to make it happen. I will also say I've not written multi-user applications that needed to access the API, so I've not needed to go that route, so I'm not an expert on oAuth, but it seems like worth researching using oAuth instead of an access token.

Without doing additional research into what is or isn't accomplishable, my recommendation would be to have your server make the calls to the API. You could get an access token with elevated privileges or developer key (since you self host, it should be easier to justify) for your app and then have the script in the browser interact with your server that makes the calls to Canvas after some kind of authentication/authorization to make sure they're legit and then forwards the appropriate information back to your script. That's an extra step in the process.

There may be a way in the browser to not send the headers, but when I was researching things last night, there was a special section on the authorization header in the standard that isn't processed the same way. It may be part of the Fetch API and you might have to digress to using a different library like jQuery or something rather than the web API functions.

[dramus]

I definitely do plan on using oAuth for this, the token is temporary and only for my account while developing. I'll ALSO see what sort of results I can get with jQuery ajax. Thanks!