cancel
Showing results for 
Search instead for 
Did you mean: 
jtheaton
Community Participant

Long ID's in Javascript

What is the suggested way to deal with the very long ID's that Canvas creates in JavaScript?  For example, if I try to parse this response (from a list of courses), Javascript rounds the ID up to 60780000000000040, rather than the expected 60780000000000037. 

const str = '[{"id":60780000000000037,"name":"This is a test","account_id":60780000000000198}]'
console.log(JSON.parse(str)[0].id)

Tags (3)
11 Replies
jtheaton
Community Participant

Yes I saw that.  It only describes what the problem is.  Not how to solve it.  How do you parse such a large number from Canvas in Javascript.  I KNOW the issue is that JS is only 32 bit, but how does Canvas suggest dealing with the large numbers it generates.

The suggested solution seems to be to place quotes around the ID's in JSON.  I don't control what Canvas outputs, so that it not an option.  As an enhancement request (bug fix?), I would suggest that, if Canvas is going to generate such long ID's I would suggest adding a quote.  These are not really numbers, its not like you are doing any sort of math on them.  The way it is now, it makes Javascript development more difficult than it should be.

By the way, the above JSON comes from the course list at: https://canvas.instructure.com/api/v1/courses 

All of my course id's, student id's, etc are this large.

robotcars
Community Champion

I'm curious about what endpoint you found that returns the global id? Every API call I have used returns or accepts the local id. 60780000000000037 = 37 (local id)

I was trying to come up with a couple of quick fixes for you but stopped when toString() won't even convert the number to a string.

Overall it seems like a bug, see lines 9 and 463

switchman/shard_internal.rb at master · instructure/switchman · GitHub 

James
Community Champion

I wouldn't call it a bug.

See JavaScript data types and data structures - JavaScript | MDN  for restrictions on JavaScript numbers. Remember that Canvas uses Ruby for most things.

Canvas mentions the large IDs, and tells you what to do, in the very first page of the API documentation: Canvas LMS REST API Documentation 

robotcars
Community Champion

If the ruby source is set to create IDs up to 10 Trillion, wouldn't 60 Quadrillion be too much? 60,780,000,000,000,037

I'm actually very curious about this, because I want to write more shared code that can work for any instance. Since you're the math professor, please teach me. :smileygrin:

// 10 trillion - 10,000,000,000,000
typeof 10000000000000
// "number"

260000000123456 % 10000000000000
// 123456‍‍‍‍‍‍‍‍‍

607800000123456 % 10000000000000
// 7800000123456‍

60780000000000037 % 10000000000000
// 40‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Isn't it a bug if the ID's per SHARD can be more than the 10T defined by the code?

jtheaton
Community Participant

Thanks, James...  I checked the API docs on large numbers, this fixed it for me:

"To force all ids to strings add the request header Accept: application/json+canvas-string-ids This will cause Canvas to return even integer IDs as strings, preventing problems with languages (particularly JavaScript) that can't properly process large integers."

James
Community Champion

carroll-ccsd

If the ruby source is set to create IDs up to 10 Trillion, wouldn't 60 Quadrillion be too much? 60,780,000,000,000,037

You're misreading that. Each shard gets up to 10 trillion IDs. 

That means that the shard ID is multiplied by 10T to avoid conflicts and then the ID is added. If you're in shard 64, you would have 64*10T=640T. The example of Shards in the LTI Variable Substitutions uses 1234, to that would be 1234*10T=12Q 340T

According to Number.MAX_SAFE_INTEGER - JavaScript | MDN , you're safe up to about 9 quadrillion, so 60 quadrillion would experience the problem. The actual maximum safe number is 9,007,199,254,740,991. 

Basically, a Canvas ID is a string made up of digits instead of letters. If your system can't handle it as a number, then you have to treat it like a string.

We're on shard 1 and so all of our long IDs are between 10T <= ID < 20T and they fit without worrying about data loss since they're less than 9 quadrillion. Like you, all of mine come through without the extra padding and the included shard. But you're correct that I should probably watch out more for people in other Shards. It's one of those things that you don't think about because it's not your experience. Experiences drive beliefs and my experience was that all the Canvas IDs are short. Others may have trouble running some of the things I've written because I try to coerce them into numbers because I would rather work with numbers that strings (occupational hazard).

A conversation related to the problems that happened when the length of the keys occurred can be found here: Increased length of surrogate keys .

I hope that's enough explanation -- I've got to go pick up my daughter right now and have to stop writing.

robotcars
Community Champion

 @James ‌,

Thank you for the explanation and the extra link, really appreciated!

I have also been trying to force all IDs to numbers and this definitely creates portability problems.

Do we know if each shard is unique to a Canvas instance/institution or can multiple schools be on Shard 64?

James
Community Champion

I don't know about Shard 64 specifically, but I am about 99.999999% positive that multiple schools can be on the same shard. Otherwise, it would be extremely unlikely that we would be on shard 1 since we didn't come to Canvas until 2012. Our IDs are not sequential, which means that while course ID 1234 might be ours, 1235 might belong to someone else (and Canvas dutifully reports a 401 not authorized error when I try), yet given what Jeff is saying, we might be getting a much longer number if we weren't on shard 1.  It might (perhaps not) be easy for you to verify. Since you're able to use small IDs, what shard are you using? If it's 1 then we know that multiple institutions can share a shard 

Canvas is limited by the size of PostgreSQL's BIGINT to 9,223,372,036,854,775,807 (9 quintillion). That seems safe for a while, but it does limit them to only 922K shards. Maybe they are reserving the leading digits before the shard number for something else down the line?

What I can't answer right now is if each school is limited to a single shard.