Float value error? Computation is incorrect in Formula Question
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I am posting here to see if I am missing something (user error) or if this is a real error in Canvas (before I try to report it to them). I am doing a formula question. My goal is to make it so that if the sum of these two numbers ends in a 5, then it MAY round differently (this is an issue with Chemistry, where the rules for rounding are sometimes different than they are in math). You can see the formula I use at the top and I've highlighted a random set of numbers that SHOULD evaluate as 0 in the IF statement, and therefore should return a 0 instead of 500. However, that's not what's happening!
round(10*(4.6+8.45))-10*(4.6+8.45)-0.5
=round(10*(11.05))-10*(11.05)-0.5
=round(110.5)-110.5-0.5
=111-110.5-0.5
=0.5-0.5
=0
But again, even though it should evaluate as false for the IF statement, it is returning the true argument instead of 500. I am guessing either I have my math wrong somewhere or there must be an error caused by some kind of programming choice (like a float value issue), but I have no idea what would be causing that. Any help would be great.