Float value error? Computation is incorrect in Formula Question

jbagley
Community Explorer

Screen Shot 2022-10-06 at 3.13.39 PM.png

I am posting here to see if I am missing something (user error) or if this is a real error in Canvas (before I try to report it to them).  I am doing a formula question.  My goal is to make it so that if the sum of these two numbers ends in a 5, then it MAY round differently (this is an issue with Chemistry, where the rules for rounding are sometimes different than they are in math).  You can see the formula I use at the top and I've highlighted a random set of numbers that SHOULD evaluate as 0 in the IF statement, and therefore should return a 0 instead of 500.  However, that's not what's happening!

round(10*(4.6+8.45))-10*(4.6+8.45)-0.5

=round(10*(11.05))-10*(11.05)-0.5

=round(110.5)-110.5-0.5

=111-110.5-0.5

=0.5-0.5

=0

 

But again, even though it should evaluate as false for the IF statement, it is returning the true argument instead of 500.  I am guessing either I have my math wrong somewhere or there must be an error caused by some kind of programming choice (like a float value issue), but I have no idea what would be causing that.  Any help would be great.

Labels (1)
0 Likes