Quantcast
Channel: Small Basic
Viewing all articles
Browse latest Browse all 669

Small Basic – Mystery of Round Operation

$
0
0

Computer calculation always has some errors.  I wrote following program to investigate how Math.Round operation works.  Guess what type of errors occur in following case.

TextWindow.WriteLine(Math.Round("2.50000000000000017763568394"))    ' 2
TextWindow.WriteLine(Math.Round( 2.5000000000000001776356839401 ))  ' 2
TextWindow.WriteLine(Math.Round("2.500000000000000200000"))         ' 2
TextWindow.WriteLine(Math.Round("2.5000000000000002000"))           ' 2
TextWindow.WriteLine(Math.Round("2.500000000000000200"))            ' 2
TextWindow.WriteLine(Math.Round("2.5000000000000002"))              ' 2
TextWindow.WriteLine(Math.Round( 2.5000000000000051 ))              ' 2
TextWindow.WriteLine(Math.Round( 2.5000000000000052 ))              ' 3
TextWindow.WriteLine(Math.Round("2.5000000000000001776356839401"))  ' 3
TextWindow.WriteLine(Math.Round("2.50000000000000020"))             ' 3

Actuary, I can't understand yet what happens in the program above...


Viewing all articles
Browse latest Browse all 669

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>