47880

Error in float to long conversion [duplicate]

Question:

This question already has an answer here:

<ul><li> <a href="/questions/8911440/c-sharp-float-expression-strange-behavior-when-casting-the-result-float-to-int" dir="ltr" rel="nofollow">C# Float expression: strange behavior when casting the result float to int</a> <span class="question-originals-answer-count"> 7 answers </span> </li> </ul>

I'm guessing this is due to floating point and accuracy but I just wanted to check to ensure that there's nothing else that I'm missing.

I've got a float (stored in info.Amount) of 1007.62 this gets multiplied by 100 and cast to a long. At this point I get a value of 100761.

<img alt="enter image description here" class="b-lazy" data-src="https://i.stack.imgur.com/8EhGA.png" data-original="https://i.stack.imgur.com/8EhGA.png" src="https://etrip.eimg.top/images/2019/05/07/timg.gif" />

Is this just the good old rounding error that we encounter with Doubles being applied? The only way I can think this could happen is if underneath it's actually stored as 1007.6199999999999 (but rounded for display purposes). Then after the multiplication the conversion to a long ignores everything after the decimal point?

Answer1:

You can use <a href="http://msdn.microsoft.com/de-de/library/system.convert.toint64%28v=vs.110%29.aspx" rel="nofollow">Convert.ToInt64</a> instead. This produces the correct value.

LinqPad-Testcode:

float z = 1007.62f; z.Dump(); float x = z *100; x.Dump(); long l = (long) (z*100); l.Dump(); l = Convert.ToInt64(z*100); l.Dump(); l = Convert.ToInt64(x); l.Dump();

Result is:

1007,62 100762 100761 100762 100762

The cast (long) (z*100) uses the CIL-instruction conv.i8 that converts to Int64, pushes Int64 on stack, while Convert.ToInt64 uses Math.Round(double) to convert/cast to Int64.

A special note on conv.i8 is

<blockquote>

Conversion from floating-point numbers to integer values truncates the number toward zero.

</blockquote>

Answer2:

Actually I think this is due to the variable you might declared.

Take the following sample:

double amount = 1007.62; long result = (long) (amount*100);

result absolutely will be 100762 without any fraction or roundness. As a matter of fact, 1007.62 is a double number which should not be stored as float that may cause further problems like your situation.

Recommend

  • When I try and assign the output of the Toggle to an old variable it says: Binding is not convertibl
  • Get the date from each sentence in a variable
  • Why base a javascript namespace as a function?
  • Can we have static variable for initilizing driver in selenium web driver
  • Load external JSON file
  • How to change type datatable column [duplicate]
  • UIViewControllerAnimatedTransitioning: Black screen fragments after rotation change
  • Why context.Wait in StartAsync didn't stop the dialog
  • openTSDB REST API is not storing data
  • Is there way to structure a QueryExpression so that you could dynamically handle a unknown number of
  • How to open a serial port with pyserial?
  • How to Export CSV file from ASP.NET core
  • Use of x`apply` to speed up loops
  • How to populate a Mutation for a different Cassandra table in a trigger
  • Git objects SHA-1 are file contents or file names?
  • Button On Click event not firing
  • build an API for streaming audio/video from GridFS mongodb
  • Gradle refresh removes source folders in build path
  • plot dirac function in matlab
  • Validating a Firebase Key [duplicate]
  • Jekyll - How do I create pages in the root directory?
  • How can i use JQuery fadeTo() in IE 7?
  • android : speech recognition what are the technologies available
  • Google TV VideoView playing YouTube rtsp videos
  • How can I filter an array of dictionaries in 'updateSearchResultsForSearchController' to s
  • Multiple canvases (pages) in Fabric.js
  • view details for exception in vs 2017
  • content must have a ListView whose id attribute is 'android.R.id.list'
  • Java Collections.shuffle() weird behaviour [closed]
  • Reload Page with Javascript after Database changes
  • Find angle of point on circle