You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When converting to float64 using Float64(), the value can differ from the input string while shopspring/decimal seems to produce the correct result. for example
Appreciate that the value cannot fit into a float64 as per ericOk but curious why the shopspring value matches?
The text was updated successfully, but these errors were encountered:
This is because shopspring converts their decimal to a big.Rat, then converts the Rat to a float64. My library converts as-is using float64.
The upside to shopspring’s method is that it won’t round worse than the decimal’s current precision during the conversion. The downside is that converting to a rational number is expensive: you end up with huge numerators and denominators. Plus, you have to rescale the decimal to create the rational number.
The upside to my method is it’s fast and allocation-free (for small floats, that is). The downside is you occasionally have rounding issues.
There might be a better way of performing the conversion (more bit twiddling instead of float division/multiplication), but I haven’t looked too closely at it. I’m not entirely sure why people are using the Float64 method—it seems to defeat the purpose of using a decimal library.
When converting to float64 using
Float64()
, the value can differ from the input string whileshopspring/decimal
seems to produce the correct result. for exampleAppreciate that the value cannot fit into a float64 as per
ericOk
but curious why the shopspring value matches?The text was updated successfully, but these errors were encountered: