[MLton-user] IEEEReal rounding not working

[Matthew Fluet]

On Thu, 26 Jul 2007, Sean McLaughlin wrote:
> The attached program demonstrates a bug in the IEEEReal rounding mode
> setting:
>
> As you can see, the rounding modes are not having any effect.

Actually, the rounding modes are having a perfectly good effect.  The 
issue is that MLton interprets the primitive Real<N>.* operations as 
functional and, hence, amenable to common-subexpression-elimination. So, 
the subsequent "1.0 / 10.0" expressions are eliminated, and your program 
is executed as:

val mtenth_lo = (down();1.0 / 10.0)
val mtenth_hi = (up();mtenth_lo)
val mtenth_near = (near();mtenth_lo)
val mtenth_zero = (zero();mtenth_lo)

It is pretty easy to change the Real<N>.* operations to be non-functional 
(and, rather, that they depend on the current state), though I need to 
think about which ones precisely need to be switched.  It'll also have a 
performance impact, due to not eliminating (seemingly) redundant 
compuations.

On the x86, there is the additional screwy bit that you are computing at 
80-bits, and hence it is the rounding mode in effect when the value is 
spilled to memory that determines the 64-bit representation.  Now, you're 
saved by the fact that the IEEEReal.setRoundingMode function is 
implemented as a FFI call, so all real values are spilled to memory to 
abide by the C calling convention.  So, I don't think you'll end up with 
any artifacts due to that; but, it may very well be the case that a 
sequence of floating-point computations carried out at 80-bits under 
rounding mode m and then converted to 64-bits via rounding mode m could 
yield a different answer than performing the sequence of floating-point 
computations at 64-bits under rounding mode m.  But, I don't think 
MLton/SML will be any different than gcc/C in this regard.

BTW, would you expect the following output:

3FB9999999999999
3FB999999999999A
3FB999999999999A
3FB9999999999999

which is what I get by making the Real_div primitive depend on state, 
rather than functional.