[MLton] [fwd] bug report, strange floating point behavior

[Matthew Fluet]

>> So, I think the behavior is pretty easily explained by the fact that at 
>> 80-bit precision, the expression (x*y/3.0, x*9.0) doesn't quite turn out 
>> to be (1/3,3), but rather one of them comes out just a few bits less 
>> than the required value, which, after many iterations, tends towards 0.
>> (You can see this more explicitly by changing the cutoff value from 
>> 10000 to 75, where you can see some precision being eroded.)
>>
>> While it is curious that floating-point register allocation is being 
>> affected by the code at the loop termination, it isn't totally surprising.
>>
> 
> Thanks for looking at this. I'm aware of inexact floating point stuff
> causing small errors, but I was very surprised by different results
> after changing a print statement.
> 
> I still don't totally understand how register allocation causes this. As
> I understand it, sometimes MLton uses a 32 bit floating point register
> under some circumstances, but not others. Is that right?

No.  In MLton, the SML type real is (intended to denote) an IEEE 64-bit 
floating-point number.  Now, it happens that on the x86 (or, more 
properly, on the 387 floating point instruction set architecture) there 
is no such thing as a '64-bit fp' register, nor is there a '32-bit fp' 
register.  Rather, the only thing you have is 80-bit fp registers.  As 
long as a computation stays in registers, it happens at 80-bit 
precision.  But, when register pressure forces a spill, then the value 
gets written to memory as a 64-bit fp value.  That causes some rounding, 
so when you reload the value, you get a slightly different value than 
was there before.

So, register allocation determines when the 80-bit values get rounded to 
64-bit values.