[MLton] [fwd] bug report, strange floating point behavior

[Scott Cruzen]

* Matthew Fluet <fluet at tti-c.org> [070406 09:20]:
> >I originally sent this to the mlton-user list after having problems
> >submitting it, so I thought I'd send it to the main mlton list just to
> >see if anyone besides Matthew has looked at it.
> 
> I'll throw out a couple of more data points.
> 
> I see the "good" behavior on the experimental amd64-linux; this uses the
> C codegen and on that architecture, gcc defaults to using sse
> instructions for floating point math, which performs the calculation at
> 64-bits.
> 
> I also see the "good" behavior on x86-linux with the compiler options:
> -codegen c -cc-opt -msse -ccopt -mfpmath=sse
> which should work on any modern x86 processor.  This also causes gcc to
> use sse instructions for floating point math, hence performs the
> calculation at 64-bits.
> 
> So, I think the behavior is pretty easily explained by the fact that at 
> 80-bit precision, the expression (x*y/3.0, x*9.0) doesn't quite turn out 
> to be (1/3,3), but rather one of them comes out just a few bits less 
> than the required value, which, after many iterations, tends towards 0.
> (You can see this more explicitly by changing the cutoff value from 
> 10000 to 75, where you can see some precision being eroded.)
> 
> While it is curious that floating-point register allocation is being 
> affected by the code at the loop termination, it isn't totally surprising.
> 

Thanks for looking at this. I'm aware of inexact floating point stuff
causing small errors, but I was very surprised by different results
after changing a print statement.

I still don't totally understand how register allocation causes this. As
I understand it, sometimes MLton uses a 32 bit floating point register
under some circumstances, but not others. Is that right?