[MLton] Bug in IntInf implementation

[Matthew Fluet]

On Fri, 7 Dec 2007, Wesley W. Terpstra wrote:
> The version: (at least) svn/HEAD
> boom.sml:
>> val x : IntInf.int = 1
>> val y = IntInf.<< (x, 0w838860800) (* 100MB *)
>> val () = print "unreached\n"
>
> The bug:
> initIntInfRes in runtime/gc/int-inf.c points a GMP limb buffer at the heap. 
> It has no reason to believe the current heap can contain the result. Normally 
> the mpz_* functions will gmp_realloc memory when needed. I have no idea what 
> will happen when calling this on a pointer in the ML heap. It seems to hang.

Incorrect.  initIntInfRes has every reason to believe that the current 
heap can contain the result of the operation.  (Indeed, there are numerous 
'assert's to that effect in the code.)  All of the IntInf primitives (both 
internally as manipulated by the compiler and externally as implemented in 
runtime/gc/int-inf.c) take a size_t argument which, at runtime, will be 
the maximum size of the resulting IntInf.  The limit check insertion pass 
integrates these arguments into the limit checks to ensure that there is 
always sufficient space in the heap to contain the result.

In fact, the program does not hang, but is simply computing the result 
rather slowly.  See <src>/basis-library/integer/int-inf.sml.  IntInf 
shifts are limited to 128 bits at a time.  I know I'm the one who wrote 
that code, and I have a very vague recollection that there was a reason, 
but I can't recall it now.  One issue is that if one compiles with 
'-default-type word64', then the shift argument of IntInf.~>> and 
IntInf.<< (as exported by the Basis Library) is a Word64.word, but on a 
32-bit platform, the GMP primitives will only accept a 32-bit word as the 
shift amount.  And, in any case, we've made the IntInf shift primitives 
require a 32-bit shift, so one needs a cap.  But, obviously, it could be 
much higher than 128.