[MLton] Memory problems with latest CVS?

[Matthew Fluet]

> >             xmlShrink1 starting
> > IntInf_mul (0x77359401, 0x276f35b9, 16)
> > fill (0x77359401, 0xbfffd350, 0xbfffd328)
> > fill (0x276f35b9, 0xbfffd340, 0xbfffd320)
>
> It would be interesting to know if this call to IntInf_mul returned.

It does not.  Here's the backtrace with a runtime compiled without
-fomit-frame-pointer:

#0  0xb74ee780 in _int_realloc () from /lib/tls/libc.so.6
#1  0xb74ed136 in realloc () from /lib/tls/libc.so.6
#2  0xb74f0303 in realloc_hook_ini () from /lib/tls/libc.so.6
#3  0xb74ed06c in realloc () from /lib/tls/libc.so.6
#4  0x08e24c3b in __gmp_default_reallocate ()
#5  0x08e1cdd7 in __gmpz_realloc ()
#6  0x08e1cc18 in __gmpz_mul ()
#7  0x08e0dd8a in binary (lhs=0x77359401 "", rhs=0x2808559f "", bytes=16,
    binop=0x8e1ca00 <__gmpz_mul>) at basis/IntInf.c:200
#8  0x08e0de50 in IntInf_mul (lhs=0x77359401 "", rhs=0x2808559f "", bytes=16)
    at basis/IntInf.c:222
#9  0x08674f0c in Chunk197 () at mlton-stubs.197.c:9220

I added the following to runtime/basis/IntInf.c in the fill function:

        if (DEBUG_INT_INF) {
                char buf[2048];
                char *str;
                str = mpz_get_str(buf, 10, res);
                assert(str == buf);
                fprintf (stderr, "fill (arg == %s)\n", buf);
        }

So, this prints out the base 10 representation of every bignum that we
import from ML into an __mpz_struct.  Now the last call to IntInf_mull
looks like:

IntInf_mul (0x77359401, 0x2808559f, 16)
fill (0x77359401, 0xbfffc9d0, 0xbfffc9a8)
fill (arg == 1000000000)
fill (0x2808559f, 0xbfffc9c0, 0xbfffc9a0)
fill (arg == 335817423)

I'm pretty certain that the bytes is off -- 16 bytes accounts for the
counter, size, header, and sign words, and doesn't leave any room for the
actual bignum; certainly not enough for the product of those two numbers.
That would explain why __gmpz_realloc is being called -- because we didn't
provide enough space for the bignum.  We should be concerned about
gmpz_realloc's right?  We should always be providing enough space the
results up front, so gmp shouldn't ever need to reallocate.

Henry, can you look at basis-library/integer/int-inf.sml?  Is this code
for big integer multiplication right:

      (*
       * bigInt multiplication.
       *)
      local
         val carry: Word.word ref = ref 0w0
      in
         fun bigMul (lhs: bigInt, rhs: bigInt): bigInt =
            let
               val res =
                  if areSmall (lhs, rhs)
                     then let
                             val lhsv = stripTag lhs
                             val rhs0 = zeroTag rhs
                             val ans0 = Prim.smallMul (lhsv, rhs0, carry)
                          in
                             if (! carry) = Word.~>> (ans0, 0w31)
                                then SOME (Prim.fromWord (incTag ans0))
                             else NONE
                          end
                  else NONE
            in
               case res of
                  NONE =>
                     dontInline
                     (fn () =>
                      Prim.* (lhs, rhs, reserve (size lhs +? size rhs, 0)))
                | SOME i => i
            end
      end

It seems to me that if we are in the situation described above -- two
"small" big-nums, but whose product is a "big" big-num, then we'll try the
Prim.smallMul, fail because of the carry and so try Prim.*, but the sizes
of "small" big-nums are 0, so we'll reserve only enough for the "big"
big-num header.