benchmarking Poly/ML

[Stephen Weeks]

> I don't know anything about
> the programs you've used 

The series of benchmarks (holes, fov, intercyl, ...) was one ML program run on a
variety of different inputs.  The program was the translation of an OCAML
program which was a raytracer that was was the winning entry in this year's ICFP
programming contest.

> but if they are affected by whether an
> implementation provides unsafe Array.sub and update then they are
> probably not typical ML programs.

Provided that the implementation does not provide an undue penalty for safe
array operations, that should not be a significant factor.  For example, the
cost in MLton for safe array operations on this benchmark is less than 10%.  I
guess that for Poly/ML the cost was largely irrelevant as well, given the
overhead for floating point and arrays.

> Were they translated from C or some
> other language?

Translated from OCAML.  While it isn't the best example of SML, I think it's not 
too bad.  There is lots of symbolic processing, recursion, pattern matching,
short-lived data structures, ...

> My feeling is that programs, such as Isabelle, written
> from scratch in ML where ML is the language of choice, behave
> significantly differently to more "conventional" programs.  They put
> much more emphasis on the garbage collector, closure optimisations and
> redundant tuple removal.  The sort of benchmarks I've seen in the past
> have tended to have tight inner loops which test the efficiency of
> low-level code generation and the removal of constant expressions from
> loops.  

I agree that some benchmarks (e.g. mandelbrot, matrix-multiply) do this.
However, this raytracing benchmark is not synthetic and there isn't one tight
inner loop -- the code coverage is actually fairly good, which is why I run on a
variety of different inputs.

> This kind of optimisation is crucial for the performance of
> languages such as C and Fortran but is of less concern to the ML
> programmer.  Looking at the behaviour of ML compilers on such programs
> can provide some useful insights but my feeling is that it is more
> important to focus on the immediate needs of the ML programmer.

We disagree on what kinds of programs should be (and are being) written by ML
programmers.  There was a long discussion on comp.lang.functional on the merits
of the raytracer as the contest problem.  I'll just add my two cents by saying
that I think that a raytracer is a good example of a program that ML provides
many advantages as an implementation language, provided that the compiler
produces good enough code.

> For
> example, although Poly/ML provides floating point operations these are
> all implemented in terms of calls to the run-time system.  There has
> simply never been a demand for them to be incorporated into the
> code-generator.  Similarly, assignment and array update involve calls to
> the RTS, at least in PolyML 3.3X.  

Thanks for the details on the implementation.  This explains much of the
slowdown on this benchmark (and several others).