[MLton-user] SVN r6941 MLton/MinGW32 and FFI

[Matthew Fluet]

On Tue, 11 Nov 2008, Wesley W. Terpstra wrote:
> On Tue, Nov 11, 2008 at 2:36 PM, Matthew Fluet <fluet at tti-c.org> wrote:
>> It is one of the examples from <src>/doc/examples/ffi.
>> These examples really should work 'out of the box'.
>
> I didn't know about these files. I've started updating them to use
> symbol scopes. I've noticed a couple of things:
>
> 1. ffi-import.c uses GC_getArrayLength from "platform.h"! Is this
> really supposed to be user-accessible? If yes, should it be a private
> symbol only accessible within the DSO, a public symbol (in which case
> it needs a libname_ prefix), or both?

It is a function defined in libmlton.a, so I would guess that it should be 
a private symbol, only accessible within the DSO.  (Just like all the 
other runtime functions.)  For libraries, I think the current setup is 
good -- the only visible symbols are those explicitly 'public' exported 
(and the auto-generated 'libname_open' and 'libname_close' functions).

With regards to whether users should actually use GC_getArrayLength, we 
don't really advertise it, but neither is it a particularly subtle 
operation.  And, it is genuinely useful (though, getting the length and 
passing it through the FFI along with the array pointer wouldn't been 
terribly difficult either).  I don't think there is any way to really 
distinguish between visible functions in libmlton.a that are available for 
use in mlton generated .c and .s files, but not available for user .c 
code.

> 2. iimport.c uses dlopen/dlclose/dlsym. platform/mingw.c defines
> these... which is a problem since the runtime is statically linked,
> but libdl is dynamically linked. Perhaps these should be public
> libname_ symbols, private MLton_dlopen/close/sym, or both?

To clarify: the issue is that on *NIX platforms (including MacOSX), 
dl{open,close,sym,error,...} are pulled from dynamically linked shared 
libraries.  That is, they are imported from another DSO.  Thus, for these 
platforms, the 'external' attribute should be used on the _import.

MinGW doesn't provide dl{open,close,...} in a (standard) dynamically 
linked shared library.  Vesa added rudimentary emulation in 
<src>/runtime/platform/mingw.c (r4862, 20061127); thus dl{open,close,...} 
are pulled from libmlton.a.  That is, they are imported from the same DSO. 
Thus, for this platform, the 'private' attribute should be used on the 
_import.

Personally, I think that MLton should not be attempting to provide 
non-essential functionality that is missing on a platform.  My suspicion 
is that 99% of all MLton compiled SML programs will not need 
dl{open,close,...} support.  Of the remaining 1%, most will probably be on 
*NIX platforms, where dl{open,close,...} are provided by standard system 
libraries.  (You can see from the <src>/doc/examples/ffi/Makefile iimport 
target that some *NIX platforms require extra link options to find the 
library; MLton doesn't default to including those link options, because 
they are not essential functionality (e.g., required by some of the Basis 
Library implementation).)  That leaves a very small number of programs 
that are on Windows (via MinGW, since, apparently, Cygwin provides 
dl{open,close,...}).  If they are MinGW exclusive, they can use the Win32 
specific {Load,Free}Library/GetProcAddress functions.

The remaining (infinitesmal) programs are ones that require dlopen-ing 
libraries and are trying to stay source compatible for *NIX and 
Windows/MinGW. For those situations, there are more robust solutions than 
MLton emulating dl{open,close,...} for MinGW.  One is to use MLB 
path-variables to provide a compatibility module:

dynlink.mlb:
local
   dyn-link.sig
   platform/dyn-link.$(TARGET_ARCH)-$(TARGET_OS).mlb
in
   signature DYN_LINK
   structure DynLink
end

platform/dynlink.x86-linux.mlb:
dynlink.dlfcn.mlb

platform/dyn-link.dlfcn.mlb:
dyn-link.dlfnc.sml

platform/dyn-link.dlfcn.sml:
structure DynLink :> DYN_LINK =
struct
       type hndl = MLton.Pointer.t
       type mode = Word32.word

       val dlopen =
          _import "dlopen" external : string * mode -> hndl;
       ...
end

platform/dynlink.x86-mingw.mlb:
dynlink.win32.mlb

platform/dynlink.win32.mlb:
dynlink.win32.sml

platform/dynlink.win32.sml:
structure DynLink :> DYN_LINK =
struct
       type hndl = MLton.Pointer.t

       val loadLibrary  =
          _import "LoadLibrary" stdcall external : string -> hndl;
       ...
end

Alternatively, use the GNU libtool libtldl library, whose express purpose 
is to provide a uniform interface to various system specific dynamic 
library mechanisms.  Assuming that libtldl is available as a dynamically 
linked shared library (i.e., a different DSO than the program) on all the 
platforms of interest, then its functions may all be _import-ed with 
'external' scope.

> What would make most sense to me:
> * MLton_{dlopen,dlclose,dlsym,getArrayLength} are private symbols
> available within the DSO
> * GC_getArrayLength is also a private alias for compatibility
> * libname_getArrayLength are public symbols available to users of a
> DSO. Perhaps the dl* symbols too.

I (respectfully) disagree on all counts ;-)

- MLton shouldn't be providing dl{open,close,...} in any form.
- GC_getArrayLength isn't advertised, so there is no need to 'bless' it
   with a MLton_getArrayLength name.  (We should eliminate its use from the
   <src>/doc/examples/ffi examples.)
- No functions should be public symbols available to users of a DSO other
   than those specified by the user.  If a function needs the length of an
   array, arrange the interface to pass the length along with the array
   pointer.

>>> See http://mlton.org/LibrarySupport , suggestions to improve the
>>> documentation are welcome.
>>
>> If the symbol scopes can't be given sensible defaults for executables that
>> match the behavior of previous versions, then they really need to be
>> documented as well at:
>>  http://mlton.org/ForeignFunctionInterfaceSyntax
>
> I agree. However, that page corresponds to the release version of
> MLton, which doesn't have symbol scopes.

It might be a good idea to clone the page (say, as 
NextRelease/ForeignFunctionInterfaceSyntax) to make current documentation 
available somewhere.  Also, it makes it easier to switch in when the time 
comes.