[MLton-user] How to access large amounts of data through FFI

[Ville Tuulos]

On Fri, Oct 10, 2008 at 4:06 PM, Vesa Karvonen <vesa.a.j.k at gmail.com> wrote:
> On Fri, Oct 10, 2008 at 9:24 PM, Ville Tuulos <tuulos at gmail.com> wrote:
>> How to construct and pass a valid int vector from C to SML through
>> Mlton's FFI, preferably without copying the data first?
>
> Unfortunately, you can't access a mmapped region of memory as an
> ordinary SML vector, because ordinary SML vectors live in MLton's
> garbage collected heap (and have a special header for MLton's runtime
> and may be moved by MLton's GC).

ok. Good to know.

>> The FFI interface seems pretty straighforward but I couldn't find any
>> documentation how arrays and vectors should be initialized based on a
>> C pointer and size of the data. Or, if the int vector is not a good
>> idea, what would be a better way to get access to the memory space?
>
> Like suggested by Florian Weimer, likely the most efficient approach
> is to access the data using MLton.Pointer.

ok. Thanks for the suggestion.

> The attached files contain an example of a way to do it.  The example
> contains the beginnings of a RawVector module that allows one to mmap
> a file and access it similarly to an ordinary vector.  As an example,
> it opens a file ("data"), prints the length of the resulting RawVector
> and then prints all the elements of the RawVector.
>
> For (my) convenience, the example makes (rather minimal) use of an
> extended basis library from the "mltonlib" repository.  You can check
> out the mltonlib repository with the following command:
>
>  svn co svn://mlton.org/mltonlib/trunk mltonlib
>
> You'll then need to modify the Build.sh script to point to the
> location of mltonlib in order to build the example with the script.

Perfect! This is exactly what I need.

Actually when I got the Florian's reply about MLton.Pointer, I started
to consider writing something like RawVector which would support map,
foldl etc. over mmaped memory. Going through your code will be even
better exercise for me.


Thanks!

Ville