[Proj] Virtual File Access

[Frank Warmerdam]

On Fri, Jul 26, 2013 at 6:00 AM, Greg Troxel <gdt at ir.bbn.com> wrote:

>
> Frank Warmerdam <warmerdam at pobox.com> writes:
>
> Perhaps I am totally misunderstanding what you are doing.   Is this
> about letting IO done on behalf of a user to read and write the user's
> data (not grid shift files, which are part of the library) to come from
> different places?  Or is it about grid shift files, but the goal is
> because the user has them in RAM rather than on the filesystem?   I had
> the impression that this was about a way to bundle the standard grid
> shift/etc. files into the library so that one could have just a single
> library file and not the ancilliary files, because managing mulitple
> files was somehow too hard.
>

Greg,

It is about the grid shift and init files, not user input.  PROJ.4 the
library doesn't accept user input directly.



> > Quite possibly there are many things broken about the environment I work
> > in, but running things on windows is certainly not one of them.
>
> I'm glad to hear that :-).
>
> So what motivates you to add this code, rather than just installing the
> files in the right place?  What is the essence of the problem?
>

The details involve aspects of security and methodology that I am
discouraged from discussing in public without explicit permission by my
employer.

Suffice it to say that there are other contexts in which deployments of
PROJ.4 would have been significantly easier and problem free if I could
just have the data files delivered directly in the .so file, and no need
for special installation rules, etc.

> But the point I'd like to stress is that I believe it is generally a good
> practice for low level libraries to provide a way to hook IO so that
> applications can do a variety of things around file access.  Libraries
like
> libtiff, GDAL, shapelib and libpng have done this for a long time.  I'm
> extending it to PROJ.4.

 I find that notion unusual and surprising.  I was completely unaware of
> those libraries adding what feels like a custom VFS layer.  I looked at
>   http://www.libtiff.org/libtiff.html
> and found no mention of it.
>

If you look at the TIFFClientOpen() call, it includes mechanisms to pass in
what is essentially a virtual file access api.
 http://www.libtiff.org/man/TIFFOpen.3t.html

To some extent my introduction of similar capabilities into GDAL, Shapelib,
and now PROJ.4 mirrors what I saw in libtiff, libpng and libjpeg.


> > I'm not clear exactly what information you are looking for.  Normal
> > practice with applications installers using PROJ.4 would be to install
> the
> > files somewhere in the file system, and to call pj_set_searchpath()  to
> > look in the directory in which things were installed from their
> application
> > initialization.  The pj_set_searchpath() and pj_set_finder() were
> intended
> > to make finding the data files tractable in cases where applications
> didn't
> > mind writing them to disk.
>
> Where I'm coming from is that I install proj via pkgsrc (and am the
> package maintainer).  So it's "pkg_add proj-4.8.0.tar.gz" and then the
> library is in /usr/pkg/lib, and the files in /usr/pkg/share/proj (plus
> programs in /usr/pkg/bin, man pages...).  I expect it's the same for all
> other packaging systems, modulo prefix.  The path to the files is set
> via PROJ_LIB at build time to $(pkgdatadir).  So I view it as a bug to
> have to set search paths.  There is a single correct place for the grid
> files, and they are part of the package (or a depending package if one
> wants to split things).
>

As a packager installer files is no problem for you.  That's great. There
are other deployment scenarios.


> Also, I'd suggest that you consider putting this facility on a branch
> instead of in a release until there's both a useful extension
> (presumably reading files that were crunched into the library) and
> broader consensus that it's a good addition (on the benefit/complexity
> tradeoff).
>
>
I don't recall a concern from anyone else.  If you feel this is a serious
issue you could request I write this up as an RFC and put it to a vote on
the metacrs project steering committee.  I'm a bit surprised at the concern
since in normal situations the introduction of this file indirection won't
affect PROJ.4 library users.

Best regards,
-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam,
warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | Geospatial Software Developer
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.maptools.org/pipermail/proj/attachments/20130726/50b47c29/attachment.htm