[Shapelib] UTF-8 filenames

[Bram de Greve]

Hi all,

I've added the source code to the shapelib source tree so it is easier
to review.  I've made one change to what I've described before:
SASetupUtf8Hooks is only conditionally declared (and defined) on
platforms that will support it.  Currently that's just Windows.  The
macro SHPAPI_UTF8_HOOKS is defined when it is available.

Cheers,
Bram

Bram de Greve wrote:
> Hi all,
>
> I've started to implement some basic support for UTF-8 filenames within
> the context of Thuban.  Before I port it to the shapelib source tree,
> I'd like to hear your comments.
>
> You can find the results here:
> https://wald.intevation.org/plugins/scmsvn/viewcvs.php/branches/WIP-pyshapelib-Unicode/thuban/libraries/shapelib/safileio.c?rev=2801&root=thuban&view=markup
>
> What I've done is added some mechanism to detect if we're compiling on
> windows, and defined SA_UTF8_WINDOWS if so (only within safileio.c).  Of
> course, I've also included <windows.h> and linked to the necessary
> libraries.
>
> Then, based on the existance of SA_UTF8_WINDOWS I defined a new function
> Utf8ToWideChar that will decode the UTF-8 filename to a wide character
> one.  I use it in SAUtf8WFOpen and SAUtf8WRemove to pass the wide
> character filenames to either _wfopen or _wremove.
>
> I've choosen not to use CreateFile and the like, as doing so causes some
> other issues to keep it equivalent to SADOpen.  For example, the access
> string must be parsed and converted to CreateFile flags, while making
> sure we maintain exactly the same behaviour.  I thought it was more
> important to maintain similar behavior than to push the file size limit
> beyond 2 gigs.  File hooks with CreateFile for files > 2 gigs are still
> interesting, but I believe that's matter for another discussion.
>
> Independently of SA_UTF8_WINDOWS, the function SASetupUtf8Hooks() is
> defined then will either hook FOpen to SAUtf8WFOpen or SADFOpen,
> depending on SA_UTF8_WINDOWS (and similar for Remove).  The declaration
> of SASetupUtf8Hooks is always added to shapefil.h, independent of
> SA_UTF8_WINDOWS.
>
> This way, SASetupUtf8Hooks will always be available, but if of course
> blatantly assumes that it will work on all platforms.  By construction,
> it will of course work on the windows platform.  On other platforms,
> SASetupUtf8Hooks is equivalent to SASetupDefaultHooks, and of course the
> big question is if the default hooks can grok UTF-8 filenames in the
> first place.  I know that's the case with the Apple platform as that
> platform uses UTF-8 filenames anyway.  But what about other platforms? 
> What about Linux?  Is there any expert around?
>
> If we cannot guarantee it, we shall either have to make the declaration
> of SASetupUtf8Hooks conditional, which of course will limit its use in
> the portability of shapelib, or drop it entirely.
>
> Cheers,
> Bram
> _______________________________________________
> Shapelib mailing list
> Shapelib at lists.maptools.org
> http://lists.maptools.org/mailman/listinfo/shapelib
>