[mapserver-users] XML mapfile?
Vic Kelson
vic@wittmanhydro.com
22 May 2002 15:44:30 -0500
--=-SjVFyMK1MIglgAY599KW
Content-Type: text/plain
Content-Transfer-Encoding: 7bit
Hi All!
WARNING -- I'm a now-former Lurker with a longwinded set of ideas and
commentary. If you've thought of this before and it can't be done,
please tell me why... Here goes! THANKS for mapserv!
I've read the comments about XML mapfiles, with relish, and I have some
reasons why I think it must be done (not should, must).
1) XML is not _that_ slow, if you use a good parser. Furthermore, you
can (and should) serialize the DOM once and then re-load the serialized
object, as someone mentioned earlier.
2) It's easy to "manufacture" XML programmatically by user programs
that might automate the production of sites (because there are dozens of
existing libraries that facilitate creation of XML from data).
3) It sure would be nice to have a "map on demand" server (perhaps a
web service using XML-RPC or SOAP?) that allows you to fire a
well-formed XML "mapfile" to the server and receive an image or HTML
page in response.
4) Although (4) can be done in mapscript, why bother if an XML mapfile
can do it? You can document what's on the server (or have the web
service provide XML for the "default" DOM tree for mark-up; woooo,
neato) and folks can use ANYTHING to edit / generate the XML. I can
imagine a client-side app in, e.g., python, that gets the XML, marks up
the DOM tree, makes a server request to retrieve the rendered maps, then
draws additional (locally-stored) data on top. One could even make a
client-side drawing library that does coordinate conversions and the
like (hey -- how about mapscript?).
5) EVERYBODY knows how to write and validate XML already. This one
feature makes me want to switch right now. The only 'lint' we have for
mapfiles is mapserv itself. It's a PAIN to edit the mapfile and hit the
browser to check for errors (for one thing, you need a separate
development web server, and that introduces hassles with user
permissions). If we have an XML mapfile and a proper DTD, a validating
parser will automatically find poorly-formed XML and missing delimiters
and flag them (imagine not having to find the missing 'END' in a
2000-line mapfile). Furthermore, 'maplint' could take big-time
advantage of the XML for system management. I envision a python script
that loads the XML mapfile, then traverses the LAYER, CLASS, and LABEL
definitions and reports whether there are missing files, attributes in
the CLASSITEM or LABELITEM specifications that aren't in the DBF files,
etc...
5a) Wow! 'maplint' could even load the XML mapfile for, e.g. version
4.0 mapserv and convert it to a version 4.1 XML mapfile!
5b) Inheritance. You can define "new" versions by subclassing the old
versions. If it's properly implemented in mapserv, a future mapserv 4.0
could read, e.g. 4.1 XML mapfiles and SKIP over the DOM nodes for the
4.1 features that it doesn't support... Or 4.1 could "skip and warn"
(instead of failing), when an obsoleted 4.0 feature is encountered.
It's easy, because the code that processes the information in the DOM
tree is independent of the code that reads the DOM tree. Web browsers
do it, mapserv can too -- the XML reader uses the DTD to read and
validate the DOM tree; what the code does with that DOM tree is up to
the code... Fatal mapserv errors displayed on client terminals due to
typos in infrequently used (and thus poorly tested) portions of mapfiles
are bad medicine; XML can help to avoid them.
5c) Most editors are XML-aware and do stuff like syntax highlighting,
auto-inserting tags, and finding missing tags, angle brackets and the
like. Who wants to maintain a "mapfile-mode" for emacs or vim?
6) How about using the XML mapfile to populate a DOM tree, then
programmatically (e.g. mapscript python or perl) build an image in a
non-web-based tool. WOW. A "poor man's ArcView" is only a few hundred
lines of PyQT or PyGnome/PyGTK code away (or a thousand or so lines of
line noise for Perl/QT or Perl/GTK programmers. Just kidding.
Maybe.). The program could traverse the DOM tree that specifies the map
structure and construct the whole GUI on the fly, creating GUI widgets
that allow layers to be turned on/off, etc. And the same DOM tree makes
the pictures! Save the DOM tree to an XML file, and *PRESTO* your view
is on the web!
6a) Incidentally, one could document the options available for mapserv
objects using another XML file (I'm not an XSL expert, but maybe that's
the way?). Then the nifty mini-ArcView could build all the layer option
dialog boxes on the fly, radio/checkbox arrays, label options, and the
whole garbanzo...
7) At some level, with appropriate editors, XML and DTD's are
self-documenting. This is a big winner, since one presumes that new
functionality is coming all the time. Properly designed, nearly all of
the (6) and (6a) code could be independent of mapserv revisions; unless
a new major option needs to be added, the GUI code should require few
revisions to keep up with mapserv.
Although XML is not required to do any of these things, XML facilitates
the incorporation of mapserver or mapscript or any number of future
children of mapserv/mapscript into other software systems, with
consistent results. Pardon me, but I'm thinking BIG -- mapserv is a
powerful, flexible, extensible, open, facility for rendering all kinds
of geographic information, FAST. It can be a component for larger
software systems, with "not-too-much" effort. If all you're doing is
blowing pictures into web browsers, forget XML. But for everything
else, one doesn't want to have to write and maintain (a) a mapfile
parser/generator of one's own, or (b) a mapscript application for every
single part of my application.
I've done some fiddling with "XML mapfiles". I designed that work as an
input vehicle for python/mapscript. It is a simple but tedious task to
build a DTD for mapfiles, but I have no control over where mapserv goes,
so I abandoned the effort for now. If you have an XML mapfile and
python with mapscript, it should be possible (perhaps even simple but
tedious) to implement all of mapserv's functionality in an OO code that
is easily extended (by inheritance) for new applications, either on the
web or standalone. The resulting classes could be used as a starting
point for all the stuff I outlined above. I'm way too busy to write it
all myself, but I have a general design in place. This can probably all
be done outside of the mapserv development effort, but following mapserv
development would be a pain. It would be nicer to do it all together,
and make a single coherent component infrastructure.
When I imagine a "mapscriptified" OpenEV/ArcView clone with XML mapfiles
as a standard "view" file format, mapserv using the same XML files to
render web pages, python with "XML mapfile" objects plugged into Zope,
and some/most/all of the data in PostgreSQL/PostGIS, well, I could just
cry... One could even make python mapserv COM objects that plug into
Office, but why go there...
I'll shut up now. THANKS!
Vic
PS: Count me in on the conference if there is one...
On Wed, 2002-05-22 at 11:04, Puneet Kishor wrote:
I have wondered about XML myself, and thought, at times, that it would
be cool to have XML. Now, reading Daniel's reasoning, I must say what he
says makes sense... besides the cool factor, we have to question what
XML will do for us. XML facilitates inter-application communication...
other than that it is a pain in the ass. If Mapserver is the only
program gonna be reading the mapfile, we can make it whatever we want it
to, and what it is now is good enough.
However, in the spirit of discussion, there is one area where XML might
be helpful. Most code editors (Homesite, jEdit, Pepper) nowadays can
parse XML to create an XML tree. This can make editing a complex Mapfile
easier as one can check for node closures, correctness and completeness.
Other than that, I don't see any great advantage to XML.
pk/
On Wednesday, May 22, 2002, at 07:19 AM, Daniel Morissette wrote:
> C F wrote:
>>
>> It seems that this and many other problems I encounter could be solved
>> by
>> making the .map file format XML based. I searched the archives and
>> have
>> seen quite a bit of discussion on it but I didn't a conclusion. The
>> benefit
>> for me being that I could download and easily parse and retrieve any
>> information I needed using XPath. Is this being considered at all?
>> If so,
>> I'd be happy to contribute to the discussion and development
>> (wherever I
>> could, but I don't know the C language yet) if that would help. A
>> combination of an XML mapfile, XSLT templates and XPath extremely
>> powerful
>> compared the current template files.
>
>
> I think that Steve has been considering a switch to XML eventually, but
> I don't think there is any formal plan.
>
> I agree that XML might give more flexibility to the user by allowing him
> to read the mapfile from external tools, but it would be at the cost of
> performance (i.e. speed) when parsing the mapfile for every request.
> Combine this loss of performance with the fact that XML would be harder
> to edit manually than the current format, and you know where I stand
> personally.
>
> I'm not saying that if we were redesigning from scratch I wouldn't
> consider XML (we use it a lot in other apps), but since we've already
> got a very good (fast and simple) mapfile format then I won't be the one
> initiating a switch to XML mapfiles.
>
> Note that MapScript already gives you access to almost every piece of
> information from the mapfile, so when the CGI cannot do what you need
> then the logic path is to switch to MapScript, and then sky is the
> limit.
>
> My 0.02$
> --
> ------------------------------------------------------------
> Daniel Morissette morissette@dmsolutions.ca
> DM Solutions Group http://www.dmsolutions.ca/
> ------------------------------------------------------------
> Don't put for tomorrow what you can do today, because if
> you enjoy it today you can do it again tomorrow.
--=-SjVFyMK1MIglgAY599KW
Content-Type: text/html; charset=utf-8
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 TRANSITIONAL//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; CHARSET=UTF-8">
<META NAME="GENERATOR" CONTENT="GtkHTML/1.0.3">
</HEAD>
<BODY>
Hi All!
<BR>
<BR>
WARNING -- I'm a now-former Lurker with a longwinded set of ideas and commentary. If you've thought of this before and it can't be done, please tell me why... Here goes! THANKS for mapserv!
<BR>
<BR>
I've read the comments about XML mapfiles, with relish, and I have some reasons why I think it must be done (not should, must).
<BR>
<BR>
1) XML is not _that_ slow, if you use a good parser. Furthermore, you can (and should) serialize the DOM once and then re-load the serialized object, as someone mentioned earlier.
<BR>
2) It's easy to "manufacture" XML programmatically by user programs that might automate the production of sites (because there are dozens of existing libraries that facilitate creation of XML from data).
<BR>
3) It sure would be nice to have a "map on demand" server (perhaps a web service using XML-RPC or SOAP?) that allows you to fire a well-formed XML "mapfile" to the server and receive an image or HTML page in response.
<BR>
4) Although (4) can be done in mapscript, why bother if an XML mapfile can do it? You can document what's on the server (or have the web service provide XML for the "default" DOM tree for mark-up; woooo, neato) and folks can use ANYTHING to edit / generate the XML. I can imagine a client-side app in, e.g., python, that gets the XML, marks up the DOM tree, makes a server request to retrieve the rendered maps, then draws additional (locally-stored) data on top. One could even make a client-side drawing library that does coordinate conversions and the like (hey -- how about mapscript?).
<BR>
5) EVERYBODY knows how to write and validate XML already. This one feature makes me want to switch right now. The only 'lint' we have for mapfiles is mapserv itself. It's a PAIN to edit the mapfile and hit the browser to check for errors (for one thing, you need a separate development web server, and that introduces hassles with user permissions). If we have an XML mapfile and a proper DTD, a validating parser will automatically find poorly-formed XML and missing delimiters and flag them (imagine not having to find the missing 'END' in a 2000-line mapfile). Furthermore, 'maplint' could take big-time advantage of the XML for system management. I envision a python script that loads the XML mapfile, then traverses the LAYER, CLASS, and LABEL definitions and reports whether there are missing files, attributes in the CLASSITEM or LABELITEM specifications that aren't in the DBF files, etc...
<BR>
5a) Wow! 'maplint' could even load the XML mapfile for, e.g. version 4.0 mapserv and convert it to a version 4.1 XML mapfile!
<BR>
5b) Inheritance. You can define "new" versions by subclassing the old versions. If it's properly implemented in mapserv, a future mapserv 4.0 could read, e.g. 4.1 XML mapfiles and SKIP over the DOM nodes for the 4.1 features that it doesn't support... Or 4.1 could "skip and warn" (instead of failing), when an obsoleted 4.0 feature is encountered. It's easy, because the code that processes the information in the DOM tree is independent of the code that reads the DOM tree. Web browsers do it, mapserv can too -- the XML reader uses the DTD to read and validate the DOM tree; what the code does with that DOM tree is up to the code... Fatal mapserv errors displayed on client terminals due to typos in infrequently used (and thus poorly tested) portions of mapfiles are bad medicine; XML can help to avoid them.
<BR>
5c) Most editors are XML-aware and do stuff like syntax highlighting, auto-inserting tags, and finding missing tags, angle brackets and the like. Who wants to maintain a "mapfile-mode" for emacs or vim?
<BR>
6) How about using the XML mapfile to populate a DOM tree, then programmatically (e.g. mapscript python or perl) build an image in a non-web-based tool. WOW. A "poor man's ArcView" is only a few hundred lines of PyQT or PyGnome/PyGTK code away (or a thousand or so lines of line noise for Perl/QT or Perl/GTK programmers. Just kidding. Maybe.). The program could traverse the DOM tree that specifies the map structure and construct the whole GUI on the fly, creating GUI widgets that allow layers to be turned on/off, etc. And the same DOM tree makes the pictures! Save the DOM tree to an XML file, and *PRESTO* your view is on the web!
<BR>
6a) Incidentally, one could document the options available for mapserv objects using another XML file (I'm not an XSL expert, but maybe that's the way?). Then the nifty mini-ArcView could build all the layer option dialog boxes on the fly, radio/checkbox arrays, label options, and the whole garbanzo...
<BR>
7) At some level, with appropriate editors, XML and DTD's are self-documenting. This is a big winner, since one presumes that new functionality is coming all the time. Properly designed, nearly all of the (6) and (6a) code could be independent of mapserv revisions; unless a new major option needs to be added, the GUI code should require few revisions to keep up with mapserv.
<BR>
<BR>
Although XML is not required to do any of these things, XML facilitates the incorporation of mapserver or mapscript or any number of future children of mapserv/mapscript into other software systems, with consistent results. Pardon me, but I'm thinking BIG -- mapserv is a powerful, flexible, extensible, open, facility for rendering all kinds of geographic information, FAST. It can be a component for larger software systems, with "not-too-much" effort. If all you're doing is blowing pictures into web browsers, forget XML. But for everything else, one doesn't want to have to write and maintain (a) a mapfile parser/generator of one's own, or (b) a mapscript application for every single part of my application.
<BR>
<BR>
I've done some fiddling with "XML mapfiles". I designed that work as an input vehicle for python/mapscript. It is a simple but tedious task to build a DTD for mapfiles, but I have no control over where mapserv goes, so I abandoned the effort for now. If you have an XML mapfile and python with mapscript, it should be possible (perhaps even simple but tedious) to implement all of mapserv's functionality in an OO code that is easily extended (by inheritance) for new applications, either on the web or standalone. The resulting classes could be used as a starting point for all the stuff I outlined above. I'm way too busy to write it all myself, but I have a general design in place. This can probably all be done outside of the mapserv development effort, but following mapserv development would be a pain. It would be nicer to do it all together, and make a single coherent component infrastructure.
<BR>
<BR>
When I imagine a "mapscriptified" OpenEV/ArcView clone with XML mapfiles as a standard "view" file format, mapserv using the same XML files to render web pages, python with "XML mapfile" objects plugged into Zope, and some/most/all of the data in PostgreSQL/PostGIS, well, I could just cry... One could even make python mapserv COM objects that plug into Office, but why go there...
<BR>
<BR>
I'll shut up now. THANKS!
<BR>
Vic
<BR>
<BR>
PS: Count me in on the conference if there is one...
<BR>
<BR>
On Wed, 2002-05-22 at 11:04, Puneet Kishor wrote:
<BLOCKQUOTE>
<PRE><FONT COLOR="#737373"><FONT SIZE="3"><I>I have wondered about XML myself, and thought, at times, that it would </FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>be cool to have XML. Now, reading Daniel's reasoning, I must say what he </FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>says makes sense... besides the cool factor, we have to question what </FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>XML will do for us. XML facilitates inter-application communication... </FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>other than that it is a pain in the ass. If Mapserver is the only </FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>program gonna be reading the mapfile, we can make it whatever we want it </FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>to, and what it is now is good enough.</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I></FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>However, in the spirit of discussion, there is one area where XML might </FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>be helpful. Most code editors (Homesite, jEdit, Pepper) nowadays can </FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>parse XML to create an XML tree. This can make editing a complex Mapfile </FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>easier as one can check for node closures, correctness and completeness.</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I></FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>Other than that, I don't see any great advantage to XML.</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I></FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>pk/</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I></FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I></FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I></FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I></FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>On Wednesday, May 22, 2002, at 07:19 AM, Daniel Morissette wrote:</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I></FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>> C F wrote:</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>>></FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>>> It seems that this and many other problems I encounter could be solved </FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>>> by</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>>> making the .map file format XML based. I searched the archives and </FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>>> have</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>>> seen quite a bit of discussion on it but I didn't a conclusion. The </FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>>> benefit</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>>> for me being that I could download and easily parse and retrieve any</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>>> information I needed using XPath. Is this being considered at all? </FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>>> If so,</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>>> I'd be happy to contribute to the discussion and development </FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>>> (wherever I</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>>> could, but I don't know the C language yet) if that would help. A</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>>> combination of an XML mapfile, XSLT templates and XPath extremely </FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>>> powerful</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>>> compared the current template files.</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>></FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>></FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>> I think that Steve has been considering a switch to XML eventually, but</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>> I don't think there is any formal plan.</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>></FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>> I agree that XML might give more flexibility to the user by allowing him</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>> to read the mapfile from external tools, but it would be at the cost of</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>> performance (i.e. speed) when parsing the mapfile for every request.</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>> Combine this loss of performance with the fact that XML would be harder</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>> to edit manually than the current format, and you know where I stand</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>> personally.</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>></FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>> I'm not saying that if we were redesigning from scratch I wouldn't</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>> consider XML (we use it a lot in other apps), but since we've already</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>> got a very good (fast and simple) mapfile format then I won't be the one</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>> initiating a switch to XML mapfiles.</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>></FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>> Note that MapScript already gives you access to almost every piece of</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>> information from the mapfile, so when the CGI cannot do what you need</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>> then the logic path is to switch to MapScript, and then sky is the</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>> limit.</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>></FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>> My 0.02$</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>> --</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>> ------------------------------------------------------------</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>> Daniel Morissette morissette@dmsolutions.ca</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>> DM Solutions Group http://www.dmsolutions.ca/</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>> ------------------------------------------------------------</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>> Don't put for tomorrow what you can do today, because if</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I>> you enjoy it today you can do it again tomorrow.</FONT></FONT></I>
<FONT COLOR="#737373"><FONT SIZE="3"><I></FONT></FONT></I>
</PRE>
</BLOCKQUOTE>
</BODY>
</HTML>
--=-SjVFyMK1MIglgAY599KW--