[ka-Map-users] KaMap performance tuning

Tue May 22 18:46:44 EDT 2007

Jeff,

Sounds like you did all the right things. I have the whole US in Tiger 
Shapefiles http://imaptools.com/maps/demo.html and I don't have problems 
so I'm wondering what is causing the problem. This is about 22GB of data 
in about 10 data layers and a mapfile that is 2600 lines long.

I have also done Navteq data which has huge shapefiles and I use 
shp2tile to split them up and create tileindexes. I get similar 
performance to the above.

Something you might want to try is to build mapserver with DEBUG enabled 
in ./configure, then add DEBUG ON to each of your layers in the mapfile 
and watch your apache error_log.

I will spit out timing stats for each layer drawn. So you can look and 
see which is the bad boy. Then consider turning it on at a smaller 
(denominator) scale. Also watch the label-cache processing time. I have 
seen this be 50% of the image processing time. You can fix this by 
labeling latter or changing the data. label cache processing is O(n**2) 
so avoid dumping a lot of labels that you know will get filtered out. If 
it is point labels the consider splitting the point layer into two 
layers with important points in one and minor points in the other, just 
you you did with major/minor roads.

I you REALLY want a performance boost with shapefiles, like 3-5X 
improvement, put 16GB of ram in your system and create a ramdisk and 
copy your shapefiles into it. It really goes fast.

-Steve

Jeff Dege wrote:
> Is there some form of indexing on shapefiles that I didn't include in
> my description of what we'd done with these shapefiles?
> 
>> -----Original Message----- From: Steve Lime
>> [mailto:Steve.Lime at dnr.state.mn.us] Sent: Tuesday, May 22, 2007
>> 4:30 PM To: Jeff Dege; ka-map-users at lists.maptools.org Subject: RE:
>> [ka-Map-users] KaMap performance tuning
>> 
>> Did you index the shapefiles?
>> 
>>>>> On 5/22/2007 at 4:16 PM, in message
>> <34E056CD4D475841B8C9ABCE956F92F63709B2 at mrburns.KorOffice.loca l>,
>> "Jeff Dege" <jdege at korterra.com> wrote:
>>> We'd never had trouble with the performance of shapefiles, until
>>> we had to deal with one that contained the California streets.
>>> 500MB .shp file, 600MB .dbf file, and performance with it was dog
>>> slow.
>>> 
>>> We pulled out interstates, highways, and main roads, into
>>> separate shapefiles, so that we didn't have to touch the streets
>>> data until we were zoomed in tight, and performance was still dog
>>> slow when we did.
>>> 
>>> We split the streets shapefile into hundreds of separate tiles,
>>> shptreed each, built a tileindex on the bunch, then shptreed the
>>> tileindex, and performance was a good bit better, but still not
>>> acceptable.
>>> 
>>> The people I've been talking to have told me that when rendering
>>> everything, shapefiles are faster, but when rending small subsets
>>> of the data, PostGIS can be considerably faster.  Particularly if
>>> you use it's simplify() function to tune the resolution of the
>>> data returned to the scale of  the map.  I've only begun
>>> experimenting with this, but nothing in my experience so far 
>>> contradicts this.
>>> 
>>> Shp2img, rendering a map with an extent of .04 decimal degrees,
>>> took 17.198 seconds to render the streets layer using (tiled)
>>> shapefiles, it took 0.219 seconds to render the streets layer
>>> using PostGIS.
>>> 
>>> 
>>>> -----Original Message----- From: Melanie Ritzau
>>>> [mailto:MelanieRitz78 at web.de] Sent: Tuesday, May 22, 2007 4:01
>>>> PM To: Jeff Dege Subject: Re: [ka-Map-users] KaMap performance
>>>> tuning
>>>> 
>>>> Hi Jeff,
>>>> 
>>>> I am quite surprised with the results of your performance test.
>>>> In my experience (and others) shapefiles are a bit (5-10%)
>>>> faster than POSTGIS. I wonder if you optimized your shapefiles
>>>> ?
>>>> 
>>>> Was your test with one big Shapefile (what size ?) or did you 
>>>> tile your shapefile ?
>>>> 
>> (http://mapserver.gis.umn.edu/docs/reference/utilityreference/tile4ms)
>> 
>>>> Are your shapefiles indexed with shptree ?
>>>> 
>> (http://mapserver.gis.umn.edu/docs/reference/utilityreference/shptree)
>> 
>>>> To your problem: If you have large data-sets you need to 
>>>> increase time and memory limits in PHP. You can add e.g. in
>>>> tile
>> PHP:
>>>> ini_set("memory_limit","600M"); set_time_limit(1200);
>>>> 
>>>> This helped me with a similar problem.
>>>> 
>>>> 
>>>> Thanks, Mel
>>>> 
>>>> 
>>>>> -----Ursprüngliche Nachricht----- Von: "Jeff Dege"
>>>>> <jdege at korterra.com> Gesendet: 21.05.07 21:42:23 An:
>>>>> <ka-map-users at lists.maptools.org> Betreff: [ka-Map-users]
>>>>> KaMap performance tuning
>>>> 
>>>>> We've been using KaMap with Mapserver and OpenLayers, with
>>>> some success.
>>>>> We've recently begin looking at using PostGIS instead of
>> shapefiles,
>>>>> hoping to getter performance when generating maps that
>>>> haven't yet been
>>>>> tiled.
>>>>> 
>>>>> So we loaded our shapefiles into PostGIS tables, and
>>>> created a new map
>>>>> file that was a copy of the first, except that it got its
>>>> map data from
>>>>> the PostGIS tables.
>>>>> 
>>>>> Our first comparison was using shp2img, which showed a
>>>> dramatic decrease
>>>>> in the time needed to generate maps - with our data,
>>>> PostGIS was eight
>>>>> to ten times faster.
>>>>> 
>>>>> So we cleared out our tile cache, pointed KaMap at the
>>>> PostGIS map file,
>>>>> and saw very little difference.
>>>>> 
>>>>> So we created a batch file that would download a thousand
>> different
>>>>> tiles, using wget to make direct calls against tile.php,
>>>> and timed how
>>>>> long it took to run, with the shapefile and the PostGIS map
>>>> files.  The
>>>>> PostGIS test was only five percent that the shapefile test.
>>>>> 
>>>>> So we created yet another batch file that used wget to make
>>>> direct calls
>>>>> against mapserver.exe, running in CGI mode.  When
>>>> generating maps from
>>>>> PostGIS data, mapserver was eight times faster than against
>>>> shapefiles.
>>>>> So now the question:  If the image creation is so much
>>>>> faster,
>> when
>>>>> accessed through shp2img, and when accessed through
>>>> mapserver.exe, why
>>>>> is that speed improvement not visible when loading tiles
>>>> through kamap's
>>>>> tile.php?
>>>>> 
>>>>> It's clear to me that there must be some bottleneck, reducing
>>>>>  performance.  My guess would be in Apache or PHP
>>>>> configuration.
>>>>> 
>>>>> Does anyone know how I can measure what time is being spent
>>>> doing what,
>>>>> in this area?  What configuration options I should look at,
>>>> to try to
>>>>> improve performance?
>>>>> 
>>>>> Thanks.
>>>>> 
>>>>> 
>>>>> _______________________________________________ ka-Map-users
>>>>> mailing list ka-Map-users at lists.maptools.org 
>>>>> http://lists.maptools.org/mailman/listinfo/ka-map-users
>>>>> 
>>>> 
>>>> _______________________________________________________________
>>>>  SMS schreiben mit WEB.DE FreeMail - einfach, schnell und 
>>>> kostenguenstig. Jetzt gleich testen! http://f.web.de/?mc=021192
>>>> 
>>>> 
>>>> 
>>> _______________________________________________ ka-Map-users
>>> mailing list ka-Map-users at lists.maptools.org 
>>> http://lists.maptools.org/mailman/listinfo/ka-map-users
> 
> _______________________________________________ ka-Map-users mailing
> list ka-Map-users at lists.maptools.org 
> http://lists.maptools.org/mailman/listinfo/ka-map-users