[ka-Map-users] KaMap performance tuning

[Paul Ramsey]

10:1 speed advantage for PostGIS is nice and all, but seems unlikely,  
even to this PostGIS zealot...

Does your layer have a FILTER on it that uses the attributes? PostGIS  
does that great, shapefiles do it terribly.

I would expect on a very large spatial file (your 500MB example) the  
PostGIS r-tree will be balanced better than the quadtree .qix file,  
but that should provide incremental advantage, not massive  
advantage.  One possibility is that you're running out of depth in  
your quadtree, maybe tree bumping up the depth in your shptree call.

P

On 22-May-07, at 2:16 PM, Jeff Dege wrote:

> We'd never had trouble with the performance of shapefiles, until we  
> had to deal with one that contained the California streets.    
> 500MB .shp file, 600MB .dbf file, and performance with it was dog  
> slow.
>
> We pulled out interstates, highways, and main roads, into separate  
> shapefiles, so that we didn't have to touch the streets data until  
> we were zoomed in tight, and performance was still dog slow when we  
> did.
>
> We split the streets shapefile into hundreds of separate tiles,  
> shptreed each, built a tileindex on the bunch, then shptreed the  
> tileindex, and performance was a good bit better, but still not  
> acceptable.
>
> The people I've been talking to have told me that when rendering  
> everything, shapefiles are faster, but when rending small subsets  
> of the data, PostGIS can be considerably faster.  Particularly if  
> you use it's simplify() function to tune the resolution of the data  
> returned to the scale of the map.  I've only begun experimenting  
> with this, but nothing in my experience so far contradicts this.
>
> Shp2img, rendering a map with an extent of .04 decimal degrees,  
> took 17.198 seconds to render the streets layer using (tiled)  
> shapefiles, it took 0.219 seconds to render the streets layer using  
> PostGIS.
>
>
>> -----Original Message-----
>> From: Melanie Ritzau [mailto:MelanieRitz78 at web.de]
>> Sent: Tuesday, May 22, 2007 4:01 PM
>> To: Jeff Dege
>> Subject: Re: [ka-Map-users] KaMap performance tuning
>>
>> Hi Jeff,
>>
>> I am quite surprised with the results of your performance
>> test. In my experience (and others) shapefiles are a bit
>> (5-10%) faster than POSTGIS.
>> I wonder if you optimized your shapefiles ?
>>
>> Was your test with one big Shapefile (what size ?) or did you
>> tile your shapefile ?
>> (http://mapserver.gis.umn.edu/docs/reference/utilityreference/ 
>> tile4ms)
>> Are your shapefiles indexed with shptree ?
>> (http://mapserver.gis.umn.edu/docs/reference/utilityreference/ 
>> shptree)
>>
>> To your problem: If you have large data-sets you need to
>> increase time and memory limits in PHP. You can add e.g. in tile PHP:
>>
>> ini_set("memory_limit","600M");
>> set_time_limit(1200);
>>
>> This helped me with a similar problem.
>>
>>
>> Thanks,
>> Mel
>>
>>
>>> -----Ursprüngliche Nachricht-----
>>> Von: "Jeff Dege" <jdege at korterra.com>
>>> Gesendet: 21.05.07 21:42:23
>>> An: <ka-map-users at lists.maptools.org>
>>> Betreff: [ka-Map-users] KaMap performance tuning
>>
>>
>>>
>>> We've been using KaMap with Mapserver and OpenLayers, with
>> some success.
>>> We've recently begin looking at using PostGIS instead of shapefiles,
>>> hoping to getter performance when generating maps that
>> haven't yet been
>>> tiled.
>>>
>>> So we loaded our shapefiles into PostGIS tables, and
>> created a new map
>>> file that was a copy of the first, except that it got its
>> map data from
>>> the PostGIS tables.
>>>
>>> Our first comparison was using shp2img, which showed a
>> dramatic decrease
>>> in the time needed to generate maps - with our data,
>> PostGIS was eight
>>> to ten times faster.
>>>
>>> So we cleared out our tile cache, pointed KaMap at the
>> PostGIS map file,
>>> and saw very little difference.
>>>
>>> So we created a batch file that would download a thousand different
>>> tiles, using wget to make direct calls against tile.php,
>> and timed how
>>> long it took to run, with the shapefile and the PostGIS map
>> files.  The
>>> PostGIS test was only five percent that the shapefile test.
>>>
>>> So we created yet another batch file that used wget to make
>> direct calls
>>> against mapserver.exe, running in CGI mode.  When
>> generating maps from
>>> PostGIS data, mapserver was eight times faster than against
>> shapefiles.
>>>
>>> So now the question:  If the image creation is so much faster, when
>>> accessed through shp2img, and when accessed through
>> mapserver.exe, why
>>> is that speed improvement not visible when loading tiles
>> through kamap's
>>> tile.php?
>>>
>>> It's clear to me that there must be some bottleneck, reducing
>>> performance.  My guess would be in Apache or PHP configuration.
>>>
>>> Does anyone know how I can measure what time is being spent
>> doing what,
>>> in this area?  What configuration options I should look at,
>> to try to
>>> improve performance?
>>>
>>> Thanks.
>>>
>>>
>>> _______________________________________________
>>> ka-Map-users mailing list
>>> ka-Map-users at lists.maptools.org
>>> http://lists.maptools.org/mailman/listinfo/ka-map-users
>>>
>>
>>
>> _______________________________________________________________
>> SMS schreiben mit WEB.DE FreeMail - einfach, schnell und
>> kostenguenstig. Jetzt gleich testen! http://f.web.de/?mc=021192
>>
>>
>
> _______________________________________________
> ka-Map-users mailing list
> ka-Map-users at lists.maptools.org
> http://lists.maptools.org/mailman/listinfo/ka-map-users