[ka-Map-users] KaMap performance tuning

[Jeff Dege]

We'd never had trouble with the performance of shapefiles, until we had to deal with one that contained the California streets.   500MB .shp file, 600MB .dbf file, and performance with it was dog slow.

We pulled out interstates, highways, and main roads, into separate shapefiles, so that we didn't have to touch the streets data until we were zoomed in tight, and performance was still dog slow when we did.

We split the streets shapefile into hundreds of separate tiles, shptreed each, built a tileindex on the bunch, then shptreed the tileindex, and performance was a good bit better, but still not acceptable.

The people I've been talking to have told me that when rendering everything, shapefiles are faster, but when rending small subsets of the data, PostGIS can be considerably faster.  Particularly if you use it's simplify() function to tune the resolution of the data returned to the scale of the map.  I've only begun experimenting with this, but nothing in my experience so far contradicts this.

Shp2img, rendering a map with an extent of .04 decimal degrees, took 17.198 seconds to render the streets layer using (tiled) shapefiles, it took 0.219 seconds to render the streets layer using PostGIS.
 

> -----Original Message-----
> From: Melanie Ritzau [mailto:MelanieRitz78 at web.de] 
> Sent: Tuesday, May 22, 2007 4:01 PM
> To: Jeff Dege
> Subject: Re: [ka-Map-users] KaMap performance tuning
> 
> Hi Jeff,
> 
> I am quite surprised with the results of your performance 
> test. In my experience (and others) shapefiles are a bit 
> (5-10%) faster than POSTGIS.
> I wonder if you optimized your shapefiles ?
> 
> Was your test with one big Shapefile (what size ?) or did you 
> tile your shapefile ? 
> (http://mapserver.gis.umn.edu/docs/reference/utilityreference/tile4ms)
> Are your shapefiles indexed with shptree ? 
> (http://mapserver.gis.umn.edu/docs/reference/utilityreference/shptree)
> 
> To your problem: If you have large data-sets you need to 
> increase time and memory limits in PHP. You can add e.g. in tile PHP:
> 
> ini_set("memory_limit","600M");
> set_time_limit(1200);
> 
> This helped me with a similar problem.
> 
> 
> Thanks,
> Mel
> 
> 
> > -----Ursprüngliche Nachricht-----
> > Von: "Jeff Dege" <jdege at korterra.com>
> > Gesendet: 21.05.07 21:42:23
> > An: <ka-map-users at lists.maptools.org>
> > Betreff: [ka-Map-users] KaMap performance tuning
> 
> 
> > 
> > We've been using KaMap with Mapserver and OpenLayers, with 
> some success.
> > We've recently begin looking at using PostGIS instead of shapefiles,
> > hoping to getter performance when generating maps that 
> haven't yet been
> > tiled.
> > 
> > So we loaded our shapefiles into PostGIS tables, and 
> created a new map
> > file that was a copy of the first, except that it got its 
> map data from
> > the PostGIS tables.
> > 
> > Our first comparison was using shp2img, which showed a 
> dramatic decrease
> > in the time needed to generate maps - with our data, 
> PostGIS was eight
> > to ten times faster.
> > 
> > So we cleared out our tile cache, pointed KaMap at the 
> PostGIS map file,
> > and saw very little difference.
> > 
> > So we created a batch file that would download a thousand different
> > tiles, using wget to make direct calls against tile.php, 
> and timed how
> > long it took to run, with the shapefile and the PostGIS map 
> files.  The
> > PostGIS test was only five percent that the shapefile test.
> > 
> > So we created yet another batch file that used wget to make 
> direct calls
> > against mapserver.exe, running in CGI mode.  When 
> generating maps from
> > PostGIS data, mapserver was eight times faster than against 
> shapefiles.
> > 
> > So now the question:  If the image creation is so much faster, when
> > accessed through shp2img, and when accessed through 
> mapserver.exe, why
> > is that speed improvement not visible when loading tiles 
> through kamap's
> > tile.php?
> > 
> > It's clear to me that there must be some bottleneck, reducing
> > performance.  My guess would be in Apache or PHP configuration.
> > 
> > Does anyone know how I can measure what time is being spent 
> doing what,
> > in this area?  What configuration options I should look at, 
> to try to
> > improve performance?
> > 
> > Thanks.
> > 
> > 
> > _______________________________________________
> > ka-Map-users mailing list
> > ka-Map-users at lists.maptools.org
> > http://lists.maptools.org/mailman/listinfo/ka-map-users
> > 
> 
> 
> _______________________________________________________________
> SMS schreiben mit WEB.DE FreeMail - einfach, schnell und
> kostenguenstig. Jetzt gleich testen! http://f.web.de/?mc=021192
> 
>