[ka-Map-users] KaMap performance tuning

[Jeff Dege]

Is there some form of indexing on shapefiles that I didn't include in my description of what we'd done with these shapefiles?

> -----Original Message-----
> From: Steve Lime [mailto:Steve.Lime at dnr.state.mn.us] 
> Sent: Tuesday, May 22, 2007 4:30 PM
> To: Jeff Dege; ka-map-users at lists.maptools.org
> Subject: RE: [ka-Map-users] KaMap performance tuning
> 
> Did you index the shapefiles?
> 
> >>> On 5/22/2007 at 4:16 PM, in message
> <34E056CD4D475841B8C9ABCE956F92F63709B2 at mrburns.KorOffice.loca
> l>, "Jeff
> Dege"
> <jdege at korterra.com> wrote:
> > We'd never had trouble with the performance of shapefiles, until we had to 
> > deal with one that contained the California streets.   500MB .shp file, 600MB 
> > .dbf file, and performance with it was dog slow.
> > 
> > We pulled out interstates, highways, and main roads, into separate 
> > shapefiles, so that we didn't have to touch the streets data until we were 
> > zoomed in tight, and performance was still dog slow when we did.
> > 
> > We split the streets shapefile into hundreds of separate tiles, shptreed 
> > each, built a tileindex on the bunch, then shptreed the tileindex, and 
> > performance was a good bit better, but still not acceptable.
> > 
> > The people I've been talking to have told me that when rendering everything, 
> > shapefiles are faster, but when rending small subsets of the data, PostGIS 
> > can be considerably faster.  Particularly if you use it's simplify() function 
> > to tune the resolution of the data returned to the scale of  the map.  I've 
> > only begun experimenting with this, but nothing in my experience so far 
> > contradicts this.
> > 
> > Shp2img, rendering a map with an extent of .04 decimal degrees, took 17.198 
> > seconds to render the streets layer using (tiled) shapefiles, it took 0.219 
> > seconds to render the streets layer using PostGIS.
> >  
> > 
> >> -----Original Message-----
> >> From: Melanie Ritzau [mailto:MelanieRitz78 at web.de] 
> >> Sent: Tuesday, May 22, 2007 4:01 PM
> >> To: Jeff Dege
> >> Subject: Re: [ka-Map-users] KaMap performance tuning
> >> 
> >> Hi Jeff,
> >> 
> >> I am quite surprised with the results of your performance 
> >> test. In my experience (and others) shapefiles are a bit 
> >> (5-10%) faster than POSTGIS.
> >> I wonder if you optimized your shapefiles ?
> >> 
> >> Was your test with one big Shapefile (what size ?) or did you 
> >> tile your shapefile ? 
> >>
> (http://mapserver.gis.umn.edu/docs/reference/utilityreference/tile4ms)
> >> Are your shapefiles indexed with shptree ? 
> >>
> (http://mapserver.gis.umn.edu/docs/reference/utilityreference/shptree)
> >> 
> >> To your problem: If you have large data-sets you need to 
> >> increase time and memory limits in PHP. You can add e.g. in tile
> PHP:
> >> 
> >> ini_set("memory_limit","600M");
> >> set_time_limit(1200);
> >> 
> >> This helped me with a similar problem.
> >> 
> >> 
> >> Thanks,
> >> Mel
> >> 
> >> 
> >> > -----Ursprüngliche Nachricht-----
> >> > Von: "Jeff Dege" <jdege at korterra.com>
> >> > Gesendet: 21.05.07 21:42:23
> >> > An: <ka-map-users at lists.maptools.org>
> >> > Betreff: [ka-Map-users] KaMap performance tuning
> >> 
> >> 
> >> > 
> >> > We've been using KaMap with Mapserver and OpenLayers, with 
> >> some success.
> >> > We've recently begin looking at using PostGIS instead of
> shapefiles,
> >> > hoping to getter performance when generating maps that 
> >> haven't yet been
> >> > tiled.
> >> > 
> >> > So we loaded our shapefiles into PostGIS tables, and 
> >> created a new map
> >> > file that was a copy of the first, except that it got its 
> >> map data from
> >> > the PostGIS tables.
> >> > 
> >> > Our first comparison was using shp2img, which showed a 
> >> dramatic decrease
> >> > in the time needed to generate maps - with our data, 
> >> PostGIS was eight
> >> > to ten times faster.
> >> > 
> >> > So we cleared out our tile cache, pointed KaMap at the 
> >> PostGIS map file,
> >> > and saw very little difference.
> >> > 
> >> > So we created a batch file that would download a thousand
> different
> >> > tiles, using wget to make direct calls against tile.php, 
> >> and timed how
> >> > long it took to run, with the shapefile and the PostGIS map 
> >> files.  The
> >> > PostGIS test was only five percent that the shapefile test.
> >> > 
> >> > So we created yet another batch file that used wget to make 
> >> direct calls
> >> > against mapserver.exe, running in CGI mode.  When 
> >> generating maps from
> >> > PostGIS data, mapserver was eight times faster than against 
> >> shapefiles.
> >> > 
> >> > So now the question:  If the image creation is so much faster,
> when
> >> > accessed through shp2img, and when accessed through 
> >> mapserver.exe, why
> >> > is that speed improvement not visible when loading tiles 
> >> through kamap's
> >> > tile.php?
> >> > 
> >> > It's clear to me that there must be some bottleneck, reducing
> >> > performance.  My guess would be in Apache or PHP configuration.
> >> > 
> >> > Does anyone know how I can measure what time is being spent 
> >> doing what,
> >> > in this area?  What configuration options I should look at, 
> >> to try to
> >> > improve performance?
> >> > 
> >> > Thanks.
> >> > 
> >> > 
> >> > _______________________________________________
> >> > ka-Map-users mailing list
> >> > ka-Map-users at lists.maptools.org 
> >> > http://lists.maptools.org/mailman/listinfo/ka-map-users 
> >> > 
> >> 
> >> 
> >> _______________________________________________________________
> >> SMS schreiben mit WEB.DE FreeMail - einfach, schnell und
> >> kostenguenstig. Jetzt gleich testen! http://f.web.de/?mc=021192 
> >> 
> >> 
> > 
> > _______________________________________________
> > ka-Map-users mailing list
> > ka-Map-users at lists.maptools.org 
> > http://lists.maptools.org/mailman/listinfo/ka-map-users
>