AW: AW: Re: [Mapserver-users] Slow MapServer with large data

[Arnulf Christl (CCGIS)]

1. What hardware are you using?
AC:
We are running PostGreSQL/PostGIS and MapServer on a free BSD System. Its an
Intel 2.6 Ghz processor with 512 MB RAM, IDE 80 GB HD. The Webserver with
the PHP client, some additional shapes and two other WMS servers  online is
a fat dual processor Win2000 with 2 GB RAM (nice box, but by now there has
accumulated all sorts of sad software on it...).
You don't need much performance to get going, but once the data gets larger
and the requests more numerous it is helpful to get a larger machine.

2. How to "thin out the data"?
AC:
We have a special situation as we are currently migrating a customer from
another system. Therefore we have to turn around every object anyway and can
decide what to do with it.
Basically the idea is to use less points to display a line as in smaller
scales you cannot identify all the details. There are some tools around to
automatically do it with shape files I believe, it was also posted some days
(weeks, months?) ago. If I re-find it I will send it to you (maybe someone
is listening in and remenbers better than me).
Alternatively you can take "snapshots" of the shape data and include them as
raster layers for small scales. As you zoom in and the resolution of the
raster image deteriorates the vector shapes are activates (MAXSCALE)

2. We'ere always only showing one layer at a time (one shape-file to be
exact). Would the "throwing the shapes into PG/PostGIS"-thing have any
effect then??
AC:
Yes, definitely. We have an attribute on each object stating up to which
scale level it will be displayed. You can do that only if you own the data
for a prolonged period and really know it intimately. If it is regularly
updated from somewhere else this wouldn't make much sense.

Nonetheless the performance gain we achieve simply by using PostGIS instead
of shape files is flattening. If anybody has made different experiences
please tell us about it - or else I will send Jan in the wrong direction.

What kind of data do you plan to show? With a road network for example its
an easy question, just display highways at small scales and add minor roads
as you zoom in.

Cheers, Arnulf.

"Arnulf Christl \\(CCGIS\\)" <arnulf.christl@ccgis.de> wrote:
> Hi,
> 15 sec does seem a lot, but maybe thats it. Our shapefile
> sizes usually do
> not exceed 10 MB to prevent this happening. The problem is
> that shapes
> physically are sequential files and performance must go down
> when the
> filesize increases. Indexing (shptree.exe) and breaking up
> (tile4ms.exe)
> really do help a lot, strange that you don't seem to notice
> the effect.
> Maybe you try to always show the comletet extension of the
> shapefile, then
> the mentioned tools will not be of much help. If that is the
> case I would
> suggest that you set zooming limits by using MINSCALE and
> MAXSCALE to only
> access the files when at an appropriate scale. For the
> overview you could
> use a raster "screenshot" of the shapes (shp2img.exe) or
> thin out the data.
> Then the performance issue should not be a problem anymore.
> If you still cannot get enough performance you should
> consider throwing the
> shapes into PG/PostGIS. If thats still not enough, separate
> the MapServer
> software and the PostGreSQL DB to run on separate machines
> (hardware). This
> will improove performance by factor 5 to 10 (thats our
> hands-on experience).
> When you have reached this stage the map (from the same
> data) should be
> delivered in about 1 or 2 seconds.
>
> Cheers, Arnulf.
>
> -----Ursprüngliche Nachricht-----
> Von: mapserver-users-admin@lists.gis.umn.edu
> [mailto:mapserver-users-admin@lists.gis.umn.edu]Im Auftrag
> von
> jankarin@post1.dknet.dk
> Gesendet: Dienstag, 13. Mai 2003 12:25
> An: mapserver-users@lists.gis.umn.edu;
> woodbri@swoodbridge.com
> Betreff: Re: Re: [Mapserver-users] Slow MapServer with large
> data
>
>
> I have tried on af few shapefiles, but it doesn't help much.
> And it wouldn't
> have any effect on our tab-data.
>
> By the way: Is 15 sec. for a 25 MB shapefile/40 sec for a 30
> MB MapInfo
> map-file the usual speed for MapServer on a PIII, 650 Mhz,
> 262 MB RAM ?
>
> If it is, there's only one thing to do,  if not .....
> Anyone have some ideas?
>
> Jan
>
>
> woodbri@swoodbridge.com wrote:
> > Have you built shptree indexes for all your shapefiles?
> >
> > On 9 May 2003 at 13:14, jankarin@post1.dknet.dk wrote:
> >
> > > Hello
> > >
> > > I have a little problem with the speed in MapServer.
> > > We're trying to make qiuck-viewer to shape- and
> > tab-formats, but
> > > MapServer is running slow with large data: 20 seconds:
> > Shapefil 17 MB,
> > > dbf-fil 9,5 MB. 45 seconds: Map-fil 31 MB, Dat-fil 25
> MB.
> > >
> > > I have been trying with the GCI-version and PHP
> Mapscript
> > - same
> > > result.
> > >
> > > I am running MapServer 3.6.4 with PNG-output under
> Apache
> > 2.4.5 on
> > > Windows 2000. The pc is Pentim III, 650 Mhz, 262 MB
> RAM.
> > >
> > > In another installation (PII, 400 Mhz, 196 MB RAM) a
> > shapefile
> > > (shapefile 17 MB, dbf-fil 9,5 MB) took 80 seconds to
> > show.
> > >
> > > We're using MapServer in an application which is
> > dynamically creating
> > > map-files with Mapscript (only showing one layer at a
> > time). We're
> > > talknig about 17.000 layers in shape- and tabformats,
> so
> > it is not
> > > possible to improve indexfiles or anything else related
> to
> > a single
> > > layer.
> > >
> > > Can anyone help?
> > >
> > > Thanks,
> > >
> > > Jan Christiansen
> > > jankarin@post1.dknet.dk
> > > Engineering College of Copenhagen
> > >
> > > _______________________________________________
> > > Mapserver-users mailing list
> > > Mapserver-users@lists.gis.umn.edu
> > >
> http://lists.gis.umn.edu/mailman/listinfo/mapserver-users
> > >
> >
> >
>
>
> _______________________________________________
> Mapserver-users mailing list
> Mapserver-users@lists.gis.umn.edu
> http://lists.gis.umn.edu/mailman/listinfo/mapserver-users
>
>


_______________________________________________
Mapserver-users mailing list
Mapserver-users@lists.gis.umn.edu
http://lists.gis.umn.edu/mailman/listinfo/mapserver-users


_______________________________________________
Mapserver-users mailing list
Mapserver-users@lists.gis.umn.edu
http://lists.gis.umn.edu/mailman/listinfo/mapserver-users