[FWTools] Compressing geoid model?

[Mikael Rittri]

Frank wrote:

> I found I got significantly better results (around 10MB) with the command:
> 
> gdal_translate egm96-5.pgm -co COMPRESS=LZW out.tif \
>     -co PREDICTOR=2 -co TILED=YES

Great! Starting from egm2008-5.pgm, this gives me a compressed 
size of 9558 KB (48 percent less than original). From Charles's 
mail, it seems hard to do better with lossless compression.

Thanks, Frank and Charles.   

I've now experimented with using lower vertical resolution.  
In the original egm2008-5.pgm, the vertical resolution is 3 mm
(there is also an offset, to make the most of the UInt16 range). 

When I added 

    -ot Int16 -scale 0 65000 -10800 8700

to the gdal_translate command, converting to 1 cm resolution
(and no offset), the resulting file is 6484 KB (64 percent less 
than original).

And when I instead used 

    -ot Int16 -scale 0 65000 -1080 870

converting to 1 dm resolution, the resulting file is 2600 KB 
(86 percent less than original).

This is of course lossy compression, and probably not in the 
spirit of GeographicLib, but at least I know how much accuracy 
that can be lost. 

Would the -scale option round the values, or just truncate them? 

Regards,
Mikael Rittri
Carmenta 
Sweden
http://www.carmenta.com 

-----Original Message-----
From: fwtools-bounces at lists.maptools.org [mailto:fwtools-bounces at lists.maptools.org] On Behalf Of Frank Warmerdam
Sent: den 7 juni 2011 16:45
To: fwtools at lists.maptools.org
Subject: Re: [FWTools] Compressing geoid model?

On 11-06-07 09:36 AM, Mikael Rittri wrote:
> However, I have tried these commands:
>
>     gdal_translate -ot UInt16 -co "COMPRESS=NONE" egm2008-5.pgm 
> egm2008-5-uint16-nocompression.tif
>
>     gdal_translate -ot UInt16 -co "COMPRESS=LZW" egm2008-5.pgm 
> egm2008-5-uint16-lzw.tif
>
>     gdal_translate -ot UInt16 -co "COMPRESS=PACKBITS" egm2008-5.pgm 
> egm2008-5-uint16-packbits.tif
...
> So, PACKBITS makes the result 0.7 percent larger than NONE, which is 
> perhaps not a surprise, since run-length encoding should not be 
> suitable for this kind of data.
> However, LZW makes the result 29.6 percent larger than NONE, which 
> surprised me.  I would have thought the the geoid data should be 
> fairly easy to compress (although I know almost nothing about how LZW 
> works).
>
> Am I missing something obvious? Is there some other
> (GDAL-readable) compressed format that would be better for this kind 
> of data?

Mikael,

I believe the problem is that each block written includes an LZW dictionary table which can be a fair amount of overhead.  In the above case the image is written out with one scanline per block.  Also, by default the literal bytes are written.  Looking at the file it seems there are no obvious simple patterns of values or large constant areas, so the LZW compressor does poorly.  Using a horizontal differencing predictor (basically compressing the changes from pixel to pixel) helped a great deal.

I found I got significantly better results (around 10MB) with the command:

gdal_translate egm96-5.pgm -co COMPRESS=LZW out.tif \
    -co PREDICTOR=2 -co TILED=YES

Best regards,
-- 
---------------------------------------+--------------------------------
---------------------------------------+------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | Geospatial Programmer for Rent

_______________________________________________
FWTools mailing list
FWTools at lists.maptools.org
http://lists.maptools.org/mailman/listinfo/fwtools
http://fwtools.maptools.org/