[FWTools] gdaltindex - performance issue with lots of files?

[Paul McCullough]

Frank,

Many thanks for the rapid reply. The drive is a local drive. It is an
external drive on a windows xp sp 2 box.
I can live with this for now. Can you provide an estimate when this will be
released in FWTools?

Regards
Paul



Frank Warmerdam-2 wrote:
> 
> Paul McCullough wrote:
>> When I run 
>>     gdaltindex -tileindex location level0 *.tif
>> in a directory with 33672 tif files, the process takes about 24 hours.
>> While it does finish properly, 24 hours seems too long.
>> (windows xp - sp 4 - 2GB RAM; FWTools 2.0.2)
>> In other layers of my image pyramid, gdaltindex completes in what I
>> consider
>> reasonable times.
>> For example, another layer with about 2000 files, runs in about 90
>> seconds.
>> It appears that the increase in time with larger file counts is
>> non-linear.
>> I can collect more data if that would help but I suspect there is BigOh
>> kind
>> of problem here.
>> Also, it seems quite reasable to have 30000 tile polygons.
>> 
>> I have tried these forms of the cmd line on medium file counts and seen
>> little difference:
>>     gdaltindex -tileindex location test1 *.tif
>>     gdaltindex -tileindex location test2 --optfile files
>> I have not tried this unix cmd line form:
>>     find $PWD -name "*.tif" -exec gdaltindex srtm-index3 {} \;
> 
> Paul,
> 
> This is:
> 
>    http://trac.osgeo.org/gdal/ticket/2158
> 
> A work around is available now in trunk and will appear in 1.5.1. 
> Basically
> for GDAL 1.5 changes were made to read the file list from the directory
> when
> a file is opened and this turns out to be very very slow in some
> circumstances
> when there are a lot of files.  Would the directory in question happen to
> be
> on a network driver?
> 
> If you need to work around this you can use an FWTools from early in 2007
> or older.
> 
> Best regards,
> -- 
> ---------------------------------------+--------------------------------------
> I set the clouds in motion - turn up   | Frank Warmerdam,
> warmerdam at pobox.com
> light and sound - activate the windows | http://pobox.com/~warmerdam
> and watch the world go round - Rush    | President OSGeo, http://osgeo.org
> 
> _______________________________________________
> FWTools mailing list
> FWTools at lists.maptools.org
> http://lists.maptools.org/mailman/listinfo/fwtools
> http://fwtools.maptools.org/
> 
> 

-- 
View this message in context: http://www.nabble.com/gdaltindex---performance-issue-with-lots-of-files--tp15231855p15238723.html
Sent from the FWTools mailing list archive at Nabble.com.