[ka-Map-users] Best Linux file system type for ka-map cache?
plists at prometheus.org.yu
Tue Oct 23 11:26:38 EDT 2007
On Tuesday 23 October 2007 16:44:38 Stephen Woodbridge wrote:
> Our research has shown Reiserfs to be the fastest. All Linux FS suffer
> from the block allocation issues. That is the fact the files are
> allocated based based on a 4k block, but most water tiles are only 130
> bytes but use up 4k of disk. We have found some SAN devices that do
> their own internal space allocation that does not suffer from this
> effect, but I believe that is specific to the internal SAN implementations.
I don't know if it has been thought of before, but there could be a special
small-tile cache for this purpose that would benefit all platforms. You could
cache for example repeating tiles (all black or all blue most often). The
easy way would be having softlinks to these (or some multiplatform
equivalent), or better yet having a master cache file that contains the list
of these tiles (so you do not have to actually have the files in cache dirs
in the first place). Identity of small repeat tiles could be checked via some
hash function (MD5, CRC) which is not all that much time (filesystem load is
far more dangerous). The advantage is that even though reiser is fastER for
massive small file filesystems, the fastest would be not to have these files
separated/repeated at all.
More information about the ka-Map-users