The ESS block layer can do much more than
just map 4K blocks. We have implemented a prototype
file system to see what levels of performance are possible.
By combining multiple block "name-spaces" and
fully variable block sizes, we have implemented a basic file system
that behaves very differently from traditional 4K block-based
approaches. The new file system is stunningly fast, especially
for small files. Single file creates can take as little as 4 uSec
or 100,000 file creates in under one half of a second.
Directories with one billion files are actually practical.
File opens and reads require a single, direct, IO.
Variable sized blocks also mean that files only use the space of their
contents and are not rounded up to the next block boundary.
WildFire is looking for corporate partners to
help commercialize this file system as an "exascale" solution for
really "fast data". The file system has all of the features of a
local array, the flexibility and scale of file systems like ZFS, but
without the overhead and inefficiencies.