in-kernel module plans?

[baryluk]

Hi,

I discovered DwarFS today, as I was exploring switching from squashfs to erofs (or something else). I was upset about poor decompression and multithreading scaling for squashfs, so was looking for alternatives. And by accident I found DwarFS quite a bit better.

Input and output in tmpfs.

The output is for livecd based on Debian with a very large number of packages.

The input is 44GiB in 947000 files and 87671 directories. A lot of binaries, libraries, text files, config files, etc.

squashfs compresses this to about 18.75 GB (this actually is iso size, so includes bootloader, kernel and initramfs - about 350MB in total, so real squashfs is more like 18.45GB). Compression takes about an hour on my 16-core / 32-thread machine. Using zstd 19.
erofs output was 26.92GB, and took 13 minutes to build. (-zlz4hc,9)

dwarfs output was 15.95GB (14.85 GiB), and took a bit over 18 minutes.

So that is quite nice indeed and significant improvement, for building, uploading, downloading, writing to USB / BD, and booting.

dwarfs output:

user@debian:~/dwarfs/build$ sudo ./mkdwarfs --input /home/user/tmp-live-build/20220708T150035Z/chroot  --output /tmp/smooth-test.dwarfs --progress unicode
I 17:44:32.393587 scanning /home/user/tmp-live-build/20220708T150035Z/chroot
I 17:44:42.341824 assigning directory and link inodes...
I 17:44:42.465037 waiting for background scanners...
I 17:45:04.721353 scanning CPU time: 588.3s
I 17:45:04.721430 finalizing file inodes...
I 17:45:05.075697 saved 1.432 GiB / 43.29 GiB in 111244/947000 duplicate files
I 17:45:05.075818 assigning device inodes...
I 17:45:05.097679 assigning pipe/socket inodes...
I 17:45:05.119432 building metadata...
I 17:45:05.119480 building blocks...
I 17:45:05.119538 saving names and symlinks...
I 17:45:05.119540 using a 4 KiB window at 512 B steps for segment analysis
I 17:45:05.120010 bloom filter size: 64 KiB
I 17:45:05.120134 ordering 834976 inodes using nilsimsa similarity...
I 17:45:05.143907 nilsimsa: depth=20000 (1000), limit=255
I 17:45:05.479148 updating name and link indices...
I 17:45:06.538335 pre-sorted index (5362274 name, 104777 path lookups) [1.394s]
I 18:02:32.941060 834976 inodes ordered [1048s, 985.2s CPU]
I 18:02:32.941118 waiting for segmenting/blockifying to finish...
I 18:02:32.955384 segmenting/blockifying CPU time: 544.3s
I 18:02:32.955454 bloom filter reject rate: 94.970% (TPR=0.102%, lookups=38905586362)
I 18:02:32.955521 segmentation matches: good=401782, bad=29184516, total=29800681
I 18:02:32.955584 segmentation collisions: L1=0.047%, L2=0.034% [79882028 hashes]
I 18:02:32.955644 saving chunks...
I 18:02:33.341802 saving directories...
I 18:02:33.546392 saving shared files table...
I 18:02:33.773129 saving names table... [192.3ms]
I 18:02:33.781233 saving symlinks table... [8.038ms]
I 18:02:34.402424 waiting for compression to finish...
I 18:02:41.195785 compressed 43.29 GiB to 14.85 GiB (ratio=0.343085)
I 18:02:41.581912 compression CPU time: 2.908e+04s
I 18:02:41.581980 filesystem created without errors [1089s]
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
waiting for block compression to finish
87671 dirs, 73241/780 soft/hard links, 947000/947000 files, 0 other
original size: 43.29 GiB, dedupe: 1.432 GiB (111244 files), segment: 3.756 GiB
filesystem: 38.1 GiB in 2439 blocks (1420491 chunks, 834976/834976 inodes)
compressed filesystem: 2439 blocks/14.85 GiB written [depth: 1639]

so, I think dwarfs is really useful and would be nice to be able to boot from it.

I am sure I could make it work with fuse somehow in intramfs, possibly with static binaries, but would be probably slowish due to use of fuse.

Are there any plans to develop native in-kernel module for dwarfs?

Kernel already has some internal APIs and modules to use zstd, lz4, lzma, so these could be reused, but of course some other pieces might require adapting, including memory allocation, hash tables, etc).

I know this is not a trivial task, but if the feature set and format is stable, it is something worth considering? There might be some features that are not strictly required either, so could be left for future.

[mhx]

Glad you like the project!

There are currently no plans at all for kernel integration. It is most definitely not a trivial task, especially given the code is C++ and pulls in libraries like folly and fbthrift.

That being said, the format is documented, but the metadata binary format depends on fbthrift's Frozen2 library.

So it's certainly not impossible, but this is going to have a hard time coming even close to the top of my priorities. :)