Please add some details about the internal workings

[cipriancraciun]

In relation to the question regarding the --max-workers and --max-scanner-workers discussed in #24, it would be nice to have a short description about the internal workings of mkdwarfs so that one can understand how to better optimize the process for his hardware.

For example:

• what does each "worker" actually do?
• is the creation a two separate phase, one scanning, and then one building?
• or the scanning and building are executed in parallel?
• are the "builder" threads writing at the same time to the resulting image, or there is an extra thread writing the final image? (from the thread names I assume this is the case);

[cipriancraciun]

Using lsof -p 1234 on a running mkdwarfs with lots of workers, I fail to see more than one open source file at a time.

[mhx]

Using lsof -p 1234 on a running mkdwarfs with lots of workers, I fail to see more than one open source file at a time.

That is likely due to what I explained here.

[cipriancraciun]

(I'm moving the conversation for #24 in here.)

Ciprian:

The storage I'm using as source for the CDNJS experiment is an RAID-0 array with 3 top-of-the-line SATA WD disks. Although the array is capable of around ~450 MiB/s sequential access, it's a slug when it comes to random access. However in some earlier experiments of mine (https://notes.volution.ro/v1/2019/01/notes/a202c8b9/) I've identified that when the access pattern is sub-optimal, the best choice is to just swamp the disk with I/O requests, and the driver / firmware will reorder the requests, thus accomplishing almost sequential speeds.

Mhx:

Oh, that is interesting, I need to have a close look at your results! I think I've found (in my tests that were certainly a lot less scientific) that using more threads generally slowed things down, which is why the option is called --max-scanner-workers. The code checks the load per thread and reduces the number of workers if it finds that they're mostly idle. So quiet likely, it'll only use a small fraction of the 512 workers you're asking for. You can remove the worker_group::load_adaptive argument in this line if you want to try to disable this.

[cipriancraciun]

When it comes to thread management I would strongly suggest that if the user specifies some numbers, then the "automatic" behaviour should be turned off.

Perhaps there could be three options for each thread set:

• --min-workers / --max-workers which enable automatic behaviour, but force the algorithm within certain limits;
• --workers which disables the automatic behaviour and just sets the number of threads to that value;

[cipriancraciun]

Also with regard to the number of threads and the throughput, depending on the storage properties there is always a sweet-spot for the number of threads, but this depends highly on the access pattern.

For example I've wrote in Rust an recursive hasher (https://github.com/volution/volution-md5-tools) that uses multiple threads to hash the files. However to improve the performance I offer a few options to sort the access pattern (see the --workers-sort argument):

• most of the time sorting by inode yields acceptable results;
• however for Ext4 (and compliant) using the actual extent information (https://www.kernel.org/doc/Documentation/filesystems/fiemap.txt or https://github.com/lilydjwg/fiemap-rs/blob/master/fiemap/src/lib.rs) yields almost optimal performance;
• path is a safe default when working with remote file-systems;
• random if one is faced with a wide RAID array whose layout you don't know;

Thus depending on the storage and access pattern, different number of threads make sense.

[cipriancraciun]

Also I would suggest using a more descriptive argument than --max-workers; "workers of what"?

For example --max-scanner-workers is quite OK as it describes the purpose of those workers.

[mhx]

When it comes to thread management I would strongly suggest that if the user specifies some numbers, then the "automatic" behaviour should be turned off.

Perhaps there could be three options for each thread set:

• --min-workers / --max-workers which enable automatic behaviour, but force the algorithm within certain limits;
• --workers which disables the automatic behaviour and just sets the number of threads to that value;

I've actually decided to remove --max-scanner-workers and only keep --num-workers. The latter will now globally determine the number of scanner / writer worker threads. I've also removed the load-dependent adjustment, so the number of threads configured will actually be used.

However, I've found that scanning was implemented in a suboptimal way. The 4 stages (finding files, actually scanning file contents, deduplication, scanning contents again for similarity) were executed sequentially when in reality they could run in parallel. So with the next release, the stages will execute autonomously in parallel and content scanning will happen as early as possible after file discovery. Deduplication is performed on-the-fly, and hardlinks are actually resolved before even considering a content scan. Also, if files are to be scanned twice (for similarity hashing), the second scan is guaranteed to re-use the memory mapping of the first scan. (This may not be ideal for huge files, but at the same time it wouldn't be ideal to perform the second scan in parallel in case of a lot of duplicates.)

While it is reasonable to expect this to perform better, I was surprised just how much better it was:

• On an M.2 SSD, with caches dropped, scanning 47.5 GiB / 2 million files took 122.7 seconds with the 0.4.1 release. The next release will only take 39.2 seconds. I've also tried running md5-create on the same input data with different --workers-sort modes and the fastest turned out to be inode-and-size with 41.5 seconds. None of the tasks were CPU bound.

• On a (quite full, 90% of 37 TiB in use) spinning-disk RAID-6, also with caches dropped, and using the same dataset, scanning time went down from 87 minutes to 30 minutes. I've only tried the default inode sorting with md5-create, and it finished in 26 minutes.

I've tried increasing the number of mkdwarfs workers to 128 just to see if scanning was any faster, but it made no difference at all in terms of wallclock time.

[mhx]

One more thing: the --max-scanner-workers only applies during the initial "scanning" phase. As soon mkdwarfs enters the "writing" phase, only a single file will be read at a time.

[mhx]

In relation to the question regarding the --max-workers and --max-scanner-workers discussed in #24, it would be nice to have a short description about the internal workings of mkdwarfs so that one can understand how to better optimize the process for his hardware.

For example:

• what does each "worker" actually do?
• is the creation a two separate phase, one scanning, and then one building?
• or the scanning and building are executed in parallel?
• are the "builder" threads writing at the same time to the resulting image, or there is an extra thread writing the final image? (from the thread names I assume this is the case);

I've added a high-level description here.