Mounting takes a while

[M3Henry]

I decided to take the plunge on bcachefs recently, for ease of disk management and for fast data caching.

I've got a fairly modest amount of data, 723GB with a pair of SATA HDDs as the backing store and a SATA SSD as the promote target.

I noticed the boot times were a bit slower than my original ext4 root on the SSD. Since then I've noticed my boot times steadily getting longer and longer. With systemd reporting a mount time of 38s at the moment.

I'm wondering what I can do to keep the mount time down. Is this sort of time expected?

systemd-analyze plot

Drives

Device	Size	Type	Access Time	SATA	Notes
/dev/sda	500	SSD	0.18ms	6Gbps	cache drive
/dev/sdb3	978GB	HDD	17ms	6Gbps	Partition 3 is bcachefs, 1 is EFI, 2 is swap.
/dev/sdc	750GB	HDD	16ms	3Gbps	Added much later than sdb3

Kernel version

6.11.6-arch1-1 #1 SMP PREEMPT_DYNAMIC

mount info

/dev/sdb3:/dev/sda:/dev/sdc on / type bcachefs (rw,noatime,metadata_replicas=2,foreground_target=hdd,background_target=hdd,promote_target=ssd)

options

acl = 1
allocator_stuck_timeout = 30
background_compression = none
background_target = hdd
block_size = 4.00 KiB
btree_node_mem_ptr_optimization = 1
btree_node_prefetch = 1
btree_node_size = 256 KiB
compression = none
data_checksum = none [crc32c] crc64 xxhash 
data_replicas = 1
data_replicas_required = 1
degraded = 0
direct_io = 1
discard = 1
encoded_extent_max = 64.0 KiB
erasure_code = 0
errors = continue [fix_safe] panic ro 
fix_errors = exit
foreground_target = hdd
fsck = 0
fsck_memory_usage_percent = 50
gc_reserve_bytes = 0 B
gc_reserve_percent = 8
grpquota = 0
inline_data = 1
inodes_32bit = 1
inodes_use_key_cache = 1
journal_flush_delay = 1000
journal_flush_disabled = 0
journal_reclaim_delay = 100
journal_transaction_names = 1
metadata_checksum = none [crc32c] crc64 xxhash 
metadata_replicas = 2
metadata_replicas_required = 1
metadata_target = none
move_bytes_in_flight = 1.00 MiB
move_ios_in_flight = 32
nochanges = 0
nocow = 0
nocow_enabled = 1
noexcl = 0
norecovery = 0
no_splitbrain_check = 0
prjquota = 0
promote_target = ssd
ratelimit_errors = 1
read_only = 0
reconstruct_alloc = 0
recovery_pass_last = [scan_for_btree_nodes] check_topology accounting_read alloc_read stripes_read initialize_subvolumes snapshots_read check_allocations trans_mark_dev_sbs fs_journal_alloc set_may_go_rw journal_replay check_alloc_info check_lrus check_btree_backpointers check_backpointers_to_extents check_extents_to_backpointers check_alloc_to_lru_refs fs_freespace_init bucket_gens_init reconstruct_snapshots check_snapshot_trees check_snapshots check_subvols check_subvol_children delete_dead_snapshots fs_upgrade_for_subvolumes check_inodes check_extents check_indirect_extents check_dirents check_xattrs check_root check_subvolume_structure check_directory_structure check_nlinks resume_logged_ops delete_dead_inodes fix_reflink_p set_fs_needs_rebalance 
root_reserve_percent = 0
shard_inode_numbers = 1
str_hash = crc32c crc64 [siphash] 
usrquota = 0
verbose = 0
version_upgrade = [compatible] incompatible none 
very_degraded = 0
wide_macs = 0

bcachefs fs usage /

Filesystem: c8c2b85b-dcd7-4d85-939a-3ff47419466d
Size:                  2050167218176
Used:                   776016495616
Online reserved:             1982464

Data type       Required/total  Durability    Devices
reserved:       1/1                [] 27172864
btree:          1/2             2             [sdb3 sdc]        8065122304
user:           1/1             1             [sdb3]          642641985536
user:           1/1             1             [sdc]           125279776768
cached:         1/1             1             [sda]           192550227968

Btree usage:
extents:          2639265792
inodes:           1851260928
dirents:           384303104
xattrs:               524288
alloc:             601358336
reflink:            13631488
subvolumes:           524288
snapshots:            524288
lru:                35651584
freespace:           3145728
need_discard:        1048576
backpointers:     2399666176
bucket_gens:         8912896
snapshot_trees:       524288
deleted_inodes:       524288
logged_ops:          1048576
rebalance_work:      1572864
accounting:        121634816

hdd (device 0):                 sdb3              rw
                                data         buckets    fragmented
  free:                 319324946432          609064
  sb:                        3149824               7        520192
  journal:                4294967296            8192
  btree:                  4032561152           10687    1570504704
  user:                 642641985536         1237778    6310166528
  cached:                          0               0
  parity:                          0               0
  stripe:                          0               0
  need_gc_gens:                    0               0
  need_discard:                    0               0
  unstriped:                       0               0
  capacity:             978178801664         1865728

hdd (device 2):                  sdc              rw
                                data         buckets    fragmented
  free:                 607476776960         1158670
  sb:                        3149824               7        520192
  journal:                4294967296            8192
  btree:                  4032561152           10673    1563164672
  user:                 125279776768          253264    7503499264
  cached:                          0               0
  parity:                          0               0
  stripe:                          0               0
  need_gc_gens:                    0               0
  need_discard:              1572864               3
  unstriped:                       0               0
  capacity:             750155988992         1430809

ssd (device 1):                  sda              rw
                                data         buckets    fragmented
  free:                 298434691072          569219
  sb:                        3149824               7        520192
  journal:                3906994176            7452
  btree:                           0               0
  user:                            0               0
  cached:               192550227968          377201    5211729920
  parity:                          0               0
  stripe:                          0               0
  need_gc_gens:                    0               0
  need_discard:               524288               1
  unstriped:                       0               0
  capacity:             500107837440          953880

[koverstreet]

Mount with -o verbose, that will tell us which recovery pass is causing problems

[M3Henry]

Here you go, hope this helps.

dmesg.txt

[g2p]

dmesg shows the filesystem is mounted 7.7s into the boot.
The device with a composite name and a long red line in the systemd-analyze graph looks like an artefact.

[koverstreet]

No, this is a bcachefs bug - if you look at the SRCU warnings, the issue is that the first inode allocations take a long time because they have to scan.

We still don't have a real inode allocator; I was planning on adding a free inodes btree but I have a better idea now.

[keidrych]

Is this addressed by commit df245c0 ?

[koverstreet]

Yes, but the bigger one is c6b74e6 ("bcachefs: bcachefs_metadata_version_disk_accounting_big_endian")