Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bcachefs hangs on intense I/O. #787

Open
0xfk0 opened this issue Nov 26, 2024 · 9 comments
Open

Bcachefs hangs on intense I/O. #787

0xfk0 opened this issue Nov 26, 2024 · 9 comments

Comments

@0xfk0
Copy link

0xfk0 commented Nov 26, 2024

Basically, the issue is the same as #673, but I cannot reopen the latter issue.

I have attached dmesg to the issue. In my particular case, the rebuild of AOSP project caused these messages, at least.

Also, the system may become completely unresponsive with load average like 2000. New processes cannot be started in this case, all stuck at IO. I will try to add a few photos I made from the screen. What is interesting, in this state existing processes (like web browser) work fine.

The kernel version is 6.10.1. Previously, @koverstreet closed the bug #637 with the words, what "if anyone is still seeing this on 6.10 go ahead and reopen". Of course, anybody cannot reopen the bug at Github... But the bug still persists.

Need to add, that bug is definitely depending on filesystem state: the problem doesn't appear in the past.

Another problems: out of kernel fsck fixes the same errors every time (like absent or zero lru, I don't remember details and didn't take the screenshot). In-kernel fsck hangs too.:-(

Looks, I need to copy all the data to another disk. I want to ask suggestion, how should I do this: currently I have one HDD and one SSD for bcachefs. Both fully occupied by one file system, but SSD is used for cache, see
superblock.txt Basically, I want to create a new file system on another HDD, copy all the data to it, and when add first HDD as a mirror and add (single, I haven't two) SDD as a cache. How can I do this in correct way? PS: I understood, what I have problems if SDD fails, but at first I need performance, but not reliability.

dmesg.txt.gz

@0xfk0
Copy link
Author

0xfk0 commented Nov 26, 2024

Kernel messages (system hangs) IMG_20241126_234550_120

More kernel messages
IMG_20241126_234556_064

Again kernel messages
IMG_20241126_234544_153

@0xfk0
Copy link
Author

0xfk0 commented Nov 26, 2024

It hangs again. Running fsck few times doesnt help. The bug is clearly reproducible.

17326548043288829022677468778784

Sorry for the photos, but I need to write from mobile phone, as computer unresponsive now

@0xfk0
Copy link
Author

0xfk0 commented Nov 27, 2024

Need to add: after switching to kernel 6.12.0 the problem fixed. Also I found, that at the first boot file system have been migrated to new version.

@0xfk0
Copy link
Author

0xfk0 commented Nov 27, 2024

Too early. On the second try (building AOSP) it hangs again! Have no logs for now.

Moreover, situation is worse than before. With new kernel I have no graphics driver (nvidia...), and unable to load previous kernel, as it becomes converting filesystem back, and latter is never ending process. With new (6.12.0) kernel it works strange now. No more effect of the caching from SSD. All is VERY SLOW and data almost always read from HDD, not from SSD.

@0xfk0
Copy link
Author

0xfk0 commented Nov 27, 2024

I have evacuated, removed, added again SSD drive. Now caching works (I see in dstat output). But now I have following error messages in dmesg:

[ 3246.834237] bcachefs (000bfc4b-b15a-41dd-895b-a200832c047f): Journal stuck? Waited for 10 seconds...
               flags:                     replay_done,running,may_skip_flush
               dirty journal entries:     449/32768
               seq:                       70406886
               seq_ondisk:                70406886
               last_seq:                  70406438
               last_seq_ondisk:           70406438
               flushed_seq_ondisk:        70406886
               watermark:                 reclaim
               each entry reserved:       321
               nr flush writes:           3414
               nr noflush writes:         1920
               average write size:        200 KiB
               nr direct reclaim:         0
               nr background reclaim:     64713
               reclaim kicked:            1
               reclaim runs in:           0 ms
               blocked:                   0
               current entry sectors:     1024
               current entry error:       ok
               current entry:             closed
               unwritten entries:
               last buf closed
               space:
                 discarded                1024:8386560
                 clean ondisk             1024:8386560
                 clean                    1024:8386560
                 total                    1024:8388608
               dev 1:
                 nr                       8192
                 
                 ... many times ...
                 
[ 3250.328126] ------------[ cut here ]------------
[ 3250.328151] btree trans held srcu lock (delaying memory reclaim) for 13 seconds
[ 3250.328162] ------------[ cut here ]------------
[ 3250.328166] ------------[ cut here ]------------
[ 3250.328166] ------------[ cut here ]------------
[ 3250.328166] ------------[ cut here ]------------
[ 3250.328172] btree trans held srcu lock (delaying memory reclaim) for 10 seconds
[ 3250.328173] btree trans held srcu lock (delaying memory reclaim) for 10 seconds
[ 3250.328174] btree trans held srcu lock (delaying memory reclaim) for 13 seconds
[ 3250.328175] btree trans held srcu lock (delaying memory reclaim) for 12 seconds
[ 3250.328194] ------------[ cut here ]------------
[ 3250.328194] ------------[ cut here ]------------
[ 3250.328198] ------------[ cut here ]------------
[ 3250.328202] ------------[ cut here ]------------
[ 3250.328202] btree trans held srcu lock (delaying memory reclaim) for 13 seconds
[ 3250.328201] btree trans held srcu lock (delaying memory reclaim) for 13 seconds
[ 3250.328193] WARNING: CPU: 7 PID: 34849 at fs/bcachefs/btree_iter.c:3028 bch2_trans_put+0x1f7/0x230
[ 3250.328206] btree trans held srcu lock (delaying memory reclaim) for 12 seconds
[ 3250.328211] btree trans held srcu lock (delaying memory reclaim) for 13 seconds
[ 3250.328217] Modules linked in:
[ 3250.328209] WARNING: CPU: 6 PID: 35207 at fs/bcachefs/btree_iter.c:3028 bch2_trans_srcu_unlock+0x122/0x130
[ 3250.328208] WARNING: CPU: 0 PID: 8 at fs/bcachefs/btree_iter.c:3028 bch2_trans_srcu_unlock+0x122/0x130
[ 3250.328208] WARNING: CPU: 12 PID: 35203 at fs/bcachefs/btree_iter.c:3028 bch2_trans_srcu_unlock+0x122/0x130
[ 3250.328208] WARNING: CPU: 1 PID: 33725 at fs/bcachefs/btree_iter.c:3028 bch2_trans_put+0x1f7/0x230
[ 3250.328222]  nft_masq xt_MASQUERADE nft_chain_nat
[ 3250.328233] Modules linked in:
[ 3250.328233] Modules linked in:
[ 3250.328228] WARNING: CPU: 2 PID: 35157 at fs/bcachefs/btree_iter.c:3028 bch2_trans_srcu_unlock+0x122/0x130
[ 3250.328236] Modules linked in:
[ 3250.328231] WARNING: CPU: 5 PID: 35164 at fs/bcachefs/btree_iter.c:3028 bch2_trans_srcu_unlock+0x122/0x130
[ 3250.328238]  nft_masq
[ 3250.328230] WARNING: CPU: 8 PID: 33722 at fs/bcachefs/btree_iter.c:3028 bch2_trans_put+0x1f7/0x230
[ 3250.328240]  nft_masq
[ 3250.328239]  nf_nat
[ 3250.328235] WARNING: CPU: 18 PID: 33724 at fs/bcachefs/btree_iter.c:3028 bch2_trans_put+0x1f7/0x230
[ 3250.328241] Modules linked in:
[ 3250.328239]  nft_masq
[ 3250.328247]  xt_MASQUERADE
[ 3250.328247]  xt_conntrack
[ 3250.328247]  xt_MASQUERADE
[ 3250.328249] Modules linked in:
[ 3250.328248]  nft_masq
[ 3250.328249]  xt_MASQUERADE
[ 3250.328255]  nft_chain_nat
[ 3250.328257]  xt_MASQUERADE
[ 3250.328256]  nft_chain_nat
[ 3250.328259] Modules linked in:
[ 3250.328259] Modules linked in:
[ 3250.328254]  nft_chain_nat
[ 3250.328262]  nf_nat
[ 3250.328263]  nft_chain_nat
[ 3250.328264]  bridge
[ 3250.328268]  nf_nat

    ... long list of modules ...

[ 3250.329907]  <TASK>
[ 3250.329905]  ? bch2_trans_srcu_unlock+0x122/0x130
[ 3250.329907] FS:  00007d5d45b03740(0000) GS:ffff9abcdfa80000(0000) knlGS:0000000000000000
[ 3250.329905]  ? handle_bug+0x6e/0xb0
[ 3250.329905]  ? handle_bug+0x6e/0xb0
[ 3250.329908]  ? bch2_trans_srcu_unlock+0x122/0x130
[ 3250.329907] CPU: 7 UID: 0 PID: 34849 Comm: kworker/u81:5 Not tainted 6.12.0 #1
[ 3250.329913] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3250.329914] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3250.329919] Hardware name: System manufacturer System Product Name/PRIME X299-DELUXE, BIOS 1503 08/03/2018
[ 3250.329920] CR2: 00007ab015e0bd58 CR3: 0000000253d90005 CR4: 00000000003726f0
[ 3250.329917]  ? report_bug+0x182/0x1b0
[ 3250.329920] CR2: 00007d44eba57ff0 CR3: 0000000222612006 CR4: 00000000003726f0
[ 3250.329919]  bch2_trans_begin+0x56c/0x710
[ 3250.329919]  ? exc_invalid_op+0x18/0x80
[ 3250.329921]  __writeback_inodes_wb+0x54/0xf0
[ 3250.329921]  ? exc_invalid_op+0x18/0x80
[ 3250.329926] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 3250.329927] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 3250.329924]  ? show_regs+0x6c/0x80
[ 3250.329932] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 3250.329933] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 3250.329932]  bch2_do_discards_work+0x195/0x410
[ 3250.329932]  wb_writeback+0x192/0x310
[ 3250.329930]  ? handle_bug+0x6e/0xb0
[ 3250.329932]  ? asm_exc_invalid_op+0x1b/0x20
[ 3250.329933]  ? asm_exc_invalid_op+0x1b/0x20
[ 3250.329938] Call Trace:
[ 3250.329935]  ? __warn+0x8d/0x150
[ 3250.329940] Call Trace:
[ 3250.329945]  <TASK>
[ 3250.329946]  <TASK>
[ 3250.329946]  wb_workfn+0x2a3/0x3f0
[ 3250.329945]  ? exc_invalid_op+0x18/0x80
[ 3250.329946]  ? bch2_trans_srcu_unlock+0x122/0x130
[ 3250.329950]  ? show_regs+0x6c/0x80
[ 3250.329949]  process_one_work+0x178/0x3d0
[ 3250.329950]  ? bch2_trans_put+0x1f7/0x230
[ 3250.329950]  ? bch2_trans_put+0x1f7/0x230
[ 3250.329949] Workqueue: bcachefs bch2_write_point_do_index_updates
[ 3250.329953]  ? show_regs+0x6c/0x80
[ 3250.329958]  ? asm_exc_invalid_op+0x1b/0x20
[ 3250.329959]  ? report_bug+0x182/0x1b0
[ 3250.329960]  worker_thread+0x2b8/0x3e0

[ 3250.329963]  ? __warn+0x8d/0x150
[ 3250.329963]  process_one_work+0x178/0x3d0
[ 3250.329964]  bch2_data_update_index_update+0x3c/0x70
[ 3250.329964]  bch2_data_update_index_update+0x3c/0x70
[ 3250.329966]  ? __warn+0x8d/0x150
[ 3250.329971]  ? __pfx_worker_thread+0x10/0x10
[ 3250.329971]  ? handle_bug+0x6e/0xb0
[ 3250.329974]  ? bch2_trans_srcu_unlock+0x122/0x130
[ 3250.329972] RIP: 0010:bch2_trans_put+0x1f7/0x230
[ 3250.329976]  ? bch2_trans_srcu_unlock+0x122/0x130
[ 3250.329976]  __bch2_write_index+0x64/0x2d0
[ 3250.329977]  __bch2_write_index+0x64/0x2d0
[ 3250.329979]  ? bch2_trans_srcu_unlock+0x122/0x130
[ 3250.329980]  worker_thread+0x2b8/0x3e0
[ 3250.329984]  ? report_bug+0x182/0x1b0
[ 3250.329983]  ? exc_invalid_op+0x18/0x80
[ 3250.329986]  kthread+0xe1/0x110
[ 3250.329986] Code: 8f 89 f0 01 48 c7 c7 38 22 4b 93 48 b8 cf f7 53 e3 a5 9b c4 20 48 29 ca 48 d1 ea 48 f7 e2 48 89 d6 48 c1 ee 04 e8 79 15 a0 ff <0f> 0b 41 8b b4 24 b0 00 00 00 49 8d be 18 38 00 00 83 fe 01 77 1e
[ 3250.329991]  ? __pfx_worker_thread+0x10/0x10
[ 3250.329994]  ? __pfx_kthread+0x10/0x10
[ 3250.329993]  ? report_bug+0x182/0x1b0
[ 3250.329995] RSP: 0000:ffffb06b60adfd80 EFLAGS: 00010246
[ 3250.329994]  bch2_write_point_do_index_updates+0xa1/0x190
[ 3250.329995]  ? asm_exc_invalid_op+0x1b/0x20
[ 3250.329994]  bch2_write_point_do_index_updates+0xa1/0x190
[ 3250.329995]  ? handle_bug+0x6e/0xb0

[ 3250.330003]  kthread+0xe1/0x110
[ 3250.330000]  bch2_trans_begin+0x56c/0x710
[ 3250.330003]  ? handle_bug+0x6e/0xb0
[ 3250.330005]  ret_from_fork+0x44/0x70
[ 3250.330006] RAX: 0000000000000000 RBX: ffff9ab17d410000 RCX: 0000000000000000
[ 3250.330007]  ? exc_invalid_op+0x18/0x80
[ 3250.330012]  ? __pfx_kthread+0x10/0x10
[ 3250.330009]  process_one_work+0x178/0x3d0
[ 3250.330012]  ? __pfx_kthread+0x10/0x10
[ 3250.330012]  process_one_work+0x178/0x3d0
[ 3250.330018] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 3250.330017]  ? exc_invalid_op+0x18/0x80
[ 3250.330020]  ret_from_fork+0x44/0x70
[ 3250.330022]  ? asm_exc_invalid_op+0x1b/0x20
[ 3250.330020]  __bch2_create+0xed/0x410
[ 3250.330021]  ret_from_fork_asm+0x1a/0x30
[ 3250.330023]  worker_thread+0x2b8/0x3e0
[ 3250.330025]  worker_thread+0x2b8/0x3e0
[ 3250.330029]  ? __pfx_kthread+0x10/0x10
[ 3250.330027]  ? asm_exc_invalid_op+0x1b/0x20
[ 3250.330029]  ? bch2_trans_srcu_unlock+0x122/0x130
[ 3250.330031] RBP: ffffb06b60adfda0 R08: 0000000000000000 R09: 0000000000000000
[ 3250.330034]  ? __pfx_worker_thread+0x10/0x10
[ 3250.330035]  ? bch2_trans_srcu_unlock+0x122/0x130
[ 3250.330039] R10: 0000000000000000 R11: 0000000000000000 R12: ffff9ab17d410000
[ 3250.330037]  ret_from_fork_asm+0x1a/0x30
[ 3250.330039]  ? __pfx_worker_thread+0x10/0x10
[ 3250.330040]  ? bch2_trans_srcu_unlock+0x122/0x130
[ 3250.330042]  </TASK>
[ 3250.330045]  kthread+0xe1/0x110
[ 3250.330048] ---[ end trace 0000000000000000 ]---
[ 3250.330048] R13: ffff9aad8ff40000 R14: ffff9aad8ff40000 R15: dead000000000100
[ 3250.330049]  kthread+0xe1/0x110
[ 3250.330048]  bch2_trans_begin+0x56c/0x710
[ 3250.330052]  </TASK>
[ 3250.330052]  bch2_create+0x26/0x70
[ 3250.330051]  bch2_trans_begin+0x56c/0x710
[ 3250.330054]  ? __pfx_kthread+0x10/0x10
[ 3250.330055] FS:  0000000000000000(0000) GS:ffff9abcdfb80000(0000) knlGS:0000000000000000
[ 3250.330057]  ? __pfx_kthread+0x10/0x10
[ 3250.330060]  bch2_trans_begin+0x56c/0x710
[ 3250.330062]  ? bch2_create+0x26/0x70
[ 3250.330064] ---[ end trace 0000000000000000 ]---
[ 3250.330063] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3250.330063]  ret_from_fork+0x44/0x70
[ 3250.330065]  ret_from_fork+0x44/0x70
[ 3250.330065]  bch2_logged_op_start+0x1e/0xb0
[ 3250.330106]  ? __pfx_kthread+0x10/0x10
[ 3250.330107]  ? __pfx_kthread+0x10/0x10
[ 3250.330105]  bch2_bucket_io_time_reset+0x93/0xb0
[ 3250.330113]  bch2_truncate+0xa2/0x130
[ 3250.330116] CR2: 00007ab019216000 CR3: 000000026e23e003 CR4: 00000000003726f0
[ 3250.330116]  __bch2_read_extent+0x870/0x14a0
[ 3250.330117]  ret_from_fork_asm+0x1a/0x30
[ 3250.330116]  ret_from_fork_asm+0x1a/0x30
[ 3250.330118]  ? sched_clock_noinstr+0x9/0x10
[ 3250.330115]  path_openat+0x10d3/0x1360
[ 3250.330131] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 3250.330130]  bchfs_truncate+0x298/0x440
[ 3250.330131]  bchfs_read+0x74/0xba0
[ 3250.330134]  </TASK>
[ 3250.330135]  </TASK>
[ 3250.330138] ---[ end trace 0000000000000000 ]---
[ 3250.330136]  do_filp_open+0xac/0x160
[ 3250.330138] ---[ end trace 0000000000000000 ]---
[ 3250.330139]  bchfs_read+0x985/0xba0
[ 3250.330145] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 3250.330147]  ? setattr_prepare+0x30/0x2c0
[ 3250.330147]  ? mempool_alloc_slab+0x15/0x20
[ 3250.330152] Call Trace:
[ 3250.330157]  <TASK>
[ 3250.330152]  ? kmem_cache_alloc_noprof+0xf9/0x2f0
[ 3250.330158]  ? kmem_cache_alloc_noprof+0xf9/0x2f0
[ 3250.330160]  ? bio_alloc_bioset+0x20e/0x540
[ 3250.330163]  ? show_regs+0x6c/0x80
[ 3250.330165]  bch2_setattr+0x70/0xa0
[ 3250.330171]  ? bio_add_page+0x7d/0xd0
[ 3250.330171]  do_sys_openat2+0x9c/0xd0
[ 3250.330174]  ? bvec_alloc+0x89/0xc0
[ 3250.330177]  notify_change+0x1ed/0x520
[ 3250.330179]  bch2_readahead+0x29c/0x400
[ 3250.330179]  ? __warn+0x8d/0x150
[ 3250.330186]  __x64_sys_openat+0x6c/0xa0
[ 3250.330187]  ? bio_alloc_bioset+0x2db/0x540
[ 3250.330185]  ? __filemap_add_folio+0x267/0x440
[ 3250.330191]  do_truncate+0x83/0xe0
[ 3250.330192]  ? bch2_trans_put+0x1f7/0x230
[ 3250.330199]  ? bio_add_page+0x7d/0xd0
[ 3250.330198]  x64_sys_call+0x1859/0x24e0
[ 3250.330201]  ? do_truncate+0x83/0xe0
[ 3250.330201]  read_pages+0x6a/0x270
[ 3250.330209]  bch2_readahead+0x29c/0x400
[ 3250.330208]  ? report_bug+0x182/0x1b0
[ 3250.330209]  do_syscall_64+0x70/0x130
[ 3250.330215]  path_openat+0x100d/0x1360
[ 3250.330217]  ? __filemap_add_folio+0x267/0x440
[ 3250.330219]  page_cache_ra_order+0x2ca/0x3b0
[ 3250.330218]  ? do_anonymous_page+0x108/0x8f0
[ 3250.330221]  ? handle_bug+0x6e/0xb0
[ 3250.330231]  ? do_syscall_64+0x7c/0x130
[ 3250.330233]  ? exc_invalid_op+0x18/0x80
[ 3250.330234]  do_filp_open+0xac/0x160
[ 3250.330235]  filemap_fault+0x636/0xe80
[ 3250.330238]  read_pages+0x6a/0x270
[ 3250.330242]  ? __pte_offset_map+0x1c/0x1b0
[ 3250.330245]  ? filemap_map_pages+0x56b/0x6e0
[ 3250.330247]  ? asm_exc_invalid_op+0x1b/0x20
[ 3250.330249]  ? kmem_cache_alloc_noprof+0xf9/0x2f0
[ 3250.330255]  ? __handle_mm_fault+0xbf6/0x1090
[ 3250.330257]  page_cache_ra_order+0x26f/0x3b0
[ 3250.330259]  bch2_page_fault+0x70/0x1c0
[ 3250.330261]  ? bch2_trans_put+0x1f7/0x230
[ 3250.330265]  do_sys_openat2+0x9c/0xd0
[ 3250.330271]  __do_fault+0x3a/0x180
[ 3250.330273]  filemap_fault+0x636/0xe80
[ 3250.330277]  __x64_sys_openat+0x6c/0xa0
[ 3250.330276]  ? __count_memcg_events+0x85/0x160
[ 3250.330278]  bch2_data_update_index_update+0x3c/0x70
[ 3250.330283]  do_fault+0x2cd/0x4f0
[ 3250.330285]  ? filemap_map_pages+0x56b/0x6e0
[ 3250.330288]  ? count_memcg_events.constprop.0+0x2a/0x50
[ 3250.330290]  __bch2_write_index+0x64/0x2d0
[ 3250.330292]  x64_sys_call+0x1859/0x24e0
[ 3250.330297]  __handle_mm_fault+0x824/0x1090
[ 3250.330299]  bch2_page_fault+0x70/0x1c0
[ 3250.330300]  ? handle_mm_fault+0xaf/0x2e0
[ 3250.330302]  do_syscall_64+0x70/0x130
[ 3250.330307]  bch2_write_point_do_index_updates+0xa1/0x190
[ 3250.330311]  __do_fault+0x3a/0x180
[ 3250.330313]  ? vfs_read+0x16c/0x350
[ 3250.330312]  ? do_user_addr_fault+0x5a7/0x7f0
[ 3250.330315]  ? __count_memcg_events+0x85/0x160
[ 3250.330322]  do_fault+0x2cd/0x4f0
[ 3250.330325]  ? count_memcg_events.constprop.0+0x2a/0x50
[ 3250.330325]  process_one_work+0x178/0x3d0
[ 3250.330326]  handle_mm_fault+0x185/0x2e0
[ 3250.330327]  ? irqentry_exit_to_user_mode+0x33/0x180
[ 3250.330333]  ? handle_mm_fault+0xaf/0x2e0
[ 3250.330335]  __handle_mm_fault+0x824/0x1090
[ 3250.330337]  do_user_addr_fault+0x2dc/0x7f0
[ 3250.330338]  ? irqentry_exit+0x43/0x50
[ 3250.330341]  worker_thread+0x2b8/0x3e0
[ 3250.330348]  ? exc_page_fault+0x93/0x1d0
[ 3250.330349]  ? do_user_addr_fault+0x5a7/0x7f0
[ 3250.330351]  exc_page_fault+0x82/0x1d0
[ 3250.330353]  handle_mm_fault+0x185/0x2e0
[ 3250.330353]  ? __pfx_worker_thread+0x10/0x10
[ 3250.330359]  ? irqentry_exit_to_user_mode+0x33/0x180
[ 3250.330361]  asm_exc_page_fault+0x27/0x30
[ 3250.330363]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 3250.330366]  kthread+0xe1/0x110
[ 3250.330368] RIP: 0033:0x64d168906311
[ 3250.330370]  do_user_addr_fault+0x2dc/0x7f0
[ 3250.330374]  ? irqentry_exit+0x43/0x50
[ 3250.330375]  ? __pfx_kthread+0x10/0x10
[ 3250.330376] RIP: 0033:0x7895b9919e92
[ 3250.330378] Code: 8b 40 10 83 c0 04 83 7f 78 01 74 2d 53 48 89 fb 8b 7f 48 48 8d 8a 00 04 00 00 48 0f af c7 48 8d 4c 01 08 8d 04 b5 00 00 00 00 <8b> 04 01 0f c8 85 c0 78 1e 89 c0 5b c3 66 90 89 f6 48 0f af f0 8b
[ 3250.330384] RSP: 002b:00007ffe6a691550 EFLAGS: 00010206
[ 3250.330383]  ? exc_page_fault+0x93/0x1d0
[ 3250.330383]  exc_page_fault+0x82/0x1d0
[ 3250.330385] Code: 83 e2 40 75 53 89 f0 f7 d0 a9 00 00 41 00 74 48 80 3d c1 c6 0e 00 00 74 6c 89 da 48 89 ee bf 9c ff ff ff b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 0f 87 92 00 00 00 48 8b 54 24 28 64 48 2b 14 25
[ 3250.330387]  ret_from_fork+0x44/0x70

[ 3250.330393] RSP: 002b:00007ffc38678260 EFLAGS: 00000202
[ 3250.330394] RAX: 00000000002420c8 RBX: 000064d173383990 RCX: 00007d5d443205e0
[ 3250.330396]  asm_exc_page_fault+0x27/0x30
[ 3250.330394]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 3250.330398]  ? __pfx_kthread+0x10/0x10
[ 3250.330402]  ORIG_RAX: 0000000000000101
[ 3250.330402] RDX: 00007d5d42e00000 RSI: 0000000000090832 RDI: 00000000000e1569
[ 3250.330406] RIP: 0033:0x7d41e0519e92
[ 3250.330408] RAX: ffffffffffffffda RBX: 00000000000800c2 RCX: 00007895b9919e92
[ 3250.330408] RBP: 000064d1731eb688 R08: 00007ffe6a69156c R09: 0000000000000007
[ 3250.330410] RIP: 0033:0x5a9820711605
[ 3250.330409]  ret_from_fork_asm+0x1a/0x30
[ 3250.330415] R10: 000064d1731eb688 R11: 00007ffe6a691600 R12: 000064d168ae5420
[ 3250.330417] RDX: 00000000000800c2 RSI: 00005bba03e24ef0 RDI: 00000000ffffff9c
[ 3250.330417] Code: 2b 14 25 28 00 00 00 75 12 48 81 c4 98 00 00 00 5b 5d 41 5c 41 5d 41 5e 41 5f c3 e8 15 04 e1 ff 0f 1f 44 00 00 f3 0f 1e fa 53 <0f> b6 07 49 89 f1 48 89 d6 48 89 cb 89 c2 49 89 c2 c1 ea 04 41 83
[ 3250.330418] Code: 83 e2 40 75 53 89 f0 f7 d0 a9 00 00 41 00 74 48 80 3d c1 c6 0e 00 00 74 6c 89 da 48 89 ee bf 9c ff ff ff b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 0f 87 92 00 00 00 48 8b 54 24 28 64 48 2b 14 25
[ 3250.330420] R13: 000064d1731eb688 R14: 000064d173383990 R15: 0000000000000000
[ 3250.330423] RBP: 00005bba03e24ef0 R08: ffffffffffffffc0 R09: 0000000000000080
[ 3250.330425] RSP: 002b:00007fff707ceb40 EFLAGS: 00000202
[ 3250.330425] RSP: 002b:00007ffe5033ed10 EFLAGS: 00010202
[ 3250.330429]  </TASK>

[ 3250.330433]  ORIG_RAX: 0000000000000101
[ 3250.330434]  </TASK>
[ 3250.330434] ---[ end trace 0000000000000000 ]---
[ 3250.330436] RAX: 00007a3d6182f7e6 RBX: 000000000002f7e6 RCX: 00007ffe5033ed40
[ 3250.330438] RAX: ffffffffffffffda RBX: 0000000000000241 RCX: 00007d41e0519e92
[ 3250.330440] ---[ end trace 0000000000000000 ]---
[ 3250.330441] R10: 00000000000001b6 R11: 0000000000000202 R12: 00005bba03e24e30
[ 3250.330444] RDX: 0000000000000241 RSI: 00005a1110bf3160 RDI: 00000000ffffff9c
[ 3250.330448] R13: 00005bb9ff49f420 R14: 00005bba03e21300 R15: 00005bb9ff4784f0
[ 3250.330449] RDX: 00007ffe5033ed50 RSI: 00000000039a89d1 RDI: 00007a3d6182f7e6
[ 3250.330452] RBP: 00005a1110bf3160 R08: 0000000000000004 R09: 0000000000000001
[ 3250.330455] RBP: 00005a984dbef4e0 R08: 00007ffe5033ec8c R09: 0000000000000020
[ 3250.330459] R10: 00000000000001b6 R11: 0000000000000202 R12: 00005a1110bf5c00
[ 3250.330461] R10: 00005a984d9be948 R11: 00007ffe5033ed20 R12: 00005a98208ec420
[ 3250.330465]  </TASK>
[ 3250.330465] R13: 00005a1110bf3160 R14: 0000000000000003 R15: 0000000000000000
[ 3250.330467] R13: 0000000000000000 R14: 00007ffe5033ed48 R15: 00007ffe5033ef00
[ 3250.330472] ---[ end trace 0000000000000000 ]---
[ 3250.330478]  </TASK>
[ 3250.330482]  </TASK>

...

[ 3250.329907]  <TASK>
[ 3250.329905]  ? bch2_trans_srcu_unlock+0x122/0x130
[ 3250.329907] FS:  00007d5d45b03740(0000) GS:ffff9abcdfa80000(0000) knlGS:0000000000000000
[ 3250.329905]  ? handle_bug+0x6e/0xb0
[ 3250.329905]  ? handle_bug+0x6e/0xb0
[ 3250.329908]  ? bch2_trans_srcu_unlock+0x122/0x130
[ 3250.329907] CPU: 7 UID: 0 PID: 34849 Comm: kworker/u81:5 Not tainted 6.12.0 #1
[ 3250.329913] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3250.329914] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3250.329919] Hardware name: System manufacturer System Product Name/PRIME X299-DELUXE, BIOS 1503 08/03/2018
[ 3250.329920] CR2: 00007ab015e0bd58 CR3: 0000000253d90005 CR4: 00000000003726f0
[ 3250.329917]  ? report_bug+0x182/0x1b0
[ 3250.329920] CR2: 00007d44eba57ff0 CR3: 0000000222612006 CR4: 00000000003726f0
[ 3250.329919]  bch2_trans_begin+0x56c/0x710
[ 3250.329919]  ? exc_invalid_op+0x18/0x80
[ 3250.329921]  __writeback_inodes_wb+0x54/0xf0
[ 3250.329921]  ? exc_invalid_op+0x18/0x80
[ 3250.329926] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 3250.329927] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 3250.329924]  ? show_regs+0x6c/0x80
[ 3250.329932] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 3250.329933] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 3250.329932]  bch2_do_discards_work+0x195/0x410
[ 3250.329932]  wb_writeback+0x192/0x310
[ 3250.329930]  ? handle_bug+0x6e/0xb0
[ 3250.329932]  ? asm_exc_invalid_op+0x1b/0x20
[ 3250.329933]  ? asm_exc_invalid_op+0x1b/0x20
[ 3250.329938] Call Trace:
[ 3250.329935]  ? __warn+0x8d/0x150
[ 3250.329940] Call Trace:
[ 3250.329945]  <TASK>
[ 3250.329946]  <TASK>
[ 3250.329946]  wb_workfn+0x2a3/0x3f0
[ 3250.329945]  ? exc_invalid_op+0x18/0x80
[ 3250.329946]  ? bch2_trans_srcu_unlock+0x122/0x130
[ 3250.329950]  ? show_regs+0x6c/0x80
[ 3250.329949]  process_one_work+0x178/0x3d0
[ 3250.329950]  ? bch2_trans_put+0x1f7/0x230
[ 3250.329950]  ? bch2_trans_put+0x1f7/0x230
[ 3250.329949] Workqueue: bcachefs bch2_write_point_do_index_updates
[ 3250.329953]  ? show_regs+0x6c/0x80
[ 3250.329958]  ? asm_exc_invalid_op+0x1b/0x20
[ 3250.329959]  ? report_bug+0x182/0x1b0
[ 3250.329960]  worker_thread+0x2b8/0x3e0

[ 3250.329963]  ? __warn+0x8d/0x150
[ 3250.329963]  process_one_work+0x178/0x3d0
[ 3250.329964]  bch2_data_update_index_update+0x3c/0x70
[ 3250.329964]  bch2_data_update_index_update+0x3c/0x70
[ 3250.329966]  ? __warn+0x8d/0x150
[ 3250.329971]  ? __pfx_worker_thread+0x10/0x10
[ 3250.329971]  ? handle_bug+0x6e/0xb0
[ 3250.329974]  ? bch2_trans_srcu_unlock+0x122/0x130
[ 3250.329972] RIP: 0010:bch2_trans_put+0x1f7/0x230
[ 3250.329976]  ? bch2_trans_srcu_unlock+0x122/0x130
[ 3250.329976]  __bch2_write_index+0x64/0x2d0
[ 3250.329977]  __bch2_write_index+0x64/0x2d0
[ 3250.329979]  ? bch2_trans_srcu_unlock+0x122/0x130
[ 3250.329980]  worker_thread+0x2b8/0x3e0
[ 3250.329984]  ? report_bug+0x182/0x1b0
[ 3250.329983]  ? exc_invalid_op+0x18/0x80
[ 3250.329986]  kthread+0xe1/0x110
[ 3250.329986] Code: 8f 89 f0 01 48 c7 c7 38 22 4b 93 48 b8 cf f7 53 e3 a5 9b c4 20 48 29 ca 48 d1 ea 48 f7 e2 48 89 d6 48 c1 ee 04 e8 79 15 a0 ff <0f> 0b 41 8b b4 24 b0 00 00 00 49 8d be 18 38 00 00 83 fe 01 77 1e
[ 3250.329991]  ? __pfx_worker_thread+0x10/0x10
[ 3250.329994]  ? __pfx_kthread+0x10/0x10
[ 3250.329993]  ? report_bug+0x182/0x1b0
[ 3250.329995] RSP: 0000:ffffb06b60adfd80 EFLAGS: 00010246
[ 3250.329994]  bch2_write_point_do_index_updates+0xa1/0x190
[ 3250.329995]  ? asm_exc_invalid_op+0x1b/0x20
[ 3250.329994]  bch2_write_point_do_index_updates+0xa1/0x190
[ 3250.329995]  ? handle_bug+0x6e/0xb0

[ 3250.330003]  kthread+0xe1/0x110
[ 3250.330000]  bch2_trans_begin+0x56c/0x710
[ 3250.330003]  ? handle_bug+0x6e/0xb0
[ 3250.330005]  ret_from_fork+0x44/0x70
[ 3250.330006] RAX: 0000000000000000 RBX: ffff9ab17d410000 RCX: 0000000000000000
[ 3250.330007]  ? exc_invalid_op+0x18/0x80
[ 3250.330012]  ? __pfx_kthread+0x10/0x10
[ 3250.330009]  process_one_work+0x178/0x3d0
[ 3250.330012]  ? __pfx_kthread+0x10/0x10
[ 3250.330012]  process_one_work+0x178/0x3d0
[ 3250.330018] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 3250.330017]  ? exc_invalid_op+0x18/0x80
[ 3250.330020]  ret_from_fork+0x44/0x70
[ 3250.330022]  ? asm_exc_invalid_op+0x1b/0x20
[ 3250.330020]  __bch2_create+0xed/0x410
[ 3250.330021]  ret_from_fork_asm+0x1a/0x30
[ 3250.330023]  worker_thread+0x2b8/0x3e0
[ 3250.330025]  worker_thread+0x2b8/0x3e0
[ 3250.330029]  ? __pfx_kthread+0x10/0x10
[ 3250.330027]  ? asm_exc_invalid_op+0x1b/0x20
[ 3250.330029]  ? bch2_trans_srcu_unlock+0x122/0x130
[ 3250.330031] RBP: ffffb06b60adfda0 R08: 0000000000000000 R09: 0000000000000000
[ 3250.330034]  ? __pfx_worker_thread+0x10/0x10
[ 3250.330035]  ? bch2_trans_srcu_unlock+0x122/0x130
[ 3250.330039] R10: 0000000000000000 R11: 0000000000000000 R12: ffff9ab17d410000
[ 3250.330037]  ret_from_fork_asm+0x1a/0x30
[ 3250.330039]  ? __pfx_worker_thread+0x10/0x10
[ 3250.330040]  ? bch2_trans_srcu_unlock+0x122/0x130
[ 3250.330042]  </TASK>
[ 3250.330045]  kthread+0xe1/0x110
[ 3250.330048] ---[ end trace 0000000000000000 ]---
[ 3250.330048] R13: ffff9aad8ff40000 R14: ffff9aad8ff40000 R15: dead000000000100
[ 3250.330049]  kthread+0xe1/0x110
[ 3250.330048]  bch2_trans_begin+0x56c/0x710
[ 3250.330052]  </TASK>
[ 3250.330052]  bch2_create+0x26/0x70
[ 3250.330051]  bch2_trans_begin+0x56c/0x710
[ 3250.330054]  ? __pfx_kthread+0x10/0x10
[ 3250.330055] FS:  0000000000000000(0000) GS:ffff9abcdfb80000(0000) knlGS:0000000000000000
[ 3250.330057]  ? __pfx_kthread+0x10/0x10
[ 3250.330060]  bch2_trans_begin+0x56c/0x710
[ 3250.330062]  ? bch2_create+0x26/0x70
[ 3250.330064] ---[ end trace 0000000000000000 ]---
[ 3250.330063] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3250.330063]  ret_from_fork+0x44/0x70
[ 3250.330065]  ret_from_fork+0x44/0x70
[ 3250.330065]  bch2_logged_op_start+0x1e/0xb0
[ 3250.330106]  ? __pfx_kthread+0x10/0x10
[ 3250.330107]  ? __pfx_kthread+0x10/0x10
[ 3250.330105]  bch2_bucket_io_time_reset+0x93/0xb0
[ 3250.330113]  bch2_truncate+0xa2/0x130
[ 3250.330116] CR2: 00007ab019216000 CR3: 000000026e23e003 CR4: 00000000003726f0
[ 3250.330116]  __bch2_read_extent+0x870/0x14a0
[ 3250.330117]  ret_from_fork_asm+0x1a/0x30
[ 3250.330116]  ret_from_fork_asm+0x1a/0x30
[ 3250.330118]  ? sched_clock_noinstr+0x9/0x10
[ 3250.330115]  path_openat+0x10d3/0x1360
[ 3250.330131] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 3250.330130]  bchfs_truncate+0x298/0x440
[ 3250.330131]  bchfs_read+0x74/0xba0
[ 3250.330134]  </TASK>
[ 3250.330135]  </TASK>
[ 3250.330138] ---[ end trace 0000000000000000 ]---
[ 3250.330136]  do_filp_open+0xac/0x160
[ 3250.330138] ---[ end trace 0000000000000000 ]---
[ 3250.330139]  bchfs_read+0x985/0xba0
[ 3250.330145] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 3250.330147]  ? setattr_prepare+0x30/0x2c0
[ 3250.330147]  ? mempool_alloc_slab+0x15/0x20
[ 3250.330152] Call Trace:
[ 3250.330157]  <TASK>
[ 3250.330152]  ? kmem_cache_alloc_noprof+0xf9/0x2f0
[ 3250.330158]  ? kmem_cache_alloc_noprof+0xf9/0x2f0
[ 3250.330160]  ? bio_alloc_bioset+0x20e/0x540
[ 3250.330163]  ? show_regs+0x6c/0x80
[ 3250.330165]  bch2_setattr+0x70/0xa0
[ 3250.330171]  ? bio_add_page+0x7d/0xd0
[ 3250.330171]  do_sys_openat2+0x9c/0xd0
[ 3250.330174]  ? bvec_alloc+0x89/0xc0
[ 3250.330177]  notify_change+0x1ed/0x520
[ 3250.330179]  bch2_readahead+0x29c/0x400
[ 3250.330179]  ? __warn+0x8d/0x150
[ 3250.330186]  __x64_sys_openat+0x6c/0xa0
[ 3250.330187]  ? bio_alloc_bioset+0x2db/0x540
[ 3250.330185]  ? __filemap_add_folio+0x267/0x440
[ 3250.330191]  do_truncate+0x83/0xe0
[ 3250.330192]  ? bch2_trans_put+0x1f7/0x230
[ 3250.330199]  ? bio_add_page+0x7d/0xd0
[ 3250.330198]  x64_sys_call+0x1859/0x24e0
[ 3250.330201]  ? do_truncate+0x83/0xe0
[ 3250.330201]  read_pages+0x6a/0x270
[ 3250.330209]  bch2_readahead+0x29c/0x400
[ 3250.330208]  ? report_bug+0x182/0x1b0
[ 3250.330209]  do_syscall_64+0x70/0x130
[ 3250.330215]  path_openat+0x100d/0x1360
[ 3250.330217]  ? __filemap_add_folio+0x267/0x440
[ 3250.330219]  page_cache_ra_order+0x2ca/0x3b0
[ 3250.330218]  ? do_anonymous_page+0x108/0x8f0
[ 3250.330221]  ? handle_bug+0x6e/0xb0
[ 3250.330231]  ? do_syscall_64+0x7c/0x130
[ 3250.330233]  ? exc_invalid_op+0x18/0x80
[ 3250.330234]  do_filp_open+0xac/0x160
[ 3250.330235]  filemap_fault+0x636/0xe80
[ 3250.330238]  read_pages+0x6a/0x270
[ 3250.330242]  ? __pte_offset_map+0x1c/0x1b0
[ 3250.330245]  ? filemap_map_pages+0x56b/0x6e0
[ 3250.330247]  ? asm_exc_invalid_op+0x1b/0x20
[ 3250.330249]  ? kmem_cache_alloc_noprof+0xf9/0x2f0
[ 3250.330255]  ? __handle_mm_fault+0xbf6/0x1090
[ 3250.330257]  page_cache_ra_order+0x26f/0x3b0
[ 3250.330259]  bch2_page_fault+0x70/0x1c0
[ 3250.330261]  ? bch2_trans_put+0x1f7/0x230
[ 3250.330265]  do_sys_openat2+0x9c/0xd0
[ 3250.330271]  __do_fault+0x3a/0x180
[ 3250.330273]  filemap_fault+0x636/0xe80
[ 3250.330277]  __x64_sys_openat+0x6c/0xa0
[ 3250.330276]  ? __count_memcg_events+0x85/0x160
[ 3250.330278]  bch2_data_update_index_update+0x3c/0x70
[ 3250.330283]  do_fault+0x2cd/0x4f0
[ 3250.330285]  ? filemap_map_pages+0x56b/0x6e0
[ 3250.330288]  ? count_memcg_events.constprop.0+0x2a/0x50
[ 3250.330290]  __bch2_write_index+0x64/0x2d0
[ 3250.330292]  x64_sys_call+0x1859/0x24e0
[ 3250.330297]  __handle_mm_fault+0x824/0x1090
[ 3250.330299]  bch2_page_fault+0x70/0x1c0
[ 3250.330300]  ? handle_mm_fault+0xaf/0x2e0
[ 3250.330302]  do_syscall_64+0x70/0x130
[ 3250.330307]  bch2_write_point_do_index_updates+0xa1/0x190
[ 3250.330311]  __do_fault+0x3a/0x180
[ 3250.330313]  ? vfs_read+0x16c/0x350
[ 3250.330312]  ? do_user_addr_fault+0x5a7/0x7f0
[ 3250.330315]  ? __count_memcg_events+0x85/0x160
[ 3250.330322]  do_fault+0x2cd/0x4f0
[ 3250.330325]  ? count_memcg_events.constprop.0+0x2a/0x50
[ 3250.330325]  process_one_work+0x178/0x3d0
[ 3250.330326]  handle_mm_fault+0x185/0x2e0
[ 3250.330327]  ? irqentry_exit_to_user_mode+0x33/0x180
[ 3250.330333]  ? handle_mm_fault+0xaf/0x2e0
[ 3250.330335]  __handle_mm_fault+0x824/0x1090
[ 3250.330337]  do_user_addr_fault+0x2dc/0x7f0
[ 3250.330338]  ? irqentry_exit+0x43/0x50
[ 3250.330341]  worker_thread+0x2b8/0x3e0
[ 3250.330348]  ? exc_page_fault+0x93/0x1d0
[ 3250.330349]  ? do_user_addr_fault+0x5a7/0x7f0
[ 3250.330351]  exc_page_fault+0x82/0x1d0
[ 3250.330353]  handle_mm_fault+0x185/0x2e0
[ 3250.330353]  ? __pfx_worker_thread+0x10/0x10
[ 3250.330359]  ? irqentry_exit_to_user_mode+0x33/0x180
[ 3250.330361]  asm_exc_page_fault+0x27/0x30
[ 3250.330363]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 3250.330366]  kthread+0xe1/0x110
[ 3250.330368] RIP: 0033:0x64d168906311
[ 3250.330370]  do_user_addr_fault+0x2dc/0x7f0
[ 3250.330374]  ? irqentry_exit+0x43/0x50
[ 3250.330375]  ? __pfx_kthread+0x10/0x10
[ 3250.330376] RIP: 0033:0x7895b9919e92
[ 3250.330378] Code: 8b 40 10 83 c0 04 83 7f 78 01 74 2d 53 48 89 fb 8b 7f 48 48 8d 8a 00 04 00 00 48 0f af c7 48 8d 4c 01 08 8d 04 b5 00 00 00 00 <8b> 04 01 0f c8 85 c0 78 1e 89 c0 5b c3 66 90 89 f6 48 0f af f0 8b
[ 3250.330384] RSP: 002b:00007ffe6a691550 EFLAGS: 00010206
[ 3250.330383]  ? exc_page_fault+0x93/0x1d0
[ 3250.330383]  exc_page_fault+0x82/0x1d0
[ 3250.330385] Code: 83 e2 40 75 53 89 f0 f7 d0 a9 00 00 41 00 74 48 80 3d c1 c6 0e 00 00 74 6c 89 da 48 89 ee bf 9c ff ff ff b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 0f 87 92 00 00 00 48 8b 54 24 28 64 48 2b 14 25
[ 3250.330387]  ret_from_fork+0x44/0x70

[ 3250.330393] RSP: 002b:00007ffc38678260 EFLAGS: 00000202
[ 3250.330394] RAX: 00000000002420c8 RBX: 000064d173383990 RCX: 00007d5d443205e0
[ 3250.330396]  asm_exc_page_fault+0x27/0x30
[ 3250.330394]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 3250.330398]  ? __pfx_kthread+0x10/0x10
[ 3250.330402]  ORIG_RAX: 0000000000000101
[ 3250.330402] RDX: 00007d5d42e00000 RSI: 0000000000090832 RDI: 00000000000e1569
[ 3250.330406] RIP: 0033:0x7d41e0519e92
[ 3250.330408] RAX: ffffffffffffffda RBX: 00000000000800c2 RCX: 00007895b9919e92
[ 3250.330408] RBP: 000064d1731eb688 R08: 00007ffe6a69156c R09: 0000000000000007
[ 3250.330410] RIP: 0033:0x5a9820711605
[ 3250.330409]  ret_from_fork_asm+0x1a/0x30
[ 3250.330415] R10: 000064d1731eb688 R11: 00007ffe6a691600 R12: 000064d168ae5420
[ 3250.330417] RDX: 00000000000800c2 RSI: 00005bba03e24ef0 RDI: 00000000ffffff9c
[ 3250.330417] Code: 2b 14 25 28 00 00 00 75 12 48 81 c4 98 00 00 00 5b 5d 41 5c 41 5d 41 5e 41 5f c3 e8 15 04 e1 ff 0f 1f 44 00 00 f3 0f 1e fa 53 <0f> b6 07 49 89 f1 48 89 d6 48 89 cb 89 c2 49 89 c2 c1 ea 04 41 83
[ 3250.330418] Code: 83 e2 40 75 53 89 f0 f7 d0 a9 00 00 41 00 74 48 80 3d c1 c6 0e 00 00 74 6c 89 da 48 89 ee bf 9c ff ff ff b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 0f 87 92 00 00 00 48 8b 54 24 28 64 48 2b 14 25
[ 3250.330420] R13: 000064d1731eb688 R14: 000064d173383990 R15: 0000000000000000
[ 3250.330423] RBP: 00005bba03e24ef0 R08: ffffffffffffffc0 R09: 0000000000000080
[ 3250.330425] RSP: 002b:00007fff707ceb40 EFLAGS: 00000202
[ 3250.330425] RSP: 002b:00007ffe5033ed10 EFLAGS: 00010202
[ 3250.330429]  </TASK>

[ 3250.330433]  ORIG_RAX: 0000000000000101
[ 3250.330434]  </TASK>
[ 3250.330434] ---[ end trace 0000000000000000 ]---
[ 3250.330436] RAX: 00007a3d6182f7e6 RBX: 000000000002f7e6 RCX: 00007ffe5033ed40
[ 3250.330438] RAX: ffffffffffffffda RBX: 0000000000000241 RCX: 00007d41e0519e92
[ 3250.330440] ---[ end trace 0000000000000000 ]---
[ 3250.330441] R10: 00000000000001b6 R11: 0000000000000202 R12: 00005bba03e24e30
[ 3250.330444] RDX: 0000000000000241 RSI: 00005a1110bf3160 RDI: 00000000ffffff9c
[ 3250.330448] R13: 00005bb9ff49f420 R14: 00005bba03e21300 R15: 00005bb9ff4784f0
[ 3250.330449] RDX: 00007ffe5033ed50 RSI: 00000000039a89d1 RDI: 00007a3d6182f7e6
[ 3250.330452] RBP: 00005a1110bf3160 R08: 0000000000000004 R09: 0000000000000001
[ 3250.330455] RBP: 00005a984dbef4e0 R08: 00007ffe5033ec8c R09: 0000000000000020
[ 3250.330459] R10: 00000000000001b6 R11: 0000000000000202 R12: 00005a1110bf5c00
[ 3250.330461] R10: 00005a984d9be948 R11: 00007ffe5033ed20 R12: 00005a98208ec420
[ 3250.330465]  </TASK>
[ 3250.330465] R13: 00005a1110bf3160 R14: 0000000000000003 R15: 0000000000000000
[ 3250.330467] R13: 0000000000000000 R14: 00007ffe5033ed48 R15: 00007ffe5033ef00
[ 3250.330472] ---[ end trace 0000000000000000 ]---
[ 3250.330478]  </TASK>
[ 3250.330482]  </TASK>                 
                 
                 

@0xfk0
Copy link
Author

0xfk0 commented Nov 27, 2024

Another message, cleaner one:

[ 3259.831196] btree trans held srcu lock (delaying memory reclaim) for 15 seconds
[ 3259.831220] WARNING: CPU: 6 PID: 386 at fs/bcachefs/btree_iter.c:3028 bch2_trans_srcu_unlock+0x122/0x130
[ 3259.831239] Modules linked in: nft_masq xt_MASQUERADE nft_chain_nat nf_nat xt_conntrack bridge nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp stp llc xt_multiport nft_compat nf_tables nfnetlink nvme_fabrics nfsd auth_rpcgss nfs_acl lockd grace sunrpc nouveau drm_gpuvm drm_exec gpu_sched drm_ttm_helper ttm drm_display_helper drm_kms_helper cmac algif_hash algif_skcipher af_alg bnep ath10k_pci ath10k_core intel_rapl_msr intel_rapl_common intel_uncore_frequency ath intel_uncore_frequency_common mac80211 snd_soc_avs skx_edac_common snd_soc_hda_codec nfit snd_hda_ext_core x86_pkg_temp_thermal intel_powerclamp snd_soc_core coretemp snd_hda_codec_realtek snd_hda_codec_generic kvm_intel snd_compress snd_hda_scodec_component snd_hda_codec_hdmi ac97_bus snd_pcm_dmaengine snd_hda_intel snd_intel_dspcfg kvm snd_intel_sdw_acpi btusb snd_hda_codec btrtl wil6210 btintel btbcm btmtk snd_hda_core bluetooth snd_hwdep binfmt_misc cfg80211 drm snd_pcm crct10dif_pclmul crc32_pclmul polyval_clmulni igb eeepc_wmi polyval_generic
[ 3259.831432]  snd_timer ghash_clmulni_intel nls_iso8859_1 sha256_ssse3 asus_wmi snd sha1_ssse3 sparse_keymap mei_me aesni_intel e1000e crypto_simd platform_profile cryptd mei video rapl soundcore ioatdma mxm_wmi i2c_algo_bit intel_cstate intel_wmi_thunderbolt libarc4 dca wmi_bmof wmi parport_pc ppdev lp parport efi_pstore dmi_sysfs ip_tables x_tables autofs4 xhci_pci_renesas mmc_block input_leds hid_generic nvme nvme_core usbhid ahci hid libahci mac_hid
[ 3259.831531] CPU: 6 UID: 0 PID: 386 Comm: bch-reclaim/000 Tainted: G        W          6.12.0 #1
[ 3259.831541] Tainted: [W]=WARN
[ 3259.831544] Hardware name: System manufacturer System Product Name/PRIME X299-DELUXE, BIOS 1503 08/03/2018
[ 3259.831547] RIP: 0010:bch2_trans_srcu_unlock+0x122/0x130
[ 3259.831557] Code: c4 98 f0 01 48 c7 c7 38 22 4b 93 48 b8 cf f7 53 e3 a5 9b c4 20 48 29 ca 48 d1 ea 48 f7 e2 48 89 d6 48 c1 ee 04 e8 ae 24 a0 ff <0f> 0b e9 4f ff ff ff 0f 0b e9 5f ff ff ff 90 90 90 90 90 90 90 90
[ 3259.831563] RSP: 0018:ffffb06b40a7bb10 EFLAGS: 00010246
[ 3259.831570] RAX: 0000000000000000 RBX: ffff9ab05ad90000 RCX: 0000000000000000
[ 3259.831574] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 3259.831578] RBP: ffffb06b40a7bb20 R08: 0000000000000000 R09: 0000000000000000
[ 3259.831582] R10: 0000000000000000 R11: 0000000000000000 R12: ffff9aad8ff40000
[ 3259.831586] R13: ffff9ab05ad90000 R14: ffff9aad8ff40000 R15: 000002f363ce23a6
[ 3259.831590] FS:  0000000000000000(0000) GS:ffff9abcdfb00000(0000) knlGS:0000000000000000
[ 3259.831596] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3259.831600] CR2: 00007366787d70b4 CR3: 000000026e23e001 CR4: 00000000003726f0
[ 3259.831605] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 3259.831609] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 3259.831613] Call Trace:
[ 3259.831616]  <TASK>
[ 3259.831623]  ? show_regs+0x6c/0x80
[ 3259.831636]  ? __warn+0x8d/0x150
[ 3259.831646]  ? bch2_trans_srcu_unlock+0x122/0x130
[ 3259.831656]  ? report_bug+0x182/0x1b0
[ 3259.831666]  ? handle_bug+0x6e/0xb0
[ 3259.831677]  ? exc_invalid_op+0x18/0x80
[ 3259.831686]  ? asm_exc_invalid_op+0x1b/0x20
[ 3259.831699]  ? bch2_trans_srcu_unlock+0x122/0x130
[ 3259.831708]  bch2_trans_begin+0x56c/0x710
[ 3259.831716]  ? finish_task_switch.isra.0+0x8a/0x2b0
[ 3259.831729]  bch2_btree_write_buffer_flush_locked+0x84/0xef0
[ 3259.831743]  ? schedule+0x29/0x130
[ 3259.831751]  ? schedule_preempt_disabled+0x15/0x30
[ 3259.831763]  btree_write_buffer_flush_seq+0x3d7/0x4a0
[ 3259.831772]  ? local_clock_noinstr+0xe/0xd0
[ 3259.831780]  ? local_clock+0x15/0x30
[ 3259.831790]  ? __bch2_trans_get+0x1ac/0x340
[ 3259.831800]  bch2_btree_write_buffer_journal_flush+0x51/0xa0
[ 3259.831808]  ? __pfx_bch2_btree_write_buffer_journal_flush+0x10/0x10
[ 3259.831816]  journal_flush_pins.constprop.0+0x1db/0x3c0
[ 3259.831827]  ? bch2_get_next_online_dev.constprop.0+0xa4/0x140
[ 3259.831839]  __bch2_journal_reclaim+0x1dc/0x380
[ 3259.831849]  bch2_journal_reclaim_thread+0x79/0x170
[ 3259.831858]  ? __pfx_bch2_journal_reclaim_thread+0x10/0x10
[ 3259.831865]  kthread+0xe1/0x110
[ 3259.831872]  ? __pfx_kthread+0x10/0x10
[ 3259.831877]  ret_from_fork+0x44/0x70
[ 3259.831884]  ? __pfx_kthread+0x10/0x10
[ 3259.831889]  ret_from_fork_asm+0x1a/0x30
[ 3259.831903]  </TASK>

I just do commands repo init --submodules --manifest-depth=0 -u https://android.googlesource.com/platform/manifest -b android-14.0.0_r9 and repo sync. Even didn't run the build with m command as previously done.

@0xfk0
Copy link
Author

0xfk0 commented Nov 27, 2024

Looks like re-adding SSD drive fixed the problem. At least, build is running now with no problems.

@Stekke
Copy link

Stekke commented Dec 15, 2024

I've been seeing the same thing on kernel 6.11.7

[14012.531259] ------------[ cut here ]------------
[14012.531262] btree trans held srcu lock (delaying memory reclaim) for 19 seconds
[14012.531300] WARNING: CPU: 1 PID: 276 at fs/bcachefs/btree_iter.c:3000 bch2_trans_srcu_unlock+0x12f/0x150 [bcachefs]
[14012.531367] Modules linked in: cfg80211 snd_hda_codec_hdmi rfkill 8021q snd_hda_codec_realtek snd_hda_codec_generic snd_hda_scodec_component i915 intel_rapl_msr snd_soc_avs nls_iso8859_1 nls_cp437 xt_conntrack vfat nf_conntrack snd_soc_hda_codec fat iTCO_wdt intel_pmc_bxt intel_rapl_common snd_hda_ext_core watchdog ee1004 mei_hdcp mei_pxp intel_tcc_cooling snd_soc_core x86_pkg_temp_thermal nf_defrag_ipv6 nf_defrag_ipv4 evdev intel_powerclamp mac_hid coretemp crct10dif_pclmul crc32_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel sha512_ssse3 sha1_ssse3 rapl ip6t_rpfilter snd_compress ac97_bus ipt_rpfilter snd_pcm_dmaengine snd_hda_intel intel_cstate snd_intel_dspcfg snd_intel_sdw_acpi drm_buddy dm_crypt intel_uncore snd_hda_codec xt_pkttype ttm xt_LOG nf_log_syslog drm_display_helper snd_hda_core aesni_intel cec snd_hwdep ip6t_REJECT gf128mul nf_reject_ipv6 snd_pcm crypto_simd ipt_REJECT igb cryptd nf_reject_ipv4 intel_pmc_core xt_tcpudp snd_timer cbc nft_compat intel_vsec dca e1000e snd encrypted_keys
[14012.531415]  i2c_i801 pmt_telemetry tiny_power_button pmt_class intel_gtt i2c_mux soundcore acpi_pad trusted button asn1_encoder tee i2c_algo_bit i2c_smbus ptp tpm video libaescfb ecdh_generic pps_core ecc rng_core mei_me wmi edac_core backlight mei nf_tables sch_fq_codel wireguard curve25519_x86_64 libchacha20poly1305 libcurve25519_generic ip6_udp_tunnel udp_tunnel atkbd libps2 serio vivaldi_fmap loop tun tap macvlan bridge stp llc kvm_intel kvm fuse efi_pstore configfs nfnetlink efivarfs dmi_sysfs ip_tables x_tables autofs4 poly1305_generic libpoly1305 poly1305_x86_64 chacha_generic chacha_x86_64 libchacha bcachefs libcrc32c crc32c_generic lz4_compress lz4hc_compress xor raid6_pq hid_apple led_class sd_mod hid_generic usbhid hid ahci libahci xhci_pci nvme xhci_pci_renesas libata xhci_hcd nvme_core scsi_mod crc32c_intel sha256_ssse3 nvme_auth scsi_common rtc_cmos af_packet dm_mod dax
[14012.531469] CPU: 1 UID: 0 PID: 276 Comm: bch-reclaim/bec Tainted: G        W          6.11.7 #1-NixOS
[14012.531472] Tainted: [W]=WARN
[14012.531473] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./H270M-ITX/ac, BIOS P2.00 03/29/2017
[14012.531474] RIP: 0010:bch2_trans_srcu_unlock+0x12f/0x150 [bcachefs]
[14012.531526] Code: 01 d4 48 c7 c7 b0 86 95 c0 48 b8 cf f7 53 e3 a5 9b c4 20 48 29 ca 48 c1 ea 03 48 f7 e2 48 89 d6 48 c1 ee 04 e8 c2 95 4d d2 90 <0f> 0b 90 90 e9 4b ff ff ff 90 0f 0b 90 e9 58 ff ff ff 66 66 2e 0f
[14012.531528] RSP: 0018:ffffa31f0034bb90 EFLAGS: 00010246
[14012.531530] RAX: 0000000000000000 RBX: ffff8a3c8d318000 RCX: 0000000000000000
[14012.531531] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[14012.531532] RBP: ffff8a3c48680000 R08: 0000000000000000 R09: 0000000000000000
[14012.531534] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8a3c8d318000
[14012.531535] R13: ffff8a3c8d318000 R14: ffff8a3c486838f8 R15: ffff8a3c486838a8
[14012.531536] FS:  0000000000000000(0000) GS:ffff8a3cab080000(0000) knlGS:0000000000000000
[14012.531538] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[14012.531539] CR2: 00007f2cd8e5e6c8 CR3: 0000000050a22004 CR4: 00000000003706f0
[14012.531541] Call Trace:
[14012.531543]  <TASK>
[14012.531546]  ? __warn+0x80/0x120
[14012.531550]  ? bch2_trans_srcu_unlock+0x12f/0x150 [bcachefs]
[14012.531602]  ? report_bug+0x172/0x1a0
[14012.531606]  ? handle_bug+0x61/0xb0
[14012.531608]  ? exc_invalid_op+0x17/0x80
[14012.531610]  ? asm_exc_invalid_op+0x1a/0x20
[14012.531614]  ? bch2_trans_srcu_unlock+0x12f/0x150 [bcachefs]
[14012.531664]  ? bch2_trans_begin+0xb8/0x680 [bcachefs]
[14012.531713]  bch2_trans_begin+0x521/0x680 [bcachefs]
[14012.531763]  ? psi_task_switch+0xb7/0x200
[14012.531766]  ? finish_task_switch.isra.0+0x9c/0x2f0
[14012.531769]  bch2_btree_write_buffer_flush_locked+0x6e/0xe60 [bcachefs]
[14012.531825]  ? mutex_spin_on_owner+0x48/0xc0
[14012.531830]  btree_write_buffer_flush_seq+0x430/0x480 [bcachefs]
[14012.531884]  ? bch2_trans_put+0x18d/0x250 [bcachefs]
[14012.531933]  ? local_clock+0x15/0x30
[14012.531936]  ? __bch2_trans_get+0x187/0x300 [bcachefs]
[14012.531995]  ? __pfx_bch2_btree_write_buffer_journal_flush+0x10/0x10 [bcachefs]
[14012.532062]  bch2_btree_write_buffer_journal_flush+0x35/0x70 [bcachefs]
[14012.532116]  journal_flush_pins.constprop.0+0x1be/0x2d0 [bcachefs]
[14012.532184]  __bch2_journal_reclaim+0x1de/0x380 [bcachefs]
[14012.532249]  bch2_journal_reclaim_thread+0x6e/0x160 [bcachefs]
[14012.532313]  ? __pfx_bch2_journal_reclaim_thread+0x10/0x10 [bcachefs]
[14012.532376]  kthread+0xcd/0x100
[14012.532380]  ? __pfx_kthread+0x10/0x10
[14012.532384]  ret_from_fork+0x31/0x50
[14012.532386]  ? __pfx_kthread+0x10/0x10
[14012.532389]  ret_from_fork_asm+0x1a/0x30
[14012.532394]  </TASK>
[14012.532394] ---[ end trace 0000000000000000 ]---

@klardotsh
Copy link

I can pretty routinely hang the FS if I thrash the disk hard enough (which is easy to do when backing files up to it via git-annex). Kernel 6.11.11, Void Linux.

[134433.326956] INFO: task git-annex:25452 blocked for more than 122 seconds.
[134433.326957]       Tainted: P        W  OE      6.11.11_1 #1
[134433.326958] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[134433.326959] task:git-annex       state:D stack:0     pid:25452 tgid:25452 ppid:25451  flags:0x00000002
[134433.326961] Call Trace:
[134433.326962]  <TASK>
[134433.326963]  __schedule+0x423/0x1480
[134433.326966]  ? bch2_btree_iter_peek_slot+0x198/0x740 [bcachefs]
[134433.326987]  schedule+0x2a/0xf0
[134433.326988]  schedule_preempt_disabled+0x18/0x30
[134433.326990]  __mutex_lock.constprop.0+0x348/0x620
[134433.326993]  btree_key_cache_fill+0x630/0xea0 [bcachefs]
[134433.327015]  ? bch2_btree_path_traverse_cached+0x20/0x2b0 [bcachefs]
[134433.327034]  bch2_btree_path_traverse_cached+0x27e/0x2b0 [bcachefs]
[134433.327053]  bch2_btree_path_traverse_one+0x6a5/0xd10 [bcachefs]
[134433.327072]  ? srso_alias_return_thunk+0x5/0xfbef5
[134433.327074]  ? bch2_btree_path_peek_slot+0x106/0x210 [bcachefs]
[134433.327094]  ? srso_alias_return_thunk+0x5/0xfbef5
[134433.327095]  ? bch2_btree_iter_peek_slot+0x2a1/0x740 [bcachefs]
[134433.327115]  ? srso_alias_return_thunk+0x5/0xfbef5
[134433.327118]  bch2_btree_iter_peek_slot+0xf2/0x740 [bcachefs]
[134433.327137]  ? srso_alias_return_thunk+0x5/0xfbef5
[134433.327138]  ? btree_path_alloc+0x19/0x170 [bcachefs]
[134433.327157]  ? srso_alias_return_thunk+0x5/0xfbef5
[134433.327159]  ? bch2_path_get+0x182/0x390 [bcachefs]
[134433.327177]  ? bch2_inode_peek_nowarn+0xbc/0x140 [bcachefs]
[134433.327198]  bch2_inode_peek_nowarn+0xc4/0x140 [bcachefs]
[134433.327218]  bch2_inode_find_by_inum_nowarn_trans+0x4d/0x90 [bcachefs]
[134433.327238]  bch2_lookup_trans+0x52e/0x5e0 [bcachefs]
[134433.327275]  ? bch2_lookup+0xad/0x180 [bcachefs]
[134433.327294]  ? srso_alias_return_thunk+0x5/0xfbef5
[134433.327296]  bch2_lookup+0xad/0x180 [bcachefs]
[134433.327316]  __lookup_slow+0x83/0x130
[134433.327319]  walk_component+0xdb/0x150
[134433.327321]  link_path_walk.part.0.constprop.0+0x215/0x390
[134433.327324]  path_lookupat+0x3e/0x1a0
[134433.327326]  filename_lookup+0xde/0x1d0
[134433.327330]  ? srso_alias_return_thunk+0x5/0xfbef5
[134433.327332]  user_path_at+0x37/0x50
[134433.327334]  do_faccessat+0xf9/0x2e0
[134433.327337]  do_syscall_64+0x84/0x170
[134433.327340]  ? srso_alias_return_thunk+0x5/0xfbef5
[134433.327342]  ? __count_memcg_events+0x77/0x110
[134433.327343]  ? srso_alias_return_thunk+0x5/0xfbef5
[134433.327345]  ? count_memcg_events.constprop.0+0x1a/0x30
[134433.327346]  ? srso_alias_return_thunk+0x5/0xfbef5
[134433.327348]  ? handle_mm_fault+0xae/0x350
[134433.327350]  ? srso_alias_return_thunk+0x5/0xfbef5
[134433.327352]  ? preempt_count_add+0x4b/0xa0
[134433.327353]  ? srso_alias_return_thunk+0x5/0xfbef5
[134433.327355]  ? up_read+0x3b/0x80
[134433.327357]  ? srso_alias_return_thunk+0x5/0xfbef5
[134433.327359]  ? do_user_addr_fault+0x336/0x6a0
[134433.327362]  ? srso_alias_return_thunk+0x5/0xfbef5
[134433.327364]  ? fpregs_assert_state_consistent+0x25/0x50
[134433.327367]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[134433.327369] RIP: 0033:0x7fea7ae31107
[134433.327370] RSP: 002b:00007ffc084f9f28 EFLAGS: 00000246 ORIG_RAX: 0000000000000015
[134433.327372] RAX: ffffffffffffffda RBX: 00000042000dc390 RCX: 00007fea7ae31107
[134433.327373] RDX: 00000000000000c6 RSI: 0000000000000000 RDI: 00000042000dc3a0
[134433.327374] RBP: 00000042001e8a60 R08: 00000000000000c6 R09: 00000042000dc2c0
[134433.327375] R10: 00000042000dc3a0 R11: 0000000000000246 R12: 00000042000f2ef0
[134433.327375] R13: 00000000049eaa18 R14: 00000042000dc2c0 R15: 00000042001e10c0
[134433.327378]  </TASK>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants