Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[6.12] fsck not fixing FS - dirent points to own directory (repair unimplemented) #790

Open
nitinkmr333 opened this issue Nov 30, 2024 · 4 comments
Open

[6.12] fsck not fixing FS - dirent points to own directory (repair unimplemented) #790

nitinkmr333 opened this issue Nov 30, 2024 · 4 comments

Comments

@nitinkmr333
Copy link

nitinkmr333 commented Nov 30, 2024 •
edited
Loading

This issue is related to #786.

In short, I was getting error "filesystem marked as clean but have deleted inode" while running fsck on kernel 6.12. It happens when we take snapshots while unlink()ed files are still present (ex- because of docker volumes in my case).

Running fsck fixed the filesystem but there were some errors in lost+found folders. Unfortunately, I cannot show the exact error because I deleted those subvolumes. Running ls -al command in subvolume's lost+found folder was showing some folders and would give No such file or directory error after each folder name. The filesystem would always become read-only whenever I would run ls -al on that lost+found folder.

I deleted the subvolume & snapshots (which had files inside lost+found folders causing filesystem to go read-only). But background task to delete subvolume did not start. I ran fsck but it is not able to fix the filesystem anymore.

Offline fsck-

❯ sudo bcachefs fsck -v -y -p /dev/sda1:/dev/sdb1:/dev/sdc1
Running userspace offline fsck
Enter passphrase:
starting version 1.13: inode_has_child_snapshots opts=ro,metadata_replicas=2,background_compression=zstd:15,foreground_target=ssd,background_target=hdd,promote_target=ssd,degraded,verbose,fsck,fix_errors=yes,read_only
recovering from unclean shutdown
starting journal read
journal read done on device /dev/sdb1, ret 0
journal read done on device /dev/sda1, ret 0
journal read done on device /dev/sdc1, ret 0
journal read done, replaying entries 8563536-8563536
Journal keys: 0 read, 0 after sorting and compacting
accounting_read... done
alloc_read... done
stripes_read... done
snapshots_read... done
check_allocations... done
going read-write
journal_replay... done
check_alloc_info... done
check_lrus... done
check_btree_backpointers... done
check_backpointers_to_extents...backpointers_to_extents: 0%, done 22/60216 nodes, at backpointers:0:10738532352:0
backpointers_to_extents: 0%, done 139/60216 nodes, at backpointers:0:54387630080:0
backpointers_to_extents: 0%, done 152/60216 nodes, at backpointers:0:60254871552:0
backpointers_to_extents: 0%, done 157/60216 nodes, at backpointers:0:62947287040:0
backpointers_to_extents: 0%, done 251/60216 nodes, at backpointers:0:98333360128:0
backpointers_to_extents: 0%, done 267/60216 nodes, at backpointers:0:104128315392:0
backpointers_to_extents: 0%, done 271/60216 nodes, at backpointers:0:105717432320:0
backpointers_to_extents: 0%, done 272/60216 nodes, at backpointers:0:107381030912:0
backpointers_to_extents: 0%, done 273/60216 nodes, at backpointers:0:109084409856:0
backpointers_to_extents: 0%, done 274/60216 nodes, at backpointers:0:110742208512:0
backpointers_to_extents: 0%, done 283/60216 nodes, at backpointers:0:114938609664:0
backpointers_to_extents: 0%, done 297/60216 nodes, at backpointers:0:119934550016:0
backpointers_to_extents: 0%, done 312/60216 nodes, at backpointers:0:124560908288:0
backpointers_to_extents: 0%, done 318/60216 nodes, at backpointers:0:127383871488:0
backpointers_to_extents: 0%, done 328/60216 nodes, at backpointers:0:131064684544:0
backpointers_to_extents: 0%, done 368/60216 nodes, at backpointers:0:144528523264:0
backpointers_to_extents: 0%, done 475/60216 nodes, at backpointers:0:182837575680:0
backpointers_to_extents: 0%, done 516/60216 nodes, at backpointers:0:196740644864:0
backpointers_to_extents: 1%, done 611/60216 nodes, at backpointers:0:224523747328:0
backpointers_to_extents: 1%, done 682/60216 nodes, at backpointers:0:246304751616:0
backpointers_to_extents: 1%, done 739/60216 nodes, at backpointers:0:264869330944:0
backpointers_to_extents: 1%, done 789/60216 nodes, at backpointers:0:280648327168:0
backpointers_to_extents: 1%, done 841/60216 nodes, at backpointers:0:297661423616:0
backpointers_to_extents: 1%, done 953/60216 nodes, at backpointers:0:333297270784:0
backpointers_to_extents: 2%, done 1217/60216 nodes, at backpointers:0:417782038544:0
backpointers_to_extents: 2%, done 1453/60216 nodes, at backpointers:0:493640335360:0
backpointers_to_extents: 2%, done 1589/60216 nodes, at backpointers:0:535980720128:0
backpointers_to_extents: 2%, done 1670/60216 nodes, at backpointers:0:566011363328:0
backpointers_to_extents: 2%, done 1690/60216 nodes, at backpointers:0:572321292288:0
backpointers_to_extents: 3%, done 1973/60216 nodes, at backpointers:0:685021700096:0
backpointers_to_extents: 3%, done 2244/60216 nodes, at backpointers:0:790951362560:0
backpointers_to_extents: 4%, done 2483/60216 nodes, at backpointers:0:886395494400:0
backpointers_to_extents: 4%, done 2661/60216 nodes, at backpointers:0:944951468032:0
backpointers_to_extents: 4%, done 2740/60216 nodes, at backpointers:0:969310011392:0
backpointers_to_extents: 4%, done 2814/60216 nodes, at backpointers:0:992281526272:0
backpointers_to_extents: 4%, done 2870/60216 nodes, at backpointers:0:1009271767040:0
backpointers_to_extents: 4%, done 2964/60216 nodes, at backpointers:0:1039605039104:0
backpointers_to_extents: 5%, done 3104/60216 nodes, at backpointers:0:1085499727872:0
backpointers_to_extents: 5%, done 3282/60216 nodes, at backpointers:0:1146093690880:0
backpointers_to_extents: 5%, done 3496/60216 nodes, at backpointers:0:1216056451072:0
backpointers_to_extents: 6%, done 3671/60216 nodes, at backpointers:0:1287742103552:0
backpointers_to_extents: 6%, done 3740/60216 nodes, at backpointers:0:1310124982272:0
backpointers_to_extents: 6%, done 3840/60216 nodes, at backpointers:0:1342110277632:0
backpointers_to_extents: 6%, done 4144/60216 nodes, at backpointers:0:1441681989632:0
backpointers_to_extents: 7%, done 4272/60216 nodes, at backpointers:0:1482828480512:0
backpointers_to_extents: 7%, done 4456/60216 nodes, at backpointers:0:1542204252160:0
backpointers_to_extents: 7%, done 4652/60216 nodes, at backpointers:0:1606639730688:0
backpointers_to_extents: 8%, done 4831/60216 nodes, at backpointers:0:1665340121088:0
backpointers_to_extents: 8%, done 5005/60216 nodes, at backpointers:0:1723105017856:0
backpointers_to_extents: 8%, done 5147/60216 nodes, at backpointers:0:1770191675392:0
backpointers_to_extents: 8%, done 5315/60216 nodes, at backpointers:0:1828675723264:0
backpointers_to_extents: 9%, done 5469/60216 nodes, at backpointers:0:1880567054336:0
backpointers_to_extents: 9%, done 5687/60216 nodes, at backpointers:0:1955123560448:0
backpointers_to_extents: 9%, done 5840/60216 nodes, at backpointers:1:9008316416:0
backpointers_to_extents: 10%, done 6087/60216 nodes, at backpointers:1:94938382336:0
backpointers_to_extents: 10%, done 6442/60216 nodes, at backpointers:1:215423647744:0
backpointers_to_extents: 10%, done 6442/60216 nodes, at backpointers:1:216953520128:0
backpointers_to_extents: 10%, done 6443/60216 nodes, at backpointers:1:218437779456:0
backpointers_to_extents: 10%, done 6449/60216 nodes, at backpointers:1:221374316544:0
backpointers_to_extents: 10%, done 6454/60216 nodes, at backpointers:1:223841615872:0
backpointers_to_extents: 10%, done 6459/60216 nodes, at backpointers:1:226187280384:0
backpointers_to_extents: 10%, done 6464/60216 nodes, at backpointers:1:228641472512:0
backpointers_to_extents: 10%, done 6469/60216 nodes, at backpointers:1:231194755072:0
backpointers_to_extents: 10%, done 6475/60216 nodes, at backpointers:1:234049503232:0
backpointers_to_extents: 10%, done 6481/60216 nodes, at backpointers:1:236522569728:0
backpointers_to_extents: 11%, done 6908/60216 nodes, at backpointers:1:368776372224:0
backpointers_to_extents: 11%, done 7100/60216 nodes, at backpointers:1:407968497664:0
backpointers_to_extents: 12%, done 7270/60216 nodes, at backpointers:1:442472366080:0
backpointers_to_extents: 12%, done 7580/60216 nodes, at backpointers:1:509186867200:0
backpointers_to_extents: 13%, done 8025/60216 nodes, at backpointers:1:634834239488:0
backpointers_to_extents: 13%, done 8228/60216 nodes, at backpointers:1:688553197568:0
backpointers_to_extents: 13%, done 8235/60216 nodes, at backpointers:1:691434160128:0
backpointers_to_extents: 13%, done 8243/60216 nodes, at backpointers:1:694355492864:0
backpointers_to_extents: 13%, done 8252/60216 nodes, at backpointers:1:697634390016:0
backpointers_to_extents: 13%, done 8262/60216 nodes, at backpointers:1:701192208384:0
backpointers_to_extents: 13%, done 8272/60216 nodes, at backpointers:1:703846154240:0
backpointers_to_extents: 13%, done 8279/60216 nodes, at backpointers:1:706421456896:0
backpointers_to_extents: 13%, done 8286/60216 nodes, at backpointers:1:709325488128:0
backpointers_to_extents: 13%, done 8293/60216 nodes, at backpointers:1:712114176000:0
backpointers_to_extents: 13%, done 8299/60216 nodes, at backpointers:1:714383294464:0
backpointers_to_extents: 13%, done 8306/60216 nodes, at backpointers:1:717197672448:0
backpointers_to_extents: 13%, done 8312/60216 nodes, at backpointers:1:719717400576:0
backpointers_to_extents: 13%, done 8351/60216 nodes, at backpointers:1:735028830208:0
backpointers_to_extents: 14%, done 8864/60216 nodes, at backpointers:1:937457950720:0
backpointers_to_extents: 15%, done 9276/60216 nodes, at backpointers:1:1021232316416:0
backpointers_to_extents: 16%, done 9947/60216 nodes, at backpointers:1:1235580829696:0
backpointers_to_extents: 17%, done 10542/60216 nodes, at backpointers:1:1430916571136:0
backpointers_to_extents: 18%, done 11126/60216 nodes, at backpointers:1:1622067503104:0
backpointers_to_extents: 19%, done 11455/60216 nodes, at backpointers:1:1728513081344:0
backpointers_to_extents: 19%, done 11780/60216 nodes, at backpointers:1:1817744326656:0
backpointers_to_extents: 20%, done 12381/60216 nodes, at backpointers:1:2010299514880:0
backpointers_to_extents: 22%, done 13615/60216 nodes, at backpointers:1:2422600105984:0
backpointers_to_extents: 23%, done 14152/60216 nodes, at backpointers:1:2596520083456:0
backpointers_to_extents: 25%, done 15221/60216 nodes, at backpointers:1:2955467563008:0
backpointers_to_extents: 26%, done 15808/60216 nodes, at backpointers:1:3148921708544:0
backpointers_to_extents: 27%, done 16467/60216 nodes, at backpointers:1:3423136669696:0
backpointers_to_extents: 29%, done 17584/60216 nodes, at backpointers:1:3860828020736:0
backpointers_to_extents: 31%, done 18717/60216 nodes, at backpointers:1:4237484933120:0
backpointers_to_extents: 31%, done 18941/60216 nodes, at backpointers:1:4325162221568:0
backpointers_to_extents: 33%, done 20266/60216 nodes, at backpointers:1:4774239264768:0
backpointers_to_extents: 34%, done 20738/60216 nodes, at backpointers:1:4937667895296:0
backpointers_to_extents: 35%, done 21407/60216 nodes, at backpointers:1:5165120831488:0
backpointers_to_extents: 37%, done 22439/60216 nodes, at backpointers:1:5513761316864:0
backpointers_to_extents: 38%, done 22897/60216 nodes, at backpointers:1:5667563896832:0
backpointers_to_extents: 39%, done 23521/60216 nodes, at backpointers:1:5878199107584:0
backpointers_to_extents: 40%, done 24392/60216 nodes, at backpointers:1:6197504737280:0
backpointers_to_extents: 40%, done 24482/60216 nodes, at backpointers:1:6229837078528:0
backpointers_to_extents: 40%, done 24507/60216 nodes, at backpointers:1:6239752503296:0
backpointers_to_extents: 40%, done 24524/60216 nodes, at backpointers:1:6246500171776:0
backpointers_to_extents: 40%, done 24545/60216 nodes, at backpointers:1:6253978206208:0
backpointers_to_extents: 41%, done 25121/60216 nodes, at backpointers:1:6448303775744:0
backpointers_to_extents: 42%, done 25596/60216 nodes, at backpointers:1:6606674780160:0
backpointers_to_extents: 42%, done 25604/60216 nodes, at backpointers:1:6609714667520:0
backpointers_to_extents: 42%, done 25613/60216 nodes, at backpointers:1:6613114028032:0
backpointers_to_extents: 42%, done 25625/60216 nodes, at backpointers:1:6617597206528:0
backpointers_to_extents: 42%, done 25688/60216 nodes, at backpointers:1:6641064869888:0
backpointers_to_extents: 45%, done 27144/60216 nodes, at backpointers:1:7118135033856:0
backpointers_to_extents: 45%, done 27478/60216 nodes, at backpointers:1:7215468732416:0
backpointers_to_extents: 46%, done 27712/60216 nodes, at backpointers:1:7282577268736:0
backpointers_to_extents: 46%, done 27936/60216 nodes, at backpointers:1:7342226235392:0
backpointers_to_extents: 46%, done 28167/60216 nodes, at backpointers:1:7410015068160:0
backpointers_to_extents: 47%, done 28369/60216 nodes, at backpointers:1:7465226395648:0
backpointers_to_extents: 47%, done 28585/60216 nodes, at backpointers:1:7524180942848:0
backpointers_to_extents: 47%, done 28809/60216 nodes, at backpointers:1:7586539159552:0
backpointers_to_extents: 48%, done 28954/60216 nodes, at backpointers:1:7628285558784:0
backpointers_to_extents: 48%, done 29153/60216 nodes, at backpointers:1:7685159837696:0
backpointers_to_extents: 48%, done 29368/60216 nodes, at backpointers:1:7741434511360:0
backpointers_to_extents: 49%, done 29546/60216 nodes, at backpointers:1:7789453287424:0
backpointers_to_extents: 49%, done 29730/60216 nodes, at backpointers:1:7839450513408:0
backpointers_to_extents: 49%, done 30018/60216 nodes, at backpointers:3:4105699328:0
backpointers_to_extents: 49%, done 30020/60216 nodes, at backpointers:3:10668736512:0
backpointers_to_extents: 49%, done 30023/60216 nodes, at backpointers:3:15487991808:0
backpointers_to_extents: 49%, done 30026/60216 nodes, at backpointers:3:21848129536:0
backpointers_to_extents: 49%, done 30027/60216 nodes, at backpointers:3:26721910784:0
backpointers_to_extents: 49%, done 30030/60216 nodes, at backpointers:3:31523340288:0
backpointers_to_extents: 49%, done 30032/60216 nodes, at backpointers:3:36475764736:0
backpointers_to_extents: 49%, done 30034/60216 nodes, at backpointers:3:41029206016:0
backpointers_to_extents: 49%, done 30036/60216 nodes, at backpointers:3:45624590336:0
backpointers_to_extents: 49%, done 30039/60216 nodes, at backpointers:3:50309627904:0
backpointers_to_extents: 49%, done 30040/60216 nodes, at backpointers:3:56627822592:0
backpointers_to_extents: 49%, done 30042/60216 nodes, at backpointers:3:61473816576:0
backpointers_to_extents: 49%, done 30044/60216 nodes, at backpointers:3:66820505600:0
backpointers_to_extents: 49%, done 30046/60216 nodes, at backpointers:3:72094318592:0
backpointers_to_extents: 49%, done 30049/60216 nodes, at backpointers:3:78504787968:0
backpointers_to_extents: 49%, done 30051/60216 nodes, at backpointers:3:85081980928:0
backpointers_to_extents: 49%, done 30053/60216 nodes, at backpointers:3:91695874048:0
backpointers_to_extents: 49%, done 30055/60216 nodes, at backpointers:3:100467736576:0
backpointers_to_extents: 49%, done 30057/60216 nodes, at backpointers:3:105632497664:0
backpointers_to_extents: 49%, done 30060/60216 nodes, at backpointers:3:116445937664:0
backpointers_to_extents: 49%, done 30062/60216 nodes, at backpointers:3:128806551552:0
backpointers_to_extents: 49%, done 30064/60216 nodes, at backpointers:3:140106530816:0
backpointers_to_extents: 49%, done 30066/60216 nodes, at backpointers:3:150192259072:0
backpointers_to_extents: 49%, done 30068/60216 nodes, at backpointers:3:162538717184:0
backpointers_to_extents: 49%, done 30070/60216 nodes, at backpointers:3:181532622848:0
backpointers_to_extents: 49%, done 30072/60216 nodes, at backpointers:3:186823213056:0
backpointers_to_extents: 49%, done 30074/60216 nodes, at backpointers:3:194891481088:0
backpointers_to_extents: 49%, done 30076/60216 nodes, at backpointers:3:206455177216:0
backpointers_to_extents: 49%, done 30078/60216 nodes, at backpointers:3:217075154944:0
backpointers_to_extents: 49%, done 30080/60216 nodes, at backpointers:3:232672198656:0
backpointers_to_extents: 49%, done 30083/60216 nodes, at backpointers:3:395260723200:0
backpointers_to_extents: 49%, done 30085/60216 nodes, at backpointers:3:400583294976:0
backpointers_to_extents: 49%, done 30087/60216 nodes, at backpointers:3:403213123584:0
 done
check_extents_to_backpointers...bch2_check_extents_to_backpointers(): alloc info does not fit in ram, running in multiple passes with 16103 nodes per pass
check_extents_to_backpointers(): POS_MIN-1:3289971556352:0
extents_to_backpointers: 1%, done 1626/119444 nodes, at extents:36497:642296:4294963371
extents_to_backpointers: 2%, done 2828/119444 nodes, at extents:40735:19125248:4294963375
extents_to_backpointers: 3%, done 4508/119444 nodes, at extents:53934:337880:4294963375
extents_to_backpointers: 5%, done 6499/119444 nodes, at extents:56678:8080:4294963371
extents_to_backpointers: 6%, done 8322/119444 nodes, at extents:58879:159672:4294963371
extents_to_backpointers: 8%, done 10018/119444 nodes, at extents:59919:298880:4294963371
extents_to_backpointers: 10%, done 12029/119444 nodes, at extents:116553:10296:4294963294
extents_to_backpointers: 11%, done 13839/119444 nodes, at extents:536898812:87704:4294963371
extents_to_backpointers: 12%, done 15260/119444 nodes, at extents:536907429:778984:4294963371
extents_to_backpointers: 14%, done 17161/119444 nodes, at extents:536914824:764928:4294963371
extents_to_backpointers: 16%, done 19474/119444 nodes, at extents:536923130:109192:4294963371
extents_to_backpointers: 17%, done 21383/119444 nodes, at extents:536926613:256984:4294963371
extents_to_backpointers: 19%, done 23109/119444 nodes, at extents:536928091:3065728:4294963371
extents_to_backpointers: 20%, done 25062/119444 nodes, at extents:536975921:521008:4294963375
extents_to_backpointers: 22%, done 26644/119444 nodes, at extents:1073757722:780528:4294963371
extents_to_backpointers: 23%, done 28128/119444 nodes, at extents:1073774375:323168:4294963371
extents_to_backpointers: 24%, done 29839/119444 nodes, at extents:1073780157:713728:4294963371
extents_to_backpointers: 26%, done 31792/119444 nodes, at extents:1073791079:716792:4294963955
extents_to_backpointers: 28%, done 33896/119444 nodes, at extents:1073792609:37344:4294963371
extents_to_backpointers: 29%, done 35779/119444 nodes, at extents:1073794099:697456:4294963305
extents_to_backpointers: 31%, done 37484/119444 nodes, at extents:1073819507:384:4294963375
extents_to_backpointers: 32%, done 38691/119444 nodes, at extents:1073843333:475080:4294963375
extents_to_backpointers: 33%, done 39927/119444 nodes, at extents:1610621098:460664:4294963371
extents_to_backpointers: 34%, done 41283/119444 nodes, at extents:1610637637:1416:4294963371
extents_to_backpointers: 35%, done 42689/119444 nodes, at extents:1610640540:51912:4294963371
extents_to_backpointers: 37%, done 44581/119444 nodes, at extents:1610652156:177456:4294963375
extents_to_backpointers: 39%, done 46649/119444 nodes, at extents:1610653532:709336:4294963371
extents_to_backpointers: 40%, done 48701/119444 nodes, at extents:1610655726:1455512:4294963371
extents_to_backpointers: 42%, done 50456/119444 nodes, at extents:1610657190:5280:4294963371
extents_to_backpointers: 43%, done 52428/119444 nodes, at extents:1610687520:24:4294963375
extents_to_backpointers: 45%, done 54122/119444 nodes, at reflink:0:202811888:0
extents_to_backpointers: 46%, done 55908/119444 nodes, at reflink:0:560704664:0
extents_to_backpointers: 48%, done 57503/119444 nodes, at reflink:0:856526200:0
extents_to_backpointers: 48%, done 58322/119444 nodes, at reflink:0:1004421280:0
extents_to_backpointers: 49%, done 59613/119444 nodes, at reflink:0:1250204680:0
check_extents_to_backpointers(): 1:3289971556352:1-SPOS_MAX
extents_to_backpointers: 4%, done 5123/119444 nodes, at extents:55415:13853272:4294963373
extents_to_backpointers: 8%, done 10252/119444 nodes, at extents:61066:29266744:4294963373
extents_to_backpointers: 12%, done 15381/119444 nodes, at extents:536907498:107672:4294963371
extents_to_backpointers: 17%, done 20485/119444 nodes, at extents:536924122:216072:4294963371
extents_to_backpointers: 21%, done 25874/119444 nodes, at extents:1073754215:287360:4294963371
extents_to_backpointers: 25%, done 30890/119444 nodes, at extents:1073790154:15018752:4294963375
extents_to_backpointers: 30%, done 36323/119444 nodes, at extents:1073794644:2163688:4294963371
extents_to_backpointers: 34%, done 41599/119444 nodes, at extents:1610639123:1304752:4294963371
extents_to_backpointers: 39%, done 46886/119444 nodes, at extents:1610653695:27392:4294963371
extents_to_backpointers: 43%, done 52331/119444 nodes, at extents:1610678759:17232:4294963375
extents_to_backpointers: 48%, done 57348/119444 nodes, at reflink:0:828573656:0
 done
check_alloc_to_lru_refs... done
check_snapshot_trees... done
check_snapshots... done
check_subvols... done
check_subvol_children... done
delete_dead_snapshots... done
check_inodes... done
check_extents... done
check_indirect_extents... done
check_dirents...directory 118105:4294963319 with wrong i_nlink: got 2, should be 1, fixing
 done
check_xattrs... done
check_root... done
check_unreachable_inodes... done
check_subvolume_structure... done
check_directory_structure...directory structure loop
check_path() 4097:4294963303
check_path() 118105:4294963319
check_path() 4097:4294963319
check_path() 118105:4294963319
directory structure loop, fixing
directory structure loop
check_path() 4097:4294963303
check_path() 118105:4294963319
check_path() 4097:4294963319
check_path() 118105:4294963319
directory structure loop
check_path() 4097:4294963303
check_path() 118105:4294963319
check_path() 4097:4294963319
check_path() 118105:4294963319
invalid bkey u64s 7 type dirent 118105:7381901084341708069:4294963319 len 0 ver 0: 118105 -> 118105 type dir
  dirent points to own directory: delete? (repair unimplemented)
Unable to continue, halting
invalid bkey on insert from bch2_check_directory_structure -> 0x555562505d08s

4 transaction updates for bch2_check_directory_structure journal seq 8563546
  update: btree=inodes cached=1 0x5555624e8ea8S
    old u64s 19 type inode_v3 0:118105:4294963319 len 0 ver 0:
    mode=40700
    flags=has_child_snapshot(15300200)
    journal_seq=8563545
    hash_seed=63f64dd1a170398a
    hash_type=siphash
    bi_size=0
    bi_sectors=0
    bi_version=0
    bi_atime=6222148210840899
    bi_ctime=6222148210840899
    bi_mtime=6222148210840899
    bi_otime=6222148210840899
    bi_uid=0
    bi_gid=0
    bi_nlink=1
    bi_generation=0
    bi_dev=0
    bi_data_checksum=0
    bi_compression=0
    bi_project=0
    bi_background_compression=0
    bi_data_replicas=0
    bi_promote_target=0
    bi_foreground_target=0
    bi_background_target=0
    bi_erasure_code=0
    bi_fields_set=0
    bi_dir=4097
    bi_dir_offset=5928645883727690363
    bi_subvol=0
    bi_parent_subvol=0
    bi_nocow=0
    new u64s 19 type inode_v3 0:118105:4294963319 len 0 ver 0:
    mode=40700
    flags=has_child_snapshot(15300200)
    journal_seq=8563546
    hash_seed=63f64dd1a170398a
    hash_type=siphash
    bi_size=0
    bi_sectors=0
    bi_version=0
    bi_atime=6222148210840899
    bi_ctime=6222148210840899
    bi_mtime=6222148210840899
    bi_otime=6222148210840899
    bi_uid=0
    bi_gid=0
    bi_nlink=1
    bi_generation=0
    bi_dev=0
    bi_data_checksum=0
    bi_compression=0
    bi_project=0
    bi_background_compression=0
    bi_data_replicas=0
    bi_promote_target=0
    bi_foreground_target=0
    bi_background_target=0
    bi_erasure_code=0
    bi_fields_set=0
    bi_dir=118105
    bi_dir_offset=7381901084341708069
    bi_subvol=0
    bi_parent_subvol=0
    bi_nocow=0
  update: btree=dirents cached=0 0x555562525e34S
    old u64s 8 type dirent 4097:5928645883727690363:4294963375 len 0 ver 0: lost+found -> 118105 type dir
    new u64s 5 type deleted 4097:5928645883727690363:4294963375 len 0 ver 0
  update: btree=dirents cached=0 0x555562528378S
    old u64s 5 type deleted 118105:7381901084341708069:4294963294 len 0 ver 0
    new u64s 5 type whiteout 118105:7381901084341708069:4294963294 len 0 ver 0
  update: btree=dirents cached=0 0x555562505d08S
    old u64s 5 type deleted 118105:7381901084341708069:4294963319 len 0 ver 0
    new u64s 7 type dirent 118105:7381901084341708069:4294963319 len 0 ver 0: 118105 -> 118105 type dir

inconsistency detected - emergency read only at journal seq 8563546
bch2_trans_commit_write_locked(): fatal error fatal error in transaction commit: fsck_delete_bkey
bch2_check_directory_structure(): error fsck_delete_bkey
bch2_fs_recovery(): error fsck_delete_bkey
bch2_fs_start(): error starting filesystem fsck_delete_bkey
shutting down
going read-only
finished waiting for writes to stop
flushing journal and stopping allocators, journal seq 8563546
flushing journal and stopping allocators complete, journal seq 8563546
unshutdown complete, journal seq 8563546
done going read-only, filesystem not clean
shutdown complete

Online fsck-

❯ sudo bcachefs fsck -v -y -p /dev/sda1:/dev/sdb1:/dev/sdc1
Running fsck online
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): check_alloc_info... done
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): check_lrus... done
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): check_btree_backpointers... done
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): check_backpointers_to_extents...bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 0%, done 34/60216 nodes, at backpointers:0:15179718656:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 0%, done 152/60216 nodes, at backpointers:0:59952857088:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 0%, done 158/60216 nodes, at backpointers:0:63320358912:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 0%, done 256/60216 nodes, at backpointers:0:100071899136:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 0%, done 269/60216 nodes, at backpointers:0:105204678656:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 0%, done 271/60216 nodes, at backpointers:0:106880827392:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 0%, done 272/60216 nodes, at backpointers:0:108859490304:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 0%, done 273/60216 nodes, at backpointers:0:110560280576:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 0%, done 283/60216 nodes, at backpointers:0:115169853440:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 0%, done 301/60216 nodes, at backpointers:0:121305628672:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 0%, done 312/60216 nodes, at backpointers:0:125257121792:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 0%, done 321/60216 nodes, at backpointers:0:128671301632:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 0%, done 338/60216 nodes, at backpointers:0:134316908544:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 0%, done 475/60216 nodes, at backpointers:0:183098146816:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 0%, done 545/60216 nodes, at backpointers:0:204956762112:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 1%, done 669/60216 nodes, at backpointers:0:242154070016:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 1%, done 787/60216 nodes, at backpointers:0:280331083776:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 1%, done 885/60216 nodes, at backpointers:0:311642529792:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 2%, done 1298/60216 nodes, at backpointers:0:444562178048:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 2%, done 1574/60216 nodes, at backpointers:0:531639099392:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 2%, done 1673/60216 nodes, at backpointers:0:567212408832:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 3%, done 1824/60216 nodes, at backpointers:0:635833573376:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 3%, done 2244/60216 nodes, at backpointers:0:791294304256:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 4%, done 2533/60216 nodes, at backpointers:0:903584792576:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 4%, done 2793/60216 nodes, at backpointers:0:986607362048:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 4%, done 2855/60216 nodes, at backpointers:0:1004882370560:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 4%, done 2953/60216 nodes, at backpointers:0:1036503777280:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 5%, done 3103/60216 nodes, at backpointers:0:1085417259008:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 5%, done 3353/60216 nodes, at backpointers:0:1170645073920:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 6%, done 3696/60216 nodes, at backpointers:0:1296738336768:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 6%, done 3753/60216 nodes, at backpointers:0:1314413633536:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 6%, done 4134/60216 nodes, at backpointers:0:1438770569216:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 7%, done 4269/60216 nodes, at backpointers:0:1482105118720:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 7%, done 4510/60216 nodes, at backpointers:0:1560422465536:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 7%, done 4694/60216 nodes, at backpointers:0:1620990066688:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 8%, done 5020/60216 nodes, at backpointers:0:1727902310400:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 8%, done 5249/60216 nodes, at backpointers:0:1805297524736:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 8%, done 5386/60216 nodes, at backpointers:0:1853599309824:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 9%, done 5697/60216 nodes, at backpointers:0:1958718046208:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 9%, done 5845/60216 nodes, at backpointers:1:11238113280:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 10%, done 6441/60216 nodes, at backpointers:1:215794843648:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 10%, done 6442/60216 nodes, at backpointers:1:217555927040:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 10%, done 6447/60216 nodes, at backpointers:1:221148348416:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 10%, done 6455/60216 nodes, at backpointers:1:224580861952:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 10%, done 6462/60216 nodes, at backpointers:1:228076814336:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 10%, done 6469/60216 nodes, at backpointers:1:231265533952:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 10%, done 6475/60216 nodes, at backpointers:1:234134962176:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 10%, done 6481/60216 nodes, at backpointers:1:237105053696:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 11%, done 7044/60216 nodes, at backpointers:1:396797468672:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 12%, done 7247/60216 nodes, at backpointers:1:437886976000:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 12%, done 7564/60216 nodes, at backpointers:1:506175709184:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 13%, done 8203/60216 nodes, at backpointers:1:683125030912:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 13%, done 8233/60216 nodes, at backpointers:1:690732138496:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 13%, done 8240/60216 nodes, at backpointers:1:693704327168:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 13%, done 8250/60216 nodes, at backpointers:1:697219678208:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 13%, done 8261/60216 nodes, at backpointers:1:701343727616:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 13%, done 8272/60216 nodes, at backpointers:1:704463765504:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 13%, done 8279/60216 nodes, at backpointers:1:707356262400:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 13%, done 8287/60216 nodes, at backpointers:1:710775668736:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 13%, done 8297/60216 nodes, at backpointers:1:714399547392:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 13%, done 8305/60216 nodes, at backpointers:1:717708853248:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 13%, done 8312/60216 nodes, at backpointers:1:720528998400:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 14%, done 8639/60216 nodes, at backpointers:1:890198097920:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 15%, done 9168/60216 nodes, at backpointers:1:999513227264:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 16%, done 10092/60216 nodes, at backpointers:1:1283406209112:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 18%, done 11080/60216 nodes, at backpointers:1:1608444805120:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 19%, done 11608/60216 nodes, at backpointers:1:1772120236032:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 20%, done 12324/60216 nodes, at backpointers:1:1996314689536:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 22%, done 13419/60216 nodes, at backpointers:1:2356681457664:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 23%, done 14095/60216 nodes, at backpointers:1:2577601708032:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 24%, done 14724/60216 nodes, at backpointers:1:2788261101568:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 26%, done 15783/60216 nodes, at backpointers:1:3144471347200:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 27%, done 16333/60216 nodes, at backpointers:1:3373276291072:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 28%, done 17351/60216 nodes, at backpointers:1:3778047057920:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 30%, done 18440/60216 nodes, at backpointers:1:4147621920768:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 31%, done 18869/60216 nodes, at backpointers:1:4297697853440:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 32%, done 19752/60216 nodes, at backpointers:1:4604199297024:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 34%, done 20692/60216 nodes, at backpointers:1:4920794243072:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 35%, done 21365/60216 nodes, at backpointers:1:5149973659648:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 36%, done 22027/60216 nodes, at backpointers:1:5374664024064:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 37%, done 22699/60216 nodes, at backpointers:1:5602185584640:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 39%, done 23519/60216 nodes, at backpointers:1:5878177636352:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 40%, done 24309/60216 nodes, at backpointers:1:6168964571136:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 40%, done 24479/60216 nodes, at backpointers:1:6229728477184:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 40%, done 24512/60216 nodes, at backpointers:1:6242542845952:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 40%, done 24532/60216 nodes, at backpointers:1:6250094485504:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 41%, done 25120/60216 nodes, at backpointers:1:6448666263552:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 42%, done 25596/60216 nodes, at backpointers:1:6607169126400:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 42%, done 25609/60216 nodes, at backpointers:1:6612502806528:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 42%, done 25622/60216 nodes, at backpointers:1:6617305006080:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 42%, done 25692/60216 nodes, at backpointers:1:6642901049344:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 45%, done 27099/60216 nodes, at backpointers:1:7105617674240:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 45%, done 27592/60216 nodes, at backpointers:1:7246724030464:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 46%, done 27808/60216 nodes, at backpointers:1:7308473819136:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 46%, done 28094/60216 nodes, at backpointers:1:7388338380800:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 46%, done 28243/60216 nodes, at backpointers:1:7430599655424:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 47%, done 28432/60216 nodes, at backpointers:1:7484429828096:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 47%, done 28640/60216 nodes, at backpointers:1:7539121471488:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 47%, done 28877/60216 nodes, at backpointers:1:7607679115264:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 48%, done 29044/60216 nodes, at backpointers:1:7652855324672:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 48%, done 29232/60216 nodes, at backpointers:1:7706634551296:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 48%, done 29436/60216 nodes, at backpointers:1:7759956893696:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 49%, done 29588/60216 nodes, at backpointers:1:7802392772608:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 49%, done 29789/60216 nodes, at backpointers:1:7855111962624:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 49%, done 30016/60216 nodes, at backpointers:3:5606211584:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 49%, done 30019/60216 nodes, at backpointers:3:12032409600:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 49%, done 30022/60216 nodes, at backpointers:3:18577096704:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 49%, done 30025/60216 nodes, at backpointers:3:26136281088:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 49%, done 30028/60216 nodes, at backpointers:3:31776047104:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 49%, done 30031/60216 nodes, at backpointers:3:37777047552:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 49%, done 30033/60216 nodes, at backpointers:3:42612031488:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 49%, done 30035/60216 nodes, at backpointers:3:48210903040:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 49%, done 30037/60216 nodes, at backpointers:3:54382297088:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 49%, done 30040/60216 nodes, at backpointers:3:60579905536:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 49%, done 30042/60216 nodes, at backpointers:3:66870312960:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 49%, done 30044/60216 nodes, at backpointers:3:72710881280:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 49%, done 30048/60216 nodes, at backpointers:3:80115400704:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 49%, done 30049/60216 nodes, at backpointers:3:87414013952:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 49%, done 30052/60216 nodes, at backpointers:3:94435803136:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 49%, done 30054/60216 nodes, at backpointers:3:102591102976:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 49%, done 30057/60216 nodes, at backpointers:3:110056439808:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 49%, done 30059/60216 nodes, at backpointers:3:121846104064:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 49%, done 30061/60216 nodes, at backpointers:3:133167579136:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 49%, done 30063/60216 nodes, at backpointers:3:146454609920:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 49%, done 30066/60216 nodes, at backpointers:3:157184163840:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 49%, done 30067/60216 nodes, at backpointers:3:178868715520:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 49%, done 30070/60216 nodes, at backpointers:3:186466697216:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 49%, done 30072/60216 nodes, at backpointers:3:195046146048:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 49%, done 30074/60216 nodes, at backpointers:3:206734622720:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 49%, done 30076/60216 nodes, at backpointers:3:218376962048:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 49%, done 30079/60216 nodes, at backpointers:3:235518558208:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 49%, done 30081/60216 nodes, at backpointers:3:398185725952:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): backpointers_to_extents: 49%, done 30083/60216 nodes, at backpointers:3:401595170816:0
 done
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): check_extents_to_backpointers...bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): extents_to_backpointers: 2%, done 2604/119444 nodes, at extents:38174:156768:4294963371
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): extents_to_backpointers: 4%, done 5593/119444 nodes, at extents:56146:1545824:4294963955
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): extents_to_backpointers: 6%, done 7488/119444 nodes, at extents:58145:360032:4294963371
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): extents_to_backpointers: 7%, done 9161/119444 nodes, at extents:59671:662784:4294963371
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): extents_to_backpointers: 9%, done 11615/119444 nodes, at extents:110314:1517304:4294963375
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): extents_to_backpointers: 11%, done 13700/119444 nodes, at extents:536898594:560088:4294963371
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): extents_to_backpointers: 13%, done 15889/119444 nodes, at extents:536907671:873384:4294963371
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): extents_to_backpointers: 15%, done 18329/119444 nodes, at extents:536922178:1352584:4294963375
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): extents_to_backpointers: 17%, done 20378/119444 nodes, at extents:536923964:4715264:4294963371
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): extents_to_backpointers: 18%, done 22204/119444 nodes, at extents:536927326:251008:4294963371
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): extents_to_backpointers: 19%, done 23848/119444 nodes, at extents:536928326:134784:4294963371
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): extents_to_backpointers: 21%, done 26079/119444 nodes, at extents:1073754845:4101184:4294963371
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): extents_to_backpointers: 23%, done 28083/119444 nodes, at extents:1073774355:308112:4294963371
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): extents_to_backpointers: 25%, done 30517/119444 nodes, at extents:1073786749:90368:4294963375
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): extents_to_backpointers: 27%, done 32672/119444 nodes, at extents:1073791441:156864:4294963371
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): extents_to_backpointers: 28%, done 34536/119444 nodes, at extents:1073792957:634688:4294963371
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): extents_to_backpointers: 30%, done 36203/119444 nodes, at extents:1073794346:127360:4294963371
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): extents_to_backpointers: 31%, done 38089/119444 nodes, at extents:1073841863:1616520:4294963375
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): extents_to_backpointers: 33%, done 39822/119444 nodes, at extents:1610620783:1089360:4294963371
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): extents_to_backpointers: 34%, done 41678/119444 nodes, at extents:1610639163:5136:4294963371
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): extents_to_backpointers: 36%, done 43966/119444 nodes, at extents:1610645472:177128:4294963371
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): extents_to_backpointers: 39%, done 46632/119444 nodes, at extents:1610653529:385792:4294963371
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): extents_to_backpointers: 40%, done 48715/119444 nodes, at extents:1610655742:331592:4294963371
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): extents_to_backpointers: 42%, done 50515/119444 nodes, at extents:1610657209:1184896:4294963371
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): extents_to_backpointers: 44%, done 52777/119444 nodes, at extents:1610696241:59640:4294963371
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): extents_to_backpointers: 45%, done 54600/119444 nodes, at reflink:0:295039616:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): extents_to_backpointers: 47%, done 56530/119444 nodes, at reflink:0:678527000:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): extents_to_backpointers: 48%, done 58349/119444 nodes, at reflink:0:1009160456:0
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): extents_to_backpointers: 3%, done 3749/119444 nodes, at extents:45123:1088256:4294963371
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): extents_to_backpointers: 6%, done 7687/119444 nodes, at extents:58280:82688:4294963371
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): extents_to_backpointers: 9%, done 11498/119444 nodes, at extents:103659:1510648:4294963371
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): extents_to_backpointers: 12%, done 15160/119444 nodes, at extents:536907404:499600:4294963371
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): extents_to_backpointers: 15%, done 18571/119444 nodes, at extents:536922285:1439904:4294963375
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): extents_to_backpointers: 18%, done 22403/119444 nodes, at extents:536927458:352808:4294963371
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): extents_to_backpointers: 21%, done 26163/119444 nodes, at extents:1073756132:9848:4294963371
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): extents_to_backpointers: 25%, done 29881/119444 nodes, at extents:1073780203:95688:4294963371
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): extents_to_backpointers: 28%, done 33576/119444 nodes, at extents:1073792213:588424:4294963371
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): extents_to_backpointers: 31%, done 37459/119444 nodes, at extents:1073817408:280:4294963375
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): extents_to_backpointers: 34%, done 41239/119444 nodes, at extents:1610635383:12800:4294963371
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): extents_to_backpointers: 37%, done 45092/119444 nodes, at extents:1610652607:551280:4294963371
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): extents_to_backpointers: 41%, done 49102/119444 nodes, at extents:1610655961:81280:4294963371
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): extents_to_backpointers: 44%, done 53106/119444 nodes, at extents:1610703234:105032:4294963375
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): extents_to_backpointers: 47%, done 56865/119444 nodes, at reflink:0:741474240:0
 done
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): check_alloc_to_lru_refs... done
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): check_snapshot_trees... done
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): check_snapshots... done
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): check_subvols... done
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): check_subvol_children... done
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): delete_dead_snapshots... done
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): check_root... done
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): check_unreachable_inodes... done
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): check_subvolume_structure... done
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): check_directory_structure...bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): directory structure loop
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): bch2_check_directory_structure(): error erofs_trans_commit
bcachefs (e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c): bch2_fsck_online_thread_fn(): error erofs_trans_commit

(this is same error from #786 but now only shows up while running online fsck)

Errors in dmesg (after running online fsck)-

[34363.271296] bcachefs: check_path() 4097:4294963303
[34363.271302] bcachefs: check_path() 118105:4294963319
[34363.271304] bcachefs: check_path() 4097:4294963319
[34363.271306] bcachefs: check_path() 118105:4294963319

I can still mount and use the filesystem but fsck does not fix it. I can share the current filesystem metdata dump if required.

show-super-

❯ sudo bcachefs show-super /dev/sda1
Device:                                     (unknown device)
External UUID:                             e18ee4c8-86bd-48f5-8b2f-e5cdaf32dd7c
Internal UUID:                             18b84341-3a47-4b7c-837a-8fdf390f8056
Magic number:                              c68573f6-66ce-90a9-d96a-60cf803df7ef
Device index:                              1
Label:                                     BACKUP
Version:                                   1.13: inode_has_child_snapshots
Version upgrade complete:                  1.13: inode_has_child_snapshots
Oldest version on disk:                    1.12: rebalance_work_acct_fix
Created:                                   Tue Sep 17 20:55:11 2024
Sequence number:                           534
Time of last write:                        Sat Nov 30 12:16:17 2024
Superblock size:                           5.91 KiB/1.00 MiB
Clean:                                     1
Devices:                                   3
Sections:                                  members_v1,crypt,replicas_v0,disk_groups,clean,journal_seq_blacklist,journal_v2,counters,members_v2,errors,ext,downgrade
Features:                                  lz4,zstd,journal_seq_blacklist_v3,reflink,new_siphash,inline_data,new_extent_overwrite,btree_ptr_v2,reflink_inline_data,new_varint,journal_no_flush,alloc_v2,extents_across_btree_nodes
Compat features:                           alloc_info,alloc_metadata,extents_above_btree_updates_done,bformat_overflow_done

Options:
  block_size:                              4.00 KiB
  btree_node_size:                         256 KiB
  errors:                                  continue [fix_safe] panic ro 
  metadata_replicas:                       2
  data_replicas:                           1
  metadata_replicas_required:              1
  data_replicas_required:                  1
  encoded_extent_max:                      64.0 KiB
  metadata_checksum:                       none [crc32c] crc64 xxhash 
  data_checksum:                           none [crc32c] crc64 xxhash 
  compression:                             none
  background_compression:                  zstd:15
  str_hash:                                crc32c crc64 [siphash] 
  metadata_target:                         none
  foreground_target:                       ssd
  background_target:                       hdd
  promote_target:                          ssd
  erasure_code:                            0
  inodes_32bit:                            1
  shard_inode_numbers:                     1
  inodes_use_key_cache:                    1
  gc_reserve_percent:                      8
  gc_reserve_bytes:                        0 B
  root_reserve_percent:                    0
  wide_macs:                               0
  promote_whole_extents:                   1
  acl:                                     1
  usrquota:                                0
  grpquota:                                0
  prjquota:                                0
  journal_flush_delay:                     1000
  journal_flush_disabled:                  0
  journal_reclaim_delay:                   100
  journal_transaction_names:               1
  allocator_stuck_timeout:                 30
  version_upgrade:                         [compatible] incompatible none 
  nocow:                                   0

members_v2 (size 592):
Device:                                    0
  Label:                                   wd_passport (10)
  UUID:                                    562d68a2-f982-4123-8708-86e332a7feb5
  Size:                                    931 GiB
  read errors:                             1
  write errors:                            29
  checksum errors:                         0
  seqread iops:                            0
  seqwrite iops:                           0
  randread iops:                           0
  randwrite iops:                          0
  Bucket size:                             256 KiB
  First bucket:                            0
  Buckets:                                 3815340
  Last mount:                              Sat Nov 30 12:02:58 2024
  Last superblock write:                   534
  State:                                   rw
  Data allowed:                            journal,btree,user
  Has data:                                journal,btree,user,cached
  Btree allocated bitmap blocksize:        32.0 MiB
  Btree allocated bitmap:                  0000000000000000000000000001000000000001011001101100000111101111
  Durability:                              1
  Discard:                                 0
  Freespace initialized:                   1
Device:                                    1
  Label:                                   wd_black (8)
  UUID:                                    54913cb0-990d-459d-a03e-0e876aad92d1
  Size:                                    3.64 TiB
  read errors:                             1179417
  write errors:                            402
  checksum errors:                         8
  seqread iops:                            0
  seqwrite iops:                           0
  randread iops:                           0
  randwrite iops:                          0
  Bucket size:                             256 KiB
  First bucket:                            0
  Buckets:                                 15261652
  Last mount:                              Sat Nov 30 12:02:58 2024
  Last superblock write:                   534
  State:                                   rw
  Data allowed:                            journal,btree,user
  Has data:                                journal,btree,user,cached
  Btree allocated bitmap blocksize:        64.0 MiB
  Btree allocated bitmap:                  0000000000010000010000000000000000000000000000000001110000001011
  Durability:                              1
  Discard:                                 0
  Freespace initialized:                   1
Device:                                    3
  Label:                                   adata (7)
  UUID:                                    858f7f3f-5d6d-48d6-ba36-699f8691d484
  Size:                                    238 GiB
  read errors:                             0
  write errors:                            0
  checksum errors:                         0
  seqread iops:                            0
  seqwrite iops:                           0
  randread iops:                           0
  randwrite iops:                          0
  Bucket size:                             256 KiB
  First bucket:                            0
  Buckets:                                 976788
  Last mount:                              Sat Nov 30 12:02:58 2024
  Last superblock write:                   534
  State:                                   rw
  Data allowed:                            journal,btree,user
  Has data:                                journal,btree,user,cached
  Btree allocated bitmap blocksize:        8.00 MiB
  Btree allocated bitmap:                  0000000000000000010000000000000111111111111111111111111111111111
  Durability:                              1
  Discard:                                 1
  Freespace initialized:                   1

errors (size 120):
journal_entry_replicas_not_marked           1               Sat Oct  5 11:04:04 2024
btree_node_data_missing                     36              Thu Oct 10 17:15:04 2024
snapshot_tree_to_missing_subvol             1               Wed Nov 13 13:19:48 2024
inode_dir_wrong_nlink                       27              Fri Nov 29 16:22:58 2024
inode_unreachable                           35              Thu Nov 28 21:17:39 2024
deleted_inode_but_clean                     32              Thu Nov 28 21:17:55 2024
accounting_mismatch                         17              Fri Nov 29 23:41:49 2024

Host device:

Device- Raspberry Pi 5 (8GB)
Kernel- Linux rpl 6.12.1-v8-16k+ #1 SMP PREEMPT Fri Nov 29 20:14:53 IST 2024 aarch64 GNU/Linux
bcachefs-tools version- 1.13.0
@nitinkmr333 nitinkmr333 mentioned this issue Nov 30, 2024
Closed
@nitinkmr333
Copy link
Author

This is the same error I faced- #786 (comment)

@koverstreet
Copy link
Owner

Can one of you get me a metadata dump?

Use the 'bcachefs dump' command, then hop on the IRC channel and magic wormhole the file(s) to me; something funny is going on with directory structure repair.

@nitinkmr333
Copy link
Author

For some reason, fsck now completes fine and does not show any error. I did not upgrade the kernel (it is still on 6.12.1) during this time. Only thing I did was to delete some folders and delete older snapshots to free up some space.

@koverstreet Since you have fixed #786 (thanks btw), I doubt we will face this issue again with your master branch (since #786 was a prerequisite to it).

Still, I will try to reproduce it.

@koverstreet
Copy link
Owner

Alright, let me know if you're successful: I suspect there is indeed still a corner case lurking in the loop checking/reattach path. Not the most urgent thing right now, but would be good to find.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@koverstreet @nitinkmr333