Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Extremely slow system update (ubuntu, apt-get upgrade, dpkg). #789

Open
0xfk0 opened this issue Nov 29, 2024 · 2 comments
Open

Extremely slow system update (ubuntu, apt-get upgrade, dpkg). #789

0xfk0 opened this issue Nov 29, 2024 · 2 comments

Comments

@0xfk0
Copy link

0xfk0 commented Nov 29, 2024

During system upgrade, dpkg unpacks packages EXTREMELY SLOW. Like one packet in a tens of seconds. Need to update ~2000 packets. Looks like system is completely broken and only way to reinstall it without bcachefs. This is most worses bug among others.

The problem is in bcachefs, see backtrace of dpkg program:

root@jupiter:~# cat /proc/1580987/stack
[<0>] bch2_journal_flush_seq+0x114/0x150
[<0>] bch2_flush_inode.part.0+0xd8/0x1a0
[<0>] bch2_fsync+0xff/0x160
[<0>] vfs_fsync_range+0x42/0xa0
[<0>] do_fsync+0x41/0x90
[<0>] __x64_sys_fsync+0x13/0x20
[<0>] x64_sys_call+0x2181/0x24e0
[<0>] do_syscall_64+0x70/0x130
[<0>] entry_SYSCALL_64_after_hwframe+0x76/0x7e

No any messages from bcachefs in kernel log (dmesg).

See strace output:

22:09:40 fsync(10)                      = 0
22:09:40 close(10)                      = 0
22:09:40 rename("/usr/share/locale/de/LC_MESSAGES/konsole.mo.dpkg-new", "/usr/share/locale/de/LC_MESSAGES/konsole.mo") = 0
22:09:40 openat(AT_FDCWD, "/usr/share/locale/el/LC_MESSAGES/konsole.mo.dpkg-new", O_WRONLY) = 10
22:09:40 fsync(10)                      = 0
22:09:40 close(10)                      = 0
22:09:40 rename("/usr/share/locale/el/LC_MESSAGES/konsole.mo.dpkg-new", "/usr/share/locale/el/LC_MESSAGES/konsole.mo") = 0
22:09:40 openat(AT_FDCWD, "/usr/share/locale/en_GB/LC_MESSAGES/konsole.mo.dpkg-new", O_WRONLY) = 10
22:09:40 fsync(10)                      = 0
22:09:41 close(10)                      = 0
22:09:41 rename("/usr/share/locale/en_GB/LC_MESSAGES/konsole.mo.dpkg-new", "/usr/share/locale/en_GB/LC_MESSAGES/konsole.mo") = 0
22:09:41 openat(AT_FDCWD, "/usr/share/locale/eo/LC_MESSAGES/konsole.mo.dpkg-new", O_WRONLY) = 10
22:09:41 fsync(10)                      = 0
22:09:41 close(10)                      = 0
22:09:41 rename("/usr/share/locale/eo/LC_MESSAGES/konsole.mo.dpkg-new", "/usr/share/locale/eo/LC_MESSAGES/konsole.mo") = 0
22:09:41 openat(AT_FDCWD, "/usr/share/locale/es/LC_MESSAGES/konsole.mo.dpkg-new", O_WRONLY) = 10
22:09:41 fsync(10)                      = 0
22:09:41 close(10)                      = 0
22:09:41 rename("/usr/share/locale/es/LC_MESSAGES/konsole.mo.dpkg-new", "/usr/share/locale/es/LC_MESSAGES/konsole.mo") = 0
22:09:41 openat(AT_FDCWD, "/usr/share/locale/et/LC_MESSAGES/konsole.mo.dpkg-new", O_WRONLY) = 10
22:09:41 fsync(10)                      = 0
22:09:42 close(10)                      = 0
22:09:42 rename("/usr/share/locale/et/LC_MESSAGES/konsole.mo.dpkg-new", "/usr/share/locale/et/LC_MESSAGES/konsole.mo") = 0
22:09:42 openat(AT_FDCWD, "/usr/share/locale/eu/LC_MESSAGES/konsole.mo.dpkg-new", O_WRONLY) = 10
22:09:42 fsync(10)                      = 0
22:09:42 close(10)                      = 0
22:09:42 rename("/usr/share/locale/eu/LC_MESSAGES/konsole.mo.dpkg-new", "/usr/share/locale/eu/LC_MESSAGES/konsole.mo") = 0
22:09:42 openat(AT_FDCWD, "/usr/share/locale/fa/LC_MESSAGES/konsole.mo.dpkg-new", O_WRONLY) = 10
22:09:42 fsync(10)                      = 0
22:09:42 close(10)                      = 0
22:09:42 rename("/usr/share/locale/fa/LC_MESSAGES/konsole.mo.dpkg-new", "/usr/share/locale/fa/LC_MESSAGES/konsole.mo") = 0
22:09:42 openat(AT_FDCWD, "/usr/share/locale/fi/LC_MESSAGES/konsole.mo.dpkg-new", O_WRONLY) = 10
22:09:42 fsync(10)                      = 0
22:09:42 close(10)                      = 0
22:09:42 rename("/usr/share/locale/fi/LC_MESSAGES/konsole.mo.dpkg-new", "/usr/share/locale/fi/LC_MESSAGES/konsole.mo") = 0
22:09:42 openat(AT_FDCWD, "/usr/share/locale/fr/LC_MESSAGES/konsole.mo.dpkg-new", O_WRONLY) = 10
22:09:42 fsync(10)                      = 0
22:09:42 close(10)                      = 0
22:09:42 rename("/usr/share/locale/fr/LC_MESSAGES/konsole.mo.dpkg-new", "/usr/share/locale/fr/LC_MESSAGES/konsole.mo") = 0
22:09:42 openat(AT_FDCWD, "/usr/share/locale/fy/LC_MESSAGES/konsole.mo.dpkg-new", O_WRONLY) = 10
22:09:42 fsync(10)                      = 0
22:09:42 close(10)                      = 0
22:09:42 rename("/usr/share/locale/fy/LC_MESSAGES/konsole.mo.dpkg-new", "/usr/share/locale/fy/LC_MESSAGES/konsole.mo") = 0
22:09:42 openat(AT_FDCWD, "/usr/share/locale/ga/LC_MESSAGES/konsole.mo.dpkg-new", O_WRONLY) = 10
22:09:42 fsync(10)                      = 0
22:09:42 close(10)                      = 0
22:09:42 rename("/usr/share/locale/ga/LC_MESSAGES/konsole.mo.dpkg-new", "/usr/share/locale/ga/LC_MESSAGES/konsole.mo") = 0
22:09:42 openat(AT_FDCWD, "/usr/share/locale/gl/LC_MESSAGES/konsole.mo.dpkg-new", O_WRONLY) = 10
22:09:42 fsync(10)                      = 0
22:09:42 close(10)                      = 0
22:09:42 rename("/usr/share/locale/gl/LC_MESSAGES/konsole.mo.dpkg-new", "/usr/share/locale/gl/LC_MESSAGES/konsole.mo") = 0
22:09:42 openat(AT_FDCWD, "/usr/share/locale/gu/LC_MESSAGES/konsole.mo.dpkg-new", O_WRONLY) = 10
22:09:42 fsync(10)                      = 0
22:09:42 close(10)                      = 0
22:09:42 rename("/usr/share/locale/gu/LC_MESSAGES/konsole.mo.dpkg-new", "/usr/share/locale/gu/LC_MESSAGES/konsole.mo") = 0
22:09:42 openat(AT_FDCWD, "/usr/share/locale/he/LC_MESSAGES/konsole.mo.dpkg-new", O_WRONLY) = 10
22:09:42 fsync(10)                      = 0
22:09:42 close(10)                      = 0
22:09:42 rename("/usr/share/locale/he/LC_MESSAGES/konsole.mo.dpkg-new", "/usr/share/locale/he/LC_MESSAGES/konsole.mo") = 0
22:09:42 openat(AT_FDCWD, "/usr/share/locale/hi/LC_MESSAGES/konsole.mo.dpkg-new", O_WRONLY) = 10
22:09:42 fsync(10)                      = 0
22:09:42 close(10)                      = 0
22:09:42 rename("/usr/share/locale/hi/LC_MESSAGES/konsole.mo.dpkg-new", "/usr/share/locale/hi/LC_MESSAGES/konsole.mo") = 0
22:09:42 openat(AT_FDCWD, "/usr/share/locale/hne/LC_MESSAGES/konsole.mo.dpkg-new", O_WRONLY) = 10
22:09:42 fsync(10)                      = 0
22:09:42 close(10)                      = 0
22:09:42 rename("/usr/share/locale/hne/LC_MESSAGES/konsole.mo.dpkg-new", "/usr/share/locale/hne/LC_MESSAGES/konsole.mo") = 0
22:09:42 openat(AT_FDCWD, "/usr/share/locale/hr/LC_MESSAGES/konsole.mo.dpkg-new", O_WRONLY) = 10
22:09:42 fsync(10)                      = 0
22:09:42 close(10)                      = 0
22:09:42 rename("/usr/share/locale/hr/LC_MESSAGES/konsole.mo.dpkg-new", "/usr/share/locale/hr/LC_MESSAGES/konsole.mo") = 0
22:09:42 openat(AT_FDCWD, "/usr/share/locale/hsb/LC_MESSAGES/konsole.mo.dpkg-new", O_WRONLY) = 10
22:09:42 fsync(10)                      = 0
22:09:42 close(10)                      = 0
22:09:42 rename("/usr/share/locale/hsb/LC_MESSAGES/konsole.mo.dpkg-new", "/usr/share/locale/hsb/LC_MESSAGES/konsole.mo") = 0
22:09:42 openat(AT_FDCWD, "/usr/share/locale/hu/LC_MESSAGES/konsole.mo.dpkg-new", O_WRONLY) = 10
22:09:42 fsync(10)                      = 0
22:09:42 close(10)                      = 0
22:09:42 rename("/usr/share/locale/hu/LC_MESSAGES/konsole.mo.dpkg-new", "/usr/share/locale/hu/LC_MESSAGES/konsole.mo") = 0
22:09:42 openat(AT_FDCWD, "/usr/share/locale/ia/LC_MESSAGES/konsole.mo.dpkg-new", O_WRONLY) = 10
22:09:42 fsync(10)                      = 0
22:09:42 close(10)                      = 0
22:09:42 rename("/usr/share/locale/ia/LC_MESSAGES/konsole.mo.dpkg-new", "/usr/share/locale/ia/LC_MESSAGES/konsole.mo") = 0
22:09:42 openat(AT_FDCWD, "/usr/share/locale/id/LC_MESSAGES/konsole.mo.dpkg-new", O_WRONLY) = 10
22:09:42 fsync(10)                      = 0
22:09:42 close(10)                      = 0
22:09:42 rename("/usr/share/locale/id/LC_MESSAGES/konsole.mo.dpkg-new", "/usr/share/locale/id/LC_MESSAGES/konsole.mo") = 0
22:09:42 openat(AT_FDCWD, "/usr/share/locale/ie/LC_MESSAGES/konsole.mo.dpkg-new", O_WRONLY) = 10
22:09:42 fsync(10)                      = 0
22:09:42 close(10)                      = 0
22:09:42 rename("/usr/share/locale/ie/LC_MESSAGES/konsole.mo.dpkg-new", "/usr/share/locale/ie/LC_MESSAGES/konsole.mo") = 0
22:09:42 openat(AT_FDCWD, "/usr/share/locale/is/LC_MESSAGES/konsole.mo.dpkg-new", O_WRONLY) = 10
22:09:42 fsync(10)                      = 0
22:09:42 close(10)                      = 0
22:09:42 rename("/usr/share/locale/is/LC_MESSAGES/konsole.mo.dpkg-new", "/usr/share/locale/is/LC_MESSAGES/konsole.mo") = 0
22:09:42 openat(AT_FDCWD, "/usr/share/locale/it/LC_MESSAGES/konsole.mo.dpkg-new", O_WRONLY) = 10
22:09:42 fsync(10)                      = 0
22:09:42 close(10)                      = 0
22:09:42 rename("/usr/share/locale/it/LC_MESSAGES/konsole.mo.dpkg-new", "/usr/share/locale/it/LC_MESSAGES/konsole.mo") = 0
22:09:42 openat(AT_FDCWD, "/usr/share/locale/ja/LC_MESSAGES/konsole.mo.dpkg-new", O_WRONLY) = 10
22:09:42 fsync(10)                      = 0
22:09:42 close(10)                      = 0
22:09:42 rename("/usr/share/locale/ja/LC_MESSAGES/konsole.mo.dpkg-new", "/usr/share/locale/ja/LC_MESSAGES/konsole.mo") = 0
22:09:42 openat(AT_FDCWD, "/usr/share/locale/ka/LC_MESSAGES/konsole.mo.dpkg-new", O_WRONLY) = 10
22:09:42 fsync(10)                      = 0
22:09:44 close(10)                      = 0
22:09:44 rename("/usr/share/locale/ka/LC_MESSAGES/konsole.mo.dpkg-new", "/usr/share/locale/ka/LC_MESSAGES/konsole.mo") = 0
22:09:44 openat(AT_FDCWD, "/usr/share/locale/kk/LC_MESSAGES/konsole.mo.dpkg-new", O_WRONLY) = 10
22:09:44 fsync(10)                      = 0
22:09:44 close(10)                      = 0
22:09:44 rename("/usr/share/locale/kk/LC_MESSAGES/konsole.mo.dpkg-new", "/usr/share/locale/kk/LC_MESSAGES/konsole.mo") = 0
22:09:44 openat(AT_FDCWD, "/usr/share/locale/km/LC_MESSAGES/konsole.mo.dpkg-new", O_WRONLY) = 10
22:09:44 fsync(10)                      = 0
22:09:44 close(10)                      = 0
22:09:44 rename("/usr/share/locale/km/LC_MESSAGES/konsole.mo.dpkg-new", "/usr/share/locale/km/LC_MESSAGES/konsole.mo") = 0
22:09:44 openat(AT_FDCWD, "/usr/share/locale/kn/LC_MESSAGES/konsole.mo.dpkg-new", O_WRONLY) = 10
22:09:44 fsync(10)                      = 0
22:09:44 close(10)                      = 0
22:09:44 rename("/usr/share/locale/kn/LC_MESSAGES/konsole.mo.dpkg-new", "/usr/share/locale/kn/LC_MESSAGES/konsole.mo") = 0
22:09:44 openat(AT_FDCWD, "/usr/share/locale/ko/LC_MESSAGES/konsole.mo.dpkg-new", O_WRONLY) = 10
22:09:44 fsync(10)                      = 0
22:09:47 close(10)                      = 0
22:09:47 rename("/usr/share/locale/ko/LC_MESSAGES/konsole.mo.dpkg-new", "/usr/share/locale/ko/LC_MESSAGES/konsole.mo") = 0
22:09:47 openat(AT_FDCWD, "/usr/share/locale/ku/LC_MESSAGES/konsole.mo.dpkg-new", O_WRONLY) = 10
22:09:47 fsync(10)                      = 0
22:09:47 close(10)                      = 0
22:09:47 rename("/usr/share/locale/ku/LC_MESSAGES/konsole.mo.dpkg-new", "/usr/share/locale/ku/LC_MESSAGES/konsole.mo") = 0
22:09:47 openat(AT_FDCWD, "/usr/share/locale/lt/LC_MESSAGES/konsole.mo.dpkg-new", O_WRONLY) = 10
22:09:47 fsync(10)                      = 0
22:09:47 close(10)                      = 0
22:09:47 rename("/usr/share/locale/lt/LC_MESSAGES/konsole.mo.dpkg-new", "/usr/share/locale/lt/LC_MESSAGES/konsole.mo") = 0
22:09:47 openat(AT_FDCWD, "/usr/share/locale/lv/LC_MESSAGES/konsole.mo.dpkg-new", O_WRONLY) = 10
22:09:47 fsync(10^Cstrace: Process 1580987 detached

Very slow execution is caused by the fact, that dpkg calls fsync after every operation on the files. This not causes significant problem on another file systems, but causes significant slowdown on bcachefs.
@0xfk0
Copy link
Author

0xfk0 commented Nov 29, 2024

Tried to "evacuate" data from SSD. 0%, nothing done, 100% cpu load. File system is unusable. Everything is very slow.

# bcachefs device evacuate /dev/nvme0n1p3
Setting /dev/nvme0n1p3 readonly
0% complete: current position btree extents:604196383:82380928

Only last two numbers changed once per second.

Looks like "a new version" (starting from 6.12 kernel) of bcachefs is completely broken and unusable. Trying to reboot, not expecting, that I will be able to boot up...

@0xfk0
Copy link
Author

0xfk0 commented Nov 29, 2024

Tried call sync in console. It takes 10 seconds.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@0xfk0