compression leads to filesystem hangs

[hyperfekt]

When I create a filesystem with bcachefs format --compression=zstd --background_compression=zstd --foreground_target=/dev/sda1 --background_target=/dev/sdb1 --promote_target=/dev/sda1 --encrypted --label=nixos --force /dev/sdb1 --discard /dev/sda1 and attempt to install NixOS on it the install hangs on an fdatasync syscall with 100% CPU usage on one core not attributed to any process in top.
Here is the strace -f log, with anything from line 76676 onwards trigged by sending SIGINT, which does not stop the hang.

[hyperfekt]

I'm also getting hangs trying to copy data off of the filesystem or trying to get a git status afterwards - it seems like trying to install nixos makes the entire file system unusable.

Mind that these hangs happen both with the reverted commits and the current head.

[koverstreet]

Which commits were you reverting?

I'm not seeing anything with xfstests, and I wouldn't expect a hang in fdatasync to have anything to do with the format options, so something weird is going on...

Can you try without zstd?

[hyperfekt]

Sorry, I should have used the term 'amended', it seems you made this change at some point and I was referring to it happening even with it:

diff --git a/fs/bcachefs/btree_gc.h b/fs/bcachefs/btree_gc.h
index 1905acfa028a..8af5f841a537 100644
--- a/fs/bcachefs/btree_gc.h
+++ b/fs/bcachefs/btree_gc.h
@@ -109,7 +109,7 @@ static inline bool gc_visited(struct bch_fs *c, struct gc_pos pos)
 
        do {
                seq = read_seqcount_begin(&c->gc_pos_lock);
-               ret = gc_pos_cmp(pos, c->gc_pos) < 0;
+               ret = gc_pos_cmp(pos, c->gc_pos) <= 0;
        } while (read_seqcount_retry(&c->gc_pos_lock, seq));
 
        return ret;

I'm still installing but I got past the point where it usually hangs by omitting zstd, looks like that is where the problem lies.

[hyperfekt]

The issue seems to also occur with lz4, but I can confirm it does not happen without compression.

[koverstreet]

Oh, it might be because of the background_compression option. Can you try without that?

…

On Sun, Feb 10, 2019 at 5:00 PM hyperfekt ***@***.***> wrote: The issue seems to also occur with lz4, but I can confirm it does not happen without compression. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#16 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB5r7qAs20vZ9Nd90YEofv9AxuvmTbVhks5vMJaGgaJpZM4ays8k> .

[hyperfekt]

Happens with only --compression, too.

[koverstreet]

Can you get me some backtraces from when it happens? perf top output? cat /proc/pid/stack is useful for getting backtraces. I've got a torture test running with lz4 - copying some data to a filesystem, doing a sync (which hits the same paths as fdatasync), deleting and starting over - but I'm not seeing anything so far. Can you tell me anything more about how to repro it?

…

On Sun, Feb 10, 2019 at 5:19 PM hyperfekt ***@***.***> wrote: Happens with only --compression, too. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#16 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB5r7hqbq780dlT_vspjDXyVV9-geOkUks5vMJrugaJpZM4ays8k> .

[hyperfekt]

perf top output does not show whatever is maxing out the core.
backtrace:

[<0>] io_schedule+0x12/0x40
[<0>] wait_on_page_bit_common+0xfe/0x210
[<0>] __filemap_fdatawait_range+0xd5/0x130
[<0>] file_write_and_wait_range+0x68/0x90
[<0>] bch2_fsync+0x28/0xa0 [bcachefs]
[<0>] do_fsync+0x38/0x60
[<0>] __x64_sys_fdatasync+0x13/0x20
[<0>] do_syscall_64+0x4e/0x100
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[<0>] 0xffffffffffffffff

To reproduce, first, we need a NixOS ISO with bcachefs support, which can be created by installing nix, cloning https://github.com/hyperfekt/nix-exp, doing nix-channel --add http://nixos.org/channels/nixos-unstable nixos-unstable and nix-channel --add http://nixos.org/channels/nixos-18.09 nixos, and then doing nix-build '<nixpkgs/nixos>' -A config.system.build.isoImage -I nixos-config=nix-exp/installation-cd-bcachefs-graphical-kde-new-kernel-git.nix. The ISO will be located in a directory named result/iso.

After starting from the NixOS ISO, creating the filesystem, mounting it under /mnt, doing nixos-generate-config --root /mnt, copying this file to /mnt/etc/nixos, add it to the imports list of the /mnt/etc/nixos/configuration.nix file. Then do nix-channel --add http://nixos.org/channels/nixos-unstable nixos-unstable and run nixos-install --root /mnt. The installation should hang with the output of querying info about missing paths. Make sure you've provided networking.

It might be possible to get the nixos-install tool without the ISO and test the whole thing without NixOS, but I'm not able to figure out on the quick how.

[koverstreet]

I managed to repro it - working on it now.

…

On Mon, Feb 11, 2019 at 1:15 AM hyperfekt ***@***.***> wrote: perf top output <https://github.com/koverstreet/bcachefs/files/2849984/perf-top.log> backtrace <https://github.com/koverstreet/bcachefs/files/2849985/backtrace.log> You should be able to reproduce it by starting a NixOS ISO, creating the filesystem, mounting it under /mnt, doing nixos-generate-config --root /mnt, copying this file to /mnt/etc/nixos and adding it to the imports list of the /mnt/etc/nixos/configuration.nix file. Then do nix-channel --add http://nixos.org/channels/nixos-unstable nixos-unstable and run nixos-install --root /mnt. The installation should hang with the output of querying info about missing paths. Make sure you've provided networking. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#16 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB5r7gUV3_Kkmu8nQFIeTMysT6IYE21iks5vMQpygaJpZM4ays8k> .

[koverstreet]

I think I've got it fixed - can you test it? On Mon, Feb 11, 2019 at 3:46 PM Kent Overstreet <[email protected]> wrote:

…

I managed to repro it - working on it now. On Mon, Feb 11, 2019 at 1:15 AM hyperfekt ***@***.***> wrote: > perf top output > <https://github.com/koverstreet/bcachefs/files/2849984/perf-top.log> > backtrace > <https://github.com/koverstreet/bcachefs/files/2849985/backtrace.log> > > You should be able to reproduce it by starting a NixOS ISO, creating the > filesystem, mounting it under /mnt, doing nixos-generate-config --root > /mnt, copying this file to /mnt/etc/nixos and adding it to the imports > list of the /mnt/etc/nixos/configuration.nix file. Then do nix-channel > --add http://nixos.org/channels/nixos-unstable nixos-unstable and run nixos-install > --root /mnt. The installation should hang with the output of querying > info about missing paths. Make sure you've provided networking. > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <#16 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AB5r7gUV3_Kkmu8nQFIeTMysT6IYE21iks5vMQpygaJpZM4ays8k> > . >

[hyperfekt]

The install works fine now, thanks for the quick fix!