Instance: Fix deadlock during failed snapshot creation

[MggMuggins]

Fixes #13466

We need a non-locking delete on an Instance; otherwise a failure while mounting the instance/updating the backup file will deadlock.

I don't love the type assertion here. However, since CreateInternal returns Instance, we don't have a lot of options beyond enriching that interface or providing an additional one.

In order to avoid using type assertions we'd need to define an internal interface instance for delete and possibly a few other functions in the drivers/ package and implement it for lxc/qemu. Alternatively, moving the Interface interface into the same package as the drivers would allow us to define some private methods on Interface instead.

(Updating the backup file on the full volume was very likely the failure in #13466; this is after applying the fix):

$ lxc snapshot lucky-bulldog
Error: Failed to create file "/var/lib/lxd/containers/microcloud-test_lucky-bulldog/backup.yaml": open /var/lib/lxd/containers/microcloud-test_lucky-bulldog/backup.yaml: disk quota exceeded

I will throw a snapshot failure test on here this afternoon.

LXD-1117

[MggMuggins]

lxc stop also fails on zfs when the disk is full:
Error: Stopping the instance failed: Failed clearing ownership: chown /home/wesley/Workspace/lxd/test/tmp.KFM/iaA/containers/c1: disk quota exceeded

The container does become STOPPED, but this prevents lxc delete -f c1 from removing the container, hence the separate call to lxc stop. I'm happy to open another bug for this if it's undesired; it's inconvenient but easy to work around.

I tried rm /root/big.bin before deleting the instance and while I was able to remove the file, the dataset USED/AVAIL percentages didn't change. I doubt that there was a process still holding the file open (no lsof in testimage but only two processes, init and ps from lxc exec c1 -- ps) so I'm sure this is some zfs feature/behavior I'm not considering/don't know about.

Other backends:

Backend	Result
dir	❌ :lxc snapshot c1 succeeds
btrfs	❌ lxc snapshot c1 succeeds; 3 MiB root disk required
lvm	❌ lxc snapshot c1 succeeds; 9 MiB root disk required

Didn't try Ceph as I don't have a cluster handy.

Let me know if you'd like other changes; thanks!

[tomponline]

lxc stop also fails on zfs when the disk is full:
Error: Stopping the instance failed: Failed clearing ownership: chown /home/wesley/Workspace/lxd/test/tmp.KFM/iaA/containers/c1: disk quota exceeded

I think we should log a warning but continue the onstop hook function if the error contains "disk quota exceeded" (or if there is a better way to detect that error).

[MggMuggins]

Correct me if I'm wrong, but the failure of those chown/chmod would only leave the mount point owned by a uid that could potentially be reused for another container. All that does is allow the mount point to be traversed by a process running under that uid that has access to the parent dir. Agreed, not a concern. Thanks for your patience.

[tomponline]

Correct me if I'm wrong, but the failure of those chown/chmod would only leave the mount point owned by a uid that could potentially be reused for another container. All that does is allow the mount point to be traversed by a process running under that uid that has access to the parent dir. Agreed, not a concern. Thanks for your patience.

Yes thats my understanding, and either way that doesn't succeeded whether we return an error or not, its just that by returning an error further bad things happen because the instance's devices are not cleaned up on the host side.