Add support for ZFS delegation

[stgraber]

This adds support for the ZFS dataset delegation feature found in OpenZFS 2.2.

On the LXD front, this is done through a new zfs.delegate configuration option on storage volumes.
One can therefore do:

lxc init images:ubuntu/22.04 u1
lxc storage volume set default container/u1 zfs.delegate=true
lxc start u1

To get a container that has delegated ZFS access to its dataset and anything underneath it.
The same works as expected with custom storage volumes which can therefore allow multiple containers to interact with the dataset.

As part of implementing this, the ZFS storage driver had to be updated to handle recursive operations throughout, so that those nested datasets don't get lost during a backup or migration.

Lastly, the way this is implemented is through new DelegateInstance and DelegateCustomVolume functions on the storage backend which are called after the instance is running (as the PID is required). This allowed for the work to be mostly contained to the ZFS storage driver. That's with the exception of the logic enabling /dev/zfs in the container as that did need to be directly to the LXC driver and has to be conditional on the correct ZFS version being on the system as one doesn't want to expose /dev/zfs to containers on prior versions (potential security risk there).

This work was sponsored by Buddy (https://buddy.works) who also sponsored the matching OpenZFS work.

Closes #11796

[tomponline]

@stgraber does this close #11796 ?

[stgraber]

@tomponline it does indeed

[stgraber]

Review feedback addressed

[stgraber]

@tomponline rebased following changes to main branch

[tomponline]

Thanks @stgraber I'm going to look at this once I've got the LXD 5.16 release done and dusted. Cheers

We've also got #12067 which might cause this to need to be rebased again once its resolved.

[tomponline]

Thanks @stgraber I'm looking forward to this!

I've added a few comments/questions.

WRT to the recursion changes, we could land that as a separate PR as it feels like that is a self contained precursor to the actual ZFS delegation?

[stgraber]

WRT to the recursion changes, we could land that as a separate PR as it feels like that is a self contained precursor to the actual ZFS delegation?

We could but it wouldn't be testable on its own as the only case where recursion makes sense is when delegation is also possible. So it feels like having the two merged together (in separate commits as they are here) is appropriate.

[stgraber]

@tomponline I believe that's all feedback addressed, just waiting to confirm I didn't break anything with those MountInfo changes :)

[stgraber]

@tomponline all tests passed, ready to go

[tomponline]

LGTM, thanks!

[dalbani]

First, it's really great that you implemented it 👍
Out of curiosity, which release of LXD will this feature find its way in?
And I suppose that a backport to LXD 5.0 is not to be expected?

[tomponline]

This will be in lxd 5.17.

As for backporting. As it depends on a very recent version of zfs and kernel, it'll be a while before we back port and we'll need to switch to the newer core22 base first too.