Multiple PVs referring to the same bucket cannot be consumed by one Pod

[songjiaxun]

Symptom

When multiple PVs are consuming the same bucket via the field volumeHandle, and the PVCs bound to these PVs are consumed by the same Pod, the volume mount will time out.

For example, the following Pod will be stuck in the volume mount stage.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: gcs-fuse-csi-static-pvc-1
spec:
  ...
  volumeName: gcs-fuse-csi-pv-1
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: gcs-fuse-csi-pv-1
spec:
  ...
  csi:
    driver: gcsfuse.csi.storage.gke.io
    volumeHandle: same-bucket-name
    readOnly: false
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: gcs-fuse-csi-static-pvc-2
spec:
  ...
  volumeName: gcs-fuse-csi-pv-2
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: gcs-fuse-csi-pv-2
spec:
  ...
  csi:
    driver: gcsfuse.csi.storage.gke.io
    volumeHandle: same-bucket-name
    readOnly: false
---
apiVersion: v1
kind: Pod
metadata:
  name: gcs-fuse-csi-example
  annotations:
    gke-gcsfuse/volumes: "true"
spec:
  serviceAccountName: gcs-csi
  containers:
    ...
  volumes:
  - name: gcs-fuse-csi-static-1
    persistentVolumeClaim:
      claimName: gcs-fuse-csi-static-pvc-1
  - name: gcs-fuse-csi-static-2
    persistentVolumeClaim:
      claimName: gcs-fuse-csi-static-pvc-2

Root Cause

According to the kubelet code: https://github.com/kubernetes/kubernetes/blob/8f15859afc9cfaeb05d4915ffa204d84da512094/pkg/kubelet/volumemanager/cache/desired_state_of_world.go#L296-L298

For non-attachable and non-device-mountable volumes, generate a unique name based on the pod namespace and name and the name of the volume within the pod.

In the case of using a CSI driver and a pre-provisioned PV, the volume name is specified via the volumeHandle. Different PVs will be treated as the same volume, therefore after kubelet mounts one of the PVs, the other PVs will be treated as already mounted. As a result, the Pod will be stuck in volume mount stage.

Solution

If for some reason, your Pod need to consume multiple volumes pointing to the same bucket, please consider using the following two approaches:

1. CSI ephemeral inline volume

Please use CSI ephemeral inline volume to configure Pod if the Pod needs to mount the same bucket to different mount paths.

For example, the following Pod has two volumes referring to the same bucket same-bucket-name, but the volumes are mounted to different mount paths using different mount options.

apiVersion: v1
kind: Pod
metadata:
  name: gcs-fuse-csi-example
  annotations:
    gke-gcsfuse/volumes: "true"
spec:
  serviceAccountName: gcs-csi
  containers:
    - image: busybox
      name: busybox
      command: ["sleep"]
      args: ["infinity"]
      volumeMounts:
      - name: gcs-fuse-csi-inline-1
        mountPath: "/upload"
      - name: gcs-fuse-csi-inline-2
        mountPath: "/download"
  volumes:
  - name: gcs-fuse-csi-inline-1
    csi:
      driver: gcsfuse.csi.storage.gke.io
      volumeAttributes:
        bucketName: same-bucket-name
        mountOptions: "debug_fuse,debug_fs,debug_gcs,implicit-dirs,only-dir=upload"
  - name: gcs-fuse-csi-inline-2
    csi:
      driver: gcsfuse.csi.storage.gke.io
      volumeAttributes:
        bucketName: same-bucket-name
        mountOptions: "debug_fuse,debug_fs,debug_gcs,implicit-dirs,only-dir=download"

2. SubPath feature

You can use PV/PVC with the subPath feature. For example:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: gcs-fuse-csi-static-pvc
spec:
  ...
  volumeName: gcs-fuse-csi-pv
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: gcs-fuse-csi-pv
spec:
  ...
  csi:
    driver: gcsfuse.csi.storage.gke.io
    volumeHandle: my-bucket-name
    readOnly: false
---
apiVersion: v1
kind: Pod
metadata:
  name: gcs-fuse-csi-example
  annotations:
    gke-gcsfuse/volumes: "true"
spec:
  serviceAccountName: gcs-csi
  containers:
  - name: busybox
    image: busybox
    volumeMounts:
    - name: gcs-fuse-csi-static
      mountPath: /log-data
      subPath: "data/log"
    - name: gcs-fuse-csi-static
      mountPath: /config-data
      subPath: "data/config"
  volumes:
  - name: gcs-fuse-csi-static
    persistentVolumeClaim:
      claimName: gcs-fuse-csi-static-pvc

[msau42]

Can you clarify why this scenario needed two PVCs for the same volume in the same pod? Were the mount options different?

[songjiaxun]

Can you clarify why this scenario needed two PVCs for the same volume in the same pod? Were the mount options different?

Yes, the mount options could be different. For example, different PVs may consume the same bucket, but use only-dir flags to mount different sub-folders in the same bucket. The example in the solution uses the CSI ephemeral volume to achieve the same thing without any issue.

[msau42]

And this is because the subpath feature does not work?

[msau42]

There may be frameworks/operators that only know how to use PVC/PV and not csi ephemeral volumes, so it would be good to see if we can get similar feature parity.

[songjiaxun]

And this is because the subpath feature does not work?

No, this is not related to the subPath feature.

There may be frameworks/operators that only know how to use PVC/PV and not csi ephemeral volumes, so it would be good to see if we can get similar feature parity.

I remember this issue will also happen on the PD CSI driver when different PVs pointing to the same underlaying disk ID on the volumeHandle field. For example, b/259727553#comment13 (Google internal link).

To get similar feature parity, maybe we can modify the kubelet logic here https://github.com/kubernetes/kubernetes/blob/8f15859afc9cfaeb05d4915ffa204d84da512094/pkg/kubelet/volumemanager/cache/desired_state_of_world.go#L296-L298?