Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

incorrect mount option was specified - mount options doesn't work #147

Closed
Jeffwan opened this issue Apr 5, 2020 · 13 comments
Closed

incorrect mount option was specified - mount options doesn't work #147

Jeffwan opened this issue Apr 5, 2020 · 13 comments
Labels
kind/support Categorizes issue or PR as a support question. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed.

Comments

@Jeffwan
Copy link

Jeffwan commented Apr 5, 2020

I have a pod with non root user to run. I plan to mount to the user's home disk. The problem that user doesn't have permission to write files because it needs root permission on volumes.

I plan to add mountOptions in the PV field and I assume this will use this options to mount the volume.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: efs-pv
spec:
  capacity:
    storage: 5Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  mountOptions:
    - dir_mode=0777
    - file_mode=0777
  persistentVolumeReclaimPolicy: Retain
  storageClassName: efs-sc
  csi:
    driver: efs.csi.aws.com
    volumeHandle: fs-fxxxxxb5c

From the logs, seem options are applied to mount command, what I am curious is why they are incorrect mount option. Seems they are pretty standard commands. Any insights here?

Events:
  Type     Reason       Age               From                                                  Message
  ----     ------       ----              ----                                                  -------
  Normal   Scheduled    12s               default-scheduler                                     Successfully assigned admin/efs-app to ip-192-168-64-63.us-west-2.compute.internal
  Warning  FailedMount  2s (x5 over 12s)  kubelet, ip-192-168-64-63.us-west-2.compute.internal  MountVolume.SetUp failed for volume "efs-pv" : rpc error: code = Internal desc = Could not mount "fs-xxxxx:/" at "/var/lib/kubelet/pods/de3cb354-dee7-4fc5-a558-8365db8f38b7/volumes/kubernetes.io~csi/efs-pv/mount": mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t efs -o dir_mode=0777,file_mode=0777 fs-f6037b5c:/ /var/lib/kubelet/pods/de3cb354-dee7-4fc5-a558-8365db8f38b7/volumes/kubernetes.io~csi/efs-pv/mount
Output: mount.nfs4: an incorrect mount option was specified

/triage support

@k8s-ci-robot k8s-ci-robot added the kind/support Categorizes issue or PR as a support question. label Apr 5, 2020
@leakingtapan
Copy link
Contributor

leakingtapan commented Apr 5, 2020

Not sure if dir_mode and file_mode is valid mount option to nfs client. Where do u see these two option?

@leakingtapan
Copy link
Contributor

And have you tried this #91 (comment)

@Jeffwan
Copy link
Author

Jeffwan commented Apr 5, 2020

@leakingtapan

I see a few different sources indicating these options are available for their storage. But I didn't find anything for EFS

Seems it can be used in PV for static provision and Storage class for dynamic provision.
kubernetes/kubernetes#54610 (comment)

I am wondering if that's because EFS doesn't support this. I found note here.

By default, new Amazon EFS file systems are owned by root:root, and only the root user (UID 0) has read-write-execute permissions. If your containers are not running as root, you must change the Amazon EFS file system permissions to allow other users to modify the file system. For more information, see Working with Users, Groups, and Permissions at the Network File System (NFS) Level in the Amazon Elastic File System User Guide.`

Currently, I just create a launcher to change permission as you attached. This is ok but brings one extra step. Since I know the uid and gid of my applications, if mountOption is supported, I can create PV for specific application with known uid and gid. As there's a workaround, this is not a big deal.

@2uasimojo
Copy link
Contributor

@Jeffwan There's another option if you want control over ownership/permissions: access points. I've proposed a doc add describing how to make this work: #153

Let me know if that works for you.

@leakingtapan
Copy link
Contributor

Using access point (without IAM) sounds a good option. Does this work for you?

@wanghao923
Copy link

wanghao923 commented May 11, 2020

I try to use iam role but not work for me.

efs policy:

{
    "Version": "2012-10-17",
    "Id": "xxxxxx",
    "Statement": [
        {
            "Sid": "allow-xxxxxx-can-use-this-efs",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws-cn:iam::xxxxxx:role/xxxxxx"
            },
            "Action": [
                "elasticfilesystem:ClientMount",
                "elasticfilesystem:ClientWrite",
                "elasticfilesystem:ClientRootAccess"
            ],
            "Resource": "arn:aws-cn:elasticfilesystem:cn-north-1:xxxxxx:file-system/fs-xxxxxx"
        }
    ]
}

pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: efs-pv
spec:
  capacity:
    storage: 5Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: efs-sc
  mountOptions:
    - tls
    - iam
  csi:
    driver: efs.csi.aws.com
    volumeHandle: fs-xxxxxx
cd aws-efs-csi-driver/examples/kubernetes/multiple_pods
kubectl apply -f specs/  # branc 0.3
kubectl describe pods app1

Error message:

Warning  FailedMount       0s               kubelet, ip-xxxxxx.cn-north-1.compute.internal  MountVolume.SetUp failed for volume "efs-pv" : kubernetes.io/csi: mounter.SetupAt failed: rpc error: code = Internal desc = Could not mount "fs-xxxxxx:/" at "/var/lib/kubelet/pods/83d8da2d-3a94-4259-9613-bc8c1d3dc7b2/volumes/kubernetes.io~csi/efs-pv/mount": mount failed: exit status 1
Mounting command: mount
Mounting arguments: -t efs -o tls,iam fs-xxxxxx:/ /var/lib/kubelet/pods/83d8da2d-3a94-4259-9613-bc8c1d3dc7b2/volumes/kubernetes.io~csi/efs-pv/mount
Output: Traceback (most recent call last):
  File "/sbin/mount.efs", line 1375, in <module>
    main()
  File "/sbin/mount.efs", line 1369, in main
    mount_tls(config, init_system, dns_name, path, fs_id, mountpoint, options)
  File "/sbin/mount.efs", line 1302, in mount_tls
    with bootstrap_tls(config, init_system, dns_name, fs_id, mountpoint, options) as tunnel_proc:
  File "/lib64/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/sbin/mount.efs", line 705, in bootstrap_tls
    security_credentials, ap_id=ap_id, base_path=state_file_dir)
  File "/sbin/mount.efs", line 848, in create_certificate
    private_key = check_and_create_private_key(base_path)
  File "/sbin/mount.efs", line 911, in check_and_create_private_key
    do_with_lock(generate_key)
  File "/sbin/mount.efs", line 894, in do_with_lock
    return function()
  File "/sbin/mount.efs", line 907, in generate_key
    subprocess_call(cmd, 'Failed to create private key')
  File "/sbin/mount.efs", line 970, in subprocess_call
    process = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE, close_fds=True)
  File "/lib64/python2.7/subprocess.py", line 394, in __init__
    errread, errwrite)
  File "/lib64/python2.7/subprocess.py", line 1047, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

I found path /var/lib/kubelet/pods/83d8da2d-3a94-4259-9613-bc8c1d3dc7b2/volumes/kubernetes.io~csi/efs-pv/mount not exist in k8s node but the path /var/lib/kubelet/pods/83d8da2d-3a94-4259-9613-bc8c1d3dc7b2/volumes/kubernetes.io~csi/ exist. Why efs-driver not to create the sub mount directory efs-pv/mount ?

[root@ip-xxxxxx kubernetes.io~csi]# ls /var/lib/kubelet/pods/83d8da2d-3a94-4259-9613-bc8c1d3dc7b2/volumes/kubernetes.io~csi/efs-pv/mount
ls: cannot access /var/lib/kubelet/pods/83d8da2d-3a94-4259-9613-bc8c1d3dc7b2/volumes/kubernetes.io~csi/efs-pv/mount: No such file or directory

[[email protected]~csi]# ls -al /var/lib/kubelet/pods/83d8da2d-3a94-4259-9613-bc8c1d3dc7b2/volumes/kubernetes.io~csi
total 0
drwxr-x--- 2 root root  6 May 11 09:09 .
drwxr-x--- 4 root root 59 May 11 08:57 ..

@2uasimojo
Copy link
Contributor

@wanghao923 It looks to me like you're using an older version of the driver. Can you please confirm whether you're running release-0.3 or latest?

@wanghao923
Copy link

wanghao923 commented May 12, 2020

It's ok if I use master branch. There is the other error when I use branch release-0.3.
My environments:

cd /root/work/eks/aws-efs-csi-driver
# git branch
  master
* release-0.3

# git status
位于分支 release-0.3
您的分支与上游分支 'origin/release-0.3' 一致。

尚未暂存以备提交的变更:
  (使用 "git add <文件>..." 更新要提交的内容)
  (使用 "git restore <文件>..." 丢弃工作区的改动)
	修改:     deploy/kubernetes/base/node.yaml
	修改:     deploy/kubernetes/overlays/stable/kustomization.yaml
	修改:     examples/kubernetes/multiple_pods/specs/pv.yaml

diff --git a/deploy/kubernetes/base/node.yaml b/deploy/kubernetes/base/node.yaml
index 532c5d7..defa205 100644
--- a/deploy/kubernetes/base/node.yaml
+++ b/deploy/kubernetes/base/node.yaml
@@ -53,7 +53,7 @@ spec:
             periodSeconds: 2
             failureThreshold: 5
         - name: csi-driver-registrar
-          image: quay.io/k8scsi/csi-node-driver-registrar:v1.1.0
+          image: xxxxxx.dkr.ecr.cn-north-1.amazonaws.com.cn/amazon/eks/driver/csi-node-driver-registrar:v1.1.0
           args:
             - --csi-address=$(ADDRESS)
             - --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)
@@ -74,7 +74,7 @@ spec:
               mountPath: /registration
         - name: liveness-probe
           imagePullPolicy: Always
-          image: quay.io/k8scsi/livenessprobe:v1.1.0
+          image: xxxxxx.dkr.ecr.cn-north-1.amazonaws.com.cn/amazon/eks/driver/livenessprobe:v1.1.0
           args:
             - --csi-address=/csi/csi.sock
             - --health-port=9809

diff --git a/deploy/kubernetes/overlays/stable/kustomization.yaml b/deploy/kubernetes/overlays/stable/kustomization.yaml
index 4f406d8..f8e2a2e 100644
--- a/deploy/kubernetes/overlays/stable/kustomization.yaml
+++ b/deploy/kubernetes/overlays/stable/kustomization.yaml
@@ -4,12 +4,12 @@ bases:
 - ../../base
 images:
 - name: amazon/aws-efs-csi-driver
-  newName: 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/aws-efs-csi-driver
+  newName: xxxxxx.dkr.ecr.cn-north-1.amazonaws.com.cn/amazon/eks/driver/aws-efs-csi-driver
   newTag: v0.2.0
 - name: quay.io/k8scsi/livenessprobe
-  newName: 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/csi-liveness-probe
+  newName: xxxxxx.dkr.ecr.cn-north-1.amazonaws.com.cn/amazon/eks/driver/livenessprobe
   newTag: v1.1.0
 - name: quay.io/k8scsi/csi-node-driver-registrar
-  newName: 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/csi-node-driver-registrar
+  newName: xxxxxx.dkr.ecr.cn-north-1.amazonaws.com.cn/amazon/eks/driver/csi-node-driver-registrar
   newTag: v1.1.0

# git diff examples/kubernetes/multiple_pods/specs/pv.yaml
diff --git a/examples/kubernetes/multiple_pods/specs/pv.yaml b/examples/kubernetes/multiple_pods/specs/pv.yaml
index 11dea7a..fab0331 100644
--- a/examples/kubernetes/multiple_pods/specs/pv.yaml
+++ b/examples/kubernetes/multiple_pods/specs/pv.yaml
@@ -10,6 +10,9 @@ spec:
     - ReadWriteMany
   persistentVolumeReclaimPolicy: Retain
   storageClassName: efs-sc
+  mountOptions:
+     - tls
+     - iam
   csi:
     driver: efs.csi.aws.com
-    volumeHandle: fs-4af69aab
+    volumeHandle: fs-xxxxxx

kubectl apply -k deploy/kubernetes/overlays/stable/
kubectl apply -f examples/kubernetes/multiple_pods/specs/

Error message:

Output: Failed to resolve "fs-xxxxxx.efs.cn-north-1.amazonaws.com" - check that your file system ID is correct.
See https://docs.aws.amazon.com/console/efs/mount-dns-name for more detail.
  Warning  FailedMount  68s (x3 over 5m42s)  kubelet, ip-xxxxxx.cn-north-1.compute.internal  Unable to attach or mount volumes: unmounted volumes=[persistent-storage], unattached volumes=[persistent-storage default-token-h6gtr]: timed out waiting for the condition

Correct DNS name is "fs-xxxxxx.efs.cn-north-1.amazonaws.com.cn", This is the other error.

Yesterday I also used branch release-0.3 but I modified the tag of image amazon/aws-efs-csi-driver. default version is 0.2.0. I can reproduce yesterday error if I use version 0.3.0 in branch release-0.3. So before error is beacuse of I updated the version of amazon/aws-efs-csi-driver to 0.3.0.

diff --git a/deploy/kubernetes/overlays/stable/kustomization.yaml b/deploy/kubernetes/overlays/stable/kustomization.yaml
index 4f406d8..9c057b9 100644
--- a/deploy/kubernetes/overlays/stable/kustomization.yaml
+++ b/deploy/kubernetes/overlays/stable/kustomization.yaml
@@ -4,12 +4,12 @@ bases:
 - ../../base
 images:
 - name: amazon/aws-efs-csi-driver
-  newName: 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/aws-efs-csi-driver
-  newTag: v0.2.0
+  newName: xxxxxx.dkr.ecr.cn-north-1.amazonaws.com.cn/amazon/eks/driver/aws-efs-csi-driver
+  newTag: v0.3.0
 - name: quay.io/k8scsi/livenessprobe
-  newName: 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/csi-liveness-probe
+  newName: xxxxxx.dkr.ecr.cn-north-1.amazonaws.com.cn/amazon/eks/driver/livenessprobe
   newTag: v1.1.0
 - name: quay.io/k8scsi/csi-node-driver-registrar
-  newName: 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/csi-node-driver-registrar
+  newName: xxxxxx.dkr.ecr.cn-north-1.amazonaws.com.cn/amazon/eks/driver/csi-node-driver-registrar
   newTag: v1.1.0

Fortunately, These errors are fixed in the master branch. Thanks

@2uasimojo
Copy link
Contributor

@wanghao923 Good to hear. I've asked for a new tagged release so we don't have to work against a moving target.

@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Aug 10, 2020
@fejta-bot
Copy link

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Sep 9, 2020
@fejta-bot
Copy link

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

@k8s-ci-robot
Copy link
Contributor

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/support Categorizes issue or PR as a support question. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed.
Projects
None yet
Development

No branches or pull requests

6 participants