Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add script & instructions for in-place upgrade test. #1304

Merged
merged 1 commit into from
Apr 15, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 54 additions & 0 deletions test/e2e/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,3 +26,57 @@ go test -v -timeout 0 ./... -report-dir=$ARTIFACTS -ginkgo.focus="\[efs-csi\]" -
```

The E2E flags that you can pass to `go test` are defined in [e2e_test.go](https://github.com/kubernetes-sigs/aws-efs-csi-driver/blob/master/test/e2e/e2e_test.go#L66-L75).


### Running Upgrade Test
In order to test upgrades from previous releases to the current development version of the driver, the following steps can be followed:

1. Ensure the EFS CSI Driver is not currently deployed on your cluster.
2. Ensure that the EFS CSI Node and Controller service accounts are deployed on your cluster. Ex:
```
$ cat ~/efs-service-account.yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
app.kubernetes.io/name: aws-efs-csi-driver
name: efs-csi-controller-sa
namespace: kube-system
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::123456789:role/test-cluster-iam-sa-role
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
app.kubernetes.io/name: aws-efs-csi-driver
name: efs-csi-node-sa
namespace: kube-system
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::123456789:role/test-cluster-iam-sa-role

$ kubectl apply -f ~/efs-service-account.yaml
```
3. Run the upgrade script, for example:
```sh
# The release version of the driver you would like to test upgrading from.
# Pulls most recent image from the release-$PREV_RELEASE branch
PREV_RELEASE=1.7
# Region of private ECR
REGION=us-east-1
# Account of private ECR
ACCOUNT=123456789
chmod +x ./upgrade_driver_version.sh
./upgrade_driver_version.sh $PREV_RELEASE $REGION $ACCOUNT
```
4. Run the e2e tests, note that $REGION should be that of the EKS cluster.
```sh
export KUBECONFIG=$HOME/.kube/config
go test -v -timeout 0 ./... -report-dir=$ARTIFACTS -ginkgo.focus="\[efs-csi\]" -ginkgo.skip="\[Disruptive\]" \
--file-system-id=$FS_ID --create-file-system=false --deploy-driver=false --region=$REGION
```
5. Clean Up: The driver + kubernetes service accounts can be cleaned up via the following command:
```sh
kubectl delete -k github.com/kubernetes-sigs/aws-efs-csi-driver/deploy/kubernetes/overlays/stable/?ref=master
```
86 changes: 86 additions & 0 deletions test/e2e/upgrade_driver_version.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
#!/bin/sh
set -eux

# Check for dependencies
if ! command -v kubectl &> /dev/null; then
echo "kubectl is not installed."
exit 1
fi
if ! command -v docker &> /dev/null; then
echo "Docker is not installed."
exit 1
fi
if ! command -v aws &> /dev/null; then
echo "AWS CLI is not installed."
exit 1
fi

prevRelease=$1
region=$2
account=$3

# The private ECR Repo is expected to take the following format:
# $account.dkr.ecr.$region.amazonaws.com/aws-efs-csi-driver
ecrRegistry="$account.dkr.ecr.$region.amazonaws.com"
ecrRepo="aws-efs-csi-driver"

# Make temp folder for temp files
mkdir ./temp
publicDriverManifest="./temp/public-driver-manifest.yaml"

# Build & push image of driver's development version
cd ../.. && make
aws ecr get-login-password --region $region | docker login --username AWS --password-stdin $ecrRegistry
docker build --pull --no-cache -t aws-efs-csi-driver .
docker tag aws-efs-csi-driver:latest $ecrRegistry/$ecrRepo:latest
docker push $ecrRegistry/$ecrRepo:latest
cd ./test/e2e

# Download starting version manifest
kubectl kustomize \
"github.com/kubernetes-sigs/aws-efs-csi-driver/deploy/kubernetes/overlays/stable/?ref=release-$prevRelease" > $publicDriverManifest

# Remove the predefined service accounts
awk '
/apiVersion: v1/ {recording=1}
recording && /app.kubernetes.io\/name: aws-efs-csi-driver/ {found=1}
recording && /name: efs-csi-controller-sa/ {controller_sa=1}
recording && /name: efs-csi-node-sa/ {node_sa=1}
recording && /---/ {
if (found && (controller_sa || node_sa)) {
recording=0; found=0; controller_sa=0; node_sa=0; next
}
}
!recording {print}
' $publicDriverManifest > ./temp/temp.yaml && mv ./temp/temp.yaml $publicDriverManifest

# Deploy starting version of driver for the upgrade test
kubectl apply -f $publicDriverManifest

# Tear down starting version
kubectl delete -f $publicDriverManifest

# Create private manifest file & modify to use the private ecr repo
privateDriverManifest="./temp/private-driver-manifest.yaml"
kubectl kustomize \
"github.com/kubernetes-sigs/aws-efs-csi-driver/deploy/kubernetes/overlays/stable/ecr/?ref=release-$prevRelease" > $privateDriverManifest

sed -i -e "s|602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/aws-efs-csi-driver:v[0-9]\+\.[0-9]\+\.[0-9]\+|$account.dkr.ecr.$region.amazonaws.com/aws-efs-csi-driver:latest|" $privateDriverManifest

# Remove the predefined service accounts
awk '
/apiVersion: v1/ {recording=1}
recording && /app.kubernetes.io\/name: aws-efs-csi-driver/ {found=1}
recording && /name: efs-csi-controller-sa/ {controller_sa=1}
recording && /name: efs-csi-node-sa/ {node_sa=1}
recording && /---/ {
if (found && (controller_sa || node_sa)) {
recording=0; found=0; controller_sa=0; node_sa=0; next
}
}
!recording {print}
' $privateDriverManifest > ./temp/temp.yaml && mv ./temp/temp.yaml $privateDriverManifest

kubectl apply -f $privateDriverManifest

rm -rf ./temp
Loading