Squashed 'release-tools/' changes from c6a88c6e..6616a6b5

kubernetes-csi/csi-release-tools@6616a6b5 Merge kubernetes-csi/csi-release-tools#146 from pohly/kubernetes-1.21 kubernetes-csi/csi-release-tools@510fb0f9 prow.sh: support Kubernetes 1.21 kubernetes-csi/csi-release-tools@c63c61b3 prow.sh: add CSI_PROW_DEPLOYMENT_SUFFIX kubernetes-csi/csi-release-tools@51ac11c3 Merge kubernetes-csi/csi-release-tools#144 from pohly/pull-jobs kubernetes-csi/csi-release-tools@dd54c926 pull-test.sh: test importing csi-release-tools into other repo kubernetes-csi/csi-release-tools@7d2643a5 Merge kubernetes-csi/csi-release-tools#143 from pohly/path-setup kubernetes-csi/csi-release-tools@6880b0c8 prow.sh: avoid creating paths unless really running tests kubernetes-csi/csi-release-tools@bc0504ad Merge kubernetes-csi/csi-release-tools#140 from jsafrane/remove-unused-k8s-libs kubernetes-csi/csi-release-tools@5b1de1ad go-get-kubernetes.sh: remove unused k8s libs kubernetes-csi/csi-release-tools@49b42693 Merge kubernetes-csi/csi-release-tools#120 from pohly/add-kubernetes-release kubernetes-csi/csi-release-tools@a1e11275 Merge kubernetes-csi/csi-release-tools#139 from pohly/kind-for-kubernetes-latest kubernetes-csi/csi-release-tools@1c0fb096 prow.sh: use KinD main for latest Kubernetes kubernetes-csi/csi-release-tools@1d77cfcb Merge kubernetes-csi/csi-release-tools#138 from pohly/kind-update-0.10 kubernetes-csi/csi-release-tools@bff2fb7e prow.sh: KinD 0.10.0 kubernetes-csi/csi-release-tools@95eac336 Merge kubernetes-csi/csi-release-tools#137 from pohly/fix-go-version-check kubernetes-csi/csi-release-tools@437e4311 verify-go-version.sh: fix check after removal of travis.yml kubernetes-csi/csi-release-tools@1748b16b Merge kubernetes-csi/csi-release-tools#136 from pohly/go-1.16 kubernetes-csi/csi-release-tools@ec844ea6 remove travis.yml, Go 1.16 kubernetes-csi/csi-release-tools@df76aba8 Merge kubernetes-csi/csi-release-tools#134 from andyzhangx/add-build-arg kubernetes-csi/csi-release-tools@e314a56d add build-arg ARCH for building multi-arch images, e.g. ARG ARCH FROM k8s.gcr.io/build-image/debian-base-${ARCH}:v2.1.3 kubernetes-csi/csi-release-tools@7bc70e52 Merge kubernetes-csi/csi-release-tools#129 from pohly/squash-documentation kubernetes-csi/csi-release-tools@e0b02e72 README.md: document usage of --squash kubernetes-csi/csi-release-tools@316cb957 Merge kubernetes-csi/csi-release-tools#132 from yiyang5055/bugfix/boilerplate kubernetes-csi/csi-release-tools@26e2ab10 fix: default boilerplate path kubernetes-csi/csi-release-tools@1add8c18 Merge kubernetes-csi/csi-release-tools#133 from pohly/kubernetes-1.20-tag kubernetes-csi/csi-release-tools@3e811d6c prow.sh: fix "on-master" prow jobs kubernetes-csi/csi-release-tools@1d60e779 Merge kubernetes-csi/csi-release-tools#131 from pohly/kubernetes-1.20-tag kubernetes-csi/csi-release-tools@9f104590 prow.sh: support building Kubernetes for a specific version kubernetes-csi/csi-release-tools@f7e7ee49 docs: steps for adding testing against new Kubernetes release kubernetes-csi/csi-release-tools@fe1f2848 Merge kubernetes-csi/csi-release-tools#121 from kvaps/namespace-check kubernetes-csi/csi-release-tools@8fdf0f78 Merge kubernetes-csi/csi-release-tools#128 from fengzixu/master kubernetes-csi/csi-release-tools@1c94220d fix: fix a bug of csi-sanity kubernetes-csi/csi-release-tools@a4c41e6a Merge kubernetes-csi/csi-release-tools#127 from pohly/fix-boilerplate kubernetes-csi/csi-release-tools@ece0f500 check namespace for snapshot-controller kubernetes-csi/csi-release-tools@dbd89672 verify-boilerplate.sh: fix path to script kubernetes-csi/csi-release-tools@9289fd16 Merge kubernetes-csi/csi-release-tools#125 from sachinkumarsingh092/optional-spelling-boilerplate-checks kubernetes-csi/csi-release-tools@ad29307f Make the spelling and boilerplate checks optional kubernetes-csi/csi-release-tools@5f06d024 Merge kubernetes-csi/csi-release-tools#124 from sachinkumarsingh092/fix-spellcheck-boilerplate-tests kubernetes-csi/csi-release-tools@48186eba Fix spelling and boilerplate errors kubernetes-csi/csi-release-tools@71690aff Merge kubernetes-csi/csi-release-tools#122 from sachinkumarsingh092/include-spellcheck-boilerplate-tests kubernetes-csi/csi-release-tools@981be3fe Adding spelling and boilerplate checks. kubernetes-csi/csi-release-tools@2bb75253 Merge kubernetes-csi/csi-release-tools#117 from fengzixu/master kubernetes-csi/csi-release-tools@3b6d17b1 Merge kubernetes-csi/csi-release-tools#118 from pohly/cloud-build-timeout kubernetes-csi/csi-release-tools@9318c6cc cloud build: double the timeout, now 1 hour kubernetes-csi/csi-release-tools@4ab8b154 use the tag to replace commit of csi-test kubernetes-csi/csi-release-tools@5d74e455 change the csi-test import path to v4 kubernetes-csi/csi-release-tools@7dcd0a99 upgrade csi-test to v4.0.2 kubernetes-csi/csi-release-tools@86ff5802 Merge kubernetes-csi/csi-release-tools#116 from andyzhangx/export-image-name kubernetes-csi/csi-release-tools@c3a96625 allow export image name and registry name git-subtree-dir: release-tools git-subtree-split: 6616a6b5294b6df39cfce37f4fce7cdce0a77583
pohly · May 5, 2021 · 4839bfe · 4839bfe
1 parent 73214ad
commit 4839bfe
Show file tree

Hide file tree

Showing 21 changed files with 644 additions and 105 deletions.
diff --git a/.prow.sh b/.prow.sh
@@ -1,7 +1,23 @@
 #! /bin/bash -e
+
+# Copyright 2021 The Kubernetes Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
 #
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 # This is for testing csi-release-tools itself in Prow. All other
 # repos use prow.sh for that, but as csi-release-tools isn't a normal
 # repo with some Go code in it, it has a custom Prow test script.
 
 ./verify-shellcheck.sh "$(pwd)"
+./verify-spelling.sh "$(pwd)"
+./verify-boilerplate.sh "$(pwd)"
diff --git a/README.md b/README.md
@@ -21,7 +21,11 @@ The expected repository layout is:
    Dockerfile in the root when only building a single command
  - `Makefile` - includes `release-tools/build.make` and sets
    configuration variables
- - `.travis.yml` - a symlink to `release-tools/.travis.yml`
+ - `.prow.sh` script which imports `release-tools/prow.sh`
+   and may contain further customization
+ - `.cloudbuild.sh` and `cloudbuild.yaml` as symlinks to
+   the corresponding files in `release-tools` or (if necessary)
+   as custom files
 
 To create a release, tag a certain revision with a name that
 starts with `v`, for example `v1.0.0`, then `make push`
@@ -44,10 +48,17 @@ is the recommended way of maintaining a copy of the rules inside the
 changes also locally, test them and then push them back to the shared
 repository at a later time.
 
+We no longer care about importing the full commit history, so `--squash` should be used
+when submitting a `release-tools` update. Also make sure that the PR for that
+contains the automatically generated commit message in the PR description.
+It contains the list of individual commits that were squashed. The script from
+https://github.com/kubernetes-csi/csi-release-tools/issues/7 can create such
+PRs automatically.
+
 Cheat sheet:
 
-- `git subtree add --prefix=release-tools https://github.com/kubernetes-csi/csi-release-tools.git master` - add release tools to a repo which does not have them yet (only once)
-- `git subtree pull --prefix=release-tools https://github.com/kubernetes-csi/csi-release-tools.git master` - update local copy to latest upstream (whenever upstream changes)
+- `git subtree add --squash --prefix=release-tools https://github.com/kubernetes-csi/csi-release-tools.git master` - add release tools to a repo which does not have them yet (only once)
+- `git subtree pull --squash --prefix=release-tools https://github.com/kubernetes-csi/csi-release-tools.git master` - update local copy to latest upstream (whenever upstream changes)
 - edit, `git commit`, `git subtree push --prefix=release-tools [email protected]:<user>/csi-release-tools.git <my-new-or-existing-branch>` - push to a new branch before submitting a PR
 
 verify-shellcheck.sh

diff --git a/SIDECAR_RELEASE_PROCESS.md b/SIDECAR_RELEASE_PROCESS.md
@@ -11,7 +11,7 @@ The release manager must:
   kubernetes/org to request membership
 * Be a top level approver for the repository. To become a top level approver,
   the candidate must demonstrate ownership and deep knowledge of the repository
-  through active maintainence, responding to and fixing issues, reviewing PRs,
+  through active maintenance, responding to and fixing issues, reviewing PRs,
   test triage.
 * Be part of the maintainers or admin group for the repository. admin is a
   superset of maintainers, only maintainers level is required for cutting a
@@ -104,3 +104,47 @@ naming convention `<hostpath-deployment-version>-on-<kubernetes-version>`.
    CSI hostpath driver with the new sidecars in the [CSI repo](https://github.com/kubernetes-csi/csi-driver-host-path/tree/master/deploy)
    and [k/k
    in-tree](https://github.com/kubernetes/kubernetes/tree/master/test/e2e/testing-manifests/storage-csi/hostpath/hostpath)
+
+## Adding support for a new Kubernetes release
+
+1. Add the new release to `k8s_versions` in
+   https://github.com/kubernetes/test-infra/blob/090dec5dd535d5f61b7ba52e671a810f5fc13dfd/config/jobs/kubernetes-csi/gen-jobs.sh#L25
+   to enable generating a job for it. Set `experimental_k8s_version`
+   in
+   https://github.com/kubernetes/test-infra/blob/090dec5dd535d5f61b7ba52e671a810f5fc13dfd/config/jobs/kubernetes-csi/gen-jobs.sh#L40
+   to ensure that the new jobs aren't run for PRs unless explicitly
+   requested. Generate and submit the new jobs.
+1. Create a test PR to try out the new job in some repo with `/test
+   pull-kubernetes-csi-<repo>-<x.y>-on-kubernetes-<x.y>` where x.y
+   matches the Kubernetes release. Alternatively, run .prow.sh in that
+   repo locally with `CSI_PROW_KUBERNETES_VERSION=x.y.z`.
+1. Optional: update to a [new
+   release](https://github.com/kubernetes-sigs/kind/tags) of kind with
+   pre-built images for the new Kubernetes release. This is optional
+   if the current version of kind is able to build images for the new
+   Kubernetes release. However, jobs require less resources when they
+   don't need to build those images from the Kubernetes source code.
+   This change needs to be tried out in a PR against a component
+   first, then get submitted against csi-release-tools.
+1. Optional: propagate the updated csi-release-tools to all components
+   with the script from
+   https://github.com/kubernetes-csi/csi-release-tools/issues/7#issuecomment-707025402
+1. Once it is likely to work in all components, unset
+   `experimental_k8s_version` and submit the updated jobs.
+1. Once all sidecars for the new Kubernetes release are released,
+   either bump the version number of the images in the existing
+   [csi-driver-host-path
+   deployments](https://github.com/kubernetes-csi/csi-driver-host-path/tree/master/deploy)
+   and/or create a new deployment, depending on what Kubernetes
+   release an updated sidecar is compatible with. If no new deployment
+   is needed, then add a symlink to document that there intentionally
+   isn't a separate deployment. This symlink is not needed for Prow
+   testing because that will use "kubernetes-latest" as fallback.
+   Update that link when creating a new deployment.
+1. Create a new csi-driver-host-path release.
+1. Bump `CSI_PROW_DRIVER_VERSION` in prow.sh to that new release and
+   (eventually) roll that change out to all repos by updating
+   `release-tools` in them. This is used when testing manually. The
+   Prow jobs override that value, so also update
+   `hostpath_driver_version` in
+   https://github.com/kubernetes/test-infra/blob/91b04e6af3a40a9bcff25aa030850a4721e2dd2b/config/jobs/kubernetes-csi/gen-jobs.sh#L46-L47
diff --git a/boilerplate/boilerplate.Dockerfile.txt b/boilerplate/boilerplate.Dockerfile.txt
@@ -0,0 +1,13 @@
+# Copyright YEAR The Kubernetes Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
diff --git a/boilerplate/boilerplate.Makefile.txt b/boilerplate/boilerplate.Makefile.txt
@@ -0,0 +1,13 @@
+# Copyright YEAR The Kubernetes Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
diff --git a/boilerplate/boilerplate.bzl.txt b/boilerplate/boilerplate.bzl.txt
@@ -0,0 +1,13 @@
+# Copyright YEAR The Kubernetes Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
diff --git a/boilerplate/boilerplate.go.txt b/boilerplate/boilerplate.go.txt
@@ -0,0 +1,15 @@
+/*
+Copyright YEAR The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
diff --git a/boilerplate/boilerplate.py b/boilerplate/boilerplate.py
@@ -0,0 +1,200 @@
+#!/usr/bin/env python
+
+# Copyright 2019 The Kubernetes Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import print_function
+
+import argparse
+import difflib
+import glob
+import os
+import re
+import sys
+from datetime import date
+
+parser = argparse.ArgumentParser()
+parser.add_argument(
+    "filenames",
+    help="list of files to check, all files if unspecified",
+    nargs='*')
+
+# Rootdir defaults to the directory **above** the repo-infra dir.
+rootdir = os.path.dirname(__file__) + "./../../"
+rootdir = os.path.abspath(rootdir)
+parser.add_argument(
+    "--rootdir", default=rootdir, help="root directory to examine")
+
+default_boilerplate_dir = os.path.abspath(os.path.dirname(__file__))
+
+parser.add_argument(
+    "--boilerplate-dir", default=default_boilerplate_dir)
+
+parser.add_argument(
+    "-v", "--verbose",
+    help="give verbose output regarding why a file does not pass",
+    action="store_true")
+
+args = parser.parse_args()
+
+verbose_out = sys.stderr if args.verbose else open("/dev/null", "w")
+
+def get_refs():
+    refs = {}
+
+    for path in glob.glob(os.path.join(args.boilerplate_dir, "boilerplate.*.txt")):
+        extension = os.path.basename(path).split(".")[1]
+
+        ref_file = open(path, 'r')
+        ref = ref_file.read().splitlines()
+        ref_file.close()
+        refs[extension] = ref
+
+    return refs
+
+def file_passes(filename, refs, regexs):
+    try:
+        f = open(filename, 'r')
+    except Exception as exc:
+        print("Unable to open %s: %s" % (filename, exc), file=verbose_out)
+        return False
+
+    data = f.read()
+    f.close()
+
+    basename = os.path.basename(filename)
+    extension = file_extension(filename)
+    if extension != "":
+        ref = refs[extension]
+    else:
+        ref = refs[basename]
+
+    # remove build tags from the top of Go files
+    if extension == "go":
+        p = regexs["go_build_constraints"]
+        (data, found) = p.subn("", data, 1)
+
+    # remove shebang from the top of shell files
+    if extension == "sh" or extension == "py":
+        p = regexs["shebang"]
+        (data, found) = p.subn("", data, 1)
+
+    data = data.splitlines()
+
+    # if our test file is smaller than the reference it surely fails!
+    if len(ref) > len(data):
+        print('File %s smaller than reference (%d < %d)' %
+              (filename, len(data), len(ref)),
+              file=verbose_out)
+        return False
+
+    # trim our file to the same number of lines as the reference file
+    data = data[:len(ref)]
+
+    p = regexs["year"]
+    for d in data:
+        if p.search(d):
+            print('File %s is missing the year' % filename, file=verbose_out)
+            return False
+
+    # Replace all occurrences of the regex "CURRENT_YEAR|...|2016|2015|2014" with "YEAR"
+    p = regexs["date"]
+    for i, d in enumerate(data):
+        (data[i], found) = p.subn('YEAR', d)
+        if found != 0:
+            break
+
+    # if we don't match the reference at this point, fail
+    if ref != data:
+        print("Header in %s does not match reference, diff:" % filename, file=verbose_out)
+        if args.verbose:
+            print(file=verbose_out)
+            for line in difflib.unified_diff(ref, data, 'reference', filename, lineterm=''):
+                print(line, file=verbose_out)
+            print(file=verbose_out)
+        return False
+
+    return True
+
+def file_extension(filename):
+    return os.path.splitext(filename)[1].split(".")[-1].lower()
+
+skipped_dirs = ['Godeps', 'third_party', '_gopath', '_output', '.git',
+                'cluster/env.sh', 'vendor', 'test/e2e/generated/bindata.go',
+                'repo-infra/verify/boilerplate/test', '.glide']
+
+def normalize_files(files):
+    newfiles = []
+    for pathname in files:
+        if any(x in pathname for x in skipped_dirs):
+            continue
+        newfiles.append(pathname)
+    return newfiles
+
+def get_files(extensions):
+    files = []
+    if len(args.filenames) > 0:
+        files = args.filenames
+    else:
+        for root, dirs, walkfiles in os.walk(args.rootdir):
+            # don't visit certain dirs. This is just a performance improvement
+            # as we would prune these later in normalize_files(). But doing it
+            # cuts down the amount of filesystem walking we do and cuts down
+            # the size of the file list
+            for d in skipped_dirs:
+                if d in dirs:
+                    dirs.remove(d)
+
+            for name in walkfiles:
+                pathname = os.path.join(root, name)
+                files.append(pathname)
+
+    files = normalize_files(files)
+
+    outfiles = []
+    for pathname in files:
+        basename = os.path.basename(pathname)
+        extension = file_extension(pathname)
+        if extension in extensions or basename in extensions:
+            outfiles.append(pathname)
+    return outfiles
+
+def get_regexs():
+    regexs = {}
+    # Search for "YEAR" which exists in the boilerplate, but shouldn't in the real thing
+    regexs["year"] = re.compile( 'YEAR' )
+    # dates can be 2014, 2015, 2016, ..., CURRENT_YEAR, company holder names can be anything
+    years = range(2014, date.today().year + 1)
+    regexs["date"] = re.compile( '(%s)' % "|".join(map(lambda l: str(l), years)) )
+    # strip // +build \n\n build constraints
+    regexs["go_build_constraints"] = re.compile(r"^(// \+build.*\n)+\n", re.MULTILINE)
+    # strip #!.* from shell scripts
+    regexs["shebang"] = re.compile(r"^(#!.*\n)\n*", re.MULTILINE)
+    return regexs
+
+
+
+def main():
+    regexs = get_regexs()
+    refs = get_refs()
+    filenames = get_files(refs.keys())
+
+    for filename in filenames:
+        if not file_passes(filename, refs, regexs):
+            print(filename, file=sys.stdout)
+
+    return 0
+
+if __name__ == "__main__":
+    sys.exit(main())
diff --git a/boilerplate/boilerplate.py.txt b/boilerplate/boilerplate.py.txt
@@ -0,0 +1,13 @@
+# Copyright YEAR The Kubernetes Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.