APT/RPM PGP key for some repositories at pkgs.k8s.io expired at 2024-11-02

[kungfooflea]

Starting around 3:30am PST we noticed our automated pipelines are failing to build the docker image.

Errors below:

#6 1.692 Err:6 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.26/deb InRelease
#6 1.692 The following signatures were invalid: EXPKEYSIG 234654DA9A296436 isv:kubernetes OBS Project isv:[email protected]
#6 1.704 Get:13 https://dl.k6.io/deb stable/main amd64 Packages [3556 B]
#6 2.267 Reading package lists...
#6 2.686 W: GPG error: https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.26/deb InRelease: The following signatures were invalid: EXPKEYSIG 234654DA9A296436 isv:kubernetes OBS Project isv:[email protected]
#6 2.686 E: The repository 'https://pkgs.k8s.io/core:/stable:/v1.26/deb InRelease' is not signed.
#6 ERROR: process "/bin/sh -c sudo gpg -k && sudo gpg --no-default-keyring --keyring /usr/share/keyrings/k6-archive-keyring.gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys C5AD17C747E3415A3642D57D77C6C491D6AC1D69 && echo "deb [signed-by=/usr/share/keyrings/k6-archive-keyring.gpg] https://dl.k6.io/deb stable main" | sudo tee /etc/apt/sources.list.d/k6.list && sudo apt update && sudo apt install -y k6 && sudo npm install --global yarn tslint" did not complete successfully: exit code: 100

[2/2] RUN sudo gpg -k && sudo gpg --no-default-keyring --keyring /usr/share/keyrings/k6-archive-keyring.gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys C5AD17C747E3415A3642D57D77C6C491D6AC1D69 && echo "deb [signed-by=/usr/share/keyrings/k6-archive-keyring.gpg] https://dl.k6.io/deb stable main" | sudo tee /etc/apt/sources.list.d/k6.list && sudo apt update && sudo apt install -y k6 && sudo npm install --global yarn tslint:
1.376 Get:10 https://deb.nodesource.com/node_16.x buster/main amd64 Packages [776 B]
1.410 Get:6 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.26/deb InRelease [1192 B]
1.533 Get:11 http://deb.debian.org/debian-security bullseye-security/main amd64 Packages [307 kB]
1.643 Get:12 http://deb.debian.org/debian bullseye-updates/main amd64 Packages [18.8 kB]
1.692 Err:6 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.26/deb InRelease
1.692 The following signatures were invalid: EXPKEYSIG 234654DA9A296436 isv:kubernetes OBS Project isv:[email protected]
1.704 Get:13 https://dl.k6.io/deb stable/main amd64 Packages [3556 B]

2.686 W: GPG error: https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.26/deb InRelease: The following signatures were invalid: EXPKEYSIG 234654DA9A296436 isv:kubernetes OBS Project isv:[email protected]
2.686 E: The repository 'https://pkgs.k8s.io/core:/stable:/v1.26/deb InRelease' is not signed.

debian:bullseye is the base

[petedyerarm]

I see the same issue with v1.27

[Creteil]

« apt dist-upgrade » under ubuntu 22.04 output same error related...

oem@oem-ThinkPad-P15-Gen-1:~$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.5 LTS
Release:	22.04
Codename:	jammy
oem@oem-ThinkPad-P15-Gen-1:~$

Err:16 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.31/deb  InRelease                                                                                                          
  The following signatures were invalid: EXPKEYSIG 234654DA9A296436 isv:kubernetes OBS Project <isv:[email protected]>
Fetched 3917 B in 8s (491 B/s)                                                                                                                                                                                    
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
1 package can be upgraded. Run 'apt list --upgradable' to see it.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.31/deb  InRelease: The following signatures were invalid: EXPKEYSIG 234654DA9A296436 isv:kubernetes OBS Project <isv:[email protected]>
W: Failed to fetch https://pkgs.k8s.io/core:/stable:/v1.31/deb/InRelease  The following signatures were invalid: EXPKEYSIG 234654DA9A296436 isv:kubernetes OBS Project <isv:[email protected]>
W: Some index files failed to download. They have been ignored, or old ones used instead.

[xmudrii]

The original key has expired on 2024-11-02, however all repositories have been resigned with a new key. At the moment, the way forward is to download and install the new key from the latest Kubernetes repository (v1.31):

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

We'll try to figure out if we can handle this in a better way in the future.

Update 2024-11-12:

On the RPM-based systems, you need to modify the Kubernetes repository definition to point to the latest GPG key. Open /etc/yum.repos.d/kubernetes.repo in your preferred text editor, then make sure that the gpgkey URL is set to https://pkgs.k8s.io/core:/stable:/v1.31/rpm/repodata/repomd.xml.key (which is the latest Kubernetes repository at the moment).

As described in #3818 (comment), we at the moment recommend using the GPG key from the latest Kubernetes repository, as older Kubernetes repositories might not have the latest key. This GPG key from the latest Kubernetes repository should work with all other repositories under pkgs.k8s.io. We're debugging that and we'll follow up as we understand what's the best way forward.

[petedyerarm]

In my use case I have a loop which gets the Release.key for 1.27, 1.28, 1.29 and 1.30, followed by an apt update which fails for 1.27. I chose later the version I actually want to install.
So have all the keys expired and been replaced?

[drzewiec]

The original key has expired on 2024-11-02, however all repositories have been resigned with a new key. At the moment, the way forward is to download and install the new key:

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

We'll try to figure out if we can handle this in a better way in the future.

There seems to be some problem beyond that, because one of the steps in my docker build is to download that key fresh. So basically I'm doing this:

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.24/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.24/deb/ /" | tee /etc/apt/sources.list.d/kubernetes.list
apt-get update && apt-get install kubectl

That fails (with the errors mentioned in this issue) even though I am not caching the key.

[petedyerarm]

It appears that you can use the v1.31 Release.key to pull the v1.24 version, but I am doubtful this is correct.

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.24/deb/ /" | tee /etc/apt/sources.list.d/kubernetes.list
apt-get update && apt-get install kubectl

[xmudrii]

@drzewiec I was able to reproduce the issue, I'll check what's going on and report back ASAP.

[Buuuntyyy]

The original key has expired on 2024-11-02, however all repositories have been resigned with a new key. At the moment, the way forward is to download and install the new key:

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

We'll try to figure out if we can handle this in a better way in the future.

Hey, I got the same issue. Unfortunately, when i try to reset my key with the previous command, i got an error too:

File '/etc/apt/keyrings/kubernetes-apt-keyring.gpg' exists. Overwrite? (y/N) curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to pkgs.k8s.io:443
gpg: no valid OpenPGP data found.

Update: any curl -fsSL to the Release.key results to a connection refused on port 443

Waiting for your report, thanks for your reactivity.
Regards

[xmudrii]

/retitle APT key for some repositories at pkgs.k8s.io expired at 2024-11-02
I'm also going to pin this issue for time being.

[xmudrii]

/priority critical-urgent

[xmudrii]

/assign

[cameronbroe]

It looks like the RPM repositories are experiencing the same issue.

[seifrajhi]

+1
Same here

ubuntu $ echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.31/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.31/deb/ /
ubuntu $ curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
gpg: can't create '/etc/apt/keyrings/kubernetes-apt-keyring.gpg': No such file or directory
gpg: no valid OpenPGP data found.
gpg: dearmoring failed: No such file or directory
(23) Failed writing body

[BenTheElder]

Please refrain from +1 comments as opposed to upvoting existing comments. It's hard enough to keep up with the GitHub notifications and this is a volunteer effort.

[Pandey23000Om]

For installing K8s 1.27, I am having to get the key for 1.28 and then install 1.27

curl -fsSL https://pkgs.k8s.io/core:/stable:/**v1.28**/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

apt update
apt install -y kubelet=1.27.11-1.1 kubeadm=1.27.11-1.1 kubectl=1.27.11-1.1

[xmudrii]

At the moment, the GPG key is the same for all Kubernetes repositories under pkgs.k8s.io. This means that you can take the key from the v1.31 repository, and use it with let's say v1.27 or any other repository.

We're confirming if it's supposed to work that away or if there's something wrong with our service provider (OpenBuildService). At this time, you can mitigate the issue by downloading the key from the v1.31 repository as described in #3818 (comment)

We'll keep this issue updated as we figure out if there's a better way to handle this in the future, or if not, to properly document this.

Update: #3818 (comment) has been updated with the latest details and changes.

[primeos-work]

On RPM based systems it might not be enough to update the repo (e.g., /etc/yum.repos.d/kubernetes.repo) since the key was only updated to extend the expiration date instead of rotating to a new key. I opened #3840 to discuss that in more details.

For RPM based systems (in my case Fedora 40) I used the following workaround to remove the old version of the key which forces dnf update to (re)import it:

• rpm -qa | grep gpg-pubkey-9a296436 | xargs rpm --erase
• dnf update (or similar)

Then DNF will prompt to import the key based on (gpgkey in the repo definition (e.g. /etc/yum.repos.d/kubernetes.repo)):

Kubernetes                                                                                                                                                            4.7 kB/s | 1.7 kB     00:00
Importing GPG key 0x9A296436:
 Userid     : "isv:kubernetes OBS Project <isv:[email protected]>"
 Fingerprint: DE15 B144 86CD 377B 9E87 6E1A 2346 54DA 9A29 6436
 From       : https://pkgs.k8s.io/core:/stable:/v1.31/rpm/repodata/repomd.xml.key
Is this ok [y/N]: y
Key imported successfully

[BenTheElder]

/retitle APT/RPM PGP key for some repositories at pkgs.k8s.io expired at 2024-11-02

[ulidtko]

Note: the current key ID 234654DA9A296436 (full DE15B14486CD377B9E876E1A234654DA9A296436) has not changed, but the .gpg file has to be redownloaded, because it gained a reissued subkey.

Before:

~> gpg /etc/apt/trusted.gpg.d/kubernetes-OBS.234654DA9A296436.gpg
pub   rsa2048 2022-08-25 [SC] [expired: 2024-11-02]
      DE15B14486CD377B9E876E1A234654DA9A296436
uid           isv:kubernetes OBS Project <isv:[email protected]>

After:

~> gpg /etc/apt/trusted.gpg.d/kubernetes-OBS.234654DA9A296436.gpg
pub   rsa2048 2022-08-25 [SC] [expires: 2026-12-29]
      DE15B14486CD377B9E876E1A234654DA9A296436
uid           isv:kubernetes OBS Project <isv:[email protected]>

— notice [expires: 2026-12-29] VS [expired: 2024-11-02].

[saiteneffekt]

Running Ubuntu 22.04.5 LTS on WSL, I had to change the target directory from /etc/apt/keyrings/ to /etc/apt/trusted.gpg.d/ to make it work:
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/kubernetes-apt-keyring.gpg

[meyerkev]

1.27 key expired today. Mitigated via using the 1.31 key in place of the 1.27 key and we are now on k8s 1.31

[BenTheElder]

@xmudrii raised this problem the December 3rd SIG Release meeting, when we discussed it there I think one of the main action items was to figure out how other projects on OBS are handling this / "best practice". This is still TODO AFAIK and if anyone knows more, help would be welcome.

Barring that, one workaround @xmudrii raised was having a root/dummy package built frequently to emit a current key and then in Kubernetes's CDN infra copying it out over the other versions.

meeting notes: https://docs.google.com/document/d/1Fu6HxXQu8wl6TwloGUEOXVzZ1rwZ72IAhglnaAMCPqA/edit?tab=t.0