Read secrets for client-onboarding-token-validation #2827

mrudraia1 · 2024-09-30T09:51:14Z

This PR reads the secrets instead of reading the secrets from the volume mounts.
whenever the new onboarding secrets are created, it takes more time to read the secrets from the volume mounts,
The user clicks the rotate onboarding keys, the kubernetes still uses the old public, private keys , the new keys are mounted later, So this PR will read the secrets directly from the kubernetes secrets.

controllers/storagecluster/storageclient.go

controllers/util/provider.go

services/ux-backend/handlers/common.go

services/ux-backend/handlers/onboarding/clienttokens/handler.go

services/ux-backend/handlers/onboarding/peertokens/handler.go

services/ux-backend/main.go

tools/csv-merger/csv-merger.go

leelavg · 2024-10-01T05:42:26Z

a suggestion, we are seeing this PR for third time, second time it's fine that you weren't able to recover GH (remember you can't create new a/c every-time though) but last time it's better if you can focus on rebasing properly.

yes, GH doesn't have any issue w/ closing & opening a new PR but for reviewers it's kinda hard to relook from the start.

mrudraia1 · 2024-10-15T07:37:00Z

I have tested this PR, with the latest 4.18 build. I could see the keys getting exchanged when the rotate signing key is clicked in storageclient page.

services/ux-backend/handlers/onboarding/clienttokens/handler.go

leelavg · 2024-10-15T07:52:10Z

I have tested this PR, with the latest 4.18 build. I could see the keys getting exchanged when the rotate signing key is clicked in storageclient page.

ack, only minor comments now. @nb-ohad possible for a final review?

controllers/storagecluster/storageclient.go

nb-ohad · 2024-10-15T08:27:36Z

controllers/storagecluster/storagecluster_controller.go

@@ -225,7 +225,7 @@ func (r *StorageClusterReconciler) SetupWithManager(mgr ctrl.Manager) error {
 		Owns(&appsv1.Deployment{}, builder.WithPredicates(predicate.GenerationChangedPredicate{})).
 		Owns(&corev1.Service{}, builder.WithPredicates(predicate.GenerationChangedPredicate{})).
 		Owns(&corev1.ConfigMap{}, builder.MatchEveryOwner, builder.WithPredicates(predicate.GenerationChangedPredicate{})).
-		Owns(&corev1.Secret{}, builder.WithPredicates(predicate.GenerationChangedPredicate{})).
+		Owns(&corev1.Secret{}, builder.MatchEveryOwner, builder.WithPredicates(predicate.GenerationChangedPredicate{})).


Why do we need a MatchEveryOwner here? Isn't StorageCluster the controller owner of the secret we want to watch?

controller owner reference set to secrets, ack

This comment was not addressed as tall! You are matching by controller ownership, it does not make sense to match every owner

Owner reference were set during creation of secrets in on-boarding job, when the rotate signing key is clicked the re-consiliation is not happening, The onboarding-token needs to be created which is failing.

Any suggestion for this issue.

If there is a controller owner ref on the secret and then someone deletes that secret then a delete event will be sent here. And because the item had an owner ref (with controller set) then a reconcile request will be queued. Now if that is not happening there can be a bug in a couple of places:

The predicate might be the one the drops the event (should not, but might be)

The reconcile is initiated but the reconcile code might not re-create the secret (might happen because of stale cache)

Are we sure we are adding a controller reference on the secret? if not and we are only adding an owner ref then that is your bug

@mrudraia1 You need to identify the issue, and only after you fully understand what is going on you can suggest a fix.

Owner reference were set during creation of secrets in on-boarding job, when the rotate signing key is clicked the re-consiliation is not happening, The onboarding-token needs to be created which is failing.
[...]
Are we sure we are adding a controller reference on the secret? if not and we are only adding an owner ref then that is your bug

ouch, so the issue all along is storagecluster not getting a reconcile event when rotate keys is invoked (ie, deletion of secrets) but not the delay b/n rotation and kubelet updating the secret as diagnosed in https://bugzilla.redhat.com/show_bug.cgi?id=2311012#c5?

@rchikatw was there a discussion w/ @mrudraia1 along these lines when he started to work on this? @mrudraia1 did you observe the same behavior from when the work started on this PR, end of July and in latest revision I don't see this change which basically means we are just refactoring the code w/o any observable behavior and not fixing the issue.

@leelavg earlier secret keys are read from the volume mounts , which is now changed to read directly by reading the secrets.

OwnerReference of the secrets changed to ControllerReference.

secrets are read from volumemounts is correct but the bug was when the rotation was clicked new secrets aren't realized which is a 2 step process.

storagecluster controller gets an event and onboarding job will be rerun creating new secrets

presence of new secrets are known by kubelet and get updated in ux-backend

the description in BZ mentioned that the problem was in (2) but the observation in this comment says the root cause probably is (1), ie, storagecluster controller isn't getting an event

yes, know with this PR build I tested, I can observe the

creation of new on-boarding_token immediately after the clicking of Rotate signing key.

earlier I was using MatchEveryOwner to watch the secrets, Know the ControllerOwner is set to watch the resources.

I am not aware of any discussions about changing the predicate, and I'm not sure why we are changing from OwnerRef to ControllerRef: Here. Because of this change, you might be experiencing issues with job creation not occurring when keys are rotated—this is just a guess. To my knowledge, a job is typically created when keys are rotated, and the QE team has tested this. You can also verify this in your cluster without implementing your change to see if the job is triggered when keys are rotated.

controllers/util/provider.go

services/ux-backend/handlers/onboarding/clienttokens/handler.go

services/ux-backend/main.go

openshift-ci · 2024-10-16T05:45:30Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: mrudraia1
Once this PR has been reviewed and has the lgtm label, please ask for approval from nb-ohad. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

deepsm007 · 2024-11-06T15:10:59Z

infra issue
/test images

nb-ohad · 2024-11-12T13:01:46Z

controllers/storagecluster/storagecluster_controller.go

@@ -225,7 +225,7 @@ func (r *StorageClusterReconciler) SetupWithManager(mgr ctrl.Manager) error {
 		Owns(&appsv1.Deployment{}, builder.WithPredicates(predicate.GenerationChangedPredicate{})).
 		Owns(&corev1.Service{}, builder.WithPredicates(predicate.GenerationChangedPredicate{})).
 		Owns(&corev1.ConfigMap{}, builder.MatchEveryOwner, builder.WithPredicates(predicate.GenerationChangedPredicate{})).
-		Owns(&corev1.Secret{}, builder.WithPredicates(predicate.GenerationChangedPredicate{})).
+		Owns(&corev1.Secret{}, builder.MatchEveryOwner, builder.WithPredicates(predicate.GenerationChangedPredicate{})).


This comment was not addressed as tall! You are matching by controller ownership, it does not make sense to match every owner

controllers/util/provider.go

services/ux-backend/handlers/onboarding/clienttokens/handler.go

services/ux-backend/handlers/onboarding/peertokens/handler.go

services/ux-backend/main.go

nb-ohad · 2024-11-19T11:58:39Z

controllers/util/provider.go

 	}

-	Block, _ := pem.Decode(pemString)
+	Block, _ := pem.Decode(privateSecretKey)


We should not ignore decode errors. What if the key is corrupted?

controllers/util/provider.go

services/ux-backend/handlers/onboarding/clienttokens/handler.go

services/ux-backend/handlers/onboarding/peertokens/handler.go

services/ux-backend/main.go

leelavg · 2024-11-29T09:37:23Z

controllers/util/provider.go

+	Block, err1 := pem.Decode(privateSecretKey)
+	if err1 != nil {
+		return nil, fmt.Errorf("Failed to decode private key: %v", err1)


lower case Block and as per pem.Decode func signature the second return value isn't err but a slice of bytes which should be empty.

Suggested change

Block, err1 := pem.Decode(privateSecretKey)

if err1 != nil {

return nil, fmt.Errorf("Failed to decode private key: %v", err1)

block, rest := pem.Decode(privateSecretKey)

if len(rest) > 0 {

return nil, fmt.Errorf("PEM block not found in private key: %s", rest)

pem.Decode returns byte, changed as suggested by Leela

I also mentioned to lowercase Block as it doesn't have any extra significance.

leelavg · 2024-11-29T09:38:43Z

services/ux-backend/handlers/onboarding/clienttokens/handler.go

+	klog.Info("Loading onboarding validation private Key")
+	privateKey, err := util.LoadOnboardingValidationPrivateKey(r.Context(), cl, namespace)
+	if err != nil {
+		http.Error(w, fmt.Sprintf("Failed loading onboarding validation private: %v", err), http.StatusBadRequest)


Suggested change

http.Error(w, fmt.Sprintf("Failed loading onboarding validation private: %v", err), http.StatusBadRequest)

http.Error(w, fmt.Sprintf("Failed loading onboarding validation private key: %v", err), http.StatusBadRequest)

leelavg · 2024-11-29T09:39:26Z

services/ux-backend/handlers/onboarding/peertokens/handler.go

+	klog.Info("Loading onboarding validation private Key")
+	privateKey, err := util.LoadOnboardingValidationPrivateKey(r.Context(), cl, namespace)
+	if err != nil {
+		http.Error(w, fmt.Sprintf("Failed loading onboarding validation private: %v", err), http.StatusBadRequest)


Suggested change

http.Error(w, fmt.Sprintf("Failed loading onboarding validation private: %v", err), http.StatusBadRequest)

http.Error(w, fmt.Sprintf("Failed loading onboarding validation private key: %v", err), http.StatusBadRequest)

leelavg

pls run make deps-update to update vendor.

rewantsoni · 2024-12-04T12:03:24Z

onboarding-validation-keys-generator/main.go

@@ -81,7 +81,7 @@ func main() {
 		klog.Exitf("failed to delete public secret: %v", err)
 	}

-	err = controllerutil.SetOwnerReference(storageCluster, privateSecret, cl.Scheme())
+	err = controllerutil.SetControllerReference(storageCluster, privateSecret, cl.Scheme())


You would need additional permissions inorder to add a ControllerReference, please make sure that you add the required permissions

Added the permissions for ControllerReference

I believe Rewant mentioned adding additional permissions, not removing existing ones, which suggests Owner and Controller are needed. @rewantsoni am I right here?

leelavg · 2024-12-04T15:59:31Z

deploy/ocs-operator/manifests/onboarding-validation-keys-generator-role.yaml

Proposed rbac will provide update access to storagecluster which isn't good, use a new section for only update on finalizers and the job doesn't has controller and watch isn't required.

Signed-off-by: Mrudraia1 <[email protected]>

rewantsoni suggested changes Oct 1, 2024

View reviewed changes

openshift-ci bot assigned rewantsoni Oct 1, 2024

mrudraia1 force-pushed the onboarding branch 3 times, most recently from 674b54c to dc8a56b Compare October 3, 2024 05:37

Nikhil-Ladha mentioned this pull request Oct 10, 2024

Read secrets for onboarding-token validation #2715

Closed

mrudraia1 requested review from leelavg and rewantsoni October 15, 2024 07:15

leelavg reviewed Oct 15, 2024

View reviewed changes

services/ux-backend/handlers/onboarding/clienttokens/handler.go Outdated Show resolved Hide resolved

services/ux-backend/handlers/onboarding/clienttokens/handler.go Outdated Show resolved Hide resolved

nb-ohad requested changes Oct 15, 2024

View reviewed changes

openshift-ci bot assigned nb-ohad Oct 15, 2024

mrudraia1 force-pushed the onboarding branch from dc8a56b to c90cc2f Compare October 16, 2024 05:45

mrudraia1 force-pushed the onboarding branch 6 times, most recently from 4dac720 to 8198c71 Compare October 16, 2024 07:06

mrudraia1 requested review from nb-ohad and leelavg October 16, 2024 09:05

sunilheggodu assigned mrudraia1 Oct 16, 2024

mrudraia1 force-pushed the onboarding branch 2 times, most recently from ecddbf0 to d31f826 Compare November 6, 2024 14:07

mrudraia1 force-pushed the onboarding branch from d31f826 to e627033 Compare November 6, 2024 15:21

nb-ohad requested changes Nov 12, 2024

View reviewed changes

mrudraia1 force-pushed the onboarding branch from e627033 to bbae1f7 Compare November 12, 2024 16:39

openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 12, 2024

mrudraia1 force-pushed the onboarding branch from bbae1f7 to 36162f4 Compare November 12, 2024 16:59

openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 12, 2024

mrudraia1 force-pushed the onboarding branch from 36162f4 to e3bba8e Compare November 12, 2024 17:08

mrudraia1 requested a review from nb-ohad November 13, 2024 08:00

mrudraia1 force-pushed the onboarding branch from e3bba8e to f6ee295 Compare November 13, 2024 10:40

nb-ohad requested changes Nov 19, 2024

View reviewed changes

mrudraia1 force-pushed the onboarding branch 2 times, most recently from 4f84a13 to 64dfa2f Compare November 19, 2024 13:54

leelavg reviewed Nov 20, 2024

View reviewed changes

services/ux-backend/main.go Outdated Show resolved Hide resolved

mrudraia1 force-pushed the onboarding branch 3 times, most recently from 8d4a61d to 1a8e140 Compare November 22, 2024 06:05

leelavg reviewed Nov 29, 2024

View reviewed changes

mrudraia1 force-pushed the onboarding branch 5 times, most recently from dc1cc8c to dc950bc Compare December 4, 2024 11:24

rewantsoni suggested changes Dec 4, 2024

View reviewed changes

mrudraia1 force-pushed the onboarding branch from dc950bc to 34594b6 Compare December 4, 2024 13:28

leelavg reviewed Dec 4, 2024

View reviewed changes

Read secrets for client-onboarding-token-validation

9c8aa70

Signed-off-by: Mrudraia1 <[email protected]>

mrudraia1 force-pushed the onboarding branch from 34594b6 to 9c8aa70 Compare December 4, 2024 16:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Read secrets for client-onboarding-token-validation #2827

Read secrets for client-onboarding-token-validation #2827

mrudraia1 commented Sep 30, 2024

leelavg commented Oct 1, 2024

mrudraia1 commented Oct 15, 2024

leelavg commented Oct 15, 2024

nb-ohad Oct 15, 2024

mrudraia1 Oct 16, 2024

nb-ohad Nov 12, 2024

mrudraia1 Nov 12, 2024

nb-ohad Nov 20, 2024

leelavg Dec 3, 2024 •

edited

Loading

mrudraia1 Dec 4, 2024

leelavg Dec 4, 2024

mrudraia1 Dec 4, 2024

rchikatw Dec 11, 2024 •

edited

Loading

openshift-ci bot commented Oct 16, 2024

deepsm007 commented Nov 6, 2024

nb-ohad Nov 12, 2024

nb-ohad Nov 19, 2024

mrudraia1 Dec 4, 2024

leelavg Nov 29, 2024

mrudraia1 Dec 4, 2024

leelavg Dec 4, 2024

mrudraia1 Dec 4, 2024

leelavg Nov 29, 2024

mrudraia1 Dec 4, 2024

leelavg Nov 29, 2024

mrudraia1 Dec 4, 2024

leelavg left a comment

rewantsoni Dec 4, 2024

mrudraia1 Dec 4, 2024

rchikatw Dec 11, 2024

leelavg Dec 4, 2024

mrudraia1 Dec 4, 2024

	http.Error(w, fmt.Sprintf("Failed loading onboarding validation private: %v", err), http.StatusBadRequest)
	http.Error(w, fmt.Sprintf("Failed loading onboarding validation private key: %v", err), http.StatusBadRequest)

Read secrets for client-onboarding-token-validation #2827

Are you sure you want to change the base?

Read secrets for client-onboarding-token-validation #2827

Conversation

mrudraia1 commented Sep 30, 2024

leelavg commented Oct 1, 2024

mrudraia1 commented Oct 15, 2024

leelavg commented Oct 15, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

leelavg Dec 3, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rchikatw Dec 11, 2024 • edited Loading

Choose a reason for hiding this comment

openshift-ci bot commented Oct 16, 2024

deepsm007 commented Nov 6, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

leelavg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

leelavg Dec 3, 2024 •

edited

Loading

rchikatw Dec 11, 2024 •

edited

Loading