Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cleanups and refactor of protosanitizer #184

Open
wants to merge 6 commits into
base: master
Choose a base branch
from

Conversation

huww98
Copy link

@huww98 huww98 commented Nov 23, 2024

What type of PR is this?
/kind cleanup

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespaces from that line:

/kind api-change
/kind bug
/kind cleanup
/kind design
/kind documentation
/kind failing-test
/kind feature
/kind flake

What this PR does / why we need it:

This PR does a series cleanups and refactors about protosanitizer:

  • Update the referenced CSI spec package to 1.10
  • Remove all the reference to old github.com/golang/protobuf package
  • Refactor the protosanitizer to use the new protoreflect package
  • Sync the Makefile for csitest.proto with csi spec repo

About the output change of protosanitizer:

  • All kinds of fields now supported: including those in array, in map, or in oneof.
  • Enum now outputs their name instead of number, I think this is more suitable for logging purpose.
  • The nesting level of oneof is reduced by one.
    Example diff of the output (manually formatted for diff):
--- old.json
+++ new.json
@@ -12,22 +12,18 @@
     "secrets": "***stripped***",
     "volume_capabilities": [
         {
-            "AccessType": {
-                "Mount": {
-                    "fs_type": "ext4",
-                    "mount_flags": [
-                        "flag1",
-                        "flag2",
-                        "flag3"
-                    ]
-                }
-            },
             "access_mode": {
-                "mode": 5
+                "mode": "MULTI_NODE_MULTI_WRITER"
+            },
+            "mount": {
+                "fs_type": "ext4",
+                "mount_flags": [
+                    "flag1",
+                    "flag2",
+                    "flag3"
+                ]
             }
         }
     ],
-    "volume_content_source": {
-        "Type": null
-    }
+    "volume_content_source": {}
 }

Also added a benchmark to justify the new implementation:
Old:

goos: darwin
goarch: arm64
pkg: github.com/kubernetes-csi/csi-lib-utils/protosanitizer
cpu: Apple M2 Pro
BenchmarkStrip             98264             11801 ns/op            7488 B/op        149 allocs/op
BenchmarkStripLarge           22          49160320 ns/op        39795573 B/op     620137 allocs/op

New:

BenchmarkStrip            141199              7608 ns/op            4816 B/op         89 allocs/op
BenchmarkStripLarge          122          12642834 ns/op         9955752 B/op     140009 allocs/op

The new implementation is significantly faster, and allocates less memory

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

protosanitizer now imports csi spec directly, instead of copy over the e_CsiSecret. Since this module already requires the csi spec module, this should not break anyone.

Support for CSI spec v0.3 is dropped, since the csi.pb.go file in that version does not implement the new protoreflect interface. And again, since this module already requires the csi spec module v1, users cannot use this module with v0.3 even without this PR. So, this is not a new breakage.

Does this PR introduce a user-facing change?:

The output format of protosanitizer is slightly refined.

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. labels Nov 23, 2024
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: huww98
Once this PR has been reviewed and has the lgtm label, please assign saad-ali for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Nov 23, 2024
@k8s-ci-robot
Copy link
Contributor

Welcome @huww98!

It looks like this is your first PR to kubernetes-csi/csi-lib-utils 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-csi/csi-lib-utils has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

@k8s-ci-robot k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Nov 23, 2024
@k8s-ci-robot
Copy link
Contributor

Hi @huww98. Thanks for your PR.

I'm waiting for a kubernetes-csi member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Nov 23, 2024
@huww98
Copy link
Author

huww98 commented Nov 23, 2024

/cc @pohly @mowangdk

@huww98 huww98 force-pushed the update-proto branch 2 times, most recently from 819e5e4 to 278086b Compare November 23, 2024 18:27
@mowangdk
Copy link

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Nov 26, 2024
It now works with all kinds of nested fields, and don't require json.Marshal() twice.

The output also looks better for oneof and enum.
@huww98
Copy link
Author

huww98 commented Dec 14, 2024

@jsafrane Can you also take a look at this one? It should speedup several times for cases like kubernetes-csi/external-provisioner#1310 when the log level is high.

}
}

func stripValue(field protoreflect.FieldDescriptor, v protoreflect.Value) any {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can be stripValue() and stripSingleValue() combined into a single function with a bigger switch?

switch {
case field.IsList():
...

case field.IsMap():
...

case field.Kind() == protoreflect.MessageKind:
...
case field.Kind() == ...

default: return v.Interface()

Comment on lines +81 to +82
for i := range l.Len() {
res[i] = stripSingleValue(field, l.Get(i))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For example, what happens when a list has a list? Will it call stripSingleValue() with a list, which then leads to default: branch there and thus print the list with potential secrets?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants