Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CAPI Operator: Increase resource quotas for pull-cluster-api-operator-make-* jobs #29808

Merged
merged 1 commit into from
Jun 14, 2023
Merged

CAPI Operator: Increase resource quotas for pull-cluster-api-operator-make-* jobs #29808

merged 1 commit into from
Jun 14, 2023

Conversation

furkatgofurov7
Copy link
Member

@furkatgofurov7 furkatgofurov7 commented Jun 14, 2023 •
edited
Loading

  1. even after the bump to 4Gi memory for the main jobs in 4c8e457 running on EKS they're still failing . This bumps the jobs (make and e2e) resource quotas to 4 CPUs and 8 GB RAM based on the https://kubernetes.slack.com/archives/CCK68P2Q2/p1686748417608999?thread_ts=1686748157.365979&cid=CCK68P2Q2 from @xmudrii
  2. reverts e2e jobs back fully by removing the resource quotas introduced in CAPI Operator: move jobs to EKS prow cluster #29743

Follow-up to: #29804

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jun 14, 2023
@k8s-ci-robot k8s-ci-robot added area/config Issues or PRs related to code in /config area/jobs sig/testing Categorizes an issue or PR as relevant to SIG Testing. labels Jun 14, 2023
@furkatgofurov7
Copy link
Member Author

/assign @ameukam
/cc @rjsadow @alexander-demicev @Fedosin @JoelSpeed

@k8s-ci-robot k8s-ci-robot requested review from Fedosin and rjsadow June 14, 2023 14:04
@furkatgofurov7 furkatgofurov7 force-pushed the increase-resource-quotas-capi-operator branch from b3a1e1d to 9b45825 Compare June 14, 2023 14:15
@furkatgofurov7
Increase resource quotas for e2e and make jobs to avoid cpu throttlin…
39b3775 
…g and extended build duration

Signed-off-by: Furkat Gofurov <[email protected]>
@furkatgofurov7 furkatgofurov7 force-pushed the increase-resource-quotas-capi-operator branch from 9b45825 to 39b3775 Compare June 14, 2023 14:16
@furkatgofurov7 furkatgofurov7 requested a review from rjsadow June 14, 2023 14:18
@rjsadow
Copy link
Contributor

rjsadow commented Jun 14, 2023

/lgtm Thanks @furkatgofurov7

@rjsadow
Copy link
Contributor

rjsadow commented Jun 14, 2023

/ok-to-test

@k8s-ci-robot k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Jun 14, 2023
@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 14, 2023
@furkatgofurov7
Copy link
Member Author

furkatgofurov7 commented Jun 14, 2023 •
edited
Loading

/hold

Looks like, not only above jobs but other build durations have increased also considerably, investigating..

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 14, 2023
@furkatgofurov7
Copy link
Member Author

Looks like, not only above jobs but other build durations have increased also considerably, investigating..

Here is the quick summary of jobs build duration (the last and longest time taken jobs compared) before and after moving to EKS cluster and excluding this PR change:

  • pull-cluster-api-operator-test-main: 3m7s vs 7m50s
  • pull-cluster-api-operator-verify-main: 45s vs 1m56s
  • pull-cluster-api-operator-make-main: 6m17s vs 9m32s
  • pull-cluster-api-operator-e2e-main: 7m25s vs 15m37s
  • pull-cluster-api-operator-build-main: 1m23s vs 6m46s
  • pull-cluster-api-operator-apidiff-main: 2m1s vs 6m31s

@rjsadow
Copy link
Contributor

rjsadow commented Jun 14, 2023

It looks like they're all still CPU pegged https://monitoring-eks.prow.k8s.io/d/96Q8oOOZk/builds?var-org=kubernetes-sigs&var-repo=cluster-api-operator&var-build=All&from=now-24h&to=now&orgId=1&var-job=pull-cluster-api-operator-apidiff-main&refresh=30s

@ameukam
Copy link
Member

ameukam commented Jun 14, 2023

/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ameukam, furkatgofurov7, rjsadow

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 14, 2023
@furkatgofurov7
Copy link
Member Author

It looks like they're all still CPU pegged https://monitoring-eks.prow.k8s.io/d/96Q8oOOZk/builds?var-org=kubernetes-sigs&var-repo=cluster-api-operator&var-build=All&from=now-24h&to=now&orgId=1&var-job=pull-cluster-api-operator-apidiff-main&refresh=30s

@ameukam yeah, so I think I have to increase the quotas for other jobs as well with the same PR if there are no objections

@rjsadow
Copy link
Contributor

rjsadow commented Jun 14, 2023

They can be separated into different PRs if you'd like. Your call.

@furkatgofurov7
Copy link
Member Author

They can be separated into different PRs if you'd like. Your call.

Thank you for helping with reviews, will follow-up then.

/hold cancel

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 14, 2023
@k8s-ci-robot k8s-ci-robot merged commit d041afe into kubernetes:master Jun 14, 2023
@k8s-ci-robot
Copy link
Contributor

@furkatgofurov7: Updated the job-config configmap in namespace default at cluster test-infra-trusted using the following files:

  • key cluster-api-operator-presubmits-main.yaml using file config/jobs/kubernetes-sigs/cluster-api-operator/cluster-api-operator-presubmits-main.yaml
  • key cluster-api-operator-presubmits-release-0-3.yaml using file config/jobs/kubernetes-sigs/cluster-api-operator/cluster-api-operator-presubmits-release-0-3.yaml

In response to this:

  1. even after the bump to 4Gi memory for the main jobs in 4c8e457 running on EKS they're still failing . This bumps the jobs (make and e2e) resource quotas to 4 CPUs and 8 GB RAM based on the https://kubernetes.slack.com/archives/CCK68P2Q2/p1686748417608999?thread_ts=1686748157.365979&cid=CCK68P2Q2 from @xmudrii
  2. reverts e2e jobs back fully by removing the resource quotas introduced in CAPI Operator: move jobs to EKS prow cluster #29743

Follow-up to: #29804

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@furkatgofurov7 furkatgofurov7 deleted the increase-resource-quotas-capi-operator branch June 14, 2023 15:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rjsadow rjsadow rjsadow approved these changes

@alexander-demicev alexander-demicev Awaiting requested review from alexander-demicev

@JoelSpeed JoelSpeed Awaiting requested review from JoelSpeed

@Fedosin Fedosin Awaiting requested review from Fedosin

Assignees

@ameukam ameukam

@rjsadow rjsadow

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/config Issues or PRs related to code in /config area/jobs cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@furkatgofurov7 @rjsadow @ameukam @k8s-ci-robot