libct/cg/v1: work around CPU quota period set failure #3090

kolyshkin · 2021-07-14T22:39:34Z

This is based on (and currently includes) #3175.

I. As reported in issue #3084, sometimes setting CPU quota period fails
when a new period is lower and a parent cgroup has CPU quota limit set.

This happens as in cgroup v1 the quota and the period can not be set
together (this is fixed in v2), and since the period is being set first,
new_limit = old_quota/new_period may be higher than the parent
cgroup limit.

The fix is to retry setting the period after the quota, to cover all
possible scenarios.

Add a test case to cover a regression caused by an earlier version of
this patch (ignoring a failure of setting invalid period when quota is
not set).

Fixes: #3084
Closes: #2044

II. Add a test case for an issue fixed by the previous commit.

Unfortunately, this is somewhat complicated as there's no easy way to
create a transient unit, so a binary, sd-helper, had to be added. On top
of that, an ability to create a parent/pod cgroup is added to
helpers.bash, which might be useful for future integration tests.

History

v2: rearranged, cleanups split to tests/int/helpers: cleanup, enable shellcheck #3175, simplified sd-helper to not depend on urlfave/cli.

1.0 backport: #3115

kolyshkin · 2021-07-15T01:39:07Z

A repro (which took me a while to write) fails as expected!

time="2021-07-15T01:24:25Z" level=error msg="failed to write "10000": write /sys/fs/cgroup/cpu,cpuacct/runc-cgroups-integration-test/pod_31883/test-cgroup-14305/cpu.cfs_period_us: invalid argument"
152

Adding the fix now.

kolyshkin · 2021-07-15T19:55:34Z

Don't know how to create a transient scope from bash :( and this is required for centos 7's systemd.

kolyshkin · 2021-07-15T21:12:38Z

Don't know how to create a transient scope from bash :( and this is required for centos 7's systemd.

In theory, I should be able to use dbus-send to call org.freedesktop.systemd1.Manager.StartTransientUnit but I can't figure out a way to.

Another way is to use libcontainer and have a small go binary to do that.

kolyshkin · 2021-07-20T02:47:10Z

Another way is to use libcontainer and have a small go binary to do that.

Trying it as I can't think of any better way right now.

kolyshkin · 2021-08-11T23:03:56Z

Ughm; the backport (#3115) is already merged, and this one has no LGTMs :-\ @opencontainers/runc-maintainers PTAL 🙏🏻

kolyshkin · 2021-08-19T02:30:47Z

@opencontainers/runc-maintainers PTAL 🙏🏻

thaJeztah

Didn't do a full review yet, but left a comment.

Also looks like the last commit is somewhat unrelated (could make sense separately)

thaJeztah · 2021-08-19T14:31:15Z

contrib/cmd/sd-helper/helper.go

+// version will be populated by the Makefile, read from
+// VERSION file of the source code.
+var version = ""
+
+// gitCommit will be the hash that the binary was built from
+// and will be populated by the Makefile.
+var gitCommit = ""


Will this utility only be used for tests? Wondering if adding version information is a bit too much for that (or even using urfave/cli for this).

Reason I'm looking at the version variable, is also that if we want to switch to //go:embed (#3163), the path to files to embed cannot traverse to parent directories (so //go:embed ../../VERSION won't work) 😅

I just copied this bit (as well as using urlfave/cli) from recvtty.

Let me try to drop both.

Fixed, PTAL @thaJeztah

kolyshkin · 2021-08-23T22:47:25Z

Also looks like the last commit is somewhat unrelated (could make sense separately)

That's right, but it's harder to work on it if it's separate (since other commits in this PR also change the same file), and I don't want to forget about it.

Let me try to rework it so the shellcheck fixes are separate.

kolyshkin · 2021-08-23T23:49:27Z

Let me try to rework it so the shellcheck fixes are separate.

Those fixes separated out to #3175 but this one depends on (and currently includes) it.

We can either review/merge #3175 first, and then I'll rebase, or merge this one as is and close #3175.

As reported in issue 3084, sometimes setting CPU quota period fails when a new period is lower and a parent cgroup has CPU quota limit set. This happens as in cgroup v1 the quota and the period can not be set together (this is fixed in v2), and since the period is being set first, new_limit = old_quota/new_period may be higher than the parent cgroup limit. The fix is to retry setting the period after the quota, to cover all possible scenarios. Add a test case to cover a regression caused by an earlier version of this patch (ignoring a failure of setting invalid period when quota is not set). Signed-off-by: Kir Kolyshkin <[email protected]>

Add a test case for an issue fixed by the previous commit. Unfortunately, this is somewhat complicated as there's no easy way to create a transient unit, so a binary, sd-helper, had to be added. On top of that, an ability to create a parent/pod cgroup is added to helpers.bash, which might be useful for future integration tests. Signed-off-by: Kir Kolyshkin <[email protected]>

kolyshkin · 2021-08-26T20:12:01Z

Rebased on top of merged #3175, PTAL @thaJeztah

thaJeztah

LGTM

kolyshkin · 2021-08-30T18:32:32Z

This fixes a real bug, and the backport to 1.0 was already merged and released in 1.0.2. @opencontainers/runc-maintainers PTAL

kolyshkin · 2023-04-06T22:22:06Z

For the reference, the same fix for crun: containers/crun#1188

kolyshkin force-pushed the cfq_quota_period branch from 0763c54 to 4a951b1 Compare July 15, 2021 01:21

kolyshkin force-pushed the cfq_quota_period branch 3 times, most recently from 4d77f91 to 8dac477 Compare July 15, 2021 19:04

kolyshkin changed the title ~~[WIP] fix for #3084~~ [WIP] libct/cg/v1: workaround CPU quota period set failure Jul 15, 2021

kolyshkin mentioned this pull request Jul 16, 2021

Reducing CPU period fails for subsystems if existing parent has quota>0 with systemd driver #3084

Closed

kolyshkin force-pushed the cfq_quota_period branch 3 times, most recently from b284685 to 956b9a8 Compare July 20, 2021 02:46

kolyshkin force-pushed the cfq_quota_period branch 8 times, most recently from 5e6f41f to c1ebddf Compare July 20, 2021 23:21

kolyshkin added area/cgroupv1 area/ci area/systemd kind/bug labels Jul 20, 2021

kolyshkin marked this pull request as ready for review July 20, 2021 23:42

kolyshkin requested review from AkihiroSuda, cyphar and mrunalp July 20, 2021 23:42

kolyshkin changed the title ~~[WIP] libct/cg/v1: workaround CPU quota period set failure~~ libct/cg/v1: workaround CPU quota period set failure Jul 21, 2021

kolyshkin force-pushed the cfq_quota_period branch from 7c0ab98 to 8d2c6c3 Compare July 28, 2021 17:46

kolyshkin added the backport/1.0-todo A PR in main branch which needs to be backported to release-1.0 label Jul 28, 2021

kolyshkin force-pushed the cfq_quota_period branch from 8d2c6c3 to 395628b Compare July 28, 2021 17:51

kolyshkin mentioned this pull request Jul 28, 2021

[1.0] libct/cg/v1: work around CPU quota period set failure #3115

Merged

kolyshkin added this to the 1.1.0 milestone Aug 11, 2021

kolyshkin mentioned this pull request Aug 12, 2021

Release 1.0.2 #3149

Closed

thaJeztah reviewed Aug 19, 2021

View reviewed changes

kolyshkin force-pushed the cfq_quota_period branch from 395628b to 0ff9030 Compare August 23, 2021 23:41

kolyshkin requested a review from thaJeztah August 23, 2021 23:50

kolyshkin mentioned this pull request Aug 23, 2021

tests/int/helpers: cleanup, enable shellcheck #3175

Merged

kolyshkin added 2 commits August 26, 2021 13:11

kolyshkin force-pushed the cfq_quota_period branch from 0ff9030 to 6394457 Compare August 26, 2021 20:11

thaJeztah approved these changes Aug 27, 2021

View reviewed changes

AkihiroSuda approved these changes Aug 30, 2021

View reviewed changes

AkihiroSuda merged commit 11d141b into opencontainers:master Aug 30, 2021

kolyshkin added backport/1.0-done A PR in main branch which has been backported to release-1.0 and removed backport/1.0-todo A PR in main branch which needs to be backported to release-1.0 labels Nov 30, 2021

kolyshkin mentioned this pull request Apr 25, 2024

libct/cg/fs: fix setting rt_period vs rt_runtime #4258

Merged

kolyshkin mentioned this pull request May 21, 2024

[1.1] libct/cg/fs: fix setting rt_period vs rt_runtime #4284

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

libct/cg/v1: work around CPU quota period set failure #3090

libct/cg/v1: work around CPU quota period set failure #3090

kolyshkin commented Jul 14, 2021 •

edited

Loading

kolyshkin commented Jul 15, 2021

kolyshkin commented Jul 15, 2021

kolyshkin commented Jul 15, 2021

kolyshkin commented Jul 20, 2021

kolyshkin commented Aug 11, 2021

kolyshkin commented Aug 19, 2021

thaJeztah left a comment

thaJeztah Aug 19, 2021

kolyshkin Aug 23, 2021

kolyshkin Aug 23, 2021

kolyshkin commented Aug 23, 2021

kolyshkin commented Aug 23, 2021

kolyshkin commented Aug 26, 2021

thaJeztah left a comment

kolyshkin commented Aug 30, 2021

kolyshkin commented Apr 6, 2023

libct/cg/v1: work around CPU quota period set failure #3090

libct/cg/v1: work around CPU quota period set failure #3090

Conversation

kolyshkin commented Jul 14, 2021 • edited Loading

History

kolyshkin commented Jul 15, 2021

kolyshkin commented Jul 15, 2021

kolyshkin commented Jul 15, 2021

kolyshkin commented Jul 20, 2021

kolyshkin commented Aug 11, 2021

kolyshkin commented Aug 19, 2021

thaJeztah left a comment

Choose a reason for hiding this comment

thaJeztah Aug 19, 2021

Choose a reason for hiding this comment

kolyshkin Aug 23, 2021

Choose a reason for hiding this comment

kolyshkin Aug 23, 2021

Choose a reason for hiding this comment

kolyshkin commented Aug 23, 2021

kolyshkin commented Aug 23, 2021

kolyshkin commented Aug 26, 2021

thaJeztah left a comment

Choose a reason for hiding this comment

kolyshkin commented Aug 30, 2021

kolyshkin commented Apr 6, 2023

kolyshkin commented Jul 14, 2021 •

edited

Loading