Fix inconsistent Prometheus cAdvisor metrics

[bboreham]

What this PR does / why we need it:

We need this because otherwise kubelet is exposing different sets of Prometheus metrics that randomly include or do not include containers.

See also google/cadvisor#1704; quoting here:

Prometheus requires that all metrics in the same family have the same labels, so we arrange to supply blank strings for missing labels

The function containerPrometheusLabels() conditionally adds various metric labels from container labels - pod name, image, etc. However, when it receives the metrics, Prometheus checks that all metrics in the same family have the same label set, and rejects those that do not.

Since containers are collected in (somewhat) random order, depending on which kind is seen first you get one set of metrics or the other.

Changing the container labels function to always add the same set of labels, adding "" when it doesn't have a real value, eliminates the issue in my testing.

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #

Fixes #50151

Special notes for your reviewer:

I have made the same fix in two places. I am 98% sure the one in cadvisor_linux.go isn't used and indeed cannot be used, but have not gone fully down that rabbit-hole.

Release note:

Fix inconsistent Prometheus cAdvisor metrics

[k8s-ci-robot]

Hi @bboreham. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[dashpole]

/ok-to-test

[dashpole]

/test pull-kubernetes-federation-e2e-gce

[dashpole]

This change looks good, although it doesnt appear to fix all issues. At least for me, it looks like diskio metrics are still not consistent. I'll have to look into that...
/lgtm

[dashpole]

/assign @Random-Liu
for approval. This fixes google/cadvisor#1704

[dashpole]

/test pull-kubernetes-federation-e2e-gce

[Random-Liu]

/approve

[k8s-github-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Random-Liu, bboreham, dashpole

Associated issue: 50151

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

• pkg/kubelet/OWNERS [Random-Liu]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to @fejta).

Review the full test history for this PR.

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to @fejta).

Review the full test history for this PR.

[k8s-github-robot]

Automatic merge from submit-queue (batch tested with PRs 51471, 50561, 50435, 51473, 51436)

[sylr]

This needs to be cherry picked into release-1.7 before the next release.

[sylr]

@Random-Liu or anyone could launch the release-1.7 cherry-pick process ?

[luxas]

@sylr if you want to cherrypick this; just propose doing so.
Install https://github.com/github/hub and run

GITHUB_USER=sylr hack/cherry-pick-pull.sh upstream/release-1.7 51473

(also see the doc: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md)

No need to wait for others to do this.
Thanks!

[k8s-cherrypick-bot]

Removing label cherrypick-candidate because no release milestone was set. This is an invalid state and thus this PR is not being considered for cherry-pick to any release branch. Please add an appropriate release milestone and then re-add the label.

[sylr]

@luxas Thanks

Note to self:

GITHUB_USER=sylr UPSTREAM_REMOTE=origin FORK_REMOTE=sylr hack/cherry_pick_pull.sh sylr/release-1.7 51473

[luxas]

yeah, if that are the git remote names you're using

[AndreaGiardini]

@luxas I think you need to add a release milestone to get this cherry-pick go on

[luxas]

@AndreaGiardini AFAIK, that's not totally mandatory, but makes it much more clear and the normal flow, so will do. Thanks!

[k8s-cherrypick-bot]

Commit found in the "release-1.7" branch appears to be this PR. Removing the "cherrypick-candidate" label. If this is an error find help to get your PR picked.