Fix error spam on AKS

[rdner]

This was happening due to the error level logging when the log path matcher detected a log.file.path that does not start with a standard Docker container log folder /var/lib/docker/containers because AKS dropped support for Docker in September 2022 and switched to containerd.

It looks like this message was not supposed to be on the error level in the first place since it just means that the matcher didn't match and it's not an error. But it was mistakenly promoted from the debug level in #16866 most likely because the message started with Error and looked confusing.

This a partial fix to unblock our customers, but we still need to come up with the full AKS/containerd support in a follow up change.

Why is it important?

There is a lot feedback from customers that Elastic Agent becomes unhealthy because of the overwhelming error messages from Filebeat due to the issue.

Checklist

• My code follows the style guidelines of this project
  - [ ] I have commented my code, particularly in hard-to-understand areas
  - [ ] I have made corresponding changes to the documentation
  - [ ] I have made corresponding change to the default configuration files
  - [ ] I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

• Closes Using Elastic Agent with the k8s integration on AKS causes error spam elastic-agent#1614

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b fix-container-id-error-spam upstream/fix-container-id-error-spam
git merge upstream/main
git push upstream fix-container-id-error-spam

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-11-16T16:22:10.127+0000

• Duration: 71 min 22 sec

Test stats 🧪

Test	Results
Failed	0
Passed	7083
Skipped	738
Total	7821

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[faec]

looks good (subject to the lint errors)

[rodrigc]

@rdner I think you can repro this in any kubernetes setup (not just AKS) which uses CRI-O
instead of docker. For example, Red Hat's Openshift.

If you do:

kubectl get nodes -o wide

if you see something like:

NAME          STATUS   ROLES   AGE   VERSION   OS-IMAGE                  KERNEL-VERSION                      CONTAINER-RUNTIME
host1   Ready    node    19d   v1.24.1      Oracle Linux Server 8.6   5.4.17-2136.314.6.2.el8uek.x86_64   cri-o://1.24.1-76.el8
host2   Ready    node    19d   v1.24.1      Oracle Linux Server 8.6   5.4.17-2136.314.6.2.el8uek.x86_64   cri-o://1.24.1-76.el8
host3   Ready    node    19d   v1.24.1      Oracle Linux Server 8.6   5.4.17-2136.314.6.2.el8uek.x86_64   cri-o://1.24.1-76.el8

where the latest column says cri-o instead of docker, then you will hit this problem