Report test results for multiple suites #2747

narrieta · 2023-01-31T00:46:25Z

Currently we report test results to the junit notifier as a single test suite (AgentTestSuite). This PR fixes that by reporting results for individual test suites.

I also includes further simplification and restructuring of the source code tree that I started in my previous PR.

codecov · 2023-01-31T00:53:33Z

Codecov Report

Merging #2747 (09702da) into develop (3aebcdd) will increase coverage by 0.06%.
The diff coverage is 82.29%.

@@             Coverage Diff             @@
##           develop    #2747      +/-   ##
===========================================
+ Coverage    71.97%   72.04%   +0.06%     
===========================================
  Files          103      104       +1     
  Lines        15692    15832     +140     
  Branches      2486     2265     -221     
===========================================
+ Hits         11295    11406     +111     
- Misses        3881     3912      +31     
+ Partials       516      514       -2

Impacted Files	Coverage Δ
azurelinuxagent/ga/collect_logs.py	`81.15% <0.00%> (-0.36%)`	⬇️
azurelinuxagent/common/logcollector.py	`88.35% <33.33%> (+0.04%)`	⬆️
azurelinuxagent/common/osutil/factory.py	`91.11% <33.33%> (-2.00%)`	⬇️
azurelinuxagent/daemon/main.py	`71.42% <33.33%> (+0.29%)`	⬆️
azurelinuxagent/common/osutil/fedora.py	`46.51% <46.51%> (ø)`
azurelinuxagent/common/cgroupconfigurator.py	`72.38% <56.36%> (-1.20%)`	⬇️
azurelinuxagent/common/protocol/hostplugin.py	`87.76% <75.00%> (+0.09%)`	⬆️
azurelinuxagent/common/protocol/wire.py	`77.37% <84.90%> (-1.18%)`	⬇️
azurelinuxagent/ga/exthandlers.py	`86.06% <87.50%> (+0.37%)`	⬆️
azurelinuxagent/common/protocol/goal_state.py	`95.59% <94.52%> (+0.27%)`	⬆️
... and 22 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

narrieta · 2023-01-31T00:48:35Z

tests_e2e/orchestrator/lib/agent_junit.py

@@ -28,7 +28,7 @@
 from lisa import schema  # pylint: disable=E0401
 from lisa.messages import (  # pylint: disable=E0401
    MessageBase,
-    TestResultMessageBase,
+    TestResultMessage,


Since now we are generating our own messages, I am restricting this to TestResultMessage instead of the base class, which also would include subtests (I may try subtests on subsequent PRs, but I need to see first if/how Azure Pipelines handles those)

narrieta · 2023-01-31T00:51:43Z

tests_e2e/orchestrator/lib/agent_junit.py

@@ -48,8 +48,11 @@ def type_schema(cls) -> Type[schema.TypedSchema]:
        return AgentJUnitSchema

    def _received_message(self, message: MessageBase) -> None:
-        if isinstance(message, TestResultMessageBase):
+        if isinstance(message, TestResultMessage) and message.type != "AgentTestResultMessage":


AgentTestResultMessages are the messages generated for our test suites. They will include the name of each suite.

On the next line I change the name of other TestResultMessages to "Setup". Those are the messages generated by LISA and include the creation, setup and deletion of the test VMs. I chose "Setup" because they are similar to the current setup logs in DCR.

narrieta · 2023-01-31T00:53:12Z

tests_e2e/orchestrator/lib/agent_test_loader.py

@@ -20,7 +20,7 @@
 from pathlib import Path
 from typing import Any, Dict, List, Type

-from tests_e2e.scenarios.lib.agent_test import AgentTest


I removed "scenarios" from the source code tree. Now "tests" and "subtests" are directly under "tests_e2e". The previous "scenarios/lib" is now "tests/lib".

narrieta · 2023-01-31T00:53:39Z

tests_e2e/orchestrator/lib/agent_test_loader.py

@@ -69,7 +69,7 @@ def load(self, test_suites: str) -> List[TestSuiteDescription]:
            pass

        # Else, it should be a comma-separated list of description files
-        description_files: List[Path] = [self._root/"testsuites"/f"{t.strip()}.json" for t in test_suites.split(',')]
+        description_files: List[Path] = [self._root/"test_suites"/f"{t.strip()}.json" for t in test_suites.split(',')]


added an "_" to the name

narrieta · 2023-01-31T00:54:07Z

tests_e2e/orchestrator/lib/agent_test_suite.py

@@ -97,8 +100,8 @@ class CollectLogs(Enum):
 @TestSuiteMetadata(area="waagent", category="", description="")
 class AgentTestSuite(TestSuite):
    """
-    Base class for Agent test suites. It provides facilities for setup, execution of tests and reporting results. Derived


I forgot to update this comment in my previous PR

narrieta · 2023-01-31T00:55:21Z

tests_e2e/orchestrator/lib/agent_test_suite.py

+                message.full_name = f"{suite_name}-{self.context.image_name}"
+                message.name = message.full_name
+                message.elapsed = 0
+                notifier.notify(message)


We send those messages to our junit notifier. This first message indicates that a new test suite is running

Sending the message at this point doesn't help us like we don't know live status. The pipeline only sends the report after test execution and at end of test execution we report with new status and that status override this running status. when do you think it's useful?

It is part of the protocol needed by the implementation of the junit reporter

(this is required by LISA)

narrieta · 2023-01-31T00:55:52Z

tests_e2e/orchestrator/lib/agent_test_suite.py

+                    message.stacktrace = traceback.format_exc()
+                finally:
+                    message.elapsed = (datetime.datetime.now() - start_time).total_seconds()
+                    notifier.notify(message)


This second message indicates that the suite completed (either PASSED or FAILED)

narrieta · 2023-01-31T00:56:54Z

tests_e2e/orchestrator/lib/agent_test_suite.py

@@ -326,57 +329,84 @@ def execute(self, node: Node, variables: Dict[str, Any], log: Logger) -> None:
            finally:
                self._clean_up()

-        # Fail the entire test suite if any test failed; this exception is handled by LISA
-        if len(failed) > 0:
-            fail(f"{[self.context.image_name]} One or more tests failed: {failed}")


We don't need to report this failure to LISA anymore. Now we report Pass/Fail for each individual test suite

nagworld9 · 2023-02-01T02:24:02Z

tests_e2e/orchestrator/lib/agent_test_suite.py

+                        message.status = TestStatus.PASSED
+                    else:
+                        message.status = TestStatus.FAILED
+                        message.message = f"Tests failed: {failed}"


This just logs the failed test_names but if we could add some debug info or error message would be helpful. This would avoid downloads the log file especially if it's too many test failures and most of them known issues. If we know if it's known issue upfront, we could save time and makes it easy for people for monitors this.

Thanks. Failures should not be frequent, but I'll consider that for a different iteration of the code.

nagworld9 · 2023-02-01T02:31:46Z

tests_e2e/orchestrator/lib/agent_test_suite.py

+                message.full_name = f"{suite_name}-{self.context.image_name}"
+                message.name = message.full_name
+                message.elapsed = 0
+                notifier.notify(message)


Sending the message at this point doesn't help us like we don't know live status. The pipeline only sends the report after test execution and at end of test execution we report with new status and that status override this running status. when do you think it's useful?

Report test results for multiple suites

09702da

narrieta requested review from ZhidongPeng, nagworld9 and maddieford as code owners January 31, 2023 00:46

maddieford approved these changes Jan 31, 2023

View reviewed changes

narrieta commented Jan 31, 2023

View reviewed changes

nagworld9 reviewed Feb 1, 2023

View reviewed changes

nagworld9 approved these changes Feb 1, 2023

View reviewed changes

narrieta merged commit becea90 into Azure:develop Feb 1, 2023

narrieta deleted the multi-suite branch February 1, 2023 02:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Report test results for multiple suites #2747

Report test results for multiple suites #2747

narrieta commented Jan 31, 2023

codecov bot commented Jan 31, 2023

narrieta Jan 31, 2023

narrieta Jan 31, 2023

narrieta Jan 31, 2023

narrieta Jan 31, 2023

narrieta Jan 31, 2023

narrieta Jan 31, 2023

nagworld9 Feb 1, 2023 •

edited

Loading

narrieta Feb 1, 2023

narrieta Feb 1, 2023

narrieta Jan 31, 2023

narrieta Jan 31, 2023

nagworld9 Feb 1, 2023

narrieta Feb 1, 2023

nagworld9 Feb 1, 2023 •

edited

Loading

Report test results for multiple suites #2747

Report test results for multiple suites #2747

Conversation

narrieta commented Jan 31, 2023

codecov bot commented Jan 31, 2023

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nagworld9 Feb 1, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nagworld9 Feb 1, 2023 • edited Loading

Choose a reason for hiding this comment

nagworld9 Feb 1, 2023 •

edited

Loading

nagworld9 Feb 1, 2023 •

edited

Loading