CI Build should run UnitTests

[rdeveen]

What's the Problem?

The automated build does not run the Unit Tests.

Solution/Idea

Change the Azure Pipelines to automaticly run the Unit Tests on every build.

Alternatives

(remove the tests ;))

Priorities

Capability	Priority
This proposal will allow developers to use a better DevToys	Must

DevToys Version

No response

Comments

No response

[veler]

Definitely! I didn't do it so far because we were using a CI plan with a limited amount of minutes, but now that this repo use a CI plan with unlimited minutes and that we have more contributors, I agree the CI should run the tests!

[rlm96]

@veler I can take this one :)

[veler]

@veler I can take this one :)

Hello @rlm96 , sorry for the late response. Sure thing, feel free to try this one out :)

[rlm96]

Hello, an update: I'm using @veler branch, the .yml files seemed to be correct so I tried to run .\vstest.console.exe using ...\DevToys.Tests.build.appxrecipe file. It failed (something similar to this vstest issue):

The active test run was aborted. Reason: Unable to communicate with test host process.
Closing app with package full name 'b99a209c-7a2d-42fa-ba34-b16c8dee0379_1.0.0.0_x64__amhbm8v6a514r'.

Test Run Aborted with error System.AggregateException: One or more errors occurred. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.IO.Stream.ReadByte()
   at System.IO.BinaryReader.ReadByte()
   at System.IO.BinaryReader.Read7BitEncodedInt()
   at System.IO.BinaryReader.ReadString()
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.LengthPrefixCommunicationChannel.NotifyDataAvailable()
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.TcpClientExtensions.MessageLoopAsync(TcpClient client, ICommunicationChannel channel, Action`1 errorHandler, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---.
Total tests: Unknown
     Passed: 152
 Total time: 29.0509 Seconds

That's probably the reason the build has failed.

[astegi-dev]

For me, when I build with Release configuration, the unit tests fail for some reason.
The error message is the same as what is logged in the build:
DEP7100: Failed to activate app 'b99a209c-7a2d-42fa-ba34-b16c8dee0379_amhbm8v6a514r!vstest.executionengine.universal.App' with parameters '--port 63354 --endpoint 127.0.0.1:063354 --role client --parentprocessid 4164 --datacollectionport 63353 --telemetryoptedin false'. COMException - The text associated with this error code could not be found.

However, if I build with Debug configuration, the unit tests can be executed. Might worth a try setting the Debug configuration in the CI.

[veler]

However, if I build with Debug configuration, the unit tests can be executed. Might worth a try setting the Debug configuration in the CI.

Perhaps it's acceptable to run unit tests only in Debug configuration? That would require the CI to build in Debug and Release mode, but it should be fine. Debug build is quite fast compared to Release. @rlm96 , would you like to try that?

[rlm96]

@veler - The failure explained in my previous comment was with Debug configuration. @astegi-dev - Did all your test finished succesfully with Debug configuration using vstest.exe?

We can try to use the Debug config in the pipeline but as it is failing in our locals I'm not sure it will fix the pipeline build

[veler]

Ah, makes sense! Let's try to make it work on the local machine first before trying in the CI then.

[rlm96]

Btw, also tried to launch vstest using the build with Release config and got the same error:

DeployAsync: END (Success, 0:00:03.84)
Deployment operation succeeded, package moniker: 'b99a209c-7a2d-42fa-ba34-b16c8dee0379_1.0.0.0_x64__amhbm8v6a514r'.
Deploy: END (Success, 0:00:03.856)
Attempting to launch app with app user model Id 'b99a209c-7a2d-42fa-ba34-b16c8dee0379_amhbm8v6a514r!vstest.executionengine.universal.App' and arguments '--port 56182 --endpoint 127.0.0.1:056182 --role client --parentprocessid 9868 --telemetryoptedin false'.
DEP7100: Failed to activate app 'b99a209c-7a2d-42fa-ba34-b16c8dee0379_amhbm8v6a514r!vstest.executionengine.universal.App' with parameters '--port 56182 --endpoint 127.0.0.1:056182 --role client --parentprocessid 9868 --telemetryoptedin false'. COMException - The text associated with this error code could not be found.

The text associated with this error code could not be found. [0x80040904]
App activation failed.
Failed to initialize client proxy: could not connect to test process.
Closing app with package full name 'b99a209c-7a2d-42fa-ba34-b16c8dee0379_1.0.0.0_x64__amhbm8v6a514r'.

Test Run Aborted.

Probably those errors (Debug and Release) may be related, thoughts?

[astegi-dev]

At first, the vstest.console.exe gave me errors as well. But then, I updated as many nuget packages as I could. After that, the test execution with Release configuration still fail, but with the Debug show no errors; all-green via console execution.

[rlm96]

@astegi-dev Indeed, so I think what is making the tests fail are MSTest.TestAdapter and MSTest.TestFramework references. I updated only both packages to 2.2.8 version and the tests passed. @veler should we update those dependencies and then try again to build in Azure DevOps (with Debug config, I also tried with Release and still failing as mentioned by @astegi-dev)?

[veler]

That's interesting! Thank you for finding this out! Yes, definitely, let's try this out. 😊

[veler]

Thank you so much @astegi-dev and @rlm96 ! It works great now 😁😁😁