When publishing verification results, only the errors are sent

[uglyog]

When publishing verification results, only the errors are being sent. This causes the Pact broker to see the passing interactions as un-verified.

For a test with 3 interactions, if two pass and one fails, the following entries are logged:

Received test result 'au.com.dius.pact.core.pactbroker.TestResult$Ok@3f693160' for Pact pactflow-example-provider-springboot-pactflow-example-consumer and a request to get a product (Pact Broker https://testdemo.pactflow.io/pacts/provider/pactflow-example-provider-springboot/consumer/pactflow-example-consumer/pact-version/4a6c2017e4f913c5b18ee0b526eae842567bf489)
2020-12-08 12:27:12.818 DEBUG 33876 --- [    Test worker] a.c.d.p.p.DefaultTestResultAccumulator   : Number of interactions #3 and results: [au.com.dius.pact.core.pactbroker.TestResult$Ok@3f693160]
2020-12-08 12:27:12.818  WARN 33876 --- [    Test worker] a.c.d.p.p.DefaultTestResultAccumulator   : Not all of the 3 were verified. The following were missing:
2020-12-08 12:27:12.818  WARN 33876 --- [    Test worker] a.c.d.p.p.DefaultTestResultAccumulator   :     a request to get a product
2020-12-08 12:27:12.818  WARN 33876 --- [    Test worker] a.c.d.p.p.DefaultTestResultAccumulator   :     a request to get all products

2020-12-08 12:27:13.029 DEBUG 33876 --- [    Test worker] a.c.d.p.p.DefaultTestResultAccumulator   : Received test result 'au.com.dius.pact.core.pactbroker.TestResult$Ok@3f693160' for Pact pactflow-example-provider-springboot-pactflow-example-consumer and a request to get all products (Pact Broker https://testdemo.pactflow.io/pacts/provider/pactflow-example-provider-springboot/consumer/pactflow-example-consumer/pact-version/4a6c2017e4f913c5b18ee0b526eae842567bf489)
2020-12-08 12:27:13.030 DEBUG 33876 --- [    Test worker] a.c.d.p.p.DefaultTestResultAccumulator   : Number of interactions #3 and results: [au.com.dius.pact.core.pactbroker.TestResult$Ok@3f693160, Failed(results=[{interactionId=ba08b56e3fecb8a49c7f7a2d509d9fe3183a44cf, attribute=status, description=expected status of 404 but was 200}], description=Response status did not match), au.com.dius.pact.core.pactbroker.TestResult$Ok@3f693160]
2020-12-08 12:27:13.030 DEBUG 33876 --- [    Test worker] a.c.d.p.p.DefaultTestResultAccumulator   : All interactions for Pact pactflow-example-provider-springboot-pactflow-example-consumer have a verification result

Then when the results are published:

2020-12-08 12:27:13.447 DEBUG 33876 --- [    Test worker] a.c.d.p.c.pactbroker.PactBrokerClient    : Test result = Failed(results=[{interactionId=ba08b56e3fecb8a49c7f7a2d509d9fe3183a44cf, attribute=status, description=expected status of 404 but was 200}], description=Response status did not match)
2020-12-08 12:27:13.448 DEBUG 33876 --- [    Test worker] a.c.dius.pact.core.pactbroker.HalClient  : Posting JSON to https://testdemo.pactflow.io/pacts/provider/pactflow-example-provider-springboot/consumer/pactflow-example-consumer/pact-version/4a6c2017e4f913c5b18ee0b526eae842567bf489/verification-results
{"providerApplicationVersion":"cdbbf3f+1607390821","success":false,"testResults":[{"interactionId":"ba08b56e3fecb8a49c7f7a2d509d9fe3183a44cf","mismatches":[{"attribute":"status","description":"expected status of 404 but was 200"}],"success":false}]}

Only the error is sent.

[anto-ac]

On which version did this start to happen?

[uglyog]

Not sure, but it has been reported with 4.0.10 and 4.1.8.

[uglyog]

4.1.12 released

[PhilHardwick]

Hi, thanks for fixing this ^ I've just opened #1276 to fix a small part of the refactor which came from this issue.

[Rafaesp]

@uglyog I'm still seeing something related to this in 4.1.14.
I have a case where a pending pact is failing and another pact tagged with master is passing.
It seems that what gets published is the following:

Published verification result of 'Failed(results=[{exception=java.lang.RuntimeException: Failed to invoke provider method 'updateTest', description=Request to provider method failed with an exception, interactionId=d7b7da1207e7db9dd47db7f01a1334baef01c303}, {interactionId=443bb987c3f94c905c58a38769407ed00725add1}, {interactionId=434ed5a5dbd99c9625d6424af897cc6bc26d32e8}, {interactionId=380cb209e8d00e36a330e77133b5091b30ef76a5}], description=Request to provider method failed with an exception)' for consumer 'Consumer(name=api)

Which in pact broker marks all interactions as failed even when all individual tests for the interaction marked as master are marked green and passing. This leads to can-i-deploy failing the build even when master pacts are passing.

When I use pact 4.1.6, the results are published one by one and I don't hit this problem.

[Rafaesp]

#1266 (comment) More logs of the issue above:

With Pact 4.1.14: https://pastebin.com/cTqBdx4V
With Pact 4.1.6: https://pastebin.com/94m9g70D

[mefellows]

@Rafaesp the pastebin logs appear to have been removed, are you able to please re-add them?

[Rafaesp]

@mefellows Done!
With Pact 4.1.14: https://pastebin.com/cTqBdx4V
With Pact 4.1.6: https://pastebin.com/94m9g70D
Thank you

[uglyog]

@Rafaesp are you using JUnit 4 or 5?

[Rafaesp]

JUnit5

[uglyog]

@Rafaesp I've released 4.1.19 which hopefully resolves your issue

[Rafaesp]

@uglyog Thank you very much. We have upgraded to the latest and made a few tests. The issue is hard to reproduce, the current status of our pacts is stable and we don't have new contracts so we can't be 100% sure it was fixed. However, we can still see the following behaviour in the Pact Broker that I expected 4.1.19 to have fixed:

• When there is a failure and the Pact status is marked as Red:
  

• When the verifications passed (there might be pending pacts that fail) and Pact status is marked as Green:
  

As you can see, Pact Broker doesn't know about the status of the interactions and I'm concerned this could lead to the same problem we were seeing when we had new contracts and pending pacts.
Did you expect Pact Broker to show a green badge for successful pact interactions?

[uglyog]

Unknown status means the broker did not receive a result for that interaction.

[Rafaesp]

@uglyog I can confirm that's still happening with 4.1.19 😞

[Rafaesp]

@uglyog

Unknown status means the broker did not receive a result for that interaction.

Does that make sense? The broker has marked the pact as verified (it's shown in the screenshot) and the logs show that the verifications have been published. But still, all individual interactions are marked as unknown.

@bethesque @mefellows Did you expect that? All interactions in our Pactflow instance (https://imanage.pact.dius.com.au/) are marked as unknown. However, can-i-deploy works as expected but this is very concerning :(

[mefellows]

@bethesque @mefellows Did you expect that? All interactions in our Pactflow instance (https://imanage.pact.dius.com.au/) are marked as unknown. However, can-i-deploy works as expected but this is very concerning :(

I can't speak for the implementation details of Pact JVM. I'm hopeful that perhaps there is a bug or detail I'm missing about why interaction level success/fail is not sent through, but I would assume the overall verification cannot be "successful" if the requests are not satisfied.

This feedback aside, are you seeing any evidence of the pact verification actually failing but still passing can-i-deploy?

[Rafaesp]

@mefellows When verifications fail, can-i-deploy also fails. That works as expected.

[uglyog]

This is actually a Pactflow issue. The verification results have a status for the verification, and then a status for each interaction. The UI is displaying an unknown result where the overall verification is a success, which does not make sense to me. If the overall verification is a success, all the interactions have to be successful, regardless if they have an individual status or not.

[bethesque]

I've made a Pactflow trello card to fix this.

[mefellows]

I believe this is fixed now, are you able to confirm?

[Rafaesp]

I can confirm the verification status is not unknown anymore! thanks!

[mefellows]

Awesome, thanks Rafa!