Pact provider reports when there are multiple consumers tested

[anddreiu]

I have multiple consumers (7, at the moment) for the same provider and I am running all the provider tests in a test suite using Junit 4.

The generated Pact provider report (the markdown report) contains currently 130 interactions and it's a little bit hard to search in it if there are failed tests.

It would be a great enhancement if the report could be improved a bit, maybe to have like a summary of all the interactions run based on the consumer names or to group the interactions based on the consumer and then to have all the run interactions.

Thanks!

[anto-ac]

@anddreiu Which version of pact-jvm are you using? This side of things has been recently improved.

[anddreiu]

@anto-ac I am using pact provider junit 4.1.1. I know it has been improved recently. What I am suggesting here is to enhance more the reports which might help when there are a lot of interactions. As I said in the description, to have like a summary, a table, and then to have the interactions grouped based on the consumer name. Because what we have now is a big list of interactions that were run and we have to check for each one if it's OK or Failed.

[uglyog]

A summary would be a really useful feature.

[anddreiu]

Hi guys. Are there any updates related to this? It would be really nice and useful to have a summary in the report.
Thanks!

[uglyog]

4.1.9 released with the summary added

[anddreiu]

Hey @uglyog . Just tried the new version and the summary. It looks like just the first consumer is added in the summary table. Then there are a lot of blank lines and afterwards we have the normal report information added. Please check below the output from an .md report file generated after running the Pact provider tests using Junit runner.

In my case I have different test classes for each consumer and then I am running all of them using a test suite. But I've tried also having the tests for all the consumers in a single test class, but the report looks the same.

# providerName
|  Description   |                      Value                      |
|----------------|-------------------------------------------------|
| Date Generated | 2020-10-19T10:06:17.957+03:00[Europe/Bucharest] |
| Pact Version   | 4.1.9                                           |
## Summary
|       Consumer        | Result |
|-----------------------|--------|
| consumerName1         | OK     |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
|                       |        |
## Verifying a pact between _consumerName1_ and _providerName_

[uglyog]

How many consumers do you have?

[anddreiu]

I have 8 consumers for the same provider.

[uglyog]

@anddreiu version 4.1.10 has been released with the fix

[anddreiu]

Thanks @uglyog. I've just upgraded the version in my project and tested this. Now the summary looks good, but the results are not quite alright. I always get OK result even if there are failed interactions for few of the consumers.

[anddreiu]

I can see that the summary contains correct result when a provider state is failed: State change call failed. But it has to show also failed message when the actual interaction failed (when the status code is not alright or when matching body or rules are failed)

[uglyog]

If the provider state change has failed, it will not attempt to verify the interaction so there will be no message to display for that.

[anddreiu]

That case is alright. The problem appears when there isn’t a provider state and the interaction fails, this is not reported ok in the summary.

[uglyog]

If the interaction fails, why would you want it to report it as ok in the summary?

[anddreiu]

I was misunderstood. I wanted to say that if the interaction fails it is not reported correctly in the summary (it should be failed, but it's reported to be ok). Not sure how to reproduce this, because for some of the consumers it's reported fine, but for others not.

I am using now 4.1.13 version.

[uglyog]

It might be a concurrency issue. I'll try adding a file lock on the report file to see if that fixes the issue.

[uglyog]

4.2.8 is released

[uglyog]

Released 4.1.24 and 4.2.9

[YOU54F]

Closing as complete, if anyone is encountering similar issues, please raise a new one and you can always reference this one.

Thanks