Incorrect "validated" state on Broker when provider test fails with runtime error

[csbiggar]

When the provider verification test fails due to a runtime error - most commonly we have MissingStateChangeMethod due to getting the state match up wrong - the results still get published to the broker and incorrectly show as verified.

This is pretty misleading, because actually the verification test didn't run so we don't know if it would have passed or not - we've had cases where it wouldn't.

Could you help me problem solve this? Is it a bug or is there a setting I can tweak so that this shows as "not run" or something else on the broker? Our main requirement is that it doesn't show as "passed"

Here's an example using the public broker as suggested - nb the consumer and provider tests are both in the same project for convenience here!

I've added steps to recreate to the readme, let me know if anything is unclear.

https://github.com/csbiggar/pact-example/blob/master/README.md

Many thanks for any help

[csbiggar]

following pact slack chats here
https://pact-foundation.slack.com/archives/C9VPNUJR2/p1584800617098000
and here
https://pact-foundation.slack.com/archives/C9UN99H24/p1585165517013600
thanks!

[csbiggar]

Just wondered if you had any thoughts on this at all? It's continuing to cause confusion for us, would love some advice about how we might work around it to force the broker to show "not run"

[tdshipley]

(FYI: I work in the same team as @csbiggar)

I have been caught out a couple of times while working with API consumers by this. It is also made a bit worse for me as consumers typically check the broker for validation status rather than our pipeline. This leads to false positives without the opportunity for us to correct them on the result unless we are extra diligent (which I try to be!).

[joaofslopes]

👍

[uglyog]

Thanks for the example project, it made it easy to diagnose the issue.

The problem is you're using an old version of the pact-jvm-provider-junit5 library (au.com.dius:pact-jvm-provider-junit5_2.12:3.6.11. This has been fixed in a later version.

Changing the dependency in the example project to

    testImplementation "au.com.dius:pact-jvm-provider-junit5:4.0.8"

fixes the issue.

[csbiggar]

Hi @uglyog thanks very much for looking

I've upgraded as you suggested (apologies, should have checked versions first thing!) and I'm still able to recreate the problem , with a subtle difference ...

(1) If the provider verification would pass if the @State was present, and the only issue is the missing @State, then the verification still fails with MissingStateChangeMethod but the result is still published to the broker as success. I've updated the example project to reflect this, so the readme steps to recreate will still recreate this.

(2) If however the provider verification would fail (should the @State be present) and also, the @State is missing - then it looks like it's running the verification first and not worrying about the state check; the verification fails, and shows as failed on the broker. You can recreate this with the example project by changing line 23 in the ProviderVerificationTest to something else , for example

context.target = HttpsTestTarget("www.google.com/abc", 443, "/")

This is better than previously, but, I think it's inconsistent in its judgement on whether missing state is a problem.

If the idea is to force the provider to acknowledge the consumer's "givens" / state requirements, shouldn't a missing state error show up as failed on the broker , even if the test passes without it?

Alternatively, make state truly optional and don't fail the provider verification if it is missing? (log a warning or something instead?)

Hope this makes sense, thanks again, Carolyn

[uglyog]

With my testing of your example project, it posted a failed result when the missing state change exception occurred. You can see the result of that here: https://test.pact.dius.com.au/matrix/provider/ron.google/consumer/ron.pretend-consumer

It runs all the state change methods first. If any of those fail, it should abort at that point with an error.

Can you enable debug level logs and post them here?

[csbiggar]

Yep let me see if I can figure out how to do that ;)

But also ... your verification is from some hours ago, perhaps you could pull the latest of the example code and try again? Until I tweaked about an hour ago, I think the verification was giving a genuine failure - ie (2) in my comment above - when I upgraded it just now i also changed it so it should pass if state is present - ie (1) in my comment above.

Will update when I've figured out debugging ...

[csbiggar]

Here's the logs:

1-pact-verification-should-pass-but-missing-state.txt

• scenario 1 above - you can recreate this with the example project (pull the latest). State is missing, but broker shows green

2-pact-verification-genuine-failure.txt

• scenario 2 above - make a small change to the target url in the example project to see this, eg
  context.target = HttpsTestTarget("www.google.com/abc", 443, "/"). State is missing but also there is a genuine verification error, broker show red

[uglyog]

I think I have replicated your issue. It looks like you are correct, the missing state change method results in a failure being published, but then the test is run and the results are overwritten with a success.

Thanks for the example project, it made it easy to diagnose.

BTW, the debug logging needed to be enabled for the tests. I did this by adding

testImplementation "ch.qos.logback:logback-core:1.2.3",
            "ch.qos.logback:logback-classic:1.2.3"

to your project dependencies. Then I could see the HTTP requests that were made in the test output.

[csbiggar]

Thanks so much for fixing this so fast! Have just upgraded our provider tests to use 4.0.10 and it's working perfectly. Much appreciated!