Mismatch between Branch Coverage in Cobertura and HTML reports

[ardalis]

We are generating code coverage using dotnet test, coverlet, and cobertura. We are using ReportGenerator to combine the coverage results and generate both Cobertura and HTML reports. We are also integrating with Build Quality Checks to gate the builds if line or branch coverage dips below a certain threshold.

We would very much like for the HTML reports to show the same numbers that the build gates will trigger off of. It's obviously very confusing to see a build fail because branch coverage is too low, but look at the Code Coverage tab and see that it's clearly above the require threshold.

I know this is related to other issues that have been closed related to how RG counts branches vs. how cobertura does. However, since RG is responsible for generating the combined cobertura.xml file, couldn't it use its own count when generating the coverage element in that file so that it matches the HTML output?

See related issue here with screenshots and more details:
MicrosoftPremier/VstsExtensions#120

Thanks!

[ardalis]

Here's how we're combining the coverage files from 3 different test runs:

    - task: Palmmedia.reportgenerator.reportgenerator-build-release-task.reportgenerator@4
      displayName: Test Code Coverage Report Generator
      inputs:
        reports: '$(Build.SourcesDirectory)/tests/{unittests}/coverage.cobertura.xml;$(Build.SourcesDirectory)/tests/{integrationtests}/coverage.cobertura.xml;$(Build.SourcesDirectory)/tests/{functionaltests}/coverage.cobertura.xml;'
        targetdir: '$(Build.SourcesDirectory)/CodeCoverage'
        reporttypes: 'HtmlInline_AzurePipelines;Cobertura;Badges'
        assemblyfilters: '-xunit*'  

[danielpalme]

The HTML and the Cobertura file generated by ReportGenerator should be consistent.
Before we have a look at the concrete files, let's make sure that the problem is not related to Azure Devops.

You are using the "PublishCodeCoverageResults" task:

    - task: PublishCodeCoverageResults@1
      displayName: 'Publish code coverage'
      inputs:
        codeCoverageTool: Cobertura
        summaryFileLocation: '$(Build.SourcesDirectory)/CodeCoverage/Cobertura.xml'
        reportDirectory: '$(Build.SourcesDirectory)/CodeCoverage'

This task uses the Coberatura file generated by ReportGenerator, but it does not use the HTML output of ReportGenerator.
Instead it regenerates the HTML report based on the Coberatura file.
To disable that behavior, you have to set an environment variable:

disable.coverage.autogenerate: 'true'

See also: https://github.com/danielpalme/ReportGenerator/wiki/Integration#attention

Does this fix your problem?

It might be helpful to pack the output of ReportGenerator (directory $(Build.SourcesDirectory)/CodeCoverage) into an artifact which you can download. Then you can inspect the HTML and Cobertura files directly.

[ardalis]

We are not using

disable.coverage.autogenerate: 'true'

You're saying if we add it, we should see consistency between the HTML and cobertura.xml files?

I have already downloaded the CodeCoverage artifact and determined that the HTML and cobertura.xml files in there do not match currently.

[danielpalme]

You're saying if we add it, we should see consistency between the HTML and cobertura.xml files?

At least the HTML files won't get regenerated, which might cause additional inconsistencies.

I have already downloaded the CodeCoverage artifact and determined that the HTML and cobertura.xml files in there do not match currently.

Can you share the three Cobertura files that you are passing to ReportGenerator?
You can send them by email: [email protected]

[danielpalme]

Can you share the cobertura xml files?

[ardalis]

Yes, I'll email them today from my client computer. We did manage to get consistent results between HTML and cobertura.xml files by setting the disable.autogenerate to true. Unfortunately, it's now reflecting the low coverage numbers in both places, because it's not taking into account our exclusions.

[ardalis]

Problems solved!

In working with the team that had this problem I discovered that while they were combining 3 cobertura files, they were only running 2 tests (the 3rd was commented out). However, the 3rd had its cobertura.xml file checked into source control (some time ago) so it was being combined via RG with the other 2.

The lack of the disable.coverage.autogenerate: 'true' hid this problem since the HTML seemed to show the right result. Adding that in got the HTML report to match the RG-generated cobertura.xml file. Removing the old cobertura coverage file from source eliminated the problem entirely.