Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PublishCodeCoverageResults is slow with many files in the generated report #4945

Open
octatone opened this issue Aug 2, 2017 · 44 comments
Open

Comments

@octatone
Copy link

octatone commented Aug 2, 2017

related: #1271

Hi there, we have a project whose coverage report (like the above closed issue) generates over 900 files as its HTML coverage summary output. The PublishCodeCoverageResults task takes about as much time as it takes to generate the coverage report effectively doubling the time to complete our CI tasks. It takes over 2 minutes to publish our code coverage output (which in total is around 4MB).

Is it possible that the task is uploading each file one at a time? Can this be changed to zip the files (which are all text files), then upload the zip, and finally expand on the VSTS side?

@myarotskaya
Copy link

Hi there, is there any updates? I face this problem as well - it takes unacceptable long time to publish code coverage results.

@iwillspeak
Copy link

Seeing 1700+ files take over 3 minutes to publish. I'd really like to use the code coverage feature of pipelines but as it stands it's taking longer to publish the coverage than it takes to build and test our project.

We're able to upload far larger artifacts with the publish: yaml step in far less time (e.g. 140M in ~3s)

@matthoulb
Copy link

This is completely unusable. We have a simple .NET Framework solution and this took 20 minutes until I gave up and cancelled it. It was written 70mb+ HTML files for every single class and would have taken HOURS.

What is wrong with this? It seems very broken.

@iwillspeak
Copy link

It would be nice to just publish coverage results like normal artifacts but have a way to mark up on the publish step that they should be used as coverage results. Normal publish steps are more than fast enough to upload.

@richardgavel
Copy link

richardgavel commented Aug 31, 2020

Agree this is unusable. Even though it's using Task.WhenAll and possibly working in parallel, that's a lot of individual HTTP requests that have to be processed.

https://github.com/microsoft/azure-pipelines-agent/blob/1500e376855708134b1662d17ceddfba9519f0ba/src/Agent.Worker/CodeCoverage/CodeCoveragePublisher.cs#L51

Couldn't this be changed to use the PipelineArtifactServer like the publish pipeline artifact task does?

https://github.com/microsoft/azure-pipelines-agent/blob/e80c3261724f3256141076ae666bd0ca47e267c7/src/Agent.Plugins/PipelineArtifact/PipelineArtifactPluginV1.cs#L140

@DerAlbertCom
Copy link

This becomes more and more a problem. It's hard when to upload times takes over the build and test times. This is even worse on premises with the added latency.

@kachkaev
Copy link

Task performance is indeed disappointing. The task takes over three minutes for a project with ≈2K files. That’s about a third of how long the tests take to run 😞 Not much joy for PR authors and reviewers.

@ahaleiii
Copy link

ahaleiii commented Feb 9, 2021

Would like to echo what everyone else has said. Adding code coverage to our pipeline (.NET Core 2.1, coverlet.collector, cobertura) added 2 minutes to what was a 3 minute pipeline (1463 files).

I don't suppose there is a way to publish a coverage file from one build to another..? Could create an async workflow:

  • main build completes, coverage file is artifact
  • second, generic build is triggered
  • downloads coverage file from main build
  • generates report and publishes coverage back to main build

For now, I think we will disable report generation as suggested by @danielpalme #10354 (comment):

disable.coverage.autogenerate: 'true'

@tldavies
Copy link

We have this problem where publishing the code coverage report increased the build by 50%. Disabling the autogenerate didnt help decrease the time.

Currently to get around this we zip and upload the whole coverage report as an artifact of the build. Thus allowing engineers to access it. All of this takes less than a 1 minute. While publishing the coverage report(8500+ files) takes 10 minutes.

@kachkaev
Copy link

Currently to get around this we zip and upload the whole coverage report as an artifact of the build.

Does this mean that you don’t have the Code Coverage tab in Azure UI and your devs need to download the archive?

@tldavies
Copy link

Currently to get around this we zip and upload the whole coverage report as an artifact of the build.

Does this mean that you don’t have the Code Coverage tab in Azure UI and your devs need to download the archive?

Yes, the devs have to download the report if they want more detail. We do have the Code Coverage tab in the Azure UI, it just empty. We do fail our builds for code coverage and the devs only care about the report when the builds fails.

@vviotto
Copy link

vviotto commented Apr 20, 2021

Adding my voice to this, code coverage is taking around 6 minutes to upload 649 files, we're going to remove this step because its really slowing down our pipeline.
image

@danielpalme
Copy link
Contributor

You can use the following approach to speed things up:

Generate coverage report with this task: https://marketplace.visualstudio.com/items?itemName=Palmmedia.reportgenerator

Use the PublishCodeCoverageResults task only to publish the report and disable report regeneration.

Relevant YAML looks like this:

variables:
  disable.coverage.autogenerate: 'true'

- task: reportgenerator@4
  inputs:
    reports: 'coverage.cobertura.xml'
    targetdir: 'coveragereport'

- task: PublishCodeCoverageResults@1
  displayName: 'Publish code coverage results'
  inputs:
    codeCoverageTool: 'Cobertura'
    summaryFileLocation: 'coveragereport/Cobertura.xml'
    reportDirectory: 'coveragereport'

@kachkaev
Copy link

kachkaev commented Apr 21, 2021

@danielpalme We generate the report with nyc and use PublishCodeCoverageResults only to upload files. Because this task sends the HTMLs individually and not in the form of a single zip archive, the upload takes minutes. Each file is a separate HTTP request, so it’s thousands of requests to make.

This seems to be a limitation of the API endpoint, so we can’t fix it without a change in the Azure backend. When it can accept a zipped bundle of htmls and still show the result in the code coverage tab, the problem is solved.

@elliot-nelson
Copy link

elliot-nelson commented May 18, 2021

Huge +1 to this issue -- if this were task changes I'd happily put up a PR, but it seems like this might not be fixable without server-side changes.

In my case, although I only have about 1600 source files, it takes 1m30s to upload the Cobertura XML file and generate the coverage HTML; if I generate the HTML files myself with nyc and disable autogeneration as per the suggestion above, it takes longer (2m30s) due to uploading all those generated HTML files individually.

I would really, really like "click into" HTML coverage reports from my builds for the developers, but not at that cost. It seems like the workaround at this point might be uploading the files myself (maybe to an S3 bucket via s3 sync) and just linking to it from the Azure coverage report.

@kadian1470
Copy link

Has there been any updates around zipping these files yet?

@jsiegmund
Copy link

Another +1 from our side, we're having the exact same problem. Uploading the results takes up about 50% of our total build time (around 12 minutes). So cutting this down would make a huge difference and does not seem to be that hard. There really should be a way to upvote these issues btw, give some sort of feedback as to how many community members need something to be fixed (I also understand this would not necessarily be the same priority as the product team has).

@anatolybolshakov
Copy link
Contributor

@nadesu @ganesp could you please take a look?

@anatolybolshakov anatolybolshakov assigned nadesu and ganesp and unassigned leantk Jan 10, 2022
@MatiSanchez
Copy link

Any updates??

@blogcraft
Copy link

I also find uploading the code coverage result to be extremely slow making me even considering to remove it.

@a1i3n37
Copy link

a1i3n37 commented Aug 15, 2022

I would like to add my voice here as well.

The task takes minutes to complete. Is there any update on this issue? Simply throwing the files in a package and uploading it as a single file and unpacking on the other end should fix this issue. The package doesn't even need to be compressed (though, that would be nice as well).

At the current state, I am also considering in disabling this feature, because it is unusably slow.

@PeetersJeroen
Copy link

We have moved code coverage results to a dedicated build because it takes so long, hoping it will be fixed eventually

@Nodnarb3
Copy link

Nodnarb3 commented Nov 3, 2022

How is this thread over 5 years old and this is still an issue?? hello?

@blogcraft
Copy link

Nothing yet?

@brianpham93
Copy link

It's still very slow as of today

@MatteoSp
Copy link

MatteoSp commented Aug 7, 2023

It's 6 years now...

@andreypopov
Copy link

+1 too slow )

@bart-medfar
Copy link

bart-medfar commented Sep 5, 2023

Really slow, did any of you get an alternative running ?

@Pentadome
Copy link

ignore game is strong

@basit1327
Copy link

basit1327 commented Oct 17, 2023

PublishCodeCoverageResults randomly skip last few files after few minutes as a result no html report on available on Azure DevOps tab.
Screenshot 2023-10-17 at 11 28 33 AM

While for task where files are less then 500 it work fine
Screenshot 2023-10-17 at 11 28 33 AM

@ptemmer
Copy link

ptemmer commented Dec 15, 2023

Adding this parameter made it work for me:
DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.ExcludeByAttribute=GeneratedCodeAttribute

Without it I even got to agent pipeline hard disk to run out of space

@Oldervoll
Copy link

+1, please improve!

@blogcraft
Copy link

Looks like no contributor cares about this.

@tripleacoder
Copy link

I have this problem too, uploading 1754 files takes 2:36 minutes.

@andreypopov
Copy link

resolved with task v2

PublishCodeCoverageResults@2

@tripleacoder
Copy link

resolved with task v2

PublishCodeCoverageResults@2

OK, good to hear.
Unfortunately, PublishCodeCoverageResults@2 does not support detailed code coverage reports, it only shows a short summary.

@chrisarmand
Copy link

PublishCodeCoverageResults randomly skip last few files after few minutes as a result no html report on available on Azure DevOps tab. Screenshot 2023-10-17 at 11 28 33 AM

While for task where files are less then 500 it work fine Screenshot 2023-10-17 at 11 28 33 AM

seems there is a 2min timeout on underling CoveragePublisher.Console
see https://github.com/microsoft/azure-pipelines-coveragepublisher/blob/master/src/CoveragePublisher/ArgumentsProcessor.cs
image

unfortunately, AzDevOps task wrapping CoveragePublisher.Console does not seem to handle this timeout parameter
see https://github.com/microsoft/azure-pipelines-tasks/blob/master/Tasks/Common/coveragepublisher/coveragepublisher.ts

@victorgomezdejuan
Copy link

Hi there, we were facing the same issue and finally found a way to check the percentage of the code coverage and publish the report in less than 4 seconds (it took over 2 minutes before).

This is what we had (over 2 minutes):

- task: reportgenerator@5
    displayName: "Generate Reports"
    inputs:
      reports: '$(Build.SourcesDirectory)\**\*.Cobertura.xml'
      targetdir: '$(Build.SourcesDirectory)\coverlet\reports'
      reporttypes: 'HtmlInline_AzurePipelines_Dark;Cobertura'

  - task: JakubRumpca.azure-pipelines-html-report.PublishHtmlReport.PublishHtmlReport@1
    displayName: "Publish Unit Tests Report"
    inputs:
      reportDir: '$(Build.SourcesDirectory)\coverlet\reports\index.html'

  - task: PublishCodeCoverageResults@2
    displayName: "Publish Code Coverage Report"
    inputs:
      summaryFileLocation: '$(Build.SourcesDirectory)\coverlet\reports\Cobertura.xml'
      failIfCoverageEmpty: true  

  - task: BuildQualityChecks@9
    displayName: "Code coverage quality check"
    inputs:
      checkCoverage: true
      coverageFailOption: 'fixed'
      coverageType: 'lines'
      coverageThreshold: '85.5'

And this what we have now (less than 4 seconds):

  - task: reportgenerator@5
    displayName: "Generate code coverage report"
    inputs:
      reports: '$(Agent.TempDirectory)/*/*.cobertura.xml'
      targetdir: '$(Build.SourcesDirectory)\code-coverage-report'
      reporttypes: 'HtmlInline_AzurePipelines_Dark;Cobertura'
      riskhotspotassemblyfilters: '-*'
      customSettings: 'minimumCoverageThresholds:lineCoverage=85'

  - task: JakubRumpca.azure-pipelines-html-report.PublishHtmlReport.PublishHtmlReport@1
    displayName: "Publish code coverage report"
    condition: always() 
    inputs:
      reportDir: '$(Build.SourcesDirectory)\code-coverage-report\index.html'
      tabName: 'Code Coverage (Complete)'

The actual leverage point was using the customSettings property of reportgenerator@5 task, so that we did not need to publish the "standard" coverage report to be able to check the code coverage percentage.

I hope this help you as well.

@danielpalme
Copy link
Contributor

@victorgomezdejuan
You could further simplify your build process by using the setting publishCodeCoverageResults: true

  - task: reportgenerator@5
    displayName: "Generate code coverage report"
    inputs:
      reports: '$(Agent.TempDirectory)/*/*.cobertura.xml'
      targetdir: '$(Build.SourcesDirectory)\code-coverage-report'
      reporttypes: 'HtmlInline_AzurePipelines_Dark;Cobertura'
      riskhotspotassemblyfilters: '-*'
      customSettings: 'minimumCoverageThresholds:lineCoverage=85'
      publishCodeCoverageResults: true

@victorgomezdejuan
Copy link

@danielpalme I tried that solution but it is much slower (~ 1 minute) and it does not generate the report exactly the same way (it doesn't allow for project collapse, for example).

@danielpalme
Copy link
Contributor

@victorgomezdejuan
You are right, the "official" Code coverage tab blocks all Javascript and therefore does not have all features available.
I wasn't aware of the JakubRumpca.azure-pipelines-html-report.PublishHtmlReport.PublishHtmlReport extension. Thanks for that!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests