[ci] Improve maestro artifact publishing #20604

pjcollins · 2024-05-15T18:04:58Z

Context: https://github.com/dotnet/arcade/blob/efc3da96e5ac110513e92ebd9ef87c73f44d8540/Documentation/DependencyFlowOnboardingWithoutArcade.md

The steps used to publish build asset information to maestro have been updated.

With the new PushToAzureDevOpsArtifacts task the build pipeline should now create all of the artifacts required for maestro artifact publishing. The add-build-to-channel darc command will now trigger a Build Promotion Pipeline that pushes build assets to the feed that corresponds to the maestro channel that is being updated. We should no longer need to push assets to various NuGet feeds in a separate step.

Context: https://github.com/dotnet/arcade/blob/efc3da96e5ac110513e92ebd9ef87c73f44d8540/Documentation/DependencyFlowOnboardingWithoutArcade.md The steps used to publish build asset information to maestro have been updated. With the new `PushToAzureDevOpsArtifacts` task the build pipeline should now create all of the artifacts required for maestro artifact publishing. The `add-build-to-channel` darc command will now trigger a [Build Promotion Pipeline][0] that pushes build assets to the feed that corresponds to the maestro channel that is being updated. We should no longer need to push assets to various NuGet feeds in a separate step. [0]: https://dev.azure.com/devdiv/DevDiv/_build/results?buildId=9577012&view=logs&j=ba23343f-f710-5af9-782d-5bd26b102304&t=c7a8693b-2f9c-5ea8-c909-cde9405ac2e1&l=238

rolfbjarne

LGTM; but note that some of the changes won't be tested until the PR is merged (because we don't publish to maestro from PRs).

pjcollins · 2024-05-28T20:14:36Z

tools/devops/automation/templates/release/vs-insertion-prep.yml

+          $darcVersion = $(Invoke-WebRequest -Uri $versionEndpoint -UseBasicParsing).Content
+          $arcadeServicesSource = 'https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-eng/nuget/v3/index.json'
+          & dotnet tool update microsoft.dotnet.darc --version "$darcVersion" --add-source "$arcadeServicesSource" --tool-path $(Agent.ToolsDirectory)\darc -v n
+          & $(Agent.ToolsDirectory)\darc\darc add-build-to-channel --default-channels --id $(BARBuildId) --publishing-infra-version 3 --password $(MaestroAccessToken) --azdev-pat $(publishing-dnceng-devdiv-code-r-build-re)


@rolfbjarne @mandel-macaque I am trying to simplify things here by removing the separate platform specific maestro manifest generation steps that happen during the build. I am also having a hard time remembering why we needed a separate post-build pipeline to run the publishing steps, so I am trying to move them back into the "Prepare release" stage of the core pipeline to see if anything breaks.

I had a question with respect to the platform-specific publishing that we are currently doing -- if the build is configured to only build a subset of platforms (let's say iOS and tvOS), will it only produce NuGet packages for that platform? Or will the not-signed-package artifact still contain packages for everything?

Is there a way to manually queue an official build to try to test some of these changes?

I had a question with respect to the platform-specific publishing that we are currently doing -- if the build is configured to only build a subset of platforms (let's say iOS and tvOS), will it only produce NuGet packages for that platform? Or will the not-signed-package artifact still contain packages for everything?

We'll only build packages for platforms that are enabled. So if iOS is not enabled, then not-signed-package won't contain any iOS packages.

Is there a way to manually queue an official build to try to test some of these changes?

I ran the post-build pipeline manually, maybe that works: https://dev.azure.com/devdiv/DevDiv/_build/results?buildId=9647369&view=results

Is there a way to manually queue an official build to try to test some of these changes?

I ran the post-build pipeline manually, maybe that works: dev.azure.com/devdiv/DevDiv/_build/results?buildId=9647369&view=results

No, it didn't work. Then the next option is to (re)name your branch release-test/* and push to origin (not a fork). There's no need to create a PR, CI should pick the branch up automatically (although sometimes I've had to manually trigger a build - in https://devdiv.visualstudio.com/DevDiv/_build?definitionId=14411 - for new branches).

I'm planning to remove the post-build pipeline as part of these changes if all goes well to simplify things, let me try to push these changes to a new release-test branch and see what falls out 😄

Trying another test run here: https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=9651926&view=results

I'm going to close this in favor of the other test branch I have been working out of which is now working: #20665

vs-mobiletools-engineering-service2 · 2024-05-28T21:45:01Z

📚 [PR Build] Artifacts 📚