DeploymentSettings doesn't handle the agent pool id if it's a secret and fails/ignore silently

[aureq]

What happened?

I have stack a that creates an AgentPool() ✔. That stack exports the agent_pool as an object that contains id, name and token (secret) ✔.
Another stack output agent-pool-id-secret exports the pool id as a secret ✔.
The stack output is like this.

Current stack outputs (2):
    OUTPUT                VALUE
    agent-pool            {"id":"12e730d9-f499-429f-b72e-cd2455f61f1b","name":"agent-pool-resource","token":"[secret]"}
    agent-pool-id-secret  [secret]

I have a stack b that retrieves that retrieves the 2 stack outputs as stack references from stack a. ✔

Stack b creates a DeploymentSettings() and uses the agent-pool.id (which should be a string) as part of the resource creation.
The DeploymentSettings() is created BUT the agent_pool_id is never set ❌.

After additional investigation, I wondered if passing the secret instead of an output could be the issue (step 5 onward) and it appears this seems to be the case. If the agent_pool_id receives a secret, then the value somehow isn't used ❌

Steps to reproduce

1. Deploy stack a ✔
2. Deploy stack b (don't forget to adapt the stack reference location) ✔
3. Check the Deployment setting for stack b
   • The Source control settings are updated correctly ✔
   • The Deployment runner pool remains the same (Pulumi hosted pool) ❌
   • No indication of a failure that the Deployment hosted pool wasn't changed ⁉
4. Destroy stack b
5. Update the code to use agent_pool_id_secret instead of agent_pool['id'] for the agent_pool_id property
6. Deploy stack b ✔
7. Check the Deployment setting for stack b
   • The Source control settings are updated correctly ✔
   • The Deployment runner pool remains the same (Pulumi hosted pool) ❌
   • No indication of a failure that the Deployment hosted pool wasn't changed ⁉
8. Running pulumi up --refresh correctly shows diff: +agentPoolId ✔
9. Check the Deployment setting for stack b
   • The Deployment runner pool remains the same (Pulumi hosted pool) ❌
   • No indication of a failure that the Deployment hosted pool wasn't changed ⁉

Workaround

For now, it's possible to workaround this temporarily by remove the secrets and using plain text values. However, this is problematic because using an object as a stack reference accounts for 1 resource only where using multiple stack references will account for more resources and ultimately increase the cost passed on the customer.

Example

Stack a

"""A Python Pulumi program"""

import pulumi
import pulumi_pulumiservice as pulumiservice

pulumi_agent_pool = pulumiservice.AgentPool("agentPoolResource",
    name="agent-pool-resource",
    organization_name="menfin",
    description="this is to test agent pool id"
)

pulumi.export("agent-pool", {
    'id': pulumi_agent_pool.agent_pool_id,
    'name': pulumi_agent_pool.name,
    'token': pulumi_agent_pool.token_value
})

pulumi.export("agent-pool-id-secret", pulumi.Output.secret(pulumi_agent_pool.agent_pool_id))

Stack b

"""A Python Pulumi program"""

import pulumi
import pulumi_pulumiservice as pulumiservice

stack_ref = pulumi.StackReference("menfin/zendesk/0001")
agent_pool = stack_ref.require_output('agent-pool')
agent_pool_id_secret = stack_ref.require_output("agent-pool-id-secret")

deployment_settings = pulumiservice.DeploymentSettings(
    resource_name = "deployment-settings",
    organization = pulumi.get_organization(),
    project = pulumi.get_project(),
    stack = pulumi.get_stack(),
    agent_pool_id=agent_pool['id'],
    # agent_pool_id=agent_pool_id_secret,
    source_context = pulumiservice.DeploymentSettingsSourceContextArgs(
        git = pulumiservice.DeploymentSettingsGitSourceArgs(
            branch = "main",
            repo_url = "aureq/aws-py-eks-helm",
            git_auth = pulumiservice.DeploymentSettingsGitSourceGitAuthArgs(
                basic_auth = pulumiservice.DeploymentSettingsGitAuthBasicAuthArgs(
                    username = "aureq",
                    password = "<redacted>"
                )
            )
        )
    ),
    operation_context = pulumiservice.DeploymentSettingsOperationContextArgs(
        environment_variables = {
            "SECRET_ENV_VAR": pulumi.Output.secret('This should NOT be visible')
        }
    )
)

Output of pulumi about

CLI          
Version      3.134.1
Go Version   go1.23.1
Go Compiler  gc

Plugins
KIND      NAME           VERSION
resource  pulumiservice  0.26.1
language  python         unknown

Host     
OS       debian
Version  12.7
Arch     x86_64

This project is written in python: executable='/home/aureq/work/customers/zendesk/0001-xxxxx/b/venv/bin/python' version='3.11.9'

Current Stack: menfin/zendesk/0002

TYPE                                    URN
pulumi:pulumi:Stack                     urn:pulumi:0002::zendesk::pulumi:pulumi:Stack::zendesk-0002
pulumi:providers:pulumi                 urn:pulumi:0002::zendesk::pulumi:providers:pulumi::default
pulumi:pulumi:StackReference            urn:pulumi:0002::zendesk::pulumi:pulumi:StackReference::menfin/zendesk/0001
pulumi:providers:pulumiservice          urn:pulumi:0002::zendesk::pulumi:providers:pulumiservice::default_0_26_1
pulumiservice:index:DeploymentSettings  urn:pulumi:0002::zendesk::pulumiservice:index:DeploymentSettings::deployment-settings


Found no pending operations associated with 0002

Backend        
Name           pulumi.com
URL            https://app.pulumi.com/aureq
User           aureq
Organizations  aureq, team-ce, menfin, menfin-team, demo
Token type     personal

Dependencies:
NAME                  VERSION
pip                   24.0
pulumi_pulumiservice  0.26.1
setuptools            65.5.0

Pulumi locates its logs in /tmp by default

Additional context

In part related to pulumi/pulumi#17422

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[IaroslavTitov]

This is solved in #467, should be out in the next release
Thank you for the detailed description!