[Failing Test]: Python Precommits Failing with "Cloud IAM check failed"

[jrmccluskey]

What happened?

Precommits are failing on PRs and tests executing on master with the same IAM permissions error.

Example error:

==================================== ERRORS ====================================
_____________ ERROR at teardown of RecommendationAIIT.test_predict _____________
[gw5] linux -- Python 3.9.18 /runner/_work/beam/beam/sdks/python/test-suites/tox/py39/build/srcs/sdks/python/target/.tox-py39-cloud/py39-cloud/bin/python

args = (parent: "projects/apache-beam-testing/locations/global/catalogs/default_catalog"
,)
kwargs = {'metadata': [('x-goog-request-params', 'parent=projects/apache-beam-testing/locations/global/catalogs/default_catalog'), ('x-goog-api-client', 'gl-python/3.9.18 grpc/1.59.0 gax/2.12.0 gapic/0.10.5')], 'timeout': 600.0}

    @functools.wraps(callable_)
    def error_remapped_callable(*args, **kwargs):
        try:
>           return callable_(*args, **kwargs)

target/.tox-py39-cloud/py39-cloud/lib/python3.9/site-packages/google/api_core/grpc_helpers.py:75: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
target/.tox-py39-cloud/py39-cloud/lib/python3.9/site-packages/grpc/_channel.py:1161: in __call__
    return _end_unary_response_blocking(state, call, False, None)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

state = <grpc._channel._RPCState object at 0x7ddedc078ac0>
call = <grpc._cython.cygrpc.SegregatedCall object at 0x7dded7fb5ac0>
with_call = False, deadline = None

    def _end_unary_response_blocking(
        state: _RPCState,
        call: cygrpc.SegregatedCall,
        with_call: bool,
        deadline: Optional[float],
    ) -> Union[ResponseType, Tuple[ResponseType, grpc.Call]]:
        if state.code is grpc.StatusCode.OK:
            if with_call:
                rendezvous = _MultiThreadedRendezvous(state, call, None, deadline)
                return state.response, rendezvous
            else:
                return state.response
        else:
>           raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
E           grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
E           	status = StatusCode.PERMISSION_DENIED
E           	details = "Cloud IAM check failed for the following permission: 'automlrecommendations.catalogItems.list' on the following resource: 'projects/apache-beam-testing/locations/global/catalogs/default_catalog'. Check if you have the appropriate roles assigned to you in https://console.cloud.google.com/iam-admin/iam."
E           	debug_error_string = "UNKNOWN:Error received from peer ipv4:172.253.114.95:443 {created_time:"2023-10-18T14:41:38.167376066+00:00", grpc_status:7, grpc_message:"Cloud IAM check failed for the following permission: \'automlrecommendations.catalogItems.list\' on the following resource: \'projects/apache-beam-testing/locations/global/catalogs/default_catalog\'. Check if you have the appropriate roles assigned to you in https://console.cloud.google.com/iam-admin/iam."}"
E           >

target/.tox-py39-cloud/py39-cloud/lib/python3.9/site-packages/grpc/_channel.py:1004: _InactiveRpcError

Issue Failure

Failure: Test is continually failing

Issue Priority

Priority: 1 (unhealthy code / failing or flaky postcommit so we cannot be sure the product is healthy)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner