repro: Fails to update dvc.lock when AWS credentials expired during execution

[aschuh-hf]

Bug Report

Description

Commands such as dvc status and dvc repro now require valid AWS credentials when using dvc-s3. This wasn't the case with DVC 2 I belief. The issue with dvc repro requiring those credentials is that this may result in the following error when the credentials expired for a long running repro task.

Updating lock file 'dvc.lock'                                                                                                                                                                                                                                                                                                                                                      

ERROR: failed to reproduce 'stage': [Errno 22] Bad Request: An error occurred (400) when calling the HeadObject operation: Bad Request

Note that the 'stage' was not being executing because of this error, i.e., the error occurs at the start of a stage when DVC tries to determine whether the stage needs to re-run.

Reproduce

Run dvc status in a DVC project which has a S3 bucket configured as remote without having valid AWS credentials. Same for dvc repro.

Expected

AWS credentials should only be required for operations which must read the remote bucket (dvc push, dvc pull, dvc gc etc).

Environment information

Output of dvc doctor:

$ dvc doctor
DVC version: 3.19.0 (conda)
---------------------------
Platform: Python 3.10.6 on Linux-3.10.0-1127.8.2.el7.x86_64-x86_64-with-glibc2.17
Subprojects:
        dvc_data = 2.16.1
        dvc_objects = 1.0.1
        dvc_render = 0.5.3
        dvc_task = 0.3.0
        scmrepo = 1.3.1
Supports:
        http (aiohttp = 3.8.5, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.8.5, aiohttp-retry = 2.8.3),
        s3 (s3fs = 2023.6.0, boto3 = 1.26.76)
Config:
        Global: /home/aschuh/.config/dvc
        System: /etc/xdg/dvc
Cache types: hardlink, symlink
Cache directory: xfs on /dev/sda1
Caches: local
Remotes: s3, s3
Workspace directory: xfs on /dev/sda1
Repo: dvc (subdir), git
Repo.site_cache_dir: /var/tmp/dvc/repo/b7a036677a9e20e73a791fb023238555

Additional Information (if any):

[pmrowla]

This is probably the same issue as #9912 which is fixed in DVC 3.20.0 and later, please try updating to the latest release

[aschuh-hf]

I've just upgraded to DVC 3.22.1 using conda and still get this error.

$ dvc status
ERROR: unexpected error - [Errno 22] Bad Request: An error occurred (400) when calling the HeadObject operation: Bad Request

My .dvc/config

[core]
    remote = s3store
    analytics = false
[cache]
    dir = cache
    s3 = s3cache
    shared = group
    type = symlink
[hydra]
    enabled = False
['remote "s3cache"']
    url = s3://.../data/cache
['remote "s3store"']
    url = s3://.../data/store

[dberenbaum]

@aschuh-hf I'm not able to quickly reproduce it, so if you are able to put together some minimal example, that would help. Is s3 = s3cache under the cache section left from an old dvc version? I don't think it's supported anymore. Can you drop it, and does that make a difference?

[aschuh-hf]

I just deleted .dvc/tmp/exps/cache to fix #9860 as part of the upgrade to DVC 3.22.1 and also wanted to look into this issue again. However, dvc status now does give me the expected answer without any Bad Request error message even though I currently have expired AWS credentials. Seems this issue was indeed fixed in 3.20 as suggested by @pmrowla, but I just needed to clear the cache?!