Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Transient 429 error fail upload cache cause workflow failure #543

Closed
1 of 5 tasks
Abacn opened this issue Sep 26, 2023 · 5 comments
Closed
1 of 5 tasks

Transient 429 error fail upload cache cause workflow failure #543

Abacn opened this issue Sep 26, 2023 · 5 comments
Assignees
Labels
bug Something isn't working

Comments

@Abacn
Copy link

Abacn commented Sep 26, 2023

Description:

We are using this workflow to setup Java environment. Occasionally, the workflow fails at Post step:

Post job cleanup.
/usr/bin/tar --posix --use-compress-program zstdmt -cf cache.tzst --exclude cache.tzst -P -C /runner/_work/beam/beam --files-from manifest.txt
Warning: Failed to save: Cache service responded with 429 during upload chunk.
/runner/_work/_actions/actions/setup-java/v3.8.0/dist/cleanup/index.js:401
                        throw new Error(`Cache upload failed because file read failed with ${error.message}`);
                        ^

Error: Cache upload failed because file read failed with EBADF: bad file descriptor, read
    at ReadStream.<anonymous> (/runner/_work/_actions/actions/setup-java/v3.8.0/dist/cleanup/index.js:401:31)
    at ReadStream.emit (node:events:513:28)
    at emitErrorNT (node:internal/streams/destroy:157:8)
    at errorOrDestroy (node:internal/streams/destroy:220:7)
    at node:internal/fs/streams:262:9
    at FSReqCallback.wrapper [as oncomplete] (node:fs:671:5)

Example run: https://github.com/apache/beam/actions/runs/6307333506/job/17123805683?pr=28079

A solution could be ignore this transient error similar to here: coursier/cache-action#131

Task version:
3.8.0 - search that not found similar issue reported to this repo before and not found a fix in recent versions

Platform:

  • Ubuntu
  • macOS
  • Windows

Runner type:

  • Hosted
  • Self-hosted

Repro steps:

https://github.com/apache/beam/actions/runs/6307333506/job/17123805683?pr=28079

@IvanZosimov
Copy link
Contributor

Hi, @Abacn 👋 Thanks for the issue, we will take a look! How often do you face it?

@Abacn
Copy link
Author

Abacn commented Sep 26, 2023

Hi @IvanZosimov thanks for prompt reply. It was low probability but appears more often (probably several percent flakiness) as we are migrating whole CI to GitHub Action.

@dsame dsame self-assigned this Nov 27, 2023
@dsame
Copy link
Contributor

dsame commented Nov 27, 2023

The problem relates to Windows and is not a subject of the action support, but as in actions/setup-go#431 It can be resolved by disabling caching:

- uses: actions/setup-java@v3
  with:
    distribution: 'temurin'
    java-version: '17'
    cache: false

Self-hosted builds retain their state, caching is unnecessary; furthermore, it slows down the build process.

@Abacn did this help?

@damccorm
Copy link
Contributor

Hey @dsame I don't think this is quite right - caching can be helpful if you're using ephemeral runners which is recommended for autoscaling since they only run one job per runner (https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/autoscaling-with-self-hosted-runners#using-ephemeral-runners-for-autoscaling).

Also, the job in question is running on Ubuntu, not Windows

@dsame
Copy link
Contributor

dsame commented Nov 30, 2023

Hello @damccorm, Thank you a lot for your input. Now the problem is clear but it does not relate to the action. I've created the Issue in the toolkit repo actions/toolkit#1589 and closing this issue as a duplicate.

Please use the reference to track the status of the problem, also you can reopen this issue or create the new one if needed/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants