Create documentation page for Python SDK unrecoverable errors #28702
Conversation
|
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control |
jrmccluskey
changed the title
website/www/site/content/en/documentation/sdks/python-unrecoverable-errors.md
Outdated
Show resolved
Hide resolved
website/www/site/content/en/documentation/sdks/python-unrecoverable-errors.md
Show resolved
Hide resolved
website/www/site/content/en/documentation/sdks/python-unrecoverable-errors.md
Outdated
Show resolved
Hide resolved
website/www/site/content/en/documentation/sdks/python-unrecoverable-errors.md
Outdated
Show resolved
Hide resolved
|
|
||
| When additional dependencies like torch, transformers, etc are not | ||
| specified via requirements_file or preinstalled with a custom container | ||
| then the worker may go into a restart loop trying to install dependencies |
There was a problem hiding this comment.
Mismatch happens after installation, when worker already started; at this point we won't attempt more installations.
There was a problem hiding this comment.
Possible suggestion:
When additional dependencies like torch, transformers, etc. are not
specified via requirements_file or preinstalled in a custom container
then the worker might fail to deserialize (unpickle) the user code. This can result in ModuleNotFound errors.
If dependencies are installed but their versions don't match the versions in submission environment, pipeline might have AttributeError messages.
|
@tvalentyn ready for a second pass |
website/www/site/content/en/documentation/sdks/python-unrecoverable-errors.md
Outdated
Show resolved
Hide resolved
|
|
||
| ## Common Unrecoverable Errors | ||
|
|
||
| ### Job Submission/Runtime Python Version Mismatch |
There was a problem hiding this comment.
If you plan to reference these errors in logs, let's add markdown anchors for better linkability, as titles might change but we can keep the same anchors, so links will be preserved.
|
|
||
| When additional dependencies like torch, transformers, etc are not | ||
| specified via requirements_file or preinstalled with a custom container | ||
| then the worker may go into a restart loop trying to install dependencies |
There was a problem hiding this comment.
Possible suggestion:
When additional dependencies like torch, transformers, etc. are not
specified via requirements_file or preinstalled in a custom container
then the worker might fail to deserialize (unpickle) the user code. This can result in ModuleNotFound errors.
If dependencies are installed but their versions don't match the versions in submission environment, pipeline might have AttributeError messages.
…rable-errors.md Co-authored-by: tvalentyn <[email protected]>
Add a site page to cover what we consider unrecoverable errors as part of the effort to reduce detection times.
Staged at https://apache-beam-website-pull-requests.storage.googleapis.com/28702/documentation/sdks/python-unrecoverable-errors/index.html
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.