Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cloud upload performance issues #19

Closed
joeyparrish opened this issue Oct 1, 2019 · 1 comment
Closed

Cloud upload performance issues #19

joeyparrish opened this issue Oct 1, 2019 · 1 comment
Assignees
@joeyparrish
Labels
status: archived Archived and locked; will not be updated type: bug Something isn't working correctly
Milestone
v1.0

Comments

@joeyparrish
Copy link
Member

The cloud upload node is much slower than gsutil, since gsutil -m rsync can parallelize much of the work.

Using threading in Python is also hairy, since the global interpreter lock may cause issues. (It's not yet clear if GIL is part of the problem.)

Should we find a way to improve performance using the python libraries? Or rely on gsutil, which is already optimized?
@joeyparrish joeyparrish added the type: bug Something isn't working correctly label Oct 1, 2019
@shaka-bot shaka-bot added this to the v1.0 milestone Oct 1, 2019
@joeyparrish
Copy link
Member Author

We've decided to move to using gsutil. This has many advantages, not only for performance, but also in simpler authentication to cloud storage.

@joeyparrish joeyparrish self-assigned this Oct 3, 2019
shaka-bot pushed a commit that referenced this issue Oct 7, 2019
@joeyparrish
Factor out threading functionality to a base class
3c87c88 
Currently, only the CloudNode is using a thread.  But it is easier to
read and understand the functionality of CloudNode if the threading
code is moved out to a base class.

This refactor is groundwork for improving CloudNode and switching to
gsutil for rsync/upload.

Issue #19

Change-Id: Iaa712d32ca7854cda0799011b5ca72cf6ec59b02
@shaka-bot shaka-bot added the status: archived Archived and locked; will not be updated label Dec 6, 2019
@shaka-project shaka-project locked and limited conversation to collaborators Dec 6, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@joeyparrish joeyparrish

Labels
status: archived Archived and locked; will not be updated type: bug Something isn't working correctly
Projects
None yet
Milestone
v1.0
Development

No branches or pull requests
2 participants
@joeyparrish @shaka-bot