feat(cloud): Upload through HTTP proxy node

[mariocynicys]

Changes:

• Disable multiperiod when using cloud upload (for now)
• Use default credentials from GCS or S3 Python libraries to upload
• Remove gsutil dependency and use threaded HTTP server, without temp files

Closes #47

[joeyparrish]

I'm going to play with this locally for a bit. I might be able to make some concrete suggestions or contribute to your branch.

[joeyparrish]

My -H "Authorization=Bearer $(gcloud auth print-access-token)" hack failed after about one hour of running. I started getting HTTP 429 responses from Google Cloud, which makes me think it wasn't an issue with the token itself, but something more fundamental we will need to resolve.

Google's docs say:

If you run into any issues such as increased latency or error rates, pause your ramp-up or reduce the request rate temporarily in order to give Cloud Storage more time to scale your bucket. You should use exponential backoff to retry your requests when:

Receiving errors with 408 and 429 response codes.
Receiving errors with 5xx response codes.

[joeyparrish]

I'm going to revive this and push changes into your branch so you are properly credited for the PR. Changes I'm planning for this PR:

1. Go back to using the cloud URL in -c for compatibility for now
2. Revert extra headers feature for now (use case unclear for now)
3. Remove upload to HTTP without gs:// or s3:// (use case unclear for now)
4. Ignore -o if -c given (no need for local files if uploading to cloud)
5. Revert changes to periodconcat_node and mark it as incompatible with cloud upload (already can't be used for live, helps avoid local temp files)
6. Remove use_local_proxy from pipeline config, implied by -c
7. Pipeline directly from incoming request to outgoing request in proxy_node
8. Try to handle authentication via default credentials in gcs and s3 python modules

I'll follow up later with new changes to:

1. Allow cloud URLs to be placed into -o
2. Make -c a deprecated alias to -o (overrides -o if both given)
3. Add command line arg to pass initial access tokens for oauth via file, maybe also environment variable

[joeyparrish]

I've started by rolling up your original commits, rebasing them, and force-pushing.

[joeyparrish]

@mariocynicys, still a WIP since I haven't finished the S3 upload yet, but it's working well enough for GCS that I'm going to deploy it to my lab to revive our live stream. Without this, my live stream was running into errors where gsutil would leave files behind in /tmp and fill up the disk.

Please take a look and give me your thoughts.

[joeyparrish]

Okay, S3 is working now, too.

[joeyparrish]

This is working in our lab deployment now!

[jspizziri]

@joeyparrish now that 1.0.0 is out we started using it and discovered that HTTP was removed

Remove upload to HTTP without gs:// or s3:// (use case unclear for now)

This was actually how we were using the streamer, as we use storage that is neither S3 or GCS. If you're doing that, the only option was HTTP. If there was some kind of storage adapter that would be nice and we'd write our own adapter, however in the absence of that HTTP was our only option.

Any possibility to get it added back?

[joeyparrish]

Yes, definitely. What are your requirements?

Do you need specific headers or auth tokens, or just for Packager to write output directly to an HTTP(S) URL?

[joeyparrish]

@jspizziri, I filed #210 to discuss your requirements for HTTP upload. Please let us know what you need!