Request gzipped data to speed up transfers from Picasa Web #13
Conversation
request gzipped data, instead of uncompressed data. To achieve this goal, I propose the use of the gem HTTParty to handle all of the HTTP connections. Not only because HTTParty already knows how to handle compressed data properly, but also because it's a mature gem for handling HTTP connections in general. And it's use can simplify the connection code. I created a new class Picasa::HTTP which sets all of the defaults for HTTParty related to: * base uri * proxy * headers * format So this doesn't need to be done at Picasa::Connection anymore. And since all of the requests are similar, the logic was centralized in the new method Picasa::Connection#exec_request for http methods get, post and delete. OBS: * To request gzipped data from Picasa API it's necessary to have " (gzip)" in the user agent url and to set the appropriate Accept-Encoding header. See: https://developers.google.com/picasa-web/faq_gdata#gzip * I moved the constants API_URL, API_AUTH_URL and API_VERSION to the new class Picasa::HTTP, because it looks like a more appropriate place.
|
Thanks. I didn't know about that option in Picasa API. |
|
Hi Wojciech, I agree with you that dealing with gzip is simple and I understand your preference for not adding another dependency. Still, I believe the use of HTTParty can really help to simplify the code of the your gem. Actually there are a few more simplifications that I'm working on right now and will be pushing soon for your consideration. It has to do with the parsing process, that can also be completely delegated to HTTParty. One last thing, related to the parsing. I've run a few benchmarks and I realized that parsing a JSON response from Picasa Web is quite faster than a XML response. When writing to Picasa Web, we can only use XML at the present moment. But JSON can be used for reading. So, I'd like to try a solution where we use JSON for all of the readings. This, together with the use of gzip, has generated speed differences in my tests of up to an order of magnitude. Anyway, I'll be sending a few more changes during the day for your consideration. |
HTTParty's response object has a method #parsed_response that returns a hash with the response properly parsed. Internally, it uses MultiJSON or MultiXML, depending on the format you've chosen to use in the request. I believe that delegating this to HTTParty is interesting because the HTTP request and it's corresponding parsing are all handled by this HTTParty, which already does it well, and the Picasa gem can be dedicated to dealing only with Picasa issues.
|
Reading via JSON sounds interesting - I wanted to switch to JSON, but fact that POSTing is not supported discouraged me. Looking forward to your changes - I will merge when finished. |
|
Nice. Good to know that you're also interested in using JSON. I'm working on it and hopefully will be pushing it soon. |
|
Switching to JSON can be a lot of work - we can do it in second step if you like. |
|
Yeah, I agree with you. Actually I do have a code of my own that already deals with JSON for Picasa Web. But changing the gem to use JSON really seems to take some work, because the parsed response has some important differences. In particular, where the XML parser generates something like I believe it's better to tackle this in a different branch and to just leave the current one as it is. If you decide to merge it, I'd only ask you to please review carefully the changes. They didn't break the tests, but I'm not sure how comprehensive the tests are at this point. You certainly have far more experience with the code to evaluate it. |
|
I noticed some problems with parsing xml. |
|
Sorry about that. :-/ I'll take a look as well and let you know if I find something. |
Wojciech, thanks a lot for this gem!
In order to speed up the transfer of data from Picasa Web, we can request gzipped data, instead of uncompressed data.
To achieve this goal, I propose the use of the gem HTTParty to handle
all of the HTTP connections. Not only because HTTParty already knows how
to handle compressed data properly, but also because it's a mature gem
for handling HTTP connections in general. And it's use can simplify the
connection code.
I created a new class Picasa::HTTP which sets all of the defaults for
HTTParty related to:
So this doesn't need to be done at Picasa::Connection anymore. And since
all of the requests are similar, the logic was centralized in the new
method Picasa::Connection#exec_request for http methods get, post and
delete.
OBS:
https://developers.google.com/picasa-web/faq_gdata#gzip
class Picasa::HTTP, because it looks like a more appropriate place.