Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

3Box-Integration: Fix Profile Data Query Timeout for Kevin's Account #6239

Merged
merged 8 commits into from
Apr 8, 2020

Conversation

think-in-universe
Copy link
Contributor

Description

Fix the query timeout failure in #5981 reported by @owocki , when backing up profile data to 3Box.

im still getting 503s trying to upload my data in production https://bits.owocki.com/5zuyYLwW

Resume the work in #5955

Refers/Fixes

Fix the query timeout failure in #5981 reported by @owocki ; resume the work in #5955

Testing

Tested locally with GitCoin development environment.

@think-in-universe
Copy link
Contributor Author

@octavioamu @owocki hey guys, please have a look whether this fix will resolve the issues met by Kevin previously. Any more tests and feedback are welcome.

@codecov
Copy link

codecov bot commented Mar 20, 2020

Codecov Report

Merging #6239 into stable will decrease coverage by <.01%.
The diff coverage is 0%.

Impacted file tree graph

@@            Coverage Diff             @@
##           stable    #6239      +/-   ##
==========================================
- Coverage   28.26%   28.26%   -0.01%     
==========================================
  Files         278      278              
  Lines       25235    25410     +175     
  Branches     3694     3724      +30     
==========================================
+ Hits         7132     7181      +49     
- Misses      17818    17948     +130     
+ Partials      285      281       -4
Impacted Files Coverage Δ
app/dashboard/views.py 11.3% <0%> (-0.06%) ⬇️
app/dashboard/export.py 60.36% <0%> (ø) ⬆️
app/townsquare/models.py 56.45% <0%> (-1.33%) ⬇️
app/dashboard/tasks.py 37.5% <0%> (-1.14%) ⬇️
app/chat/views.py 29.78% <0%> (-0.65%) ⬇️
app/dashboard/models.py 50.44% <0%> (-0.51%) ⬇️
app/dataviz/views.py 9.03% <0%> (-0.12%) ⬇️
app/retail/views.py 21.19% <0%> (-0.11%) ⬇️
app/kudos/utils.py 19.78% <0%> (-0.08%) ⬇️
app/dashboard/tip_views.py 15.41% <0%> (-0.07%) ⬇️
... and 16 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update eb9ea5d...dc6a579. Read the comment docs.

@think-in-universe
Copy link
Contributor Author

Could you please help check why the unit test fails? @octavioamu @thelostone-mc

https://travis-ci.org/github/gitcoinco/web/jobs/664757656

image

@owocki
Copy link
Contributor

owocki commented Mar 20, 2020 via email

@think-in-universe
Copy link
Contributor Author

@owocki cool. thanks

@owocki
Copy link
Contributor

owocki commented Mar 23, 2020 via email

@think-in-universe
Copy link
Contributor Author

cool. thanks Kevin @owocki

  1. Could you please test the latest commit in this PR first? The data for portfolio and tips are fixed, as I mentioned in Continue to fix data serialization error in 3Box integration #5981
  2. And could you please share the timeout requests as cURL? I'd like to confirm which models have failed due to timeout.

A bit occupied so didn't catch up the two timeout issues yet, will take a look today.


Also from #5981

BTW, in another project of Cerebro (gitcoinco/data-ops#42), @frankchen07 and I are reaching out to @octavioamu about whether it's possible to get access to the production data securely for someone outside the team (like me). This may also help assist the testing of this issue.

Any suggestions for this?

@owocki
Copy link
Contributor

owocki commented Mar 24, 2020

woooooooooooooooooooooooooooo no 500 errors! https://bits.owocki.com/7KuRZjP6

though the 3dbox calls did get 404s/401s
https://bits.owocki.com/NQuZA4o6
https://bits.owocki.com/o0uDL9P7

i did sign all the messages that wree prompted to me

heres the console https://bits.owocki.com/6quBoNE7

@think-in-universe
Copy link
Contributor Author

think-in-universe commented Mar 25, 2020

looks like a good news!! @owocki

However, after checking your log, it seems the source code (or at least the static JS files) in your local development environment is not updated to the latest version in the current PR #6239 . Or at least the 3box-sync.js file is not in its latest version, which renamed one option from models to model now. That's why we saw a undefined data in the message.

image

@michaelsena 's team may help take a look at the 3Box connection issue I think. To me, the 3box network is not very good often, and I cannot open 3box hub sometimes.

@msterle
Copy link

msterle commented Mar 25, 2020

The 404s are expected behaviour @owocki - the 3box client polls the address server until the address can be correctly resolved.

@danlipert
Copy link
Contributor

@owocki looks like once the static files are updated this is good to merge? @think-in-universe @msterle we don't use any kind of js package management, just the static files, so sounds like we just need to update that library and it should be working error free?

@think-in-universe
Copy link
Contributor Author

@danlipert I'm not sure actually. I would prefer @owocki test again locally before we merge.

And I'm not sure whether it works for others except Kevin? How many people have tested for their profile?

It works for my profile and works for some mock data I build, but I can hardly know how it looks like for others, and there're some corner cases or performance issue that is hard for me to test locally without accessing to production data (or sufficient testing data).

@think-in-universe
Copy link
Contributor Author

think-in-universe commented Mar 25, 2020

I'm actually upset and disappointed about how we're doing testing for this issue. It's really inefficient.

I have pointed out this challenge of testing data from very early (around Feb 6th, when the team asked me to help fix the failure in #5947), but no one can really help take actions to bridge the gap, and shorten the feedback loop. So every time, we have to wait for the PR being merged, to validate the correctness, or waiting for Kevin's (who is busy often) response. I would suggest we learn from this issue, and figure out how to make it easy for generating testing data or how to access production safely for a contributor, or maybe the internal team who can access to the production data could do more testing and fix the failures.

@owocki
Copy link
Contributor

owocki commented Mar 25, 2020

i think your frustrations are valid - if sanitizing the test data was an easy lift (without a lot of risk), we'd have done it already... maybe the solution is better fixtures.

@danlipert i think this is good to merge from what i saw last testing session

@think-in-universe
Copy link
Contributor Author

@owocki thanks Kevin. I agree better fixtures should definitely help.

and what's your opinion about how to prepare data for Cerebro (gitcoinco/data-ops#42)? a bit similar to the case here.

@owocki
Copy link
Contributor

owocki commented Mar 25, 2020 via email

@think-in-universe
Copy link
Contributor Author

@owocki yeah. I did manually create accounts from GitHub for my local testing. (I didn't know there're other ways...)

But I think it didn't work well for both the 3Box integration and Cerebro, since the manual creation of test accounts with GitHub cannot fully reflect the real scenarios, and it's low efficient and difficult for creating sufficient testing data manually as I tried. Basically we cannot ensure the testing coverage, or don't have enough training/testing data for ML.

And all test data are cleaned up after refresh the dev environment (the DB schemas update often), maybe I need to back it up before refresh, but it increase even more cost for one-time tasks.

Anyway, better fixtures should definitely be helpful.

@owocki
Copy link
Contributor

owocki commented Mar 26, 2020

thanks @think-in-universe - will think on this a bit

@gitcoinco/engineers this is 👍 to merge from my perspective (though we probably need the tests passing huh?)

@think-in-universe
Copy link
Contributor Author

hey guys, could you please move faster for this issue? The feedback loop is indeed to long.

@danlipert
Copy link
Contributor

@think-in-universe I'm sure you can imagine that we have a lot of different priorities right now, asking us to "move faster" is not helpful. Also, why is this against stable instead of master?

@think-in-universe
Copy link
Contributor Author

think-in-universe commented Apr 2, 2020

@danlipert ok. considering the issue has been mainly pending on your team's actions for around 3 to 4 months (3Box team and I have been waiting for this change for 3~4 months), I want to say I hope it can move faster. and you should know the test feedback loop is indeed long as you can see from the conversation with Kevin @owocki above (and it's already 7 days after that)

the reason it's against stable is because the PR I sent previously to master is changed by your team to stable. So I would rather set it to stable by myself. If you think it's more appropriate to set it to master, please do it. I have no preference.

@owocki
Copy link
Contributor

owocki commented Apr 3, 2020

valid feedback @think-in-universe - thanks for passing it along

i think that the lack of good fixtures is certainly slowing things down here, so thats to some extent an extenuating circumstance. i wish we had the cleansed test data available.

but based off what i've seen it does seem that what we've done on this PR is working, so i'd love to get it merged and into an upcoming release. @gitcoinco/engineers i've added this to the 'preview pending' part of the issue board

@thelostone-mc thelostone-mc merged commit 9a6475c into gitcoinco:stable Apr 8, 2020
@think-in-universe
Copy link
Contributor Author

thanks for merging the feature.

Is it already available in production or only available in alpha? (with https://gitcoin.co/settings/account?cb=alphatester)

@gitcoinbot
Copy link
Member

⚡️ A tip worth 0.65000 ETH (110.44 USD @ $169.91/ETH) has been granted to @think-in-universe for this issue from @owocki. ⚡️

Nice work @think-in-universe! Your tip has automatically been deposited in the ETH address we have on file.

@gitcoinbot
Copy link
Member

Turing Talker! ⚡️ A *Turing Talker!* Kudos has been sent to @think-in-universe for this issue from @owocki. ⚡️

The sender had the following public comments:

thanks for the work on 3box!

Nice work @think-in-universe!
Your Kudos has automatically been sent in the ETH address we have on file.

@think-in-universe
Copy link
Contributor Author

thanks for merging the PR.

talked with @RachBLondon from 3Box team, and it would be nice to see when it could be available in production.

@owocki
Copy link
Contributor

owocki commented Apr 8, 2020

its live now!

@owocki
Copy link
Contributor

owocki commented Apr 8, 2020

link to use it https://gitcoin.co/settings/account

seems to be workin got me .. at least insofar as i dont see any obvious errors. though the ('loading' indicator)[https://bits.owocki.com/QwuKv54G] in the UI doesnt ever seem to go awy.

https://bits.owocki.com/yAu2GyyW
https://bits.owocki.com/QwuKv55N

@think-in-universe
Copy link
Contributor Author

@owocki it seems being blocked at the openSpace step (it also happened to me sometimes, but not often. I thought that was due to my network only ... )

@RachBLondon Hi Rachel, do you have any advice on this? sometimes it takes a long time to openSpace or just blocked there. I reported that before, but I thought that was due to my network condition.

@owocki
Copy link
Contributor

owocki commented Apr 9, 2020 via email

@think-in-universe
Copy link
Contributor Author

@owocki usually I need to refresh the web page and click the 3Box button again, and it will usually work.

@RachBLondon may know better about this issue, which is related to 3Box APIs and network.

@RachBLondon
Copy link

@think-in-universe Thanks for reporting this, will investigate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants