release (#340)

* Fabo/fix gas (#250) * fix gas estimate * linted * fixed test * do not keep data sources (#251) * track failing transactions in Sentry (#249) * correctly set the tx schema for a failing tx (#248) * Fabo/remove per block caching as not working (#247) * remove per block caching as not working * fix memoized results Co-authored-by: Ana G. <[email protected]> * delete perblockcachedatasource (#253) * Ana/fix balances in actionmodal (#255) * fix action modal available balance * include regen * use dictionary for denomlookup * use correct events for received txs (#257) * enable account creation for some networks (#252) * network update time metric added (#256) * network update time metric added * added missing dep Co-authored-by: Fabian <[email protected]> * Fix proposal deposit (#261) * Remove denom handling from getDeposit() * Revert undesired change * delete package-lock.json * localtestnet config change (#265) * Ana/handle "address not from this network" error (#263) * add check address function for all queries * apply suggestions * Ana/add fiatvalue to balances query (e-Money) (#262) * preparation * more preparation * add fiatvalue field to balances query * fix get account info * apply suggestions * apply one last suggestion * suggestions+ Co-authored-by: Fabian <[email protected]> * Ana/emoney fix expected returns with inflation and totalbacked (#243) * fix expected returns with inflation and supply * minor fixes. dictionary * query exchange rates from emoney api * fix infinite expected returns * convert api url to const * add eur value to totalbackedvalue. totalngm gains * add important comment * finish calculation * lint * catch errors with sentry Co-authored-by: Fabian <[email protected]> * readd coin conversion (#268) * delete amount field (#274) * Fabo/increase gas again (#271) * icrease gas again * fixed test * Fabo/load all txs (even if more then first page in response) (#270) * load all txs (even if more then first page in response) * improved handling of txs * missing renaming * fixed paginated load * add pagination fix also to cosmosV0-source Co-authored-by: iambeone <[email protected]> Co-authored-by: Ana G. <[email protected]> * fixing issue with multiple senders in one event (#273) * fixing issue with multiple senders in one event * Update lib/source/cosmosV2-source.js Co-authored-by: Fabian <[email protected]> * Fabo/allow signing for terra + emoney (#267) * allow signing for terra * readd coin conversion * enable actions for terra * fix correct terra testnet url * comments and guards * enabled more txs for emoney and fixed broadcasting * added a catch for wrongly formatted broadcast urls * recover default field. change some network titles (#277) * Fabo/add network data to API (#278) * non desctructive introduction of better address prefix wording * added address creator to API * adjusted test * added ledger app to networks config * add icon property to schema (#281) * add icon property to schema * fix network schema validation Co-authored-by: Ana G. <[email protected]> * filter out validator specific txs (#279) * Ana/balances coinreducer good fix (#269) * balances coinreducer good fix * refactored fiat value logic Co-authored-by: Fabian <[email protected]> * Create network_integration.md * Update network_integration.md * Update network_integration.md * Fabo/avoid 500 errors (#288) * avoid using the latest query * cleanup * Ana/filter validator tx cross network and add txvalue reducer (#285) * filter validators cross network * add value reducer. necessary for multi claim txs * add validator txs filter also for cosmosv0 source * filter and make array only claim rewards msg value * filter txs by whitelist * change length in multi claim reward reducer * add withdrawvalidators * replace dictionary for set * refactor transaction snippet. avoid repetition * Ana/emoney upgrade (mergeable) (#282) * update emoney api_url * fix denom. add default fiat currency * fix rpc endpoint * fix value (my bad) (#293) * fix value (my bad) * trigger another ci flow * erase space * set correct new chain id (#294) * restart API * restart API * fix pr alert (#297) * Fabo/298 tendermint reconnect (#300) * reconnect on tendermint disconnect * cleanup * comments * Update cosmos-node-subscription.js * Fabo/299 trigger a chain hangup error (#301) * trigger a chain hangup error * increase chain hangup time * Apply suggestions from code review * Fabo/store validator addresses (#296) * add validator addresses to db * linted * ignore in local dev * revert * fixed fetch * comment * refactored db into constructor * cleanup * add clearTimeout to avoid reconnection hell (#306) * add clearTimeout to avoid reconnection hell * removed console.log * Aleksei/luniedb replaced (#303) * add validator addresses to db * linted * ignore in local dev * revert * fixed fetch * comment * refactored db into constructor * cleanup * replaced luniedb * linted Co-authored-by: Fabian <[email protected]> * disable reconnection logic * clear polling interval for tendermint connection * simple api fixes (#310) * Fabo/remove tendermint (#311) * remove tendermint * fixed empty blockHeight issue * small refactoring * catch on fetches to get logging * delay block updates * add retry logic * refactored getBlockByHeight * remove pm2 dep * validator profiles were returned as array (#312) * remove pr github action (#316) * fixing caching issue and more (#315) * fixing caching issue and more * clear chain hangup timeout * timeout fixed * Update lib/block-listeners/cosmos-node-subscription.js Co-authored-by: Jordan Bibla <[email protected]> * added terra_mainnet to networks (#309) * added terra_mainnet to networks * errors fix reverted * Ana/add multidenom rewards. necessary for emoney (#308) * add multidenom rewards. necessary for emoney * add rewards to overview. improve code location * lint * add filter rewards for multidenom * important fix * another important fix * fix cannot read propery constructor of undefined (#318) * change terra mainnet title * pushing something to trigger new build and to restart the node * trying out nylira's node * WIP: Ana/More Terra fixes (#323) * fixing my mess * move all rewards logic to terra * clearly signal single denom rewards * Revert "WIP: Ana/More Terra fixes (#323)" (#325) This reverts commit 408186a. * catch inside block polling function (#327) * Fabo/disable writing validator addresses for now (#329) * disable writing validator addresses for now * linted * flushing http cache (#332) * Transaction abstraction schema (#333) * tx abstraction proposal using union * Use Coin type * Cleanup, TransactionV2 * Cleanup, not multisend tx * Unsupported tx (like multisend) don't have details * Fabo/switch to docker swarm mode (#330) * swtich to docker swarm mode * typo * Update .github/workflows/development.yml * Ana/more terra fixes (***THE COMEBACK***) (#324) * fixing my mess * move all rewards logic to terra * clearly signal single denom rewards * error structure has changed. need to look into it * also return denom for single denom rewards * manage errors properly * delete comments * Update lib/reducers/terraV3-reducers.js * loop through transaction messages * doing crazy stuff * rewrite the crazy bit Co-authored-by: Fabian <[email protected]> * cosmosv0 was missing the retry logic (#334) * add back validator to db logic (#335) Co-authored-by: Ana G. <[email protected]> Co-authored-by: Aleksey Rudometov <[email protected]> Co-authored-by: Mario Pino <[email protected]> Co-authored-by: Jordan Bibla <[email protected]>
luniehq · Feb 18, 2020 · 073d044 · 073d044
1 parent eb5beaf
commit 073d044
Show file tree

Hide file tree

Showing 10 changed files with 113 additions and 46 deletions.
diff --git a/.github/workflows/development.yml b/.github/workflows/development.yml
@@ -25,6 +25,6 @@ jobs:
     - name: Installing Docker Compose
       run: ssh [email protected] "sudo curl -L \"https://github.com/docker/compose/releases/download/1.24.1/docker-compose-$(uname -s)-$(uname -m)\" -o /usr/local/bin/docker-compose; sudo chmod +x /usr/local/bin/docker-compose"
     - name: Run on Digital Ocean
-      run: ssh [email protected] "docker image prune -a -f; docker load < lunieapi.tgz; docker container stop lunieapi;docker container stop caddy; docker container stop node_exporter; docker container prune -f; export HASURA_URL="https://staging-db.lunie.io/v1/graphql"; export HASURA_ADMIN_KEY="${{ secrets.LUNIE_STAGING_DB_KEY }}"; export SENTRY_DSN="${{ secrets.SENTRY_DSN }}"; docker-compose -f docker-compose.yml up --build -d"
+      run: ssh [email protected] "docker image prune -a -f; docker load < lunieapi.tgz;export HASURA_URL="https://staging-db.lunie.io/v1/graphql"; export HASURA_ADMIN_KEY="${{ secrets.LUNIE_STAGING_DB_KEY }}"; export SENTRY_DSN="${{ secrets.SENTRY_DSN }}"; docker stack deploy -c docker-compose.yml lunieapi"
     - name: Setting up cron job for pm2 metrics export
       run: ssh [email protected] "mkdir /logs -p; touch /logs/show; line='*/1 * * * * docker exec lunieapi pm2 show 0 > /logs/show; perl /root/pm2metrics.pl'; crontab -l | grep -q 'lunieapi pm2' && true || (crontab  -l; echo "$line" ) | crontab -"
diff --git a/.github/workflows/production.yml b/.github/workflows/production.yml
@@ -25,6 +25,6 @@ jobs:
     - name: Installing Docker Compose
       run: ssh [email protected] "sudo curl -L \"https://github.com/docker/compose/releases/download/1.24.1/docker-compose-$(uname -s)-$(uname -m)\" -o /usr/local/bin/docker-compose; sudo chmod +x /usr/local/bin/docker-compose"
     - name: Run on Digital Ocean
-      run: ssh [email protected] "docker image prune -a -f; docker load < lunieapi.tgz; docker container stop lunieapi;docker container stop caddy; docker container stop node_exporter; docker container prune -f;export HASURA_URL="https://production-db.lunie.io/v1/graphql"; export HASURA_ADMIN_KEY="${{ secrets.LUNIE_PRODUCTION_DB_KEY }}"; export SENTRY_DSN="${{ secrets.SENTRY_DSN_PRODUCTION }}"; docker-compose -f docker-compose.yml up --build -d"
+      run: ssh [email protected] "docker image prune -a -f; docker load < lunieapi.tgz;export HASURA_URL="https://production-db.lunie.io/v1/graphql"; export HASURA_ADMIN_KEY="${{ secrets.LUNIE_PRODUCTION_DB_KEY }}"; export SENTRY_DSN="${{ secrets.SENTRY_DSN_PRODUCTION }}"; docker stack deploy -c docker-compose.yml lunieapi"
     - name: Setting up cron job for pm2 metrics export
       run: ssh [email protected] "mkdir /logs -p; touch /logs/show; line='*/1 * * * * docker exec lunieapi pm2 show 0 > /logs/show; perl /root/pm2metrics.pl'; crontab -l | grep -q 'lunieapi pm2' && true || (crontab  -l; echo "$line" ) | crontab -"
diff --git a/lib/block-listeners/cosmos-node-subscription.js b/lib/block-listeners/cosmos-node-subscription.js
@@ -74,7 +74,7 @@ class CosmosNodeSubscription {
 
       const validators = await this.cosmosAPI.getAllValidators(block.height)
       const validatorMap = await this.getValidatorMap(validators)
-      // this.updateDBValidatorProfiles(validators)
+      this.updateDBValidatorProfiles(validators)
       this.store.update({
         height: block.height,
         block,

diff --git a/lib/reducers/cosmosV0-reducers.js b/lib/reducers/cosmosV0-reducers.js
@@ -392,12 +392,18 @@ function formatTransactionsReducer(txs, reducers) {
   const sortedTxs = sortBy(duplicateFreeTxs, ['timestamp'])
   const reversedTxs = reverse(sortedTxs)
   // here we filter out all transactions related to validators
-  reversedTxs.forEach(tx => {
-    tx.tx.value.msg = tx.tx.value.msg.filter(msg =>
+  let filteredTxs = []
+  reversedTxs.forEach(transaction => {
+    const index = transaction.tx.value.msg.findIndex(msg =>
       cosmosWhitelistedMessageTypes.has(msg.type.split('/')[1])
     )
+    // only push transactions messages supported by Lunie
+    if (index !== -1) {
+      transaction.tx.value.msg = [transaction.tx.value.msg[index]]
+      filteredTxs.push(transaction)
+    }
   })
-  return reversedTxs.map(tx => transactionReducer(tx, reducers))
+  return filteredTxs.map(tx => transactionReducer(tx, reducers))
 }
 
 // to be able to catch all validators from a multi-claim reward tx, we need to capture

diff --git a/lib/reducers/emoneyV0-reducers.js b/lib/reducers/emoneyV0-reducers.js
@@ -5,16 +5,13 @@ const _ = require('lodash')
 const EMoneyAPIUrl = `https://api.e-money.com/v1/`
 const exchangeAPIURL = `https://api.exchangeratesapi.io/latest?`
 const Sentry = require('@sentry/node')
-const { atoms } = terraV3Reducers
 
 async function totalBackedValueReducer(totalBackedValue) {
   const exchangeRates = await fetchTokenExchangeRates()
   const token = `e`.concat(totalBackedValue.denom.substring(1))
   const ticker = totalBackedValue.denom.substring(1).toUpperCase()
   // First we calculate the fiat net value of the token's total supply in it counterpart
   // fiat currency
-  // TODO Not all the tokens have yet their rate value in the e-Money API. So the calculation
-  // is not yet 100% accurate
   const fiatValue = exchangeRates[token]
     ? BigNumber(totalBackedValue.amount)
         .div(100000000)
@@ -120,30 +117,8 @@ async function expectedRewardsPerToken(
   return expectedReturns / 100
 }
 
-function rewardReducer(rewards, validatorsDictionary) {
-  const formattedRewards = rewards.map(
-    reward =>
-      (reward = {
-        reward: reward.reward,
-        validator: validatorsDictionary[reward.validator_address]
-      })
-  )
-  let multiDenomRewardsArray = []
-  formattedRewards.map(({ reward, validator }) =>
-    reward.forEach(denomReward => {
-      multiDenomRewardsArray.push({
-        denom: denomReward.denom,
-        amount: atoms(denomReward.amount),
-        validator: validator
-      })
-    })
-  )
-  return multiDenomRewardsArray
-}
-
 module.exports = {
   ...terraV3Reducers,
   expectedRewardsPerToken,
-  totalBackedValueReducer,
-  rewardReducer
+  totalBackedValueReducer
 }
diff --git a/lib/reducers/terraV3-reducers.js b/lib/reducers/terraV3-reducers.js
@@ -1,4 +1,5 @@
 const cosmosV2Reducers = require('./cosmosV2-reducers')
+const { atoms } = cosmosV2Reducers
 
 // Terra has a slightly different structure and needs its own undelegationEndTimeReducer
 function undelegationEndTimeReducer(transaction) {
@@ -23,7 +24,29 @@ function undelegationEndTimeReducer(transaction) {
   }
 }
 
+function rewardReducer(rewards, validatorsDictionary) {
+  const formattedRewards = rewards.map(
+    reward =>
+      (reward = {
+        reward: reward.reward,
+        validator: validatorsDictionary[reward.validator_address]
+      })
+  )
+  let multiDenomRewardsArray = []
+  formattedRewards.forEach(({ reward, validator }) =>
+    reward.forEach(denomReward => {
+      multiDenomRewardsArray.push({
+        denom: denomReward.denom,
+        amount: atoms(denomReward.amount),
+        validator: validator
+      })
+    })
+  )
+  return multiDenomRewardsArray
+}
+
 module.exports = {
   ...cosmosV2Reducers,
+  rewardReducer,
   undelegationEndTimeReducer
 }
diff --git a/lib/schema.js b/lib/schema.js
@@ -155,6 +155,54 @@ const typeDefs = gql`
     undelegationEndTime: String
   }
 
+  union TransactionDetails =
+      SendTx
+    | StakeTx
+    | UnstakeTx
+    | ClaimRewardsTx
+    | VoteProposalTx
+
+  type TransactionV2 {
+    type: String!
+    hash: String!
+    height: Int!
+    details: TransactionDetails
+    timestamp: String!
+  }
+
+  type SendTx {
+    amount: Coin!
+    to: String!
+    memo: String
+  }
+
+  type StakeTx {
+    amount: Coin!
+    to: [String]!
+  }
+
+  type UnstakeTx {
+    amount: Coin!
+    from: [String]!
+  }
+
+  type ClaimRewardsTx {
+    amount: Coin!
+    from: [String]!
+  }
+
+  type SubmitProposalTx {
+    proposalType: String!
+    proposalTitle: String!
+    proposalDescription: String!
+    initialDeposit: Coin!
+  }
+
+  type VoteProposalTx {
+    proposalId: Int!
+    voteOption: String!
+  }
+
   type GovernanceParameters {
     depositDenom: String
     votingThreshold: Float

diff --git a/lib/source/cosmosV0-source.js b/lib/source/cosmosV0-source.js
@@ -1,4 +1,5 @@
-const { RESTDataSource } = require('apollo-datasource-rest')
+const { RESTDataSource, HTTPCache } = require('apollo-datasource-rest')
+const { InMemoryLRUCache } = require('apollo-server-caching')
 const BigNumber = require('bignumber.js')
 const _ = require('lodash')
 const { encodeB32, decodeB32, pubkeyToAddress } = require('../tools')
@@ -17,21 +18,33 @@ class CosmosV0API extends RESTDataSource {
     this.setReducers()
   }
 
+  initialize(config) {
+    this.context = config.context
+    // manually set cache to checking it
+    this.cache = new InMemoryLRUCache()
+    this.httpCache = new HTTPCache(this.cache, this.httpFetch)
+  }
+
   setReducers() {
     this.reducers = require('../reducers/cosmosV0-reducers')
   }
 
   async getRetry(url, intent = 0) {
+    // check cache size, and flush it if it's bigger than something
+    if ((await this.cache.getTotalSize()) > 100000) {
+      await this.cache.flush()
+    }
     // cleareing memoizedResults
     this.memoizedResults.clear()
     try {
       return await this.get(url, null, { cacheOptions: { ttl: 1 } }) // normally setting cacheOptions should be enought, but...
     } catch (error) {
       // give up
       if (intent >= 3) {
-        throw new Error(
+        console.error(
           `Error for query ${url} in network ${this.networkId} (tried 3 times)`
         )
+        throw error
       }
 
       // retry
@@ -44,7 +57,7 @@ class CosmosV0API extends RESTDataSource {
   // is overwritten in cosmos v2 to extract from a differnt result format
   // some endpoints /blocks and /txs have a different response format so they use this.get directly
   async query(url) {
-    return this.get(url)
+    return this.getRetry(url)
   }
 
   async getStakinDenom() {
@@ -152,6 +165,7 @@ class CosmosV0API extends RESTDataSource {
       )
     } catch (error) {
       // in some rare cases the validator has no self delegation so this query fails
+
       if (error.extensions.response.status === 500) {
         const parsedErrorLog = JSON.parse(error.extensions.response.body.error)
         if (parsedErrorLog.message.startsWith('no delegation for this')) {

diff --git a/lib/source/emoneyV0-source.js b/lib/source/emoneyV0-source.js
@@ -135,17 +135,6 @@ class EMoneyV0API extends TerraV3API {
         return `?`
     }
   }
-
-  async getRewards(delegatorAddress, validatorsDictionary) {
-    this.checkAddress(delegatorAddress)
-    const result = await this.query(
-      `distribution/delegators/${delegatorAddress}/rewards`
-    )
-    const rewards = (result.rewards || []).filter(
-      ({ reward }) => reward && reward.length > 0
-    )
-    return this.reducers.rewardReducer(rewards, validatorsDictionary)
-  }
 }
 
 module.exports = EMoneyV0API
diff --git a/lib/source/terraV3-source.js b/lib/source/terraV3-source.js
@@ -66,6 +66,18 @@ class TerraV3API extends CosmosV2API {
     )
     return expectedReturns
   }
+
+  // Terra will be the root source for functions specific to Tendermint multidenom networks
+  async getRewards(delegatorAddress, validatorsDictionary) {
+    this.checkAddress(delegatorAddress)
+    const result = await this.query(
+      `distribution/delegators/${delegatorAddress}/rewards`
+    )
+    const rewards = (result.rewards || []).filter(
+      ({ reward }) => reward && reward.length > 0
+    )
+    return this.reducers.rewardReducer(rewards, validatorsDictionary)
+  }
 }
 
 module.exports = TerraV3API