release (#307)

* Fabo/fix gas (#250) * fix gas estimate * linted * fixed test * do not keep data sources (#251) * track failing transactions in Sentry (#249) * correctly set the tx schema for a failing tx (#248) * Fabo/remove per block caching as not working (#247) * remove per block caching as not working * fix memoized results Co-authored-by: Ana G. <[email protected]> * delete perblockcachedatasource (#253) * Ana/fix balances in actionmodal (#255) * fix action modal available balance * include regen * use dictionary for denomlookup * use correct events for received txs (#257) * enable account creation for some networks (#252) * network update time metric added (#256) * network update time metric added * added missing dep Co-authored-by: Fabian <[email protected]> * Fix proposal deposit (#261) * Remove denom handling from getDeposit() * Revert undesired change * delete package-lock.json * localtestnet config change (#265) * Ana/handle "address not from this network" error (#263) * add check address function for all queries * apply suggestions * Ana/add fiatvalue to balances query (e-Money) (#262) * preparation * more preparation * add fiatvalue field to balances query * fix get account info * apply suggestions * apply one last suggestion * suggestions+ Co-authored-by: Fabian <[email protected]> * Ana/emoney fix expected returns with inflation and totalbacked (#243) * fix expected returns with inflation and supply * minor fixes. dictionary * query exchange rates from emoney api * fix infinite expected returns * convert api url to const * add eur value to totalbackedvalue. totalngm gains * add important comment * finish calculation * lint * catch errors with sentry Co-authored-by: Fabian <[email protected]> * readd coin conversion (#268) * delete amount field (#274) * Fabo/increase gas again (#271) * icrease gas again * fixed test * Fabo/load all txs (even if more then first page in response) (#270) * load all txs (even if more then first page in response) * improved handling of txs * missing renaming * fixed paginated load * add pagination fix also to cosmosV0-source Co-authored-by: iambeone <[email protected]> Co-authored-by: Ana G. <[email protected]> * fixing issue with multiple senders in one event (#273) * fixing issue with multiple senders in one event * Update lib/source/cosmosV2-source.js Co-authored-by: Fabian <[email protected]> * Fabo/allow signing for terra + emoney (#267) * allow signing for terra * readd coin conversion * enable actions for terra * fix correct terra testnet url * comments and guards * enabled more txs for emoney and fixed broadcasting * added a catch for wrongly formatted broadcast urls * recover default field. change some network titles (#277) * Fabo/add network data to API (#278) * non desctructive introduction of better address prefix wording * added address creator to API * adjusted test * added ledger app to networks config * add icon property to schema (#281) * add icon property to schema * fix network schema validation Co-authored-by: Ana G. <[email protected]> * filter out validator specific txs (#279) * Ana/balances coinreducer good fix (#269) * balances coinreducer good fix * refactored fiat value logic Co-authored-by: Fabian <[email protected]> * Create network_integration.md * Update network_integration.md * Update network_integration.md * Fabo/avoid 500 errors (#288) * avoid using the latest query * cleanup * Ana/filter validator tx cross network and add txvalue reducer (#285) * filter validators cross network * add value reducer. necessary for multi claim txs * add validator txs filter also for cosmosv0 source * filter and make array only claim rewards msg value * filter txs by whitelist * change length in multi claim reward reducer * add withdrawvalidators * replace dictionary for set * refactor transaction snippet. avoid repetition * Ana/emoney upgrade (mergeable) (#282) * update emoney api_url * fix denom. add default fiat currency * fix rpc endpoint * fix value (my bad) (#293) * fix value (my bad) * trigger another ci flow * erase space * set correct new chain id (#294) * restart API * restart API * fix pr alert (#297) * Fabo/298 tendermint reconnect (#300) * reconnect on tendermint disconnect * cleanup * comments * Update cosmos-node-subscription.js * Fabo/299 trigger a chain hangup error (#301) * trigger a chain hangup error * increase chain hangup time * Apply suggestions from code review * Fabo/store validator addresses (#296) * add validator addresses to db * linted * ignore in local dev * revert * fixed fetch * comment * refactored db into constructor * cleanup * add clearTimeout to avoid reconnection hell (#306) * add clearTimeout to avoid reconnection hell * removed console.log Co-authored-by: Ana G. <[email protected]> Co-authored-by: Aleksey Rudometov <[email protected]> Co-authored-by: Mario Pino <[email protected]> Co-authored-by: Jordan Bibla <[email protected]>
luniehq · Feb 11, 2020 · 8ed1f54 · 8ed1f54
1 parent d2e06a5
commit 8ed1f54
Show file tree

Hide file tree

Showing 4 changed files with 161 additions and 6 deletions.
diff --git a/lib/block-listeners/cosmos-node-subscription.js b/lib/block-listeners/cosmos-node-subscription.js
@@ -7,10 +7,14 @@ const {
   publishEvent: publishEvent
 } = require('../subscriptions')
 const Sentry = require('@sentry/node')
+const database = require('../database')
+const config = require('../../config.js')
 
 const WAIT_FOR_BLOCK_DELAY = 5000
 const EXPECTED_MAX_BLOCK_WINDOW = 120000
 
+let reconnectionTimeout = {}
+
 // This class establishes an rpc connection to Tendermint.
 // Used for listening to events, such as new blocks.
 class CosmosNodeSubscription {
@@ -22,6 +26,8 @@ class CosmosNodeSubscription {
     this.metric = io.metric({
       name: `${this.network.id}_update`
     })
+    const networkSchemaName = this.network.id.replace(/-/g, '_')
+    this.db = new database(config)(networkSchemaName)
     this.chainHangup = undefined
 
     this.connectTendermint(this.network)
@@ -68,7 +74,12 @@ class CosmosNodeSubscription {
             scope.setExtra('rpc_url', network.rpc_url)
             Sentry.captureException(new Error(`Lost Tendermint connection`))
           })
-          setTimeout(() => this.connectTendermint(network), 3000)
+          // need to clear previous timeout to evoid connection hell
+          clearTimeout(reconnectionTimeout[network.id])
+          reconnectionTimeout[network.id] = setTimeout(
+            () => this.connectTendermint(network),
+            3000
+          )
         }
       })
       .catch(e => {
@@ -78,8 +89,12 @@ class CosmosNodeSubscription {
           Sentry.captureException(e)
         })
 
+        clearTimeout(reconnectionTimeout[network.id])
         // if can't connect, retry
-        setTimeout(() => this.connectTendermint(network), 3000)
+        reconnectionTimeout[network.id] = setTimeout(
+          () => this.connectTendermint(network),
+          3000
+        )
       })
   }
 
@@ -100,7 +115,9 @@ class CosmosNodeSubscription {
     // eslint-disable-next-line require-atomic-updates
     height = block.height
 
-    const validatorMap = await this.getValidatorMap(height)
+    const validators = await this.cosmosAPI.getAllValidators(height)
+    const validatorMap = await this.getValidatorMap(validators)
+    this.updateDBValidatorProfiles(validators)
     this.store.update({ height, block, validators: validatorMap })
     publishBlockAdded(this.network.id, block)
     // TODO remove, only for demo purposes
@@ -132,11 +149,19 @@ class CosmosNodeSubscription {
     this.cosmosAPI.memoizedResults.clear()
   }
 
-  async getValidatorMap(height) {
-    const validators = await this.cosmosAPI.getAllValidators(height)
+  async getValidatorMap(validators) {
     const validatorMap = _.keyBy(validators, 'operatorAddress')
     return validatorMap
   }
+
+  // this adds all the validator addresses to the database so we can easily check in the database which ones have an image and which ones don't
+  async updateDBValidatorProfiles(validators) {
+    const validatorRows = validators.map(({ operatorAddress, name }) => ({
+      operator_address: operatorAddress,
+      name
+    }))
+    return this.db.upsert('validatorprofiles', validatorRows)
+  }
 }
 
 module.exports = CosmosNodeSubscription
diff --git a/lib/database/helpers.js b/lib/database/helpers.js
@@ -0,0 +1,113 @@
+const fetch = require('node-fetch')
+const escape = require('escape-html')
+
+const graphQLQuery = ({ hasura_url, hasura_admin_key }) => async query => {
+  const data = await fetch(hasura_url, {
+    method: 'POST',
+    headers: {
+      'Content-Type': 'application/json',
+      'x-hasura-admin-secret': hasura_admin_key
+    },
+    body: JSON.stringify({
+      query
+    })
+  }).then(res => res.json())
+
+  if (data.errors || data.error) {
+    console.error('Query failed:', query)
+    console.error('GraphQL query failed:', data.error)
+    throw new Error('GraphQL query failed')
+  }
+
+  return data
+}
+
+function gqlKeyValue([key, value]) {
+  // escape all values as they could be malicious
+  return `${key}: "${escape(value)}"`
+}
+
+// stringify a set of row to be according to the graphQL schema
+function stringifyForGraphQL(rows, height, chainId) {
+  if (!Array.isArray(rows)) {
+    rows = [rows]
+  }
+  // set the height for every row if specified
+  if (height !== undefined) {
+    rows = rows.map(row => Object.assign(row, { height }))
+  }
+  // set the height for every row if specified
+  if (chainId !== undefined) {
+    rows = rows.map(row => Object.assign(row, { chain_id: chainId }))
+  }
+  return `[
+    ${rows
+      .map(row => {
+        return `{${Object.entries(row)
+          .map(gqlKeyValue)
+          .join(', ')}}`
+      })
+      .join(',\n')}
+    ]`
+}
+
+// get all columns for a db object
+function getColumns(objects) {
+  const obj = Array.isArray(objects) ? objects[0] : objects
+  return Object.keys(obj)
+}
+
+const insert = (
+  { hasura_url, hasura_admin_key },
+  upsert = false
+) => schema => async (table, rows, height, chainId) => {
+  if ((Array.isArray(rows) && rows.length === 0) || !rows) {
+    return
+  }
+
+  const query = `
+        mutation {
+            insert_${schema}_${table} (
+                objects: ${stringifyForGraphQL(rows, height, chainId)}${
+    upsert
+      ? `,
+                            on_conflict: {
+                                constraint: ${table}_pkey,
+                                update_columns: [${getColumns(rows)}]
+                            }
+                            `
+      : ''
+  }
+            )
+            {
+                affected_rows
+            }
+    }
+    `
+  return graphQLQuery({ hasura_url, hasura_admin_key })(query)
+}
+
+const read = ({ hasura_url, hasura_admin_key }) => schema => async (
+  table,
+  queryName,
+  keys,
+  filter
+) => {
+  keys = Array.isArray(keys) ? keys : [keys]
+
+  const query = `
+        query ${schema}_${queryName} {
+            ${schema}_${table}${filter ? `(${filter})` : ''} {
+                ${keys.join('\n')}
+            }
+        }
+    `
+
+  const res = await graphQLQuery({ hasura_url, hasura_admin_key })(query)
+  return res.data[`${schema}_${table}`]
+}
+
+module.exports = {
+  insert,
+  read
+}
diff --git a/lib/database/index.js b/lib/database/index.js
@@ -0,0 +1,17 @@
+const { insert, read } = require('./helpers')
+
+function database({ hasura_url, hasura_admin_key }) {
+  return schema => {
+    const methods = {
+      insert: insert({ hasura_url, hasura_admin_key })(schema),
+      upsert: insert({ hasura_url, hasura_admin_key }, true)(schema),
+      read: read({ hasura_url, hasura_admin_key })(schema)
+    }
+
+    return {
+      ...methods
+    }
+  }
+}
+
+module.exports = database
diff --git a/package.json b/package.json
@@ -13,7 +13,7 @@
     "dev-docker": "docker build -t lunie-api-dev . && docker run --init -it -p 4000:4000 lunie-api-dev",
     "lint": "eslint index.js config.js lib tests",
     "lint-fix": "yarn lint --fix",
-    "start": "node index.js",
+    "start": "node index.js ",
     "test": "jest"
   },
   "husky": {