Add newPendingRawTransactions to filter system to save bandwidth #24524

t-anyu · 2022-03-09T18:13:51Z

Hey,

In use cases where client subscribes to newPendingTransactions and gets txhashes would typically re-request transaction content from the node. In public nodes or nodes that serve many simultaneous clients, this would cause some additional load.

It would be nice, and quite easy to implement another subscription of newPendingRawTransactions, which would directly send the raw transaction instead of just tx hash. Then the client can decode the raw transaction without re-requesting json'd eth_getTransactionByHash.

I think this should be easily implemented. Instead of sending tx.Hash(), a tx.MarshalBinary() would be sent to the subscription channel.

(I can do the commit if this gets green light from the maintainers)

karalabe · 2022-03-10T09:23:40Z

Could you provide some more details about what you're trying to use it for? We do have a few ideas in mind too, but curious about your case.

t-anyu · 2022-03-15T13:08:24Z

Hey,

In my case, the node that I'm hosting is mostly being used for various trading related applications. But currently working on a MEV tracking case, where we need to record all the pending transactions across multiple nodes.

I believe in most use cases that uses pending transaction filter are being used in tranding / gaming related applications. And in these cases, the exact transaction data is needed.

amirilidan78 · 2022-06-09T10:43:59Z

it would be nice for example i want to listen on new pending transactions and check if receiver address is in my database , execute callback for my application ,but now because i have only txId and i should call 1 http per tx it would not be cool

trezor blockbook is good example it provider transaction detail on new pending transaction not just txId
https://github.com/trezor/blockbook/blob/master/docs/api.md
subscribeNewTransaction

example

{
    "id": "1",
    "data": {
        "txid": "ca32a8ea2a881ab508e3fa84f03a5ae5073bab977444bcfd5e75d6ce5d31b86f",
        "version": 2,
        "vin": [
            {
                "txid": "75eff4aaabfeaa7824337ef61ad12918ea2a8659918898330949efe9703840bf",
                "vout": 1,
                "sequence": 4294967295,
                "n": 0,
                "addresses": [
                    "bc1qunxnu8gv8ckdsvewqj809ajfyxxfkv6chak2j7"
                ],
                "isAddress": true,
                "value": "129911"
            },
            {
                "txid": "f973e88c40ea8462af2c79432a193c79247d44b24edbe44598e7fceb0361859c",
                "vout": 1,
                "sequence": 4294967295,
                "n": 1,
                "addresses": [
                    "bc1q9zexgk4r956yvskdlpvvz3y9h27e5pa424v4r5"
                ],
                "isAddress": true,
                "value": "98397"
            },
            {
                "txid": "3218619a5f858adf2471ef3eb4263ecdb041cd49f53553ce624f363838f528e2",
                "vout": 1,
                "sequence": 4294967295,
                "n": 2,
                "addresses": [
                    "bc1quc7yudz95x6m7dhh96nfpts4d6e9uxczyh5lx0"
                ],
                "isAddress": true,
                "value": "29939"
            },
            {
                "txid": "55413f9a41dd887762e8ae21917fa95ebed7bfcaabd316ff82035cae441505d8",
                "vout": 1,
                "sequence": 4294967295,
                "n": 3,
                "addresses": [
                    "bc1qww3n386xqfzzrygkq2gjv8jxl47ak8d8frwu2s"
                ],
                "isAddress": true,
                "value": "4229"
            }
        ],
        "vout": [
            {
                "value": "258804",
                "n": 0,
                "hex": "76a914cc75f99318cbeb65673396b53d9298729e558e7688ac",
                "addresses": [
                    "1Ke6BB3hJKzkPdLoYmWmBiuKbeJqpKgp8D"
                ],
                "isAddress": true
            }
        ],
        "blockHeight": 0,
        "confirmations": 0,
        "blockTime": 1654771422,
        "value": "258804",
        "valueIn": "262476",
        "fees": "3672",
        "hex": "02000000000104bf403870e9ef49093398889159862aea1829d11af67e332478aafeabaaf4ef750100000000ffffffff9c856103ebfce79845e4db4eb2447d24793c192a43792caf6284ea408ce873f90100000000ffffffffe228f53838364f62ce5335f549cd41b0cd3e26b43eef7124df8a855f9a6118320100000000ffffffffd8051544ae5c0382ff16d3abcabfd7be5ea97f9121aee8627788dd419a3f41550100000000ffffffff01f4f20300000000001976a914cc75f99318cbeb65673396b53d9298729e558e7688ac02483045022100ae8ddcf6b5a13aa900afb9e5fed523b6db5f7f8a43fcfff8f0904d73aa38fe3f022068ae9717d3512e763f603b0f77fc13da1c0db056f67a217f89c1aee7e80e27370121028e9e92b4719a30cf5934a7692709e63ed956a40b4eea7a93e9ac11efa26ab6d4024830450221009f6596fa13ee6896b6d6e948cf1ee5cdc047e80bcd15ef89e60cb13bb6d0b22002203410f27ac19ec5260eaf5b198fea0f344ea8c5cba8054973020e17f2a75b956201210238d4124e2afa0601cdabc73ef0f9aa0b738ba576f5bac48db8d8307c3dade51a0247304402206bb199089b582a7778eddda7c00def4561c60b617bb53379b29ec3a29d3fa378022077c4a18849e09d0e0f29933e3dacb0944b122a78b61547370e0c1e8d3254132b012102a2c4211dd67e3cff7789f9170f6082b0e80f48c9e16fc9f43bc89a6b1e0713a9024830450221008d077900f6ee0a619e6aebf93827b8df85afdf73eb67811bd924d6cd1bc41d5202205c68c608ba3929d1a12b1d241fc442067c9b011c1590f7e8f1626e67b0b686e5012102d502c2bd743743b71aaef37973f87173d58bcd9e062c581e5d504f12e85b453f00000000"
    }
}

lmittmann · 2022-06-14T09:02:52Z

@karalabe would you accept PRs that provide the functionality?

…sactions * ethereum#24524

karalabe · 2022-06-30T08:31:12Z

We should review and merge the feature I guess.

Anti83 · 2022-06-30T10:26:39Z

@lmittmann
Hi, I merged your PR into my fork, but still getting only hashes.
I am using web3 web3.eth.subscribe('pendingTransactions')
What is the right way to use it? Thanks.

t-anyu · 2022-06-30T14:41:46Z

Hey,

I have implemented newPendingRawTransactions on my test node and I suggest that raw RLP is sent instead of Json if performance is concern. RLP encoding seems to be much faster than json encode. Probably RLP is Pooled and now need to ecrecover + keccak?

ofarukcaki · 2022-06-30T14:57:25Z

Hey,

I have implemented newPendingRawTransactions on my test node and I suggest that raw RLP is sent instead of Json if performance is concern. RLP encoding seems to be much faster than json encode. Probably RLP is Pooled and now need to ecrecover + keccak?

Hey, can you share the implementation details with us? Thanks

lmittmann · 2022-06-30T15:31:33Z

@lmittmann Hi, I merged your PR into my fork, but still getting only hashes. I am using web3 web3.eth.subscribe('pendingTransactions') What is the right way to use it? Thanks.

@Anti83 You have to update your RPC client to send the additional fullTx flag. See

go-ethereum/ethclient/gethclient/gethclient.go

Lines 178 to 181 in 4a69d76

    
           // SubscribePendingTransactions subscribes to new pending transactions. 
        
           func (ec *Client) SubscribePendingTransactions(ctx context.Context, ch chan<- *types.Transaction) (*rpc.ClientSubscription, error) { 
        
           	return ec.c.EthSubscribe(ctx, ch, "newPendingTransactions", true) 
        
           }

for reference

ofarukcaki · 2022-06-30T18:30:48Z

@lmittmann Hi, I merged your PR into my fork, but still getting only hashes. I am using web3 web3.eth.subscribe('pendingTransactions') What is the right way to use it? Thanks.

Did you figure out what is the right way?

…sactions * ethereum#24524

holiman · 2022-10-12T09:55:56Z

Fixed by #25186

amirilidan78 · 2022-10-24T06:51:20Z

thanks 😍

jwelch-qn · 2023-07-27T18:26:50Z

Hey all, is it possible to toggle this back to receiving only tx hashes with the latest update? @holiman

jsvisa · 2023-07-28T15:14:41Z

@jwelch-qn It did have supported, ref https://geth.ethereum.org/docs/interacting-with-geth/rpc/pubsub#newpendingtransactions

eg:

$ wscat -c ws://localhost:8546
Connected (press CTRL+C to quit)

> {"id": 1, "jsonrpc": "2.0", "method": "eth_subscribe", "params": ["newPendingTransactions", false]}
< {"jsonrpc":"2.0","id":1,"result":"0xb94e3d49419b39284360092ec6eec2ea"}

# here comes the new tx with hash only
< {"jsonrpc":"2.0","method":"eth_subscription","params":{"subscription":"0xb94e3d49419b39284360092ec6eec2ea","result":"0xf73d808cb819267b97dadb98d83a7348f296807c058a526b40cc7594b8f2d66a"}}
< {"jsonrpc":"2.0","method":"eth_subscription","params":{"subscription":"0xb94e3d49419b39284360092ec6eec2ea","result":"0x7e032b3fa572484f9416a95d4b0a0228c51ccb9f48de9e58c8c24ba38128afd4"}}

damartripamungkas · 2023-07-28T20:17:55Z

@jwelch-qn It did have supported, ref https://geth.ethereum.org/docs/interacting-with-geth/rpc/pubsub#newpendingtransactions

eg:

$ wscat -c ws://localhost:8546
Connected (press CTRL+C to quit)

> {"id": 1, "jsonrpc": "2.0", "method": "eth_subscribe", "params": ["newPendingTransactions", false]}
< {"jsonrpc":"2.0","id":1,"result":"0xb94e3d49419b39284360092ec6eec2ea"}

# here comes the new tx with hash only
< {"jsonrpc":"2.0","method":"eth_subscription","params":{"subscription":"0xb94e3d49419b39284360092ec6eec2ea","result":"0xf73d808cb819267b97dadb98d83a7348f296807c058a526b40cc7594b8f2d66a"}}
< {"jsonrpc":"2.0","method":"eth_subscription","params":{"subscription":"0xb94e3d49419b39284360092ec6eec2ea","result":"0x7e032b3fa572484f9416a95d4b0a0228c51ccb9f48de9e58c8c24ba38128afd4"}}

what is version your geth?

jwelch-qn · 2023-07-28T21:25:59Z

Hmm, this is helpful, but this change is causing issues with eth_newPendingTransactionFilter and eth_getFilterChanges which is breaking downstream libraries.
e.g. https://github.com/web3j/web3j/release

@damartripamungkas @jsvisa

damartripamungkas · 2023-07-28T21:51:30Z

you must be extend class Web3Subscription, refrence : https://docs.web3js.org/guides/events_subscriptions/custom_subscriptions/

or use my package for subscription :
https://github.com/damartripamungkas/simple-client-eth-rpc/tree/master

jsvisa · 2023-07-29T03:29:39Z

@jwelch-qn It did have supported, ref https://geth.ethereum.org/docs/interacting-with-geth/rpc/pubsub#newpendingtransactions

eg:
$ wscat -c ws://localhost:8546

Connected (press CTRL+C to quit)

{"id": 1, "jsonrpc": "2.0", "method": "eth_subscribe", "params": ["newPendingTransactions", false]}

< {"jsonrpc":"2.0","id":1,"result":"0xb94e3d49419b39284360092ec6eec2ea"}

here comes the new tx with hash only

< {"jsonrpc":"2.0","method":"eth_subscription","params":{"subscription":"0xb94e3d49419b39284360092ec6eec2ea","result":"0xf73d808cb819267b97dadb98d83a7348f296807c058a526b40cc7594b8f2d66a"}}

< {"jsonrpc":"2.0","method":"eth_subscription","params":{"subscription":"0xb94e3d49419b39284360092ec6eec2ea","result":"0x7e032b3fa572484f9416a95d4b0a0228c51ccb9f48de9e58c8c24ba38128afd4"}}
what is version your geth?

v1.12.0

jsvisa · 2023-07-29T03:31:02Z

Hmm, this is helpful, but this change is causing issues with eth_newPendingTransactionFilter and eth_getFilterChanges which is breaking downstream libraries.

e.g. https://github.com/web3j/web3j/release

@damartripamungkas @jsvisa

imo this is the web3j’s issue, not geth’s, and you should file an issue in web3j’s repo

t-anyu added the type:feature label Mar 9, 2022

ligi added the status:triage label Mar 10, 2022

ligi removed the status:triage label Mar 17, 2022

s1na mentioned this issue Jun 27, 2022

How to subscribe to entire tx objects? #25173

Closed

s1na added the status:triage label Jun 27, 2022

lmittmann added a commit to lmittmann/go-ethereum that referenced this issue Jun 28, 2022

eth/filter, ethclient/gethclient: added fullTx-flag to NewPendingTran…

e36519c

…sactions * ethereum#24524

lmittmann mentioned this issue Jun 28, 2022

eth/filters, ethclient/gethclient: add fullTx option to pending tx filter #25186

Merged

lmittmann added a commit to lmittmann/go-ethereum that referenced this issue Jun 30, 2022

eth/filter, ethclient/gethclient: added fullTx-flag to NewPendingTran…

d696e43

…sactions * ethereum#24524

karalabe self-assigned this Jun 30, 2022

s1na removed the status:triage label Jul 7, 2022

fjl pushed a commit to lmittmann/go-ethereum that referenced this issue Oct 12, 2022

eth/filter, ethclient/gethclient: added fullTx-flag to NewPendingTran…

7b7f40b

…sactions * ethereum#24524

holiman closed this as completed Oct 12, 2022

yuichiroaoki mentioned this issue Jun 1, 2023

Support SubscribeFullPendingTransactions from go-ethereum maticnetwork/bor#884

Closed

18 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add newPendingRawTransactions to filter system to save bandwidth #24524

Add newPendingRawTransactions to filter system to save bandwidth #24524

t-anyu commented Mar 9, 2022

karalabe commented Mar 10, 2022

t-anyu commented Mar 15, 2022 •

edited

Loading

amirilidan78 commented Jun 9, 2022 •

edited

Loading

lmittmann commented Jun 14, 2022

karalabe commented Jun 30, 2022 •

edited

Loading

Anti83 commented Jun 30, 2022

t-anyu commented Jun 30, 2022

ofarukcaki commented Jun 30, 2022

lmittmann commented Jun 30, 2022 •

edited

Loading

ofarukcaki commented Jun 30, 2022

holiman commented Oct 12, 2022

amirilidan78 commented Oct 24, 2022

jwelch-qn commented Jul 27, 2023 •

edited

Loading

jsvisa commented Jul 28, 2023

damartripamungkas commented Jul 28, 2023

jwelch-qn commented Jul 28, 2023

damartripamungkas commented Jul 28, 2023

jsvisa commented Jul 29, 2023

here comes the new tx with hash only

jsvisa commented Jul 29, 2023

Add newPendingRawTransactions to filter system to save bandwidth #24524

Add newPendingRawTransactions to filter system to save bandwidth #24524

Comments

t-anyu commented Mar 9, 2022

karalabe commented Mar 10, 2022

t-anyu commented Mar 15, 2022 • edited Loading

amirilidan78 commented Jun 9, 2022 • edited Loading

lmittmann commented Jun 14, 2022

karalabe commented Jun 30, 2022 • edited Loading

Anti83 commented Jun 30, 2022

t-anyu commented Jun 30, 2022

ofarukcaki commented Jun 30, 2022

lmittmann commented Jun 30, 2022 • edited Loading

ofarukcaki commented Jun 30, 2022

holiman commented Oct 12, 2022

amirilidan78 commented Oct 24, 2022

jwelch-qn commented Jul 27, 2023 • edited Loading

jsvisa commented Jul 28, 2023

damartripamungkas commented Jul 28, 2023

jwelch-qn commented Jul 28, 2023

damartripamungkas commented Jul 28, 2023

jsvisa commented Jul 29, 2023

here comes the new tx with hash only

jsvisa commented Jul 29, 2023

t-anyu commented Mar 15, 2022 •

edited

Loading

amirilidan78 commented Jun 9, 2022 •

edited

Loading

karalabe commented Jun 30, 2022 •

edited

Loading

lmittmann commented Jun 30, 2022 •

edited

Loading

jwelch-qn commented Jul 27, 2023 •

edited

Loading