Feature/grpc execution api (#1)

* add buf config and generated grpc code

* poc e2e grpc communication

* comment out panicking code

* logging help

* now reads cli args. now stops grpc server on shutdown.

* add mutex to GRPCServerHandler

update readme

containerize w/ github action to build and push to ghcr (#3)

fix gh action syntax (#4)

* containerize w/ github action to build and push to ghcr

* fix gh action yml syntax

* enable manual trigger

* fix syntax

build and push manually (#5)

* containerize w/ github action to build and push to ghcr

* fix gh action yml syntax

* enable manual trigger

* fix syntax

* dont use github action, do it ourselves

* push manually

Feature/containerize (#6)

* containerize w/ github action to build and push to ghcr

* fix gh action yml syntax

* enable manual trigger

* fix syntax

* dont use github action, do it ourselves

* push manually

* correct multi line

update SubmitTransaction to send tx to metro

ethclient: ensure returned subscription is nil on error (ethereum#26976)

core/state, trie: remove Try prefix in Trie accessors (ethereum#26975)

This change renames StateTrie methods to remove the Try* prefix.

We added the Trie methods with prefix 'Try' a long time ago, working
around the problem that most existing methods of Trie did not return the
database error. This weird naming convention has persisted until now.

Co-authored-by: Gary Rong <[email protected]>

metrics/librato: ensure resp.body closed (ethereum#26969)

This change ensures that we call Close on a http response body, in various places in the source code (mostly tests)

core/vm: use atomic.Bool (ethereum#26951)

Make use of new atomic types
---------

Co-authored-by: Felix Lange <[email protected]>
Co-authored-by: Martin Holst Swende <[email protected]>

core/bloombits: use atomic type (ethereum#26993)

core/state: use atomic.Bool (ethereum#26992)

graphql: fix data races (ethereum#26965)

Fixes multiple data races caused by the fact that resolving fields are done concurrently by the graphql library. It also enforces caching at the stateobject level for account fields.

eth/tracers/native: prevent panic for LOG edge-cases (ethereum#26848)

This PR fixes OOM panic in the callTracer as well as panicing on
opcode validation errors (e.g. stack underflow) in callTracer and
prestateTracer.

Co-authored-by: Martin Holst Swende <[email protected]>

internal/debug: add log.logfmt flag to set logging to use logfmt (ethereum#26970)

docs: update outdated DeriveSha docs comment (ethereum#26968)

add localnet genesis

update metro-transactions dep

Adding features to mempool to support our pre-ordered txs (#2)

* Adding features to mempool to support our pre-ordered txs

* Add framing for gRPC to execute blocks

* Remove public engine API call

* Fix circular dependencies

* Updated to use new Init, and fill out starting attributes

* Add clear astriaordered, cleanup

* readme fix

* add bash and jq to final docker image

* make txpool interface

* no more panics, update DoBlock to also update state + store block

* cleanup

* set post-merge at genesis

* cleanup

* doc update

* remove txpool interface changes

* cleanup

* cleanup

* build and push images wih tags defined by git tags

* build for multiple architectures. use docker-metadata action for semver

* add push: true

* only build arm for git tags/releases

---------

Co-authored-by: Jesse Snyder <[email protected]>
Co-authored-by: elizabeth <[email protected]>

tag image with latest for builds from astria branch (#8)

rename state_root to block_hash

use FROM --platform=$BUILDPLATFORM to build arm images correctly (#9)

use v4 of build and push action (#10)

* use FROM --platform=$BUILDPLATFORM to build arm images correctly

* use v4 of docker build and push

fix arm builds. only build arm for tags and merges to default branch. (#12)

* fix arm builds. only build arm for tags and merges to default branch.

* fix gh action function syntax

* more correct comment

* build for semver tags

Features and fixes needed for contract deployment test (#13)

* add and remove tx from geth mempool so forge can deploy a contract

* pass metro addr and port by flag

* fixes from pr feedback

* formatting

implement FinalizeBlock gRPC call

bump metro-transactions dep

Update Protos (#16)

* uses new protos

* remove old protos

update deps

update deps

no more unknown/unknown image (#19)

* no more unknown/unknown image

* hardcode condition

Changing from prev_state_root to prev_block_hash

remove metro

integrate submission to sequencer

log updates

use env vars for chain id/tendermint endpoint

rename to cometbft

Make the environment variables actually work

cleaner

simple logging change

removed sleep before getPayload by removing goroutine in buildPayload

removed unneeded comment

removed for loop from buildPayload

Remove direct sequencer submission

cleanup go.mod

Migrate to the v1alpha1 API

Missed pieces

set genesis block as head/safe/final

remove all mentions of metro

Initial implementation of Execution v1alpha2 api

Integrate updates

remove need for consensus api

Updates

small updates

Update grpc/execution/server.go

Co-authored-by: noot <[email protected]>

logging

remove unused dependency

Mark the chain 'finalized' on startup

.github/workflows/astria-build-and-publish-image.yml

@@ -0,0 +1,68 @@
+name: Build and Publish Docker image
+
+# Trigger on pushes to astria branch, new semantic version tags, and pull request updates
+on:
+  workflow_dispatch:
+  push:
+    branches:
+      - "astria"
+    tags:
+      - "v[0-9]+.[0-9]+.[0-9]+"
+      - "v[0-9]+.[0-9]+.[0-9]+-alpha.[0-9]+"
+      - "v[0-9]+.[0-9]+.[0-9]+-beta.[0-9]+"
+      - "v[0-9]+.[0-9]+.[0-9]+-rc[0-9]+"
+  # trigger on pull request updates when target is `astria` branch
+  pull_request:
+    branches:
+      - "astria"
+
+jobs:
+  build-and-publish-latest:
+    runs-on: ubuntu-latest
+    steps:
+      # Checking out the repo
+      - uses: actions/checkout@v3
+      # Setting up Go
+      - uses: actions/setup-go@v4
+        with:
+          go-version: "^1.20.x" # The Go version to download (if necessary) and use.
+      - run: go version
+
+      # https://github.com/docker/setup-qemu-action
+      - name: Set up QEMU
+        uses: docker/setup-qemu-action@v2
+      # https://github.com/docker/setup-buildx-action
+      - name: Set up Docker Buildx
+        id: buildx
+        uses: docker/setup-buildx-action@v2
+      - name: Log in to GitHub Container Registry
+        uses: docker/login-action@v2
+        with:
+          registry: ghcr.io
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+      # Generate correct tabs and labels
+      - name: Docker metadata
+        id: metadata
+        uses: docker/metadata-action@v4
+        with:
+          images: |
+            ghcr.io/astriaorg/go-ethereum
+          tags: |
+            type=ref,event=pr
+            type=semver,pattern={{major}}.{{minor}}.{{patch}}
+            type=sha
+            # set latest tag for `astria` branch
+            type=raw,value=latest,enable=${{ github.ref == format('refs/heads/{0}', 'astria') }}
+      - name: Build and push
+        uses: docker/build-push-action@v4
+        with:
+          # this gets rid of the unknown/unknown image that is created without this setting
+          # https://github.com/docker/build-push-action/issues/820#issuecomment-1455687416
+          provenance: false
+          context: .
+          # It takes a long time to build the arm image right now, so we only build it on tags which is what we use for releases, or on merges to the default branch.
+          platforms: ${{ (contains(github.ref, 'refs/tags/v') || github.ref == 'refs/heads/astria') && 'linux/amd64,linux/arm64' || 'linux/amd64' }}
+          push: true
+          tags: ${{ steps.metadata.outputs.tags }}
+          labels: ${{ steps.metadata.outputs.labels }}

Dockerfile

@@ -20,9 +20,14 @@ RUN cd /go-ethereum && go run build/ci.go install -static ./cmd/geth
 FROM alpine:latest
 
 RUN apk add --no-cache ca-certificates
+# Astria - add bash and jq to support start-geth.sh in conductor
+RUN apk add bash jq
+# Astria - copy genesis.json so it can be used in start-geth.sh
+COPY genesis.json /genesis.json
 COPY --from=builder /go-ethereum/build/bin/geth /usr/local/bin/
 
-EXPOSE 8545 8546 30303 30303/udp
+# Astria - add 50051 for GRPC
+EXPOSE 8545 8546 30303 30303/udp 50051
 ENTRYPOINT ["geth"]
 
 # Add some metadata labels to help programatic image consumption

Dockerfile.alltools

@@ -22,7 +22,8 @@ FROM alpine:latest
 RUN apk add --no-cache ca-certificates
 COPY --from=builder /go-ethereum/build/bin/* /usr/local/bin/
 
-EXPOSE 8545 8546 30303 30303/udp
+# Astria - add 50051 for GRPC
+EXPOSE 8545 8546 30303 30303/udp 50051
 
 # Add some metadata labels to help programatic image consumption
 ARG COMMIT=""

cmd/geth/config.go

@@ -37,6 +37,7 @@ import (
 	"github.com/ethereum/go-ethereum/eth/catalyst"
 	"github.com/ethereum/go-ethereum/eth/downloader"
 	"github.com/ethereum/go-ethereum/eth/ethconfig"
+	"github.com/ethereum/go-ethereum/grpc/execution"
 	"github.com/ethereum/go-ethereum/internal/ethapi"
 	"github.com/ethereum/go-ethereum/internal/flags"
 	"github.com/ethereum/go-ethereum/internal/version"
@@ -201,6 +202,14 @@ func makeFullNode(ctx *cli.Context) (*node.Node, ethapi.Backend) {
 	if ctx.IsSet(utils.GraphQLEnabledFlag.Name) {
 		utils.RegisterGraphQLService(stack, backend, filterSystem, &cfg.Node)
 	}
+
+	// Configure gRPC if requested.
+	if ctx.IsSet(utils.GRPCEnabledFlag.Name) {
+		serviceV1a1 := execution.NewExecutionServiceServerV1Alpha1(eth)
+		serviceV1a2 := execution.NewExecutionServiceServerV1Alpha2(eth)
+		utils.RegisterGRPCExecutionService(stack, serviceV1a1, serviceV1a2, &cfg.Node)
+	}
+
 	// Add the Ethereum Stats daemon if requested.
 	if cfg.Ethstats.URL != "" {
 		utils.RegisterEthStatsService(stack, backend, cfg.Ethstats.URL)

cmd/geth/main.go

@@ -176,6 +176,9 @@ var (
 		utils.AllowUnprotectedTxs,
 		utils.BatchRequestLimit,
 		utils.BatchResponseMaxSize,
+		utils.GRPCEnabledFlag,
+		utils.GRPCHostFlag,
+		utils.GRPCPortFlag,
 	}
 
 	metricsFlags = []cli.Flag{

cmd/utils/flags.go

@@ -55,6 +55,8 @@ import (
 	"github.com/ethereum/go-ethereum/ethdb/remotedb"
 	"github.com/ethereum/go-ethereum/ethstats"
 	"github.com/ethereum/go-ethereum/graphql"
+	executionv1a1 "github.com/ethereum/go-ethereum/grpc/gen/astria/execution/v1alpha1"
+	executionv1a2 "github.com/ethereum/go-ethereum/grpc/gen/astria/execution/v1alpha2"
 	"github.com/ethereum/go-ethereum/internal/ethapi"
 	"github.com/ethereum/go-ethereum/internal/flags"
 	"github.com/ethereum/go-ethereum/les"
@@ -733,6 +735,24 @@ var (
 		Usage:    "Enables the (deprecated) personal namespace",
 		Category: flags.APICategory,
 	}
+	// grpc
+	GRPCEnabledFlag = &cli.BoolFlag{
+		Name:     "grpc",
+		Usage:    "Enable the gRPC server",
+		Category: flags.APICategory,
+	}
+	GRPCHostFlag = &cli.StringFlag{
+		Name:     "grpc.addr",
+		Usage:    "gRPC server listening interface",
+		Value:    node.DefaultGRPCHost,
+		Category: flags.APICategory,
+	}
+	GRPCPortFlag = &cli.IntFlag{
+		Name:     "grpc.port",
+		Usage:    "gRPC server listening port",
+		Value:    node.DefaultGRPCPort,
+		Category: flags.APICategory,
+	}
 
 	// Network Settings
 	MaxPeersFlag = &cli.IntFlag{
@@ -1175,6 +1195,19 @@ func setHTTP(ctx *cli.Context, cfg *node.Config) {
 	}
 }
 
+// setGRPC creates the gRPC RPC listener interface string from the set command
+// line flags, returning empty if the gRPC endpoint is disabled.
+func setGRPC(ctx *cli.Context, cfg *node.Config) {
+	if ctx.Bool(GRPCEnabledFlag.Name) {
+		if ctx.IsSet(GRPCHostFlag.Name) {
+			cfg.GRPCHost = ctx.String(GRPCHostFlag.Name)
+		}
+		if ctx.IsSet(GRPCPortFlag.Name) {
+			cfg.GRPCPort = ctx.Int(GRPCPortFlag.Name)
+		}
+	}
+}
+
 // setGraphQL creates the GraphQL listener interface string from the set
 // command line flags, returning empty if the GraphQL endpoint is disabled.
 func setGraphQL(ctx *cli.Context, cfg *node.Config) {
@@ -1415,6 +1448,7 @@ func SetNodeConfig(ctx *cli.Context, cfg *node.Config) {
 	SetP2PConfig(ctx, &cfg.P2P)
 	setIPC(ctx, cfg)
 	setHTTP(ctx, cfg)
+	setGRPC(ctx, cfg)
 	setGraphQL(ctx, cfg)
 	setWS(ctx, cfg)
 	setNodeUserIdent(ctx, cfg)
@@ -1948,6 +1982,14 @@ func RegisterGraphQLService(stack *node.Node, backend ethapi.Backend, filterSyst
 	}
 }
 
+// RegisterGRPCExecutionService adds the gRPC API to the node.
+// It was done this way so that our grpc execution server can access the ethapi.Backend
+func RegisterGRPCExecutionService(stack *node.Node, execServerV1a1 executionv1a1.ExecutionServiceServer, execServerV1a2 executionv1a2.ExecutionServiceServer, cfg *node.Config) {
+	if err := node.NewGRPCServerHandler(stack, execServerV1a1, execServerV1a2, cfg); err != nil {
+		Fatalf("Failed to register the gRPC service: %v", err)
+	}
+}
+
 // RegisterFilterAPI adds the eth log filtering RPC API to the node.
 func RegisterFilterAPI(stack *node.Node, backend ethapi.Backend, ethcfg *ethconfig.Config) *filters.FilterSystem {
 	isLightClient := ethcfg.SyncMode == downloader.LightSync

consensus/ethash/consensus.go

@@ -234,7 +234,6 @@ func (ethash *Ethash) verifyHeader(chain consensus.ChainHeaderReader, header, pa
 	}
 	// Verify the block's difficulty based on its timestamp and parent's difficulty
 	expected := ethash.CalcDifficulty(chain, header.Time, parent)
-
 	if expected.Cmp(header.Difficulty) != 0 {
 		return fmt.Errorf("invalid difficulty: have %v, want %v", header.Difficulty, expected)
 	}

core/blockchain.go

@@ -319,10 +319,10 @@ func NewBlockChain(db ethdb.Database, cacheConfig *CacheConfig, genesis *Genesis
 		return nil, ErrNoGenesis
 	}
 
-	bc.currentBlock.Store(nil)
-	bc.currentSnapBlock.Store(nil)
-	bc.currentFinalBlock.Store(nil)
-	bc.currentSafeBlock.Store(nil)
+	bc.currentBlock.Store(bc.genesisBlock.Header())
+	bc.currentSnapBlock.Store(bc.genesisBlock.Header())
+	bc.currentFinalBlock.Store(bc.genesisBlock.Header())
+	bc.currentSafeBlock.Store(bc.genesisBlock.Header())
 
 	// Update chain info data metrics
 	chainInfoGauge.Update(metrics.GaugeInfoValue{"chain_id": bc.chainConfig.ChainID.String()})

core/txpool/txpool.go

@@ -72,6 +72,8 @@ type TxPool struct {
 
 	subs event.SubscriptionScope // Subscription scope to unsubscribe all on shutdown
 	quit chan chan error         // Quit channel to tear down the head updater
+
+	astria  *astriaOrdered
 }
 
 // New creates a new transaction pool to gather, sort and filter inbound
@@ -356,6 +358,108 @@ func (p *TxPool) Stats() (int, int) {
 	return runnable, blocked
 }
 
+func (pool *TxPool) SetAstriaOrdered(rawTxs [][]byte) {
+	txs := []*types.Transaction{}
+	for idx, rawTx := range rawTxs {
+		tx := new(types.Transaction)
+		err := tx.UnmarshalBinary(rawTx)
+		if err != nil {
+			log.Warn("failed to unmarshal raw astria tx bytes", rawTx, "at index", idx, "error:", err)
+			continue
+		}
+
+		err = pool.astriaValidate(tx)
+		if err != nil {
+			log.Warn("astria tx failed validation at index", idx, "error:", err)
+			continue
+		}
+
+		txs = append(txs, tx)
+	}
+
+	pool.astria = newAstriaOrdered(types.Transactions(txs))
+}
+
+func (pool *TxPool) ClearAstriaOrdered() {
+	if pool.astria == nil {
+		return
+	}
+	pool.astria.clear()
+}
+
+func (pool *TxPool) AstriaOrdered() *types.Transactions {
+	// sus but whatever
+	if pool.astria == nil {
+		return &types.Transactions{}
+	}
+	return &pool.astria.txs
+}
+
+// validateTx checks whether a transaction is valid according to the consensus
+// rules and adheres to some heuristic limits of the local node (price and size).
+func (pool *TxPool) astriaValidate(tx *types.Transaction) error {
+	// Accept only legacy transactions until EIP-2718/2930 activates.
+	if !pool.eip2718 && tx.Type() != types.LegacyTxType {
+		return core.ErrTxTypeNotSupported
+	}
+	// Reject dynamic fee transactions until EIP-1559 activates.
+	if !pool.eip1559 && tx.Type() == types.DynamicFeeTxType {
+		return core.ErrTxTypeNotSupported
+	}
+	// Reject transactions over defined size to prevent DOS attacks
+	if tx.Size() > txMaxSize {
+		return ErrOversizedData
+	}
+	// Check whether the init code size has been exceeded.
+	if pool.shanghai && tx.To() == nil && len(tx.Data()) > params.MaxInitCodeSize {
+		return fmt.Errorf("%w: code size %v limit %v", core.ErrMaxInitCodeSizeExceeded, len(tx.Data()), params.MaxInitCodeSize)
+	}
+	// Transactions can't be negative. This may never happen using RLP decoded
+	// transactions but may occur if you create a transaction using the RPC.
+	if tx.Value().Sign() < 0 {
+		return ErrNegativeValue
+	}
+	// Ensure the transaction doesn't exceed the current block limit gas.
+	if pool.currentMaxGas < tx.Gas() {
+		return ErrGasLimit
+	}
+	// Sanity check for extremely large numbers
+	if tx.GasFeeCap().BitLen() > 256 {
+		return core.ErrFeeCapVeryHigh
+	}
+	if tx.GasTipCap().BitLen() > 256 {
+		return core.ErrTipVeryHigh
+	}
+	// Ensure gasFeeCap is greater than or equal to gasTipCap.
+	if tx.GasFeeCapIntCmp(tx.GasTipCap()) < 0 {
+		return core.ErrTipAboveFeeCap
+	}
+	// Make sure the transaction is signed properly.
+	from, err := types.Sender(pool.signer, tx)
+	if err != nil {
+		return ErrInvalidSender
+	}
+	// Ensure the transaction adheres to nonce ordering
+	if pool.currentState.GetNonce(from) > tx.Nonce() {
+		return core.ErrNonceTooLow
+	}
+	// Transactor should have enough funds to cover the costs
+	// cost == V + GP * GL
+	balance := pool.currentState.GetBalance(from)
+	if balance.Cmp(tx.Cost()) < 0 {
+		return core.ErrInsufficientFunds
+	}
+	// Ensure the transaction has more gas than the basic tx fee.
+	intrGas, err := core.IntrinsicGas(tx.Data(), tx.AccessList(), tx.To() == nil, true, pool.istanbul, pool.shanghai)
+	if err != nil {
+		return err
+	}
+	if tx.Gas() < intrGas {
+		return core.ErrIntrinsicGas
+	}
+	return nil
+}
+
 // Content retrieves the data content of the transaction pool, returning all the
 // pending as well as queued transactions, grouped by account and sorted by nonce.
 func (p *TxPool) Content() (map[common.Address][]*types.Transaction, map[common.Address][]*types.Transaction) {
@@ -415,3 +519,17 @@ func (p *TxPool) Status(hash common.Hash) TxStatus {
 	}
 	return TxStatusUnknown
 }
+
+type astriaOrdered struct {
+	txs types.Transactions
+}
+
+func newAstriaOrdered(txs types.Transactions) *astriaOrdered {
+	return &astriaOrdered{
+		txs: txs,
+	}
+}
+
+func (ao *astriaOrdered) clear() {
+	ao.txs = *&types.Transactions{}
+}

eth/catalyst/api.go

@@ -172,6 +172,7 @@ func newConsensusAPIWithoutHeartbeat(eth *eth.Ethereum) *ConsensusAPI {
 // If there are payloadAttributes: we try to assemble a block with the payloadAttributes
 // and return its payloadID.
 func (api *ConsensusAPI) ForkchoiceUpdatedV1(update engine.ForkchoiceStateV1, payloadAttributes *engine.PayloadAttributes) (engine.ForkChoiceResponse, error) {
+	log.Info("ForkchoiceUpdatedV1 called")
 	if payloadAttributes != nil {
 		if payloadAttributes.Withdrawals != nil {
 			return engine.STATUS_INVALID, engine.InvalidParams.With(errors.New("withdrawals not supported in V1"))

genesis.json

@@ -0,0 +1,22 @@
+{
+    "config": {
+        "chainId": 1337,
+        "homesteadBlock": 0,
+        "eip150Block": 0,
+        "eip155Block": 0,
+        "eip158Block": 0,
+        "byzantiumBlock": 0,
+        "constantinopleBlock": 0,
+        "petersburgBlock": 0,
+        "istanbulBlock": 0,
+        "berlinBlock": 0,
+        "londonBlock": 0,
+        "terminalTotalDifficulty": 0,
+        "ethash": {}
+    },
+    "difficulty": "10000000",
+    "gasLimit": "8000000",
+    "alloc": {
+        "0x46B77EFDFB20979E1C29ec98DcE73e3eCbF64102": { "balance": "300000000000000000000" }
+    }
+}