From 828fc10ef96b643d8009f88a7114f1bf186996bc Mon Sep 17 00:00:00 2001 From: Sourabh Niyogi Date: Mon, 2 Oct 2023 19:09:01 -0700 Subject: [PATCH 1/8] RFC-0033: Support Non-Substrate EVM L2 Networks MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Polkadot+Kusama should support the possibility of having up to 10-30% of its blockspace weight allocatable to EVM and WASM Smart Contracts. This is not for Polkadot to be a yet another EVM Chain, but specifically to: 1. support the use of Polkadot's Data Availability resources to non-Substrate [largely EVM] L2 Stacks, which will bring in additional demand for DOT and through those networks, new users and developers 2. (assuming CoreJam Work Packages have some direct relation to EVM Contracts + WASM Contract *Interpretation*), support Polkadot 2.0 experimentation in its transformation into a “map reduce” computer and bring in new classes of CoreJam+CorePlay developers in addressing sync/asynchronous composability --- text/0033-support-non-substrate-evm-l2.md | 237 ++++++++++++++++++++++ 1 file changed, 237 insertions(+) create mode 100644 text/0033-support-non-substrate-evm-l2.md diff --git a/text/0033-support-non-substrate-evm-l2.md b/text/0033-support-non-substrate-evm-l2.md new file mode 100644 index 000000000..fbb9adb23 --- /dev/null +++ b/text/0033-support-non-substrate-evm-l2.md @@ -0,0 +1,237 @@ +# RFC-0033: Support Non-Substrate EVM L2 Networks + +| | | +|--|--| +| **Start Date** | 3 October 2023 | +| **Description** | Support Non-Substrate EVM L2 Networks using Polkadot Data Availability by adding EVM/WASM Contracts Pallets to Polkadot Relay Chain, limited to 10+20=30% of blockspace | +| **Authors** | Sourabh Niyogi | + + +## Summary + +Polkadot+Kusama should support the possibility of having up to 10-30% of its blockspace weight allocatable to EVM and WASM Smart Contracts. This is not for Polkadot to be a yet another EVM Chain, but specifically to: +1. support the use of Polkadot's Data Availability resources to non-Substrate [largely EVM] L2 Networks, which will bring in additional demand for DOT and through those networks, new users and developers +2. (assuming CoreJam Work Packages have some direct relation to EVM Contracts + WASM Contract *Interpretation*) support Polkadot 2.0 experimentation in its transformation into a “map reduce” computer and bring in new classes of CoreJam+CorePlay developers in addressing synchronous/asynchronous composability + +## Motivation + +Polkadot 1.0's approach to scalability since inception has been as a platform for heterogenous shards or "app chains". Asynchronous interoperability between shards was enabled through XCM through the relay chain, including two teams (Moonbeam/Moonriver, Astar/Shiden) who successfully led Substrate-first EVM Chains in 2020-2023, promoting EVM + WASM Contract platforms widely to a large user and developer community. Contemporaneously, in 2020-2023, Ethereum developed a [roll-up centric roadmap](https://ethereum-magicians.org/t/a-rollup-centric-ethereum-roadmap/4698). The dominant use of Ethereum's L2 rollups has been to replicate Ethereum's capability at lower costs, enabled by: +* using Ethereum as a storage layer, recording the L2 activity to be able to derive 100% of the L2 state and support fraud proof +* using EVM Contracts deployed on L1 Ethereum and L2 chain to manage asset transfers +* using (rarely used but necessary) EVM Contracts fraud + ZK proofs on L1 to manage security of the L2 chain. + +The success of these Ethereum L2 rollups by common measures (TVL) are seen on [L2Beat](https://l2beat.com/scaling/summary) and can be seen to exceed to 100% of Polkadot parachains with similar age and functionality (Moonbeam + Astar). The bottom line is that Arbitrum and Optimism and many OP Stack chains now dominate with over 85% of TVL that is 100x that of Polkadot, including Base (built on OP Stack) launched in July 2023. The compute cost improvements relative to Ethereum L1 are immediately visible, while storage gas costs (1 gas per byte) remain. Ethereum's [EIP-4844](https://www.eip4844.com/) introduces a new "Blob" transaction type to Ethereum that enables L2 data to be persisted in the beacon node for a short period of time, and is expected to reduce the storage cost of a L2 Transaction (recorded on L1) by 3-8x. + +This proposal is about having Polkadot's DA layer serve as a resource for non-Substrate L2 networks to use. It is believed that: +1. Polkadot DA is already likely technically superior to Ethereum's EIP-4844 solution and can offer users and non-Substrate L2 Stack devs competitive storage and end users lower costs than Ethereum L1; +2. Astar + Moonbeam's efforts in 2021-2023 paved the way for EVM+WASM Contracts to be deployed on the relay chain in 2024; +3. CoreJam's CRJA will correctly treat synchronous (contract) and asynchronous (XCM) composability patterns in Polkadot 2.0 in the next 12-18 months + +The sizing of Ethereum L2 using Ethereum can be culled from [BigQuery Public Datasets](https://twitter.com/Polkadot/status/1707052392712212676) with: +``` +SELECT count(*), to_address, sum(length(input))/(2000000000) as gbPerMonth, sum(length(input))/(3600*24*30*2000000000) as kbPerSecond FROM `bigquery-public-data.crypto_ethereum.transactions` WHERE TIMESTAMP_TRUNC(block_timestamp, DAY) >= TIMESTAMP("2023-09-01") and TIMESTAMP_TRUNC(block_timestamp, DAY) <= TIMESTAMP("2023-09-30") group by to_address order by l desc limit 10000``` +``` + +The results of Ethereum L1 September 2023 activity are compiled [here](https://docs.google.com/spreadsheets/d/1ng2-dfPxCKCelI8_OmdqTnezYWHM0hODVarQNd7-oCU/edit#gid=1164282259). Here is a summary of the top 14 (in approximate storage bytes) Ethereum L2-related activity from that compilation: + +| Contract | numTxs/mo | GB/mo | K/s | Approx % | +|------------------------------------------------------|-----------|-------|------|----------| +| [Arbitrum: Sequencer Inbox](https://etherscan.io/address/0x1c479675ad559dc151f6ec7ed3fbf8cee79582b6) | 27,839 | 2.75 | 1.06 | 8.67% | +| [Optimism: Sequencer](https://etherscan.io/address/0xff00000000000000000000000000000000000010) | 22,966 | 2.22 | 0.86 | 7.01% | +| [Zksync Era: Validator Timelock](https://etherscan.io/address/0x3db52ce065f728011ac6732222270b3f2360d919) | 114,460 | 6.66 | 2.57 | 21.01% | +| [Base: Batch Inbox](https://etherscan.io/address/0xff00000000000000000000000000000000008453) | 47,240 | 2.81 | 1.08 | 8.85% | +| [StarkNet: SHARP](https://etherscan.io/address/0xfd14567eaf9ba941cb8c8a94eec14831ca7fd1b4) | 202,389 | 6.00 | 2.32 | 18.93% | +| [StarkNet](https://etherscan.io/address/0xc662c410c0ecf747543f5ba90660f6abebd9c8c4) | 93,340 | 0.13 | 0.05 | 0.42% | +| [StarkNet: FriStatementContract](https://etherscan.io/address/0x3e6118da317f7a433031f03bb71ab870d87dd2dd) | 5,940 | 0.07 | 0.03 | 0.23% | +| [Starkware: Sharp Verifier](https://etherscan.io/address/0x47312450b3ac8b5b8e247a6bb6d523e7605bdb60) | 790 | 0.06 | 0.02 | 0.20% | +| [Linea: L1 Message Service](https://etherscan.io/address/0xd19d4b5d358258f05d7b411e21a1460d11b0876f) | 136,648 | 2.30 | 0.89 | 7.25% | +| [Polygon: Polygon zkEVM Proxy](https://etherscan.io/address/0x5132a183e9f3cb7c848b0aac5ae0c4f0491b7ab2) | 6,383 | 0.46 | 0.18 | 1.46% | +| [Mantle](https://etherscan.io/address/0xd1328c9167e0693b689b5aa5a024379d4e437858) | 2,427 | 0.23 | 0.09 | 0.72% | +| [Zksync](https://etherscan.io/address/0xabea9132b05a70803a4e85094fd0e1800777fbef) | 86,694 | 0.16 | 0.06 | 0.51% | +| [Zksync](https://etherscan.io/address/0xf8a16864d8de145a266a534174305f881ee2315e) | 8,264 | 0.14 | 0.06 | 0.45% | +| [Zksync](https://etherscan.io/address/0x32400084c286cf3e17e7b677ea9583e60a000324) | 357,981 | 0.11 | 0.04 | 0.33% | +| **Top 14 Contracts from L2s** | **1,113,361** | **24.12** | **9.30** | **76.05%** | +| **All Contracts** | | **31.71** | | | | + + + +The above 14 Contracts make up **over 76%** of the input data bytes, handily surpassing the volumes of the well-known Ethereum contracts (from uniswap, 1inch, blur, etc.). It is *this* source of robust storage demand that Polkadot should tap into and extend security from Substrate chains to non-Substrate chains. + +According to this [Polkadot DA vs competition](https://forum.polkadot.network/t/polkadot-da-vs-competition/3403) discussion from @pepyakin + @bkchr, Polkadot can _presently_ support 0.83KiB/s per core, which at 100 cores implies 83-166 MiB/s. Thus, if I understand this discussion correctly, Polkadot can _trivially_ support Ethereum's present storage throughput of 12.25K/s, and handily support 100x of growth for non-Substrate EVM L2s like the ones above. If this is correct, then Polkadot DA is a diamond in the rough, just waiting to be used for EVM L2 scalability. + +However, a high throughput Polkadot's Data Availability layer is not enough to support non-Substrate EVM L2 networks. EVM Contract functionality must be deployed on Polkadot as well to support: +* deposits from L1 => L2 and withdrawals from L2 => L1 +* fraud proofs (e.g. in Optimism and Base) + validity proofs (e.g. in zkSync Era and StarkNet) + +Both DA + EVM Contract functionality are for non-Substrate EVM L2 Businesses to run entirely on Polkadot (in the same way as Moonbeam and Astar have and did until Astar zkEVM). Fortunately, Polkadot leading parachains Moonbeam and Astar already have explored EVM thoroughly in a Substrate architecture, enabling the `ethereum` pallet to be used on the Polkadot Relay Chain. + +We do not propose that Polkadot become yet another EVM Chain for users and arbitrary Smart Contract developers. Instead, we propose that Polkadot _supports_ non-Substrate EVM L2s, just as it always done -- the difference is an expansion in focus from Substrate-only L2s to include non-Substrate L2s. + +This addition, it is hoped, may also support Polkadot 2.0’s “Map Reduce”/Collect-Refine-Join-Accumulate architecture, which envisions a different way of addressing synchronous + asynchronous composability other than Ethereum's historical synchronous-first approach and Polkadot's asynchrous-first approach to scalability. By adding both EVM+WASM Contract pallets in 2024 coincident with CoreJam we enable both revenue growth and user/developer growth. + +## Stakeholders + +- Current and Prospective EVM L2 Chain Business Owners +- EVM L2 Stack Owners +- Substrate-native Parachains Smart Contract Platforms (Moonbeam, Astar) +- Substrate-native Parachains +- Core protocol and XCM format developers +- Chain-as-Service companies +- Tooling, block explorer, and UI developers +- CoreJam/Coreplay developers +- EVM and ink!/WASM Contract Developers + +## Proposal + +### 1. Add `ethereum` Pallet to Relay Chain + +This is to support drop-in EVM Contract Deployment of: +* OP Stack Contracts +* Arbitrum Contracts +* StarkNet Contracts +* ... others that depend on L1 EVM Contracts + +for: +* L1-L2 Bridges +* Rollup Fraud proofs +* ZK Validity proofs + +### 2. Add `contracts` Pallet to Relay Chain + +This is to support WASM Contract equivalents of the above, which we would hope enjoy improved efficiency relative to their EVM Contract counterparts. However, this is secondary in importance until the connection to CoreJam Work Packages is realized. + +### 3. Impose per-block weight limits of 30% for `ethereum` and `contracts` in aggregate, and 10% for `ethereum` and 20% `contracts` specifically + +This approach to having a limited amount of blockspace allocated for smart contracts is suggested by Polkadot senior fellows: +1. @rphmeier [here](https://www.rob.tech/blog/hybrid-chains/) "What the hybrid-chain approach is likely to look like in practice is application-specific logic deployed alongside smart contracts, **with a limited amount of blockspace allocated to smart contracts**. Given that contracts would only be able to consume some of the resources per-block, the absolute costs of gas would be higher relative to chains that focus entirely on smart contracts, all else equal." +2. @joe [here](https://forum.polkadot.network/t/permissioned-pallet-contracts-deployment-to-asset-hub-for-defi-primitives/3908/3) "For interactions that absolutely **must be synchronous** … then we could explore the idea of hybrid blockspace where **we set aside say 10% of the blockspace for contract execution** If there is a lot of demand for this tranche of blockspace, then the gas price will increase a lot, motivating contract developers to migrate to a contract chain or rethink their application architecture to work asynchronously." + +The 30% limit is proposed so that Polkadot may continue to fulfill security obligations to its Bulk Coretime customers. It is believed that 10% would be too small to be taken seriously by new and existing EVM L2 Business owners (when compared to Ethereum) while 30% is ample empirically. Significantly higher than 50% (or having no limit) would cannibalize BCC and threaten new ICC customers. + +The 10% limit on `ethereum` and 20% limit on `contracts` is to incentivize higher performance translations of L1 EVM Contracts to be done with WASM Contracts instead. It is not clear if these higher performance WASM Contract translations are worth translating, especially given the critical importance of fraud proofs, relative immature of ink! vs Solidity (perceived or real). + +This can be equally weighted (15%/15%) or omitted altogether, especially if difficult to implement technically, or stand as the primary mechanism. + +However, something like the total being around 30% should be non-negotiable. + +### 4. Adjust Storage Costs on Polkadot to provide an 2-5x cost difference over Ethereum L2 Chains [Base] backed by Polkadot DA/Security but 25-50% higher than Moonbeam+Astar. + +It is widely expected in both Ethereum and Polkadot ecosystems that everyday users will interact with L2s rather than L1. However, for EVM L2s built on Polkadot DA to remain competitive with Ethereum's EIP-4844, it is proposed that Polkadot's DA to having storage costs be set to be low enough so as to have non-Substrate Polkadot EVM L2 user costs be visibly lower than an Ethereum EVM L2. On the other hand, end user pricing for Polkadot L1 activity should be higher than both Astar and Moonbeam. This treats Polkadot as a storage wholesaler competing with Ethereum but not Astar and Moonbeam as retailer. + +In Ethereum, storage costs for L1 is 1 gas per byte. [Here](https://etherscan.io/tx/0xef2ea6e6a980c05a80e5b03ef60a070415e1f53edb01001d25fdb9841c3c7fd6) is a October 2023 Optimism L2=>L1 (0.0271 ETH, $45.09) writing 115,733 bytes, representing approximately 100-200 Optimism transactions at an average of $.22-.45/tx like [this](https://optimistic.etherscan.io/tx/0x64defec295aebad526a925d198a0ed63bcae2de40556c383281c4fd095dd2360). + +In Polkadot, at present, using the `preimage.notePreimage` the exact 115,733 bytes costs 11.58 DOT (@$4.12 = $47.59), which does not offer the order of magnitude improvements desired. + +With Protodanksharding in EIP-4844 expected to offer a different kind of storage on Execution/Beacon layer, it is expected to be 3-8x lower. So for Polkadot to aim for a competitive pricing, a 10x reduction ($4.50) should be sought for. + +Needless to say, this sort of permissionless storage is directly analogous to the permissioned storage of AWS S3 vs GCP GS: It is necessary to conduct the same kind of commodities storage pricing in DOT vs ETH, perceived in users mind in USD terms. If a storage chain or alternate Storage API is required to meet the L2 end users pricing perception, a concrete proposal is needed to meet the EVM L2 Business Owners goals that serve those users. + +At the end of the day, EVM L2 Chain Owners need pricing consistency from their suppliers to run a healthy business that L2 users can depend on. A good supply chain is not overly dependent on one supplier. An ideal outcome could be that EVM L2 costs is not dominated by a single L1 costs as it now but run on multiple storage systems so that true resilience is achieved. In the meantime, we need Polkadot DA to be 10x lower on its own. + +### 5. Match Moonbeam+Astar precompiles, but keep Existential Deposits as is. + +The [precompiles](https://docs.astar.network/docs/build/EVM/precompiles/) of Astar and [xcDOT](https://docs.astar.network/docs/learn/interoperability/asset-list/) are necessary to support drop-in replacement for DOT withdrawal/deposits. It is not desirable for Polkadot to be the home of thousands of ERC20 assets but the xcDOT case must be covered. Otherwise, a permissionless design (not requiring governance) that is compatible with Assethub is desired but requires a more sophisticated plan of how XCM could be situated. + +Moonbeam is currently facing significant storage bloat due to having no storage deposit. Having the 1 DOT Existential Deposit would trivially address this. + +Polkadot fellows have significant deep knowledge of how Ethereum's gas model is superficially too simple and would benefit from a higher dimensional model of weights (see [here](https://forum.polkadot.network/t/weight-v2-discussion-and-updates/227/5)). A engineering path to bring in a more sophisticated model targeting the EVM L2 Stack developer may be desirable. + +## Explanation + +### Interfaces + +We assume the Balances pallet will continue to exist to support drop-in replacement of EVM L1 Contracts. + +We imagine CoreJam's Collect/Refine/Join/Accumulate architecture should interface with both EVM+WASM Contracts. + +### Deployment + +The importance of Polkadot DA being used for EVM L2 is critical to pursue with high urgency to have high impact in 2024. Here is an adhoc timeline: + +* Rococo - December 2023 +* Kusama - Spring 2024 +* Polkadot - Summer 2024 + +## Cultural Change: Synchronous and Asynchronous Patterns, and the Role of EVM + WASM Smart Contracts + +There is a tremendous amount of anti-EVM feelings in the Polkadot community over any use of Ethereum technology that is not running on a Substrate stack. It is not that hard to see how beautifully connected Astar + Moonbeam are to Polkadot Relay Chain, relatively to how kludgily Arbitrum + Optimism + Base are connected to Ethereum chain. Polkadot's XCM is visibly better than similarly kludgy messaging systems in Ethereum. First and foremost, for this proposal to succeed, **cultural change is needed**. Leadership of Polkadot must dispel Substrate-first tribalism (or similar Rust/ink!/WASM tribalism vs Solidity/EVM) and *welcome* non-Substrate activity in all its forms. This is very easy for outsiders (like myself) and extremely hard for insiders. + +On a more technical level, Polkadot architecture has systematically adopted the following choices: +1. a very specific thesis that having multiple asynchronous chains (system chains or parachain) is the way to address scalability; +2. a systematic taste for "separation of concerns", widely adopted in software engineering, motivating a whole parachain for { assets (Asset Hub), identity (Encointer), staking, storage ... } + +You can see this drive in PR #32's Proposal for Minimal Relay, which even proposes to remove *Balances* from the relay chain and put it into Asset Hub. Such a move would significantly increase the complexity of deployment of EVM L2 chains and require async patterns for everything. + +The simplicity of Ethereum's synchronous-first programming model is now up against the complexity of Polkadot's asynchronous-first programming model. + +CoreJam "Map reduce" programming model, being developed by Polkadot founders+senior fellows, will force a reconciliation between these two artificial extremes: +* Ethereum's synchronous-first model, embodied in EVM Contracts on L1+L2 +* Polkadot's asynchronous-first model, embodied in [PR #32 - Minimal Relay](https://github.com/polkadot-fellows/RFCs/pull/32) + +Technologists (as least me, as a relative outsider with little tribal allegiance) know that Ethereum's synchronous-first model tends towards scalability failure (L2s become mostly disconnected islands, themselves incapable of scaling due to the lack of composable async), while Polkadot's asynchronous-first model tends toward unusability (users and developers must figure out what is going on many chains). + +If we open Polkadot up to non-Substrate L2 activity aggressively, this will enable Polkadot to experience real user + developer growth on these new non-Substrate L2 chains. Anything less probably will cause Polkadot to be like Apple Lisa, Amiga, NeXT in the 1980s and early 90s: great technology, pursued with great engineering and product taste, but ultimately never obtaining the kind of ubiquitous computer imagined relative to Windows PCs and Macs that dominated and survived to the present day. + +To be clear, this proposal is not about turning Polkadot into yet another EVM chain. Instead, I believe Rob Habermeier's appeal to **hybrid chains** is the pragmatically correct answer for Polkadot in the near future: + +![](https://europe1.discourse-cdn.com/standard21/uploads/polkadot2/original/2X/7/7cca3339cc4eb1ab3971d6fe10280d4469510ea3.png) + +If we take Rob's advice of "All chains should have smart contracts" seriously, then the Polkadot Relay Chain should include Smart Contracts. L2 Stack Developers (in a wide variety of rollup architectures from OP Stack to a new ZK rollup stack) should be able to use Polkadot's DA + EVM+WASM Contract platform freely as a hybrid chain. Rather than overengineer something for a specific rollup stack, Polkadot should offer the best DA it can with competitive storage, offer the best VMs to support L2 Stack Developers generally. + +Rather than aggressively stub out everything into N system chains and force unnecessary async patterns, I believe Polkadot should thoroughly develop its solution to synchronous + asynchronous composability at the center of CoreJam's Map reduce architecture, and extend it to EVM Contract Interpretation which enjoy a mature large developer community outside Substrate. Assuming cultural change is possible, by opening the door to non-Substrate ecosystems, we should bring in new users through the L2 stacks (entirely EVM now, but potentially other VMs in the future) and position Polkadot as being more technology neutral between Substrate vs non-Substrate, EVM vs WASM, ink! vs Coreplay, or whatever. + + +## Testing, Security, and Privacy + +Standard audit/review would be appropriate for Polkadot-deployed L1 EVM Contracts. + +OP Stack is the lowest complexity EVM L2 Chain and appears to be enjoying some network effects in 2023. + +An OP Deployment could be explored with Shibuya/Shiden/Astar, Moonbase/Moonriver/Moonbeam in parallel to a Rococo+Westend/Kusama/Polkadot. + +Other L2 stacks could be explored. + +## Performance, Ergonomics, and Compatibility + +### Performance + +Significant parametric choices are being made that will have long term consequences. By enabling EVM + WASM Contracts on the relay chain, new *long-term* storage requirements are important to satisfy to EVM L2 Business Owners. This long-term storage may justify a storage chain in the future. A hybrid short-term and long-term storage solution is already present in Ethereum's EIP-4844, and it is likely necessary for Polkadot to be informed by its performance characteristics for competitiveness. Polkadot fellows should be able to offer specific plans for success. + +### Ergonomics + +EVM + WASM Contract Developers should be able to interact with Polkadot EVM Contracts in the same way as they do as they do on Astar. + +### Compatibility + +This proposal is incompatible with PR #32 Minimal Relay Chain. + +It is not clear how a new Storage Chain and CoreJam's Storage API relates to this. + +## Prior Art and References + +Numerous EVM L2 Chains exist already, here are the leading ones: +- [OP Stack](https://github.com/ethereum-optimism/optimism/blob/develop/packages/contracts-bedrock/src/L1/L1StandardBridge.sol) +- [zkSync Era](https://github.com/matter-labs/era-contracts/tree/main) +- [Arbitrum Stack](https://docs.arbitrum.io/for-devs/useful-addresses) +- StarkNet + +Several competing DA stacks exist: +- [ethStorage](https://eth-store.w3eth.io/#/) +- [Eigenlayer](https://docs.mantle.xyz/network/introduction/concepts/data-availability) used by Mantle + + +## Major Unresolved Questions + +* Is the Polkadot DA technically capable of supporting permanent long-term storage needs of rollups? If so, should it be allocated to EVM L2 Chains? What is the connection to storage chains? + + * Can Polkadot Fellows decide between the minimalism relay chain vs hybrid chains? What is the connection to storage chains and how to manage short-term (24hr - 2 weeks) and long-term storage (full derivation of an L2 from L1 storage)? + +* How does CoreJam's map-reduce architecture (that addresses synch/async patterns) relate to EVM + WASM Contracts "sync" pattern? + +* Will Polkadot fellows able to overcome their tribal instincts and well-informed technical biases against Ethereum and support non-Substrate EVM Chains and enable Polkadot to support its own revenue growth, user growth and developer user growth? + +## Future Directions and Related Material + +Assuming Optimistic rollups can be backed by Polkadot security, we believe ZK Rollups would be natural to develop as well. This would treat EVM L2 Chains largely identical to the way Ethereum L1 treats its L2 chains: relatively disconnected islands with centralized bridges (CCTP) and slow kludgy messaging protocols like LayerZero and Axelar trying to mediate between them. + +However, a more exciting outcome that may be more transformational and revolutionary is to connect [CoreJam' map-reduce CRJA architecture](https://github.com/polkadot-fellows/RFCs/blob/gav-corejam/text/0031-corejam.md) and XCM to connect multiple non-Substrate EVM L2 Chains as well as the Substrate L2 Chains. It is hoped that XCM and this programming model can be combined in 2024 for Polkadot's solution to permission compute/storage capability to enjoy the recognition and impact it so clearly deserves. From b8649b1e0781e66555a3e0ba6983bc7679b72332 Mon Sep 17 00:00:00 2001 From: Sourabh Niyogi Date: Mon, 9 Oct 2023 04:23:35 -0700 Subject: [PATCH 2/8] Kusama Network 2.0: The Hybrid Chain Network --- text/0033-kusama-hybrid-relay-chain.md | 156 ++++++++++++++ text/0033-support-non-substrate-evm-l2.md | 237 ---------------------- 2 files changed, 156 insertions(+), 237 deletions(-) create mode 100644 text/0033-kusama-hybrid-relay-chain.md delete mode 100644 text/0033-support-non-substrate-evm-l2.md diff --git a/text/0033-kusama-hybrid-relay-chain.md b/text/0033-kusama-hybrid-relay-chain.md new file mode 100644 index 000000000..9afd68b58 --- /dev/null +++ b/text/0033-kusama-hybrid-relay-chain.md @@ -0,0 +1,156 @@ +# RFC-0033: Kusama Network 2.0: The Hybrid Chain Network + +| | | +|--|--| +| **Start Date** | 9 October 2023 | +| **Description** | Evolve Kusama 2.0 to be a Hybrid Relay Chain, specifically with Smart Contracts on the Relay Chain, elevating Kusama to the level of Ethereum. | +| **Authors** | Sourabh Niyogi | + +## Summary + +Polkadot+Kusama should pursue different usability and scalability: + +1. Polkadot 2.0 should be a [Minimal Relay Chain](https://github.com/polkadot-fellows/RFCs/pull/32), maximizing cores/resources for maximum scalability of CoreJam + CorePlay's asynchronous-first patterns + +2. Kusama 2.0 should be a [Hybrid Relay Chain](https://www.rob.tech/blog/hybrid-chains/), prioritizing developer+user friendliness over scalability with full backwards compatibility with 1.0 Blockchain technology: EVM+WASM Smart Contracts + +This solution elevates Kusama to the level of Ethereum while giving developers on-ramps to Polkadot's new scalability solutions. + +## Motivation + +Polkadot 1.0's approach to scalability since inception has been as a platform for heterogenous shards or "app chains". Asynchronous interoperability between shards was enabled through XCM through the relay chain, including two teams (Moonbeam/Moonriver, Astar/Shiden) who successfully led Substrate-first EVM (+ WASM) Smart Contract dev platforms in 2020-2023, promoting them widely to a large user and developer community. Almost all other chains ignored smart contract usage entirely, in favor of Substrate pallets. Still, when not using XCM asynchronous patterns (used largely just for cross-chain asset transfers), both sorts of approaches rely on developers pursuing a blockchain-centric state transition way of operating: everything has to be done within one block with sync patterns, whether using EVM/WASM Smart Contracts or Substrate Pallets. + +In Polkadot 2.0, the state transition function is refactored by bringing asynchronous patterns to the center with CoreJam's Map-Reduce/Collect-Refine-Join-Accumulate (CJRA) + CorePlay Actor programming models. This will take most of 2024 to develop, while most blockchain technology will continue to live on with 1.0 EVM Contracts, which have non-existent treatment of async patterns: messaging is bolted on by networks like Axelar and LayerZero. + +While Polkadot aims to be a massively scalable map reduce computer, in most cases, developers do NOT need scalability right away -- it is widely recognized that both devs+users find the multichain factorization unnecessarily complex, while technologists know that complexity is required for scalability. Few parachains have scaled their user base to more than a few hundred user per day, resulting in very poor "blockspace" usage. We are left with a puzzle: if Polkadot technology is fundamentally the world's best solution to async/sync patterns at the heart of the state transition function, but embodies complex patterns, how do we give developers an on-ramp? + +Rob Habermerier's [Hybrid Chains](https://www.rob.tech/blog/hybrid-chains/) provide the answer: + +![](https://europe1.discourse-cdn.com/standard21/uploads/polkadot2/original/2X/7/7cca3339cc4eb1ab3971d6fe10280d4469510ea3.png) + +By adding Smart Contracts to the Kusama Relay Chain, developers can scale from sync-first "old" blockchain technology to "new" in this spectrum: + +1. full sync patterns in Smart Contracts on the Relay Chain. _This pattern is manifested in Ethereum but has never been seen in Polkadot or Kusama._ +2. full sync patterns in Smart Contracts on a Parachain. _This pattern is seen in Astar + Moonbeam._ +3. full sync patterns in Substrate pallets on a Parachain. _This pattern is seen in all other parachains._ +4. hybrid sync + async patterns in Smart Contracts on a Relay Chain. _This pattern is manifested in Ethereum with Axelar/LayerZero._ +5. hybrid sync + async patterns in CoreJam + CorePlay. _This pattern is under active development and will appear in Kusama 2.0 + Polkadot 2.0._ +6. hybrid sync + async patterns in Substrate Pallets. _This pattern is seen in all other parachains._ + +Developers can _start_ at any point in the above spectrum and move upwards as their needs for scale increase, culminating in CoreJam/CorePlay and Substrate pallets in app chains. + +For "core" efficiency reasons, it is highly undesirable to load sync patterns (whether Smart Contracts or Substrate pallets) onto the Relay Chain: [RFC #32 Minimal Relay Chain](https://github.com/polkadot-fellows/RFCs/pull/32) takes this the logical extreme, moving Balances, Staking, Identity, Governance off of Polkadot, with Polkadot founder @gavofyork's strong conviction that "The Relay-chain should do one thing and do it well. That means being the head of a secure decentralised multicore computer. Sticking an EVM smart contract in there and/or continuing to allow end-user-application-level transactions and general interaction goes directly against this mantra." + +So, this proposal takes Kusama 2.0 to the other extreme of Polkadot 2.0, and + +_Elevates Kusama to the level of Ethereum._ + +By putting EVM+WASM Smart Contracts on the Relay Chain, Kusama can simultaneously pursue both leading ecosystem's path to scalability: + +1. Ethereum: [Scalability from rollups](https://ethereum-magicians.org/t/a-rollup-centric-ethereum-roadmap/4698) _and_ +2. Polkadot: [CoreJam](https://github.com/polkadot-fellows/RFCs/pull/31) and [Coreplay](https://github.com/polkadot-fellows/RFCs/blob/gav-coreplay/text/coreplay.md) + + +* Developers can program EVM Contracts directly on Kusama just as they do on Ethereum. But not Polkadot. +* Developers can work on EVM L2 Stacks (e.g. OP Stack) that live on top of Kusama instead of Ethereum. But not Polkadot. +* Developers can work on appchains that does not exist on Ethereum, with highly scalable async+sync patterns. On both Polkadot and Kusama. +* Developers can work on Corejam/Coreplay that does not exist on Ethereum, with highly scalable async+sync patterns. On both Polkadot and Kusama. + + +## Stakeholders + +- EVM Smart Contract Platforms: Shiden+Moonriver +- EVM Smart Contract Developers and Users +- ink!/WASM Contract Developers +- Core protocol and XCM format developers +- Tooling, block explorer, and UI developers +- CoreJam/Coreplay developers + +## Proposal + +### 1. Add `ethereum` + `contracts` Pallet to Kusama Relay Chain + +The `ethereum` pallet is to support drop-in EVM Contract Deployment on Kusama itself as well as EVM L2 Network deployment (OP Stack Contracts, Arbitrum Contracts, StarkNet Contracts and others) that depend on L1 EVM Contracts. + +The `contracts` pallet is to support drop-in WASM Contracts. + +### 2. Impose per-block weight limits of 50% for `ethereum` + `contracts` + +The 50% limit is proposed so that Kusama may continue to fulfill security obligations to its Bulk Coretime customers. It is believed that 10% would be too small to be taken seriously by new and existing EVM L2 Business owners (when compared to Ethereum) while 50% is ample empirically. Significantly higher than 50% (or having no limit) could cannibalize BCC and threaten new ICC customers. + +### 3. Adjust Storage Costs on Kusama to provide an 2-5x cost difference over Ethereum L2 Chains [eg Base] backed by Kusama + +For EVM L2s built on Kusama to remain competitive with Ethereum's EIP-4844, it is proposed that Kusama's storage costs be set to be low enough so as to have non-Substrate Kusama EVM L2 user costs be visibly lower than an Ethereum EVM L2. + +### 4. Match Moonbeam+Astar precompiles, but keep Existential Deposits as is. + +The [precompiles](https://docs.astar.network/docs/build/EVM/precompiles/) of Astar and [xcDOT](https://docs.astar.network/docs/learn/interoperability/asset-list/) are necessary to support drop-in replacement for KSM withdrawal/deposits. + +### 5. Adjust Kusama Messaging to endorse it as a Hybrid Relay Chain + +Instead of Kusama being a _canary_ network, messaging should be adjusted that it is a Hybrid Chain, fully elevated to the level of Ethereum. + + +## Explanation + +### Interfaces + +We imagine CoreJam's Collect/Refine/Join/Accumulate architecture could interface with EVM Contracts. + +### Deployment + +* Rococo - December 2023 +* Kusama - Spring 2024 + +## Testing, Security, and Privacy + +The `ethereum` pallet has been well explored by Moonbeam/Moonriver + Astar/Shiden already. + +It may be highly desirable to maintain a existential deposit on Kusama for security. + +## Performance, Ergonomics, and Compatibility + +### Performance + +By enabling EVM + WASM Contracts on the relay chain, new *long-term* storage requirements are important to satisfy to EVM L2 Business Owners. This long-term storage may justify a storage chain in the future. A hybrid short-term and long-term storage solution is already present in Ethereum's EIP-4844, and it is likely necessary for Kusama to be informed by its performance characteristics for competitiveness. + +### Ergonomics + +EVM Contract Developers and Users should be able to interact with Kusama EVM Contracts in the same way as they do as they do on Ethereum and Astar. The degree to which this must strictly be adhered to is a matter of debate. + +WASM Contract Developers and Users should be able to interact with Kusama WASM Contracts in the same way as they do as they do on Astar. + +### Compatibility + +This proposal is compatible with PR #32 Minimal Relay Chain, if that applies to Polkadot only. + +Instead of Polkadot Fellows deciding between the minimalism relay chain vs hybrid chains, this proposal puts Polkadot and Kusama on two different paths in usage, but not in any technical underpinnings. + +It is not clear how a new Storage Chain and CoreJam's Storage API relates to this proposal. + +## Prior Art and References + +Numerous EVM L2 Chains exist already, here are the leading ones: +- [OP Stack](https://github.com/ethereum-optimism/optimism/blob/develop/packages/contracts-bedrock/src/L1/L1StandardBridge.sol) +- [zkSync Era](https://github.com/matter-labs/era-contracts/tree/main) +- [Arbitrum Stack](https://docs.arbitrum.io/for-devs/useful-addresses) +- StarkNet + +Several competing DA stacks exist: +- [ethStorage](https://eth-store.w3eth.io/#/) +- [Eigenlayer](https://docs.mantle.xyz/network/introduction/concepts/data-availability) used by Mantle + + +## Major Unresolved Questions + +* Is Kusama DA technically capable of supporting permanent long-term storage needs of EVM L2 rollups? What is the connection to storage chains? + +* How can CoreJam + Coreplay relate to EVM + WASM Contract Interpretation and provide developers a smooth migration path? + +## Future Directions and Related Material + +Here is a detailed [Proposal to develop OP Stack On Kusama](https://github.com/colorfulnotion/optimism/blob/develop/README.md) + +Assuming Optimistic rollups can be backed by Polkadot security, we believe ZK Rollups would be natural to develop as well. + +The `contracts` pallet should be added for completeness. diff --git a/text/0033-support-non-substrate-evm-l2.md b/text/0033-support-non-substrate-evm-l2.md deleted file mode 100644 index fbb9adb23..000000000 --- a/text/0033-support-non-substrate-evm-l2.md +++ /dev/null @@ -1,237 +0,0 @@ -# RFC-0033: Support Non-Substrate EVM L2 Networks - -| | | -|--|--| -| **Start Date** | 3 October 2023 | -| **Description** | Support Non-Substrate EVM L2 Networks using Polkadot Data Availability by adding EVM/WASM Contracts Pallets to Polkadot Relay Chain, limited to 10+20=30% of blockspace | -| **Authors** | Sourabh Niyogi | - - -## Summary - -Polkadot+Kusama should support the possibility of having up to 10-30% of its blockspace weight allocatable to EVM and WASM Smart Contracts. This is not for Polkadot to be a yet another EVM Chain, but specifically to: -1. support the use of Polkadot's Data Availability resources to non-Substrate [largely EVM] L2 Networks, which will bring in additional demand for DOT and through those networks, new users and developers -2. (assuming CoreJam Work Packages have some direct relation to EVM Contracts + WASM Contract *Interpretation*) support Polkadot 2.0 experimentation in its transformation into a “map reduce” computer and bring in new classes of CoreJam+CorePlay developers in addressing synchronous/asynchronous composability - -## Motivation - -Polkadot 1.0's approach to scalability since inception has been as a platform for heterogenous shards or "app chains". Asynchronous interoperability between shards was enabled through XCM through the relay chain, including two teams (Moonbeam/Moonriver, Astar/Shiden) who successfully led Substrate-first EVM Chains in 2020-2023, promoting EVM + WASM Contract platforms widely to a large user and developer community. Contemporaneously, in 2020-2023, Ethereum developed a [roll-up centric roadmap](https://ethereum-magicians.org/t/a-rollup-centric-ethereum-roadmap/4698). The dominant use of Ethereum's L2 rollups has been to replicate Ethereum's capability at lower costs, enabled by: -* using Ethereum as a storage layer, recording the L2 activity to be able to derive 100% of the L2 state and support fraud proof -* using EVM Contracts deployed on L1 Ethereum and L2 chain to manage asset transfers -* using (rarely used but necessary) EVM Contracts fraud + ZK proofs on L1 to manage security of the L2 chain. - -The success of these Ethereum L2 rollups by common measures (TVL) are seen on [L2Beat](https://l2beat.com/scaling/summary) and can be seen to exceed to 100% of Polkadot parachains with similar age and functionality (Moonbeam + Astar). The bottom line is that Arbitrum and Optimism and many OP Stack chains now dominate with over 85% of TVL that is 100x that of Polkadot, including Base (built on OP Stack) launched in July 2023. The compute cost improvements relative to Ethereum L1 are immediately visible, while storage gas costs (1 gas per byte) remain. Ethereum's [EIP-4844](https://www.eip4844.com/) introduces a new "Blob" transaction type to Ethereum that enables L2 data to be persisted in the beacon node for a short period of time, and is expected to reduce the storage cost of a L2 Transaction (recorded on L1) by 3-8x. - -This proposal is about having Polkadot's DA layer serve as a resource for non-Substrate L2 networks to use. It is believed that: -1. Polkadot DA is already likely technically superior to Ethereum's EIP-4844 solution and can offer users and non-Substrate L2 Stack devs competitive storage and end users lower costs than Ethereum L1; -2. Astar + Moonbeam's efforts in 2021-2023 paved the way for EVM+WASM Contracts to be deployed on the relay chain in 2024; -3. CoreJam's CRJA will correctly treat synchronous (contract) and asynchronous (XCM) composability patterns in Polkadot 2.0 in the next 12-18 months - -The sizing of Ethereum L2 using Ethereum can be culled from [BigQuery Public Datasets](https://twitter.com/Polkadot/status/1707052392712212676) with: -``` -SELECT count(*), to_address, sum(length(input))/(2000000000) as gbPerMonth, sum(length(input))/(3600*24*30*2000000000) as kbPerSecond FROM `bigquery-public-data.crypto_ethereum.transactions` WHERE TIMESTAMP_TRUNC(block_timestamp, DAY) >= TIMESTAMP("2023-09-01") and TIMESTAMP_TRUNC(block_timestamp, DAY) <= TIMESTAMP("2023-09-30") group by to_address order by l desc limit 10000``` -``` - -The results of Ethereum L1 September 2023 activity are compiled [here](https://docs.google.com/spreadsheets/d/1ng2-dfPxCKCelI8_OmdqTnezYWHM0hODVarQNd7-oCU/edit#gid=1164282259). Here is a summary of the top 14 (in approximate storage bytes) Ethereum L2-related activity from that compilation: - -| Contract | numTxs/mo | GB/mo | K/s | Approx % | -|------------------------------------------------------|-----------|-------|------|----------| -| [Arbitrum: Sequencer Inbox](https://etherscan.io/address/0x1c479675ad559dc151f6ec7ed3fbf8cee79582b6) | 27,839 | 2.75 | 1.06 | 8.67% | -| [Optimism: Sequencer](https://etherscan.io/address/0xff00000000000000000000000000000000000010) | 22,966 | 2.22 | 0.86 | 7.01% | -| [Zksync Era: Validator Timelock](https://etherscan.io/address/0x3db52ce065f728011ac6732222270b3f2360d919) | 114,460 | 6.66 | 2.57 | 21.01% | -| [Base: Batch Inbox](https://etherscan.io/address/0xff00000000000000000000000000000000008453) | 47,240 | 2.81 | 1.08 | 8.85% | -| [StarkNet: SHARP](https://etherscan.io/address/0xfd14567eaf9ba941cb8c8a94eec14831ca7fd1b4) | 202,389 | 6.00 | 2.32 | 18.93% | -| [StarkNet](https://etherscan.io/address/0xc662c410c0ecf747543f5ba90660f6abebd9c8c4) | 93,340 | 0.13 | 0.05 | 0.42% | -| [StarkNet: FriStatementContract](https://etherscan.io/address/0x3e6118da317f7a433031f03bb71ab870d87dd2dd) | 5,940 | 0.07 | 0.03 | 0.23% | -| [Starkware: Sharp Verifier](https://etherscan.io/address/0x47312450b3ac8b5b8e247a6bb6d523e7605bdb60) | 790 | 0.06 | 0.02 | 0.20% | -| [Linea: L1 Message Service](https://etherscan.io/address/0xd19d4b5d358258f05d7b411e21a1460d11b0876f) | 136,648 | 2.30 | 0.89 | 7.25% | -| [Polygon: Polygon zkEVM Proxy](https://etherscan.io/address/0x5132a183e9f3cb7c848b0aac5ae0c4f0491b7ab2) | 6,383 | 0.46 | 0.18 | 1.46% | -| [Mantle](https://etherscan.io/address/0xd1328c9167e0693b689b5aa5a024379d4e437858) | 2,427 | 0.23 | 0.09 | 0.72% | -| [Zksync](https://etherscan.io/address/0xabea9132b05a70803a4e85094fd0e1800777fbef) | 86,694 | 0.16 | 0.06 | 0.51% | -| [Zksync](https://etherscan.io/address/0xf8a16864d8de145a266a534174305f881ee2315e) | 8,264 | 0.14 | 0.06 | 0.45% | -| [Zksync](https://etherscan.io/address/0x32400084c286cf3e17e7b677ea9583e60a000324) | 357,981 | 0.11 | 0.04 | 0.33% | -| **Top 14 Contracts from L2s** | **1,113,361** | **24.12** | **9.30** | **76.05%** | -| **All Contracts** | | **31.71** | | | | - - - -The above 14 Contracts make up **over 76%** of the input data bytes, handily surpassing the volumes of the well-known Ethereum contracts (from uniswap, 1inch, blur, etc.). It is *this* source of robust storage demand that Polkadot should tap into and extend security from Substrate chains to non-Substrate chains. - -According to this [Polkadot DA vs competition](https://forum.polkadot.network/t/polkadot-da-vs-competition/3403) discussion from @pepyakin + @bkchr, Polkadot can _presently_ support 0.83KiB/s per core, which at 100 cores implies 83-166 MiB/s. Thus, if I understand this discussion correctly, Polkadot can _trivially_ support Ethereum's present storage throughput of 12.25K/s, and handily support 100x of growth for non-Substrate EVM L2s like the ones above. If this is correct, then Polkadot DA is a diamond in the rough, just waiting to be used for EVM L2 scalability. - -However, a high throughput Polkadot's Data Availability layer is not enough to support non-Substrate EVM L2 networks. EVM Contract functionality must be deployed on Polkadot as well to support: -* deposits from L1 => L2 and withdrawals from L2 => L1 -* fraud proofs (e.g. in Optimism and Base) + validity proofs (e.g. in zkSync Era and StarkNet) - -Both DA + EVM Contract functionality are for non-Substrate EVM L2 Businesses to run entirely on Polkadot (in the same way as Moonbeam and Astar have and did until Astar zkEVM). Fortunately, Polkadot leading parachains Moonbeam and Astar already have explored EVM thoroughly in a Substrate architecture, enabling the `ethereum` pallet to be used on the Polkadot Relay Chain. - -We do not propose that Polkadot become yet another EVM Chain for users and arbitrary Smart Contract developers. Instead, we propose that Polkadot _supports_ non-Substrate EVM L2s, just as it always done -- the difference is an expansion in focus from Substrate-only L2s to include non-Substrate L2s. - -This addition, it is hoped, may also support Polkadot 2.0’s “Map Reduce”/Collect-Refine-Join-Accumulate architecture, which envisions a different way of addressing synchronous + asynchronous composability other than Ethereum's historical synchronous-first approach and Polkadot's asynchrous-first approach to scalability. By adding both EVM+WASM Contract pallets in 2024 coincident with CoreJam we enable both revenue growth and user/developer growth. - -## Stakeholders - -- Current and Prospective EVM L2 Chain Business Owners -- EVM L2 Stack Owners -- Substrate-native Parachains Smart Contract Platforms (Moonbeam, Astar) -- Substrate-native Parachains -- Core protocol and XCM format developers -- Chain-as-Service companies -- Tooling, block explorer, and UI developers -- CoreJam/Coreplay developers -- EVM and ink!/WASM Contract Developers - -## Proposal - -### 1. Add `ethereum` Pallet to Relay Chain - -This is to support drop-in EVM Contract Deployment of: -* OP Stack Contracts -* Arbitrum Contracts -* StarkNet Contracts -* ... others that depend on L1 EVM Contracts - -for: -* L1-L2 Bridges -* Rollup Fraud proofs -* ZK Validity proofs - -### 2. Add `contracts` Pallet to Relay Chain - -This is to support WASM Contract equivalents of the above, which we would hope enjoy improved efficiency relative to their EVM Contract counterparts. However, this is secondary in importance until the connection to CoreJam Work Packages is realized. - -### 3. Impose per-block weight limits of 30% for `ethereum` and `contracts` in aggregate, and 10% for `ethereum` and 20% `contracts` specifically - -This approach to having a limited amount of blockspace allocated for smart contracts is suggested by Polkadot senior fellows: -1. @rphmeier [here](https://www.rob.tech/blog/hybrid-chains/) "What the hybrid-chain approach is likely to look like in practice is application-specific logic deployed alongside smart contracts, **with a limited amount of blockspace allocated to smart contracts**. Given that contracts would only be able to consume some of the resources per-block, the absolute costs of gas would be higher relative to chains that focus entirely on smart contracts, all else equal." -2. @joe [here](https://forum.polkadot.network/t/permissioned-pallet-contracts-deployment-to-asset-hub-for-defi-primitives/3908/3) "For interactions that absolutely **must be synchronous** … then we could explore the idea of hybrid blockspace where **we set aside say 10% of the blockspace for contract execution** If there is a lot of demand for this tranche of blockspace, then the gas price will increase a lot, motivating contract developers to migrate to a contract chain or rethink their application architecture to work asynchronously." - -The 30% limit is proposed so that Polkadot may continue to fulfill security obligations to its Bulk Coretime customers. It is believed that 10% would be too small to be taken seriously by new and existing EVM L2 Business owners (when compared to Ethereum) while 30% is ample empirically. Significantly higher than 50% (or having no limit) would cannibalize BCC and threaten new ICC customers. - -The 10% limit on `ethereum` and 20% limit on `contracts` is to incentivize higher performance translations of L1 EVM Contracts to be done with WASM Contracts instead. It is not clear if these higher performance WASM Contract translations are worth translating, especially given the critical importance of fraud proofs, relative immature of ink! vs Solidity (perceived or real). - -This can be equally weighted (15%/15%) or omitted altogether, especially if difficult to implement technically, or stand as the primary mechanism. - -However, something like the total being around 30% should be non-negotiable. - -### 4. Adjust Storage Costs on Polkadot to provide an 2-5x cost difference over Ethereum L2 Chains [Base] backed by Polkadot DA/Security but 25-50% higher than Moonbeam+Astar. - -It is widely expected in both Ethereum and Polkadot ecosystems that everyday users will interact with L2s rather than L1. However, for EVM L2s built on Polkadot DA to remain competitive with Ethereum's EIP-4844, it is proposed that Polkadot's DA to having storage costs be set to be low enough so as to have non-Substrate Polkadot EVM L2 user costs be visibly lower than an Ethereum EVM L2. On the other hand, end user pricing for Polkadot L1 activity should be higher than both Astar and Moonbeam. This treats Polkadot as a storage wholesaler competing with Ethereum but not Astar and Moonbeam as retailer. - -In Ethereum, storage costs for L1 is 1 gas per byte. [Here](https://etherscan.io/tx/0xef2ea6e6a980c05a80e5b03ef60a070415e1f53edb01001d25fdb9841c3c7fd6) is a October 2023 Optimism L2=>L1 (0.0271 ETH, $45.09) writing 115,733 bytes, representing approximately 100-200 Optimism transactions at an average of $.22-.45/tx like [this](https://optimistic.etherscan.io/tx/0x64defec295aebad526a925d198a0ed63bcae2de40556c383281c4fd095dd2360). - -In Polkadot, at present, using the `preimage.notePreimage` the exact 115,733 bytes costs 11.58 DOT (@$4.12 = $47.59), which does not offer the order of magnitude improvements desired. - -With Protodanksharding in EIP-4844 expected to offer a different kind of storage on Execution/Beacon layer, it is expected to be 3-8x lower. So for Polkadot to aim for a competitive pricing, a 10x reduction ($4.50) should be sought for. - -Needless to say, this sort of permissionless storage is directly analogous to the permissioned storage of AWS S3 vs GCP GS: It is necessary to conduct the same kind of commodities storage pricing in DOT vs ETH, perceived in users mind in USD terms. If a storage chain or alternate Storage API is required to meet the L2 end users pricing perception, a concrete proposal is needed to meet the EVM L2 Business Owners goals that serve those users. - -At the end of the day, EVM L2 Chain Owners need pricing consistency from their suppliers to run a healthy business that L2 users can depend on. A good supply chain is not overly dependent on one supplier. An ideal outcome could be that EVM L2 costs is not dominated by a single L1 costs as it now but run on multiple storage systems so that true resilience is achieved. In the meantime, we need Polkadot DA to be 10x lower on its own. - -### 5. Match Moonbeam+Astar precompiles, but keep Existential Deposits as is. - -The [precompiles](https://docs.astar.network/docs/build/EVM/precompiles/) of Astar and [xcDOT](https://docs.astar.network/docs/learn/interoperability/asset-list/) are necessary to support drop-in replacement for DOT withdrawal/deposits. It is not desirable for Polkadot to be the home of thousands of ERC20 assets but the xcDOT case must be covered. Otherwise, a permissionless design (not requiring governance) that is compatible with Assethub is desired but requires a more sophisticated plan of how XCM could be situated. - -Moonbeam is currently facing significant storage bloat due to having no storage deposit. Having the 1 DOT Existential Deposit would trivially address this. - -Polkadot fellows have significant deep knowledge of how Ethereum's gas model is superficially too simple and would benefit from a higher dimensional model of weights (see [here](https://forum.polkadot.network/t/weight-v2-discussion-and-updates/227/5)). A engineering path to bring in a more sophisticated model targeting the EVM L2 Stack developer may be desirable. - -## Explanation - -### Interfaces - -We assume the Balances pallet will continue to exist to support drop-in replacement of EVM L1 Contracts. - -We imagine CoreJam's Collect/Refine/Join/Accumulate architecture should interface with both EVM+WASM Contracts. - -### Deployment - -The importance of Polkadot DA being used for EVM L2 is critical to pursue with high urgency to have high impact in 2024. Here is an adhoc timeline: - -* Rococo - December 2023 -* Kusama - Spring 2024 -* Polkadot - Summer 2024 - -## Cultural Change: Synchronous and Asynchronous Patterns, and the Role of EVM + WASM Smart Contracts - -There is a tremendous amount of anti-EVM feelings in the Polkadot community over any use of Ethereum technology that is not running on a Substrate stack. It is not that hard to see how beautifully connected Astar + Moonbeam are to Polkadot Relay Chain, relatively to how kludgily Arbitrum + Optimism + Base are connected to Ethereum chain. Polkadot's XCM is visibly better than similarly kludgy messaging systems in Ethereum. First and foremost, for this proposal to succeed, **cultural change is needed**. Leadership of Polkadot must dispel Substrate-first tribalism (or similar Rust/ink!/WASM tribalism vs Solidity/EVM) and *welcome* non-Substrate activity in all its forms. This is very easy for outsiders (like myself) and extremely hard for insiders. - -On a more technical level, Polkadot architecture has systematically adopted the following choices: -1. a very specific thesis that having multiple asynchronous chains (system chains or parachain) is the way to address scalability; -2. a systematic taste for "separation of concerns", widely adopted in software engineering, motivating a whole parachain for { assets (Asset Hub), identity (Encointer), staking, storage ... } - -You can see this drive in PR #32's Proposal for Minimal Relay, which even proposes to remove *Balances* from the relay chain and put it into Asset Hub. Such a move would significantly increase the complexity of deployment of EVM L2 chains and require async patterns for everything. - -The simplicity of Ethereum's synchronous-first programming model is now up against the complexity of Polkadot's asynchronous-first programming model. - -CoreJam "Map reduce" programming model, being developed by Polkadot founders+senior fellows, will force a reconciliation between these two artificial extremes: -* Ethereum's synchronous-first model, embodied in EVM Contracts on L1+L2 -* Polkadot's asynchronous-first model, embodied in [PR #32 - Minimal Relay](https://github.com/polkadot-fellows/RFCs/pull/32) - -Technologists (as least me, as a relative outsider with little tribal allegiance) know that Ethereum's synchronous-first model tends towards scalability failure (L2s become mostly disconnected islands, themselves incapable of scaling due to the lack of composable async), while Polkadot's asynchronous-first model tends toward unusability (users and developers must figure out what is going on many chains). - -If we open Polkadot up to non-Substrate L2 activity aggressively, this will enable Polkadot to experience real user + developer growth on these new non-Substrate L2 chains. Anything less probably will cause Polkadot to be like Apple Lisa, Amiga, NeXT in the 1980s and early 90s: great technology, pursued with great engineering and product taste, but ultimately never obtaining the kind of ubiquitous computer imagined relative to Windows PCs and Macs that dominated and survived to the present day. - -To be clear, this proposal is not about turning Polkadot into yet another EVM chain. Instead, I believe Rob Habermeier's appeal to **hybrid chains** is the pragmatically correct answer for Polkadot in the near future: - -![](https://europe1.discourse-cdn.com/standard21/uploads/polkadot2/original/2X/7/7cca3339cc4eb1ab3971d6fe10280d4469510ea3.png) - -If we take Rob's advice of "All chains should have smart contracts" seriously, then the Polkadot Relay Chain should include Smart Contracts. L2 Stack Developers (in a wide variety of rollup architectures from OP Stack to a new ZK rollup stack) should be able to use Polkadot's DA + EVM+WASM Contract platform freely as a hybrid chain. Rather than overengineer something for a specific rollup stack, Polkadot should offer the best DA it can with competitive storage, offer the best VMs to support L2 Stack Developers generally. - -Rather than aggressively stub out everything into N system chains and force unnecessary async patterns, I believe Polkadot should thoroughly develop its solution to synchronous + asynchronous composability at the center of CoreJam's Map reduce architecture, and extend it to EVM Contract Interpretation which enjoy a mature large developer community outside Substrate. Assuming cultural change is possible, by opening the door to non-Substrate ecosystems, we should bring in new users through the L2 stacks (entirely EVM now, but potentially other VMs in the future) and position Polkadot as being more technology neutral between Substrate vs non-Substrate, EVM vs WASM, ink! vs Coreplay, or whatever. - - -## Testing, Security, and Privacy - -Standard audit/review would be appropriate for Polkadot-deployed L1 EVM Contracts. - -OP Stack is the lowest complexity EVM L2 Chain and appears to be enjoying some network effects in 2023. - -An OP Deployment could be explored with Shibuya/Shiden/Astar, Moonbase/Moonriver/Moonbeam in parallel to a Rococo+Westend/Kusama/Polkadot. - -Other L2 stacks could be explored. - -## Performance, Ergonomics, and Compatibility - -### Performance - -Significant parametric choices are being made that will have long term consequences. By enabling EVM + WASM Contracts on the relay chain, new *long-term* storage requirements are important to satisfy to EVM L2 Business Owners. This long-term storage may justify a storage chain in the future. A hybrid short-term and long-term storage solution is already present in Ethereum's EIP-4844, and it is likely necessary for Polkadot to be informed by its performance characteristics for competitiveness. Polkadot fellows should be able to offer specific plans for success. - -### Ergonomics - -EVM + WASM Contract Developers should be able to interact with Polkadot EVM Contracts in the same way as they do as they do on Astar. - -### Compatibility - -This proposal is incompatible with PR #32 Minimal Relay Chain. - -It is not clear how a new Storage Chain and CoreJam's Storage API relates to this. - -## Prior Art and References - -Numerous EVM L2 Chains exist already, here are the leading ones: -- [OP Stack](https://github.com/ethereum-optimism/optimism/blob/develop/packages/contracts-bedrock/src/L1/L1StandardBridge.sol) -- [zkSync Era](https://github.com/matter-labs/era-contracts/tree/main) -- [Arbitrum Stack](https://docs.arbitrum.io/for-devs/useful-addresses) -- StarkNet - -Several competing DA stacks exist: -- [ethStorage](https://eth-store.w3eth.io/#/) -- [Eigenlayer](https://docs.mantle.xyz/network/introduction/concepts/data-availability) used by Mantle - - -## Major Unresolved Questions - -* Is the Polkadot DA technically capable of supporting permanent long-term storage needs of rollups? If so, should it be allocated to EVM L2 Chains? What is the connection to storage chains? - - * Can Polkadot Fellows decide between the minimalism relay chain vs hybrid chains? What is the connection to storage chains and how to manage short-term (24hr - 2 weeks) and long-term storage (full derivation of an L2 from L1 storage)? - -* How does CoreJam's map-reduce architecture (that addresses synch/async patterns) relate to EVM + WASM Contracts "sync" pattern? - -* Will Polkadot fellows able to overcome their tribal instincts and well-informed technical biases against Ethereum and support non-Substrate EVM Chains and enable Polkadot to support its own revenue growth, user growth and developer user growth? - -## Future Directions and Related Material - -Assuming Optimistic rollups can be backed by Polkadot security, we believe ZK Rollups would be natural to develop as well. This would treat EVM L2 Chains largely identical to the way Ethereum L1 treats its L2 chains: relatively disconnected islands with centralized bridges (CCTP) and slow kludgy messaging protocols like LayerZero and Axelar trying to mediate between them. - -However, a more exciting outcome that may be more transformational and revolutionary is to connect [CoreJam' map-reduce CRJA architecture](https://github.com/polkadot-fellows/RFCs/blob/gav-corejam/text/0031-corejam.md) and XCM to connect multiple non-Substrate EVM L2 Chains as well as the Substrate L2 Chains. It is hoped that XCM and this programming model can be combined in 2024 for Polkadot's solution to permission compute/storage capability to enjoy the recognition and impact it so clearly deserves. From d2c40a943e3731cc89782a4c542d49f0677e8981 Mon Sep 17 00:00:00 2001 From: Sourabh Niyogi Date: Sun, 5 Nov 2023 17:22:51 -0800 Subject: [PATCH 3/8] CoreJam System Chain on Ethereum L2 This proposal adapts the CoreJam architecture to EVM L2s, specifically utilizing OP Stack + Solidity instead of Polkadot/Substrate. Almost all CoreJam concepts are retained, but CoreJam's Rust relay chain interfaces are replaced with Solidity/EVM Contracts + OP Stack's Golang, situated in a "system chain". --- text/0033-corejam-system-chain-evm-l2s.md | 528 ++++++++++++++++++++++ text/0033-kusama-hybrid-relay-chain.md | 156 ------- 2 files changed, 528 insertions(+), 156 deletions(-) create mode 100644 text/0033-corejam-system-chain-evm-l2s.md delete mode 100644 text/0033-kusama-hybrid-relay-chain.md diff --git a/text/0033-corejam-system-chain-evm-l2s.md b/text/0033-corejam-system-chain-evm-l2s.md new file mode 100644 index 000000000..804828ebb --- /dev/null +++ b/text/0033-corejam-system-chain-evm-l2s.md @@ -0,0 +1,528 @@ +# RFC-0033: CoreJam System Chain on Ethereum L2 + +| | | +| --------------- | ------------------------------------------------------------------------------------------- | +| **Start Date** | 5 November 2023 | +| **Description** | CoreJam System Chain on Ethereum L2 | +| **Authors** | Sourabh Niyogi | + +## Summary + +CoreJam (RFC [#31](https://github.com/polkadot-fellows/RFCs/pull/31)) offers a significant refactorization of the blockchain state transition to blockchain scalability with its map-reduce architecture, starting with a Polkadot/Substrate 2.0 foundation. This proposal adapts the CoreJam architecture to EVM L2s, utilizing [OP Stack](https://stack.optimism.io/) + Solidity instead of Polkadot/Substrate. The endpoint being proposed is a _CoreJam System chain on Ethereum L2_. Almost all CoreJam concepts from #31 are retained (but none of Polkadot 1.0's primary "product" of parachains), but CoreJam's interfaces are replaced with Solidity/EVM Contracts + OP Stack's Golang. + +In this adaptation of "Collect-Refine-Join-Accumulate", a Solidity/EVM Smart Contract has two primary entry-points: `refine` and `accumulate`. Work packages are processed "in-core" (and thus parallelized) via `refine`, and the refined outputs of this processing are gathered together and an on-chain state-machine progressed according to `accumulate`. + +For pedagogical reasons, the write up below retains most of the original language, with small adaptation made for the OP Stack. + +## Motivation + +* CoreJam impact outside Polkadot ecosystem +* Provide On-ramp of new developers familiar with Solidity, new users with ETH liquidity and other EVM Chains +* Reduced Time-to-market of CoreJam + +From a usability and impact perspective, this expands the set of CoreJam developers from a small but growing number Substrate+Rust developers solely in the Polkadot ecosystem to a large but growing number of Solidity+EVM Developers in a larger number of EVM ecosystems beyond Polkadot alone. It is believed Polkadot may benefit from an increasing number of CoreJam developers onboarded from Ethereum and other EVM Chains through this adaptation. + +## Requirements + +In order of importance: + +1. This proposal must be compatible with Solidity+Ethereum to have developer + user impact +2. Implementation should be practical, needing minimal changes to OP Stack. +3. Utilization of CoreJam must be permissionless. +4. Support decentralization of nodes. + +## Stakeholders + +1. Solidity/EVM Smart Contract developers already building on EVM L1+L2 Chains needing the scalability of map-reduce. +2. Anyone wanting to create decentralised/unstoppable/resilient applications. +3. Anyone with exposure to the token economies of EVM Chains, specifically Ethereum. + +## Explanation + +**CoreJam is a general model for utilization of Cores. It is a mechanism by which Work Packages are communicated, authorized, computed and verified, and their results gathered, combined and accumulated into particular parts of the CoreJam chain's state.** + +### CoreJam in Polkadot Relay Chain vs CoreJam System Chain on ETH/.. L2 + +All of CoreJams interrelated concepts: *Work Package*, *Service*, *Work Item*, *Work Output*, *Work Result*, *Work Report*, *Guarantee* and *Service Trie* are situated within OP Stack. + +| CoreJam model | Context | +| --- | --- | +| *CoreJam EVM L2 Chain* | Primary block-chain | +| *Work Package* | Untrusted data provided to ScBG | +| *Work Item* | State-transition inputs and witness | +| *Work Output* | State-transition consequence | +| *Work Report* | Target of attestation | +| *(Work Package) Attestation* | Output signed in attestation | +| *Reporting* | Placement of Attestation on-chain | +| *Integration* | Irreversible transition of state | +| *Builder* | Creator of data worthy of Attestation | + +The following two groups of participants are coordinated with OP Stack nodes: + +- *ScBG*: System-chain Backing Group, a grouping of System-chain validators who act as the initial guarantors over the result of some computation done off-chain. These are OP Stack Nodes runing the "Map" Collect-Refine using Solidity/EVM Contracts. +- *ScBA*: System-chain Block Author, the author of some particular block on the CoreJam System-chain, an EVM L2 ORU also using OP Stack, and run "Reduce" Join-Accumulate using Solidity/EVM Contracts + +Unlike Polkadot and Kusama, the CoreJam System Chain (contemplated on Ethereum at L2, and potentially other EVM Chains) is not a relay chain: it does not offer parachain security or have a messaging protocol between parachains. For this reason, CoreJam's RcBG and RcBA are just ScBG and ScBA. In addition, EVM ORUs are centralized. + +### Overview + +The essence of CoreJam is to run a map-reduce process with `refine`: + * map Work Items into Work Results using `refine` + * Work Results using `accumulate` + +This can be done permissionlessly in Solidity/EVM Smart Contracts with the following interface: + +```solidity +pragma solidity ^0.8.0; + +interface CoreJamService { + struct Tuple { + CoreJam.Authorization authorization; + CoreJam.WorkResult[] data; + } + + function refine(bytes memory payload, CoreJam.PackageInfo memory packageInfo) external pure returns (CoreJam.BoundedVecOutput memory); + + function applyRefine(CoreJam.WorkItem memory item) external pure returns (CoreJam.WorkResult memory); + + function accumulate(CoreJam.Tuple[] memory results) external; + + // TODO: prune +} + +contract MyCoreJamService is CoreJamService { + using CoreJam for *; + + // ************* Any ScBG can call this + function refine(bytes memory payload, CoreJam.PackageInfo memory packageInfo) external pure returns (CoreJam.BoundedVecOutput memory) { + // Implement refine function logic here based on the provided parameters + // Return refined output as BoundedVecOutput + // Make sure to handle the logic according to your specific use case + CoreJam.BoundedVecOutput memory output; + // Implement the refining logic here + return output; + } + + function applyRefine(CoreJam.WorkItem memory item) external pure returns (CoreJam.WorkResult memory) { + // Implement applyRefine function logic here based on the provided WorkItem + // Return WorkResult containing service, item_hash, result, and gas + // Make sure to handle the logic according to your specific use case + CoreJam.WorkResult memory result; + // Implement the applyRefine logic here + return result; + } + + // Define modifier to restrict access to authorized users + modifier onlyRCBA(uint32 authId, CoreJam.WorkPackage memory package, uint32 coreIndex) { + require(CoreJam.isAuthorized(authorizers[authId].param, package, coreIndex), "Unauthorized"); + _; + } + + // ************* only ScBA can call this, and it can mutate the state of others + function accumulate(CoreJam.Tuple[] memory results) external onlyRCBA(authId, package, coreIndex) { + // Implement accumulate function logic here based on the provided parameters + // Iterate through the input results and process each tuple as needed + for (uint256 i = 0; i < results.length; i++) { + CoreJam.Authorization memory authorization = results[i].authorization; + CoreJam.WorkResult[] memory workResults = results[i].data; + + // Process authorization and workResults as needed + // ... + // CoreJam's transfer / on transfer is accomplished with a internal contract to another accumulate + } + } +} +``` + +Under ordinary circumstances, a CoreJam developer will deploy a CoreJamService along with a mechanism to add Work Items. + +The core insights of CoreJam is: +* the "in-core" Collect-Refine (with Smart Contract entry point`refine` ) supports mass parallelization and can be done with a _scalable_ distributed nodes (ScBG) organized by "core index", +* the "on-chain" `accumulate` is done by CoreJam System Chain ScBA, fundamentally bottlenecked by the system chain + +It is thought that CoreJams's map-reduce refactorization efficiently addresses the underlying dependency graph of what can be computed in what order better than smart contracts alone, separating what can be parallelised in the `refine` step. + +A CoreJam Work Package are placed in EVM Optimistic Rollup (ORU) context by mapping the 3 components: +* Authorization - this just a contract address with a `isAuthorized` interface +* Context - this is simplified in an EVM ORU to be just a block number and block hash. This is possible because EVM L2 ORUs in OP Stack are fundamentally centralized sequencers, at least for now. +* Prerequisite - TBD + +A *Work Package* is an *Authorization* together with a series of *Work Items* and a context, limited in plurality, versioned and with a maximum encoded size. The Context includes an optional reference to a Work Package (`WorkPackageHash`) which limits the relative order of the Work Package (see **Work Package Ordering**, later). + +(The number of prerequisites of a Work Package is limited to at most one. However, we cannot trivially control the number of dependents in the same way, nor would we necessarily wish to since it would open up a griefing vector for misbehaving Work Package Builders who interrupt a sequence by introducing their own Work Packages with a prerequisite which is within another's sequence.) + +Work Items are a pair where the first item, `service`, itself identifies a pairing of code and state known as a *Service*; and the second item, `payload`, is a block of data which, through the aforementioned code, mutates said state in some presumably useful way. + +A Service IS a smart contract stored on-chain and transitioned only using on-chain logic, strictly and deterministically constrained, holding funds and call into each other synchronously. +However, the on-chain `accumulate` function cannot be transacted with and is only called by either the ScBA or a inter-service calling. Otherwise, all input data (and state progression) must come as the result of a Work Item. A Work Item is a blob of data meant for a particular Service and crafted by some source external to consensus, which can be a user or another Service. It may be thought of as akin to a transaction or internal contract call. + +The Work Item is first processed *in-core* through `refine`, which is to say an OP Stack standalone node, yielding a distillate known as a *Work Result*. It is this Work Result which is collated together with others of the same service and Accumulated into the Service on-chain. + +In short, a Service is EVM Contract on the ETH L2 whose work items are first preprocessed with cheap decentralized compute power. + +Though this process happens entirely in consensus, there are two main consensus environments at play, _in-core_ and _on-chain_. We therefore partition the progress into two pairs of stages: Collect & Refine and Join & Accumulate. + +### Processing stages of a Work Package + +A Work Package has several stages of consensus computation associated with its processing, which happen as the system becomes more certain that it represents a correct and useful transition of its Service. + +While a Work Package is being built, the *ScBG* must have access to the CoreJam System-chain state in order to supply a specific *Context*. The Context dictates a certain *Scope* for the Work Package which is used by the Initial Validation to limit which System-chain blocks it may be processed on to a small sequence of a specific fork (which is yet to be built, presumably). We define the System-chain height at this point to be `T`. + +The first consensus computation to be done is the Work Package having its Authorization checked in-core, hosted by the System-chain Backing Group. If it is determined to be authorized, then the same environment hosts the Refinement of the Work Package into a series of Work Results. This concludes the bulk of the computation that the Work Package represents. We would assume that the System-chain's height at this point is shortly after the authoring time, `T+r` where `r` could be as low as zero. + +The second consensus computation happens on-chain at the behest of the System-chain Block Author of the time `T+r+i`, where `i` is generally zero or one, the time taken for the Work Results to be transported from within the Core to get to the gateway of being on-chain. The computation done essentially just ensures that the Work Package is still in scope and that the prerequisite it relies upon (if any) has been submitted ahead of it. This is called the on-chain *Reporting* and initiates the *Availability Protocol* for this Work Package once System-chain Validators synchronize to the block. This protocol guarantees that the Work Package will be made available for as long as we allow disputes over its validity to be made. + +At some point later `T+r+i+a` (where `a` is the time to distribute the fragments of the Work Package and report their archival to the next System-chain Block Author) the Availability Protocol has concluded and the System-chain Block Author of the time brings this information on-chain in the form of a bitfield in which an entry flips from zero to one. At this point we can say that the Work Report's Package is *Available*. + +Finally, at some point later still `T+r+i+a+o`, the Results of the Work Package are aggregated into groups of Services, and then *Pruned* and *Accumulated* into the common state of the System-chain. This process is known as *Integration* (in the fixed-function parachains model, this is known as "inclusion") and is irreversible within any given fork. Additional latency from being made *Available* to being *Integrated* (i.e. the `o` component) may be incurred due to ordering requirements, though it is expected to be zero in the variant of this proposal to be implemented initially. + +### Collect-Refine + +The first two stages of the CoreJam process are *Collect* and *Refine*. *Collect* refers to the collection and authorization of Work Packages (collections of items together with an authorization) to utilize a Polkadot Core. *Refine* refers to the performance of computation according to the Work Packages in order to yield *Work Results*. Finally, each Backing Group member attests to a Work Package yielding a series of Work Results and these Attestations form the basis for bringing the Results on-chain and integrating them into the Service's state which happens in the following stages. + +#### Collection and `is_authorized` + +Collection is the means of a Backing Group member attaining a Work Package which is authorized to be performed on their assigned Core at the current time. Authorization is a prerequisite for a Work Package to be included on-chain. Computation of Work Packages which are not Authorized is not rewarded. Incorrectly attesting that a Work Package is authorized is a disputable offence and can result in substantial punishment. + +On arrival of a Work Package, after the initial decoding, a first check is that the `context` field is valid. This must reference a header hash of a known block which may yet be finalized and the additional fields must correspond to the data of that block. + +We introduce the concept of an *Authorizer* procedure, which is a smart contract. Assigning computational resources to an Authorizer implies allowing any Work Package which passes that authorization process to utilize computational resources in order to be submitted on-chain. It controls the circumstances under which ScBGs (OP Stack standalone nodes capable of watching L2 OP Stack) may be rewarded for evaluation and submission of Work Packages (and thus what Work Packages become valid to submit onto CoreJam System Chain). Authorization logic is entirely arbitrary and need not be restricted to authorizing a single collator, Work Package builder, or even a single Service. + +An *Authorizer* is a EVM Smart Contract that is executed and is expressed in this code (which must be capable of in-core VM execution). Its entry-point prototype is: + +```solidity +interface Authorizer { + function isAuthorized(WorkPackage calldata package, uint32 coreIndex) external view returns (bool); +} +``` + +This function is executed by a ScBG node in a _metered VM_ and subject to a modest system-wide limitation on execution time. If it overruns this limit or panics on some input, it is considered equivalent to returning `false`. While it is mostly stateless (e.g. isolated from any System-chain state) it is provided with the package's `context` field in order to give information about a recent System-chain block. This allows it to be provided with a concise proof over some recent System-chain state. + +A single `Authorizer` value is associated with the index of the Core at a particular System-chain block and limits in some way what Work Packages may be legally processed by that Core. + +The need of validators to be rewarded for doing work they might reasonably expect to be useful competes with that of the procurers of work to be certain to get work done which is useful to them. With CoreJam, validators have little ability to identify a high-quality Work Package builder and the permissionless design means a greater expectation of flawed code executing in-core. Because of this, we make a slightly modified approach: Work Packages must have a valid Authorization, i.e. `is_authorized` returns `true` when provided with the Work Package. However, Validators get rewarded for *any* such authorized Work Package, even one which ultimately panics or overruns on its evaluation. + +This ensures that Validators do a strictly limited amount of work before knowing whether they will be rewarded and are able to discontinue and attempt other candidates earlier than would otherwise be the case. There is the possibility of wasting computational resources by processing Work Packages which result in error, but well-written authorization procedures can mitigate this risk by making a prior validation of the Work Items. + +### Refine + +The `refine` function is implemented as an entry-point inside a Service Contract Address: + +```solidity +function refine(bytes memory payload, CoreJam.PackageInfo memory packageInfo) external pure returns (CoreJam.BoundedVecOutput memory); +``` + +Both `refine` and `isAuthorized` are only ever executed in-core. Within this environment, we need to ensure that we can interrupt computation not long after some well-specified limit and deterministically determine when an invocation of the VM exhausts this limit. Since the exact point at which interruption of computation need not be deterministic, it is expected to be executed by a streaming JIT transpiler with a means of approximate and overshooting interruption coupled with deterministic metering. + +When applying `refine` from the client code, we must allow for the possibility that the VM exits unexpectedly or does not end. Validators are always rewarded for computing properly authorized Work Packages, including those which include such broken Work Items. But they must be able to report their broken state into the System-chain in order to collect their reward. Thus we define a type `WorkResult`: + +```solidity +enum WorkError { + None, + Timeout, + Panic +} + +struct WorkResult { + uint32 service; + bytes32 item_hash; + WorkError error; + bytes result; + uint256 gas_used; +} + +function applyRefine(CoreJam.WorkItem memory item) external pure returns (CoreJam.WorkResult memory); +``` + +The amount of gas used in executing the `refine` function is noted in the `WorkResult` value, and this is used later in order to help apportion remaining gas in the Join-Accumulate process to the Services whose items appear in the Work Packages. + +```golang +/// Secure reference to a Work Package. +type WorkPackageSpec struct { + /// The hash of the SCALE encoded `EncodedWorkPackage`. + Hash WorkPackageHash + /// The erasure root of the SCALE encoded `EncodedWorkPackage`. + Root ErasureRoot + /// The length in bytes of SCALE encoded `EncodedWorkPackage`. + Len uint32 +} + +/// The specification of the underlying Work Package. +type WorkReport struct { + PackageId WorkPackageId + /// The context of the underlying Work Package. + Context Context + /// The Core index under which the Work Package was Refined to generate the Report. + CoreIndex CoreIndex + /// The results of the evaluation of the Items in the underlying Work Package. + Results []WorkResult // MaxWorkItemsInPackage +} + +/// Multiple signatures are consolidated into a single Attestation in a space-efficient +/// manner using a `Bitmap` to succinctly express which validators have attested. +type Attestation struct { + /// The Work Report which is being attested. + Report WorkReport + /// Which validators from the group have a signature in `attestations`. + Validators *bitmap.Bitmap + /// The signatures of the ScBG members set out in `validators` whose message is the + /// hash of the `report`. The order of the signatures is the same order as the validators appear in `validators`. + Attestations []Signature +} +``` + +Each System-chain block, every Backing Group representing a Core which is assigned work provides a series of Work Results coherent with an authorized Work Package. Validators are rewarded when they take part in their Group and process such a Work Package. Thus, together with some information concerning their execution context, they sign a *Report* concerning the work done and the results of it. This is also known as a *Candidate*. This signed Report is called an *Attestation*, and is provided to the System-chain block author. If no such Attestation is provided (or if the System-chain block author refuses to introduce it for Reporting), then that Backing Group is not rewarded for that block. + +WorkReports are gossiped among ScBG nodes (OP Stack nodes) to form a sufficient number attestations, which arrive at the System-chain Block Author. + +### Join-Accumulate + +Join-Accumulate is the second major stage of computation and is independent from Collect-Refine. Unlike with the computation in Collect-Refine which happens contemporaneously within one of many isolated cores, the consensus computation of Join-Accumulate is both entirely synchronous with all other computation of its stage and operates within (and has access to) the same shared state-machine. + +Being *on-chain* (rather than *in-core* as with Collect-Refine), information and computation done in the Join-Accumulate stage is carried out (initially) by the Block Author and the resultant block evaluated by all Validators and full-nodes. Because of this, and unlike in-core computation, it has full access to the System-chain's state. + +The Join-Accumulate stage may be seen as a synchronized counterpart to the parallelised Collect-Refine stage. It may be used to integrate the work done from the context of an isolated VM into a self-consistent singleton world model. In concrete terms this means ensuring that the independent work components, which cannot have been aware of each other during the Collect-Refine stage, do not conflict in some way. Less dramatically, this stage may be used to enforce ordering or provide a synchronisation point. Finally, this stage may be a sensible place to manage asynchronous interactions between subcomponents of a Service or even different Services and oversee message queue transitions. + +#### Reporting and Integration + +There are two main phases of on-chain logic before a Work Package's ramifications are irreversibly assimilated into the state of the (current fork of the) System-chain. The first is where the Work Package is *Reported* on-chain. This is proposed through an extrinsic introduced by the ScBA and implies the successful outcome of some *Initial Validation* (described next). This kicks-off an off-chain process of *Availability* which, if successful, culminates in a second extrinsic being introduced on-chain shortly afterwards specifying that the Availability requirements of the Work Report are met. + +Since this is an asynchronous process, there are no ordering guarantees on Work Reports' Availability requirements being fulfilled. There may or may not be provision for adding further delays at this point to ensure that Work Reports are processed according to strict ordering. See *Work Package Ordering*, later, for more discussion here. + +Once both Availability and any additional requirements are met (including ordering and dependencies, but possibly also including reevaluation of some of the Initial Validation checks), then the second phase is executed which is known as *Integration*. This is the irreversible application of Work Report consequences into the Service's State Trie and (via certain permissionless host functions) the wider state of the System-chain. Work Results are segregated into groups based on their Service, joined into a `Vec` and passed through the immutable Prune function and into the mutable Accumulate function. + +#### Initial Validation + +There are a number of Initial Validation requirements which the ScBA must do in order to ensure no time is wasted on further, possibly costly, computation. Since the same tests are done on-chain, then for a Block Author to expect to make a valid block these tests must be done prior to actually placing the Attestations in the System-chain Block Body. + +Firstly, any given Work Report must have enough signatures in the Attestation to be considered for Reporting on-chain. Only one Work Report may be considered for Reporting from each ScBG per block. + +Secondly, any Work Reports introduced by the ScBA must be *Recent*, defined as having a `context.header_hash` which is an ancestor of the ScBA head and whose height is less than `RECENT_BLOCKS` from the block which the ScBA is now authoring. + +```golang +const RECENT_BLOCKS uint32 = 16 +``` + +Thirdly, dependent elements of the Context (`context.block_number`) must correctly correspond to those on-chain for the block corresponding to the provided `context.header_hash`. For this to be possible, the System-chain is expected to track Recent state roots in a queue. + +Fourthly, the ScBA may not attempt to report multiple Work Reports for the same Work Package. Since Work Reports become inherently invalid once they are no longer *Recent*, then this check may be simplified to ensuring that there are no Work Reports of the same Work Package within any *Recent* blocks. + +Finally, the ScBA may not register Work Reports whose prerequisite is not itself Reported in *Recent* blocks. + +In order to ensure all of the above tests are honoured by the ScBA, a block which contains Work Reports which fail any of these tests shall panic on import. The System-chain's on-chain logic will thus include these checks in order to ensure that they are honoured by the ScBA. We therefore introduce the *Recent Reports* storage item, which retaining all Work Package hashes which were Reported in the *Recent* blocks: + +```golang +const MAX_CORES = 512 +/// Must be ordered. +type ReportSet = Items [MAX_CORES]WorkPackageHash +type WorkPackageHash common.Hash + +type BoundedVec struct { + Items []WorkPackageHash +} + +type ReportSet struct { + BoundedVec +} + +type RecentReports struct { + StorageValue BoundedVec +} +``` + +The ScBA must keep an up to date set of which Work Packages have already been Reported in order to avoid accidentally attempting to introduce a duplicate Work Package or one whose prerequisite has not been fulfilled. Since the currently authored block is considered *Recent*, Work Reports introduced earlier in the same block do satisfy the prerequisite of Work Packages introduced later. + +While it will generally be the case that ScBGs know precisely which Work Reports will have been introduced at the point that their Attestation arrives with the ScBA by keeping the head of the System-chain in sync, it will not always be possible. Therefore, ScBGs will never be punished for providing an Attestation which fails any of these tests; the Attestation will simply be kept until either: + +1. it stops being *Recent*; +2. it becomes Reported on-chain; or +3. some other Attestation of the same Work Package becomes Reported on-chain. + +#### Availability + +Once the Work Report of a Work Package is Reported on-chain, the Work Package itself must be made *Available* through the off-chain Availability Protocol, which ensures that any dispute over the correctness of the Work Report can be easily objectively judged by all validators. Being off-chain this is _not_ block-synchronized and any given Work Package may take one or more blocks to be made Available or may even fail. + +Only once a Work Report's Work Package is made Available the processing continue with the next steps of Joining and Accumulation. Ordering requirements of Work Packages may also affect this variable latency and this is discussed later in the section **Work Package Ordering**. + +#### Gas Provisioning + +Join-Accumulate is, as the name suggests, comprised of two subordinate stages. Both stages involve executing code inside a VM on-chain. Thus code must be executed in a *metered* format, meaning it must be able to be executed in a sandboxed and deterministic fashion but also with a means of providing an upper limit on the amount of gas it may consume and a guarantee that this limit will never be breached. + +Practically speaking, we may allow a similar VM execution metering system similar to that for the `refine` execution, whereby we do not require a strictly deterministic means of interrupting, but do require deterministic metering and only approximate interruption. This would mean that full-nodes and System-chain validators could be made to execute some additional margin worth of computation without payment, though any attack could easily be mitigated by attaching a fixed cost (either economically or in gas terms) to an VM invocation. + +Each Service defines some requirements it has regarding the provision of on-chain gas. Since all on-chain gas requirements must be respected of all processed Work Packages, it is important that each Work Report does not imply using more gas than its fair portion of the total available, and in doing so provides enough gas to its constituent items to meet their requirements. + +```rust +struct WorkItemGasRequirements { + prune: uint256, + accumulate: uint256, +} +type GasRequirements = StorageMap; +``` + +Each Service has two gas requirements associated with it corresponding to the two pieces of permissionless on-chain Service logic and represent the amount of gas allotted for each Work Item of this service within in a Work Package assigned to a Core. + +The total amount of gas utilizable by each Work Package (`gas_per_package`) is specified as: + +```rust +gas_per_package = system_block_gas * safety_margin / max_cores +``` + +`safety_margin` ensures that other System-chain system processes can happen and important transactions can be processed and is likely to be around 75%. + +A Work Report is only valid if all gas liabilities of all Work Items to be Accumulated fit within this limit: + +```rust +let total_gas_requirement = work_statement + .items + .map(|item| gas_requirements[item.service]) + .sum(|requirements| requirements.prune + requirements.accumulate); +total_gas_requirement <= gas_per_package +``` + +Because of this, Work Report builders must be aware of any upcoming alterations to `max_cores` and build Statements which are in accordance with it not at present but also in the near future when it may have changed. + +### Accumulate + +The next phase, which happens on-chain, is Accumulate. This governs the amalgamation of the Work Package Outputs calculated during the Refinement stage into the System-chain's overall state and in particular into the various Child Tries of the Services whose Items were refined. Crucially, since the Refinement happened in-core, and since all in-core logic must be disputable and therefore its inputs made *Available* for all future disputers, Accumulation of a Work Package may only take place *after* the Availability process for it has completed. + +The function signature to the `accumulate` entry-point in the Service's code blob is: + +```solidity +function accumulate(CoreJam.Tuple[] memory results) external; +``` + +The logic in `accumulate` may need to know how the various Work Items arrived into a processed Work Package. Since a Work Package could have multiple Work Items of the same Service, it makes sense to have a separate inner `Vec` for Work Items sharing the Authorization (by virtue of being in the same Work Package). + +Work Items are identified by their Keccak hash, known at the *Item Hash* (`ItemHash`). We provide both the Authorization of the Package and the constituent Work Item Hashes and their Results in order to allow the `refine` logic to take appropriate action in the case that an invalid Work Item was submitted (i.e. one which caused its Refine operation to panic or time-out). + +There is an amount of gas which it is allowed to use before being forcibly terminated and any non-committed state changes lost. The lowest amount of gas provided to `accumulate` is defined as the number of `WorkResult` values passed in `results` to `accumulate` multiplied by the `accumulate` field of the Service's gas requirements. + +However, the actual amount of gas may be substantially more. Each Work Package is allotted a specific amount of gas for all on-chain activity (`gas_per_package` above) and has a gas liability defined by the gas requirements of all Work Items it contains (`total_gas_requirement` above). Any gas remaining after the liability (i.e. `gas_per_package - total_gas_requirement`) may be apportioned to the Services of Items within the Report on a pro-rata basis according to the amount of gas they utilized during `refine`. Any gas unutilized by Classes within one Package may be carried over to the next Package and utilized there. + +Read-access to the System-chain state is allowed with EVM Precompiles. No direct write access may be provided since `accumulate` is untrusted code. + +Since `accumulate` is permissionless and untrusted code, we must ensure that its child trie does not grow to degrade the System-chain's overall performance or place untenable requirements on the storage of full-nodes. To this goal, we require an account sovereign to the Service to be holding an amount of funds proportional to the overall storage footprint of its Child Trie. `set_work_storage` may return an error should the balance requirement not be met. + +Host functions are provided allowing any state changes to be committed at fail-safe checkpoints to provide resilience in case of gas overrun (or even buggy code which panics). The amount of gas remaining may also be queried without setting a checkpoint. `Gas` is expressed in a regular fashion for a solo-chain (i.e. one-dimensional). + +`accumulate` of one service may call `accumulate` of another service, as is common with smart contracts This is an entirely synchronous function which transfers the execution over to a `destination` Service as well as the provided `amount` into their account. + + + +### CoreJam Storage API + +To reference large, immutable and long-term data payloads both in-core (`refine`) and on-chain (`accumulate`), we expose a *CoreJam Storage API*, accessible to untrusted code through Solidity library and to trusted System-chain code via a Golang interface. + +Internally, data is stored with a reference count so that two separate usages of `store` need not be concerned about the other. + +Every piece of data stored for an untrusted caller requires a sizeable deposit. When used by untrusted code via a host function, the `depositor` would be set to an account controlled by the executing code. + + +```golang +type Storage interface { + /// Immutable function to attempt to determine the preimage for the given `hash`. + Lookup(hash common.Hash) []byte + /// Allow a particular preimage to be `provide`d. Once provided, this will be available through `lookup` until `unrequest` is called. + Request(hash common.Hash, len int) bool + /// Remove request that some data be made available. If the data was never + /// available or the data will remain available due to another request, + /// then `false` is returned and `expunge` may be called immediately. + /// Otherwise, `true` is returned and `expunge` may be called in + /// 24 hours. + Unrequest(hash common.Hash) bool + // Functions used by implementations of untrusted functions; such as + // extrinsics or host functions. + /// Place a deposit in order to allow a particular preimage to be `provide`d. + /// Once provided, this will be available through `lookup` until + /// `unrequest_untrusted` is called. + RequestUntrusted(depositor AccountID, hash common.Hash, len int) + /// Remove request that some data be made available. If the data was never + /// available or the data will remain available due to another request, + /// then `false` is returned and `expunge_untrusted` may be called immediately. + /// Otherwise, `true` is returned and `expunge_untrusted` may be called in + /// 24 hours. + UnrequestUntrusted(depositor AccountID, hash common.Hash) bool + // Permissionless items utilizable directly by an extrinsic or task. + + /// Provide the preimage of some requested hash. Returns `Some` if its hash + /// was requested; `None` otherwise. + /// + /// Usually utilized by an extrinsic and is free if `Some` is returned. + Provide(preimage []byte) (common.Hash, bool) + + /// Potentially remove the preimage of `hash` from the chain when it was + /// unrequested using `unrequest`. `Ok` is returned iff the operation is + /// valid. + /// + /// Usually utilized by a task and is free if it returns `Ok`. + Expunge(hash common.Hash) error + /// Return the deposit associated with the removal of the request by + /// `depositor` using `unrequest_untrusted`. Potentially + /// remove the preimage of `hash` from the chain also. `Ok` is returned + /// iff the operation is valid. + /// + /// Usually utilized by a task and is free if it returns `Ok`. + ExpungeUntrusted(depositor AccountID, hash common.Hash) error + /// Equivalent to `request` followed immediately by `provide`. + Store(data []byte) common.Hash +} +``` + +New OVM opcodes for each function are as follows: + +| OVM Opcode | Go Interface | Description | +|------------|--------------|-------------| +| `APLOOKUP` | `Lookup(hash common.Hash) []byte` | Looks up the provided hash in the storage and returns the corresponding value as a byte array. If the hash is not found, returns None. | +| `APREQ` | `RequestUntrusted(depositor AccountID, hash common.Hash, len int)` | Places a deposit to request the specified hash with a given length to be made available in the storage. This function is used by untrusted sources. +| `APUNREQ` | `UnrequestUntrusted(depositor AccountID, hash common.Hash) bool` | Removes the request for the specified hash made by the specified depositor. If the request is successfully removed, returns true. Otherwise, returns false. | +| `APPROVIDE` | `Provide(preimage []byte) (common.Hash, bool)` | Provides the preimage data. If the preimage corresponds to a previously requested hash, returns the hash. Otherwise, returns None. | +| `APEXP` | `ExpungeUntrusted(depositor AccountID, hash common.Hash) error` | Removes the specified hash and its corresponding data from the storage, based on a request made by the specified depositor. Returns Ok(()) if the operation is successful, an error otherwise. | +| `APSTORE` | `Store(data []byte) common.Hash` | Stores the provided data in the storage and returns the hash of the stored data. | + + +Removing data happens in a two-phase procedure; first the data is unrequested, signalling that calling `lookup` on its hash may no longer work (it may still work if there are other requests active). 24 hours following this, the data is expunged with a second call which, actually removes the data from the chain assuming no other requests for it are active. Only once expunge is called successfuly is the deposit returned. If the data was never provided, or is additional requests are still active, then expunge may be called immediately after a successful unrequest. + + +### Work Package Ordering + +We target the soft-ordering variant: providing ordering only on a *best-effort* basis, whereby Work Reports respect the ordering requested in their Work Packages as much as possible, but it is not guaranteed. Work Reports may be Accumulated before, or even entirely without, their prerequisites. We refer to this *Soft-Ordering*. The alternative is to provide a guarantee that the Results of Work Packages will always be Accumulated no earlier than the Result of any prerequisite Work Package. As we are unable to alter the Availability Protocol, this is achieved through on-chain queuing and deferred Accumulation. + +Initial Validation are made on-chain prior to the Availability Protocol begins for any given Work Package. This ensures that the Work Package is still in scope. However, this does not ensure that the Work Package is still in scope at the point that the Work Results are actually Accumulated. It is as yet unclear whether this is especially problematic. + +### Fees + +Fees are ETH on L2, bridged from Ethereum L1. This allows a succinct *Authorization* (i.e. a small blob of data) to be included in the Work Package which, when fed into the relevant Authorizer function could verify that some Work Package is indeed allowed to utilize that Core at (roughly) that time. A simple proof system would be a regular PKI signature. More complex proof systems could include more exotic cryptography (e.g. multisignatures or zk-SNARKs). + +In this model, we would expect any authorized Work Packages which panic or overrun to result in a punishment to the specific author by the logic of the Service. + +## Performance, Ergonomics and Compatibility, Notes on Implementation in OP Stack + +We envision an initial version of this proposal with modifications to the OP Stack: + +1. We require a mapping between Work Packages and Core Index, with an equally easy to implement ScBG VRF, following [Dynamic Backing Groups: Preparing Cores for Agile Scheduling](https://forum.polkadot.network/t/dynamic-backing-groups-preparing-cores-for-agile-scheduling/3629/10) +2. We require extending the [Optimistic Virtual Machine] Availability Protocol in OP Stack, using KAGOME's erasure coding, using Go C++ bindings. It is not clear how the CoreJam Storage API relates to Optimism fraud proofs. +3. It is not claimed that the CoreJam L2 Chain is + + +## Testing, Security and Privacy + +Standard testing as per OP Stack and Solidity security auditing applies. + +The proposal introduces no new privacy concerns. + +## Future Directions and Related Material + +It is highly desirable to: +1. build multiple services demonstrating scaling performance of the CoreJam architecture in Services +2. measure performance of Availability Protocol as a function of # of cores in the CoreJam System Chain + +## Drawbacks, Alternatives and Unknowns + +1. In the case of composite Work Packages, allowing synchronous (and therefore causal) interactions between the Work Items. If this were to be the case, then some sort of synchronisation sentinel would be needed to ensure that should one subpackage result without the expected effects on its Service State (by virtue of the `accumulate` outcome for that subpackage), that the `accumulate` of any causally entangled subpackages takes appropriate account for this (i.e. by dropping it and not effecting any changes from it). + +2. Work Items may need some degree of coordination to be useful by the `accumulate` function of their Service. To a large extent this is outside of the scope of this proposal's computation model by design. Through the authorization framework we assert that it is the concern of the Service and not of the System-chain validators themselves. + +## Prior Art and References + +This is an adaptation of RFC #31. diff --git a/text/0033-kusama-hybrid-relay-chain.md b/text/0033-kusama-hybrid-relay-chain.md deleted file mode 100644 index 9afd68b58..000000000 --- a/text/0033-kusama-hybrid-relay-chain.md +++ /dev/null @@ -1,156 +0,0 @@ -# RFC-0033: Kusama Network 2.0: The Hybrid Chain Network - -| | | -|--|--| -| **Start Date** | 9 October 2023 | -| **Description** | Evolve Kusama 2.0 to be a Hybrid Relay Chain, specifically with Smart Contracts on the Relay Chain, elevating Kusama to the level of Ethereum. | -| **Authors** | Sourabh Niyogi | - -## Summary - -Polkadot+Kusama should pursue different usability and scalability: - -1. Polkadot 2.0 should be a [Minimal Relay Chain](https://github.com/polkadot-fellows/RFCs/pull/32), maximizing cores/resources for maximum scalability of CoreJam + CorePlay's asynchronous-first patterns - -2. Kusama 2.0 should be a [Hybrid Relay Chain](https://www.rob.tech/blog/hybrid-chains/), prioritizing developer+user friendliness over scalability with full backwards compatibility with 1.0 Blockchain technology: EVM+WASM Smart Contracts - -This solution elevates Kusama to the level of Ethereum while giving developers on-ramps to Polkadot's new scalability solutions. - -## Motivation - -Polkadot 1.0's approach to scalability since inception has been as a platform for heterogenous shards or "app chains". Asynchronous interoperability between shards was enabled through XCM through the relay chain, including two teams (Moonbeam/Moonriver, Astar/Shiden) who successfully led Substrate-first EVM (+ WASM) Smart Contract dev platforms in 2020-2023, promoting them widely to a large user and developer community. Almost all other chains ignored smart contract usage entirely, in favor of Substrate pallets. Still, when not using XCM asynchronous patterns (used largely just for cross-chain asset transfers), both sorts of approaches rely on developers pursuing a blockchain-centric state transition way of operating: everything has to be done within one block with sync patterns, whether using EVM/WASM Smart Contracts or Substrate Pallets. - -In Polkadot 2.0, the state transition function is refactored by bringing asynchronous patterns to the center with CoreJam's Map-Reduce/Collect-Refine-Join-Accumulate (CJRA) + CorePlay Actor programming models. This will take most of 2024 to develop, while most blockchain technology will continue to live on with 1.0 EVM Contracts, which have non-existent treatment of async patterns: messaging is bolted on by networks like Axelar and LayerZero. - -While Polkadot aims to be a massively scalable map reduce computer, in most cases, developers do NOT need scalability right away -- it is widely recognized that both devs+users find the multichain factorization unnecessarily complex, while technologists know that complexity is required for scalability. Few parachains have scaled their user base to more than a few hundred user per day, resulting in very poor "blockspace" usage. We are left with a puzzle: if Polkadot technology is fundamentally the world's best solution to async/sync patterns at the heart of the state transition function, but embodies complex patterns, how do we give developers an on-ramp? - -Rob Habermerier's [Hybrid Chains](https://www.rob.tech/blog/hybrid-chains/) provide the answer: - -![](https://europe1.discourse-cdn.com/standard21/uploads/polkadot2/original/2X/7/7cca3339cc4eb1ab3971d6fe10280d4469510ea3.png) - -By adding Smart Contracts to the Kusama Relay Chain, developers can scale from sync-first "old" blockchain technology to "new" in this spectrum: - -1. full sync patterns in Smart Contracts on the Relay Chain. _This pattern is manifested in Ethereum but has never been seen in Polkadot or Kusama._ -2. full sync patterns in Smart Contracts on a Parachain. _This pattern is seen in Astar + Moonbeam._ -3. full sync patterns in Substrate pallets on a Parachain. _This pattern is seen in all other parachains._ -4. hybrid sync + async patterns in Smart Contracts on a Relay Chain. _This pattern is manifested in Ethereum with Axelar/LayerZero._ -5. hybrid sync + async patterns in CoreJam + CorePlay. _This pattern is under active development and will appear in Kusama 2.0 + Polkadot 2.0._ -6. hybrid sync + async patterns in Substrate Pallets. _This pattern is seen in all other parachains._ - -Developers can _start_ at any point in the above spectrum and move upwards as their needs for scale increase, culminating in CoreJam/CorePlay and Substrate pallets in app chains. - -For "core" efficiency reasons, it is highly undesirable to load sync patterns (whether Smart Contracts or Substrate pallets) onto the Relay Chain: [RFC #32 Minimal Relay Chain](https://github.com/polkadot-fellows/RFCs/pull/32) takes this the logical extreme, moving Balances, Staking, Identity, Governance off of Polkadot, with Polkadot founder @gavofyork's strong conviction that "The Relay-chain should do one thing and do it well. That means being the head of a secure decentralised multicore computer. Sticking an EVM smart contract in there and/or continuing to allow end-user-application-level transactions and general interaction goes directly against this mantra." - -So, this proposal takes Kusama 2.0 to the other extreme of Polkadot 2.0, and - -_Elevates Kusama to the level of Ethereum._ - -By putting EVM+WASM Smart Contracts on the Relay Chain, Kusama can simultaneously pursue both leading ecosystem's path to scalability: - -1. Ethereum: [Scalability from rollups](https://ethereum-magicians.org/t/a-rollup-centric-ethereum-roadmap/4698) _and_ -2. Polkadot: [CoreJam](https://github.com/polkadot-fellows/RFCs/pull/31) and [Coreplay](https://github.com/polkadot-fellows/RFCs/blob/gav-coreplay/text/coreplay.md) - - -* Developers can program EVM Contracts directly on Kusama just as they do on Ethereum. But not Polkadot. -* Developers can work on EVM L2 Stacks (e.g. OP Stack) that live on top of Kusama instead of Ethereum. But not Polkadot. -* Developers can work on appchains that does not exist on Ethereum, with highly scalable async+sync patterns. On both Polkadot and Kusama. -* Developers can work on Corejam/Coreplay that does not exist on Ethereum, with highly scalable async+sync patterns. On both Polkadot and Kusama. - - -## Stakeholders - -- EVM Smart Contract Platforms: Shiden+Moonriver -- EVM Smart Contract Developers and Users -- ink!/WASM Contract Developers -- Core protocol and XCM format developers -- Tooling, block explorer, and UI developers -- CoreJam/Coreplay developers - -## Proposal - -### 1. Add `ethereum` + `contracts` Pallet to Kusama Relay Chain - -The `ethereum` pallet is to support drop-in EVM Contract Deployment on Kusama itself as well as EVM L2 Network deployment (OP Stack Contracts, Arbitrum Contracts, StarkNet Contracts and others) that depend on L1 EVM Contracts. - -The `contracts` pallet is to support drop-in WASM Contracts. - -### 2. Impose per-block weight limits of 50% for `ethereum` + `contracts` - -The 50% limit is proposed so that Kusama may continue to fulfill security obligations to its Bulk Coretime customers. It is believed that 10% would be too small to be taken seriously by new and existing EVM L2 Business owners (when compared to Ethereum) while 50% is ample empirically. Significantly higher than 50% (or having no limit) could cannibalize BCC and threaten new ICC customers. - -### 3. Adjust Storage Costs on Kusama to provide an 2-5x cost difference over Ethereum L2 Chains [eg Base] backed by Kusama - -For EVM L2s built on Kusama to remain competitive with Ethereum's EIP-4844, it is proposed that Kusama's storage costs be set to be low enough so as to have non-Substrate Kusama EVM L2 user costs be visibly lower than an Ethereum EVM L2. - -### 4. Match Moonbeam+Astar precompiles, but keep Existential Deposits as is. - -The [precompiles](https://docs.astar.network/docs/build/EVM/precompiles/) of Astar and [xcDOT](https://docs.astar.network/docs/learn/interoperability/asset-list/) are necessary to support drop-in replacement for KSM withdrawal/deposits. - -### 5. Adjust Kusama Messaging to endorse it as a Hybrid Relay Chain - -Instead of Kusama being a _canary_ network, messaging should be adjusted that it is a Hybrid Chain, fully elevated to the level of Ethereum. - - -## Explanation - -### Interfaces - -We imagine CoreJam's Collect/Refine/Join/Accumulate architecture could interface with EVM Contracts. - -### Deployment - -* Rococo - December 2023 -* Kusama - Spring 2024 - -## Testing, Security, and Privacy - -The `ethereum` pallet has been well explored by Moonbeam/Moonriver + Astar/Shiden already. - -It may be highly desirable to maintain a existential deposit on Kusama for security. - -## Performance, Ergonomics, and Compatibility - -### Performance - -By enabling EVM + WASM Contracts on the relay chain, new *long-term* storage requirements are important to satisfy to EVM L2 Business Owners. This long-term storage may justify a storage chain in the future. A hybrid short-term and long-term storage solution is already present in Ethereum's EIP-4844, and it is likely necessary for Kusama to be informed by its performance characteristics for competitiveness. - -### Ergonomics - -EVM Contract Developers and Users should be able to interact with Kusama EVM Contracts in the same way as they do as they do on Ethereum and Astar. The degree to which this must strictly be adhered to is a matter of debate. - -WASM Contract Developers and Users should be able to interact with Kusama WASM Contracts in the same way as they do as they do on Astar. - -### Compatibility - -This proposal is compatible with PR #32 Minimal Relay Chain, if that applies to Polkadot only. - -Instead of Polkadot Fellows deciding between the minimalism relay chain vs hybrid chains, this proposal puts Polkadot and Kusama on two different paths in usage, but not in any technical underpinnings. - -It is not clear how a new Storage Chain and CoreJam's Storage API relates to this proposal. - -## Prior Art and References - -Numerous EVM L2 Chains exist already, here are the leading ones: -- [OP Stack](https://github.com/ethereum-optimism/optimism/blob/develop/packages/contracts-bedrock/src/L1/L1StandardBridge.sol) -- [zkSync Era](https://github.com/matter-labs/era-contracts/tree/main) -- [Arbitrum Stack](https://docs.arbitrum.io/for-devs/useful-addresses) -- StarkNet - -Several competing DA stacks exist: -- [ethStorage](https://eth-store.w3eth.io/#/) -- [Eigenlayer](https://docs.mantle.xyz/network/introduction/concepts/data-availability) used by Mantle - - -## Major Unresolved Questions - -* Is Kusama DA technically capable of supporting permanent long-term storage needs of EVM L2 rollups? What is the connection to storage chains? - -* How can CoreJam + Coreplay relate to EVM + WASM Contract Interpretation and provide developers a smooth migration path? - -## Future Directions and Related Material - -Here is a detailed [Proposal to develop OP Stack On Kusama](https://github.com/colorfulnotion/optimism/blob/develop/README.md) - -Assuming Optimistic rollups can be backed by Polkadot security, we believe ZK Rollups would be natural to develop as well. - -The `contracts` pallet should be added for completeness. From 1eaf5372323512c16178369b2c538724076f307f Mon Sep 17 00:00:00 2001 From: Sourabh Niyogi Date: Tue, 7 Nov 2023 00:14:22 -0800 Subject: [PATCH 4/8] cleanup --- text/0033-corejam-system-chain-evm-l2s.md | 208 +++++++++++----------- 1 file changed, 100 insertions(+), 108 deletions(-) diff --git a/text/0033-corejam-system-chain-evm-l2s.md b/text/0033-corejam-system-chain-evm-l2s.md index 804828ebb..d53f7fe05 100644 --- a/text/0033-corejam-system-chain-evm-l2s.md +++ b/text/0033-corejam-system-chain-evm-l2s.md @@ -3,39 +3,40 @@ | | | | --------------- | ------------------------------------------------------------------------------------------- | | **Start Date** | 5 November 2023 | -| **Description** | CoreJam System Chain on Ethereum L2 | +| **Description** | CoreJam System Chain on Ethereum L2 with OP Stack | | **Authors** | Sourabh Niyogi | ## Summary -CoreJam (RFC [#31](https://github.com/polkadot-fellows/RFCs/pull/31)) offers a significant refactorization of the blockchain state transition to blockchain scalability with its map-reduce architecture, starting with a Polkadot/Substrate 2.0 foundation. This proposal adapts the CoreJam architecture to EVM L2s, utilizing [OP Stack](https://stack.optimism.io/) + Solidity instead of Polkadot/Substrate. The endpoint being proposed is a _CoreJam System chain on Ethereum L2_. Almost all CoreJam concepts from #31 are retained (but none of Polkadot 1.0's primary "product" of parachains), but CoreJam's interfaces are replaced with Solidity/EVM Contracts + OP Stack's Golang. +CoreJam (RFC [#31](https://github.com/polkadot-fellows/RFCs/pull/31)) offers a significant refactorization of the blockchain state transition function with its map-reduce architecture, starting with a Polkadot/Substrate 2.0 foundation. This proposal adapts the CoreJam architecture to EVM L2s, specifically utilizing [OP Stack](https://stack.optimism.io/) + Solidity instead of Polkadot/Substrate. The endpoint being proposed is a _CoreJam System Chain on Ethereum L2_. Almost all CoreJam concepts from #31 are retained (but none of Polkadot 1.0's primary "product" of parachains), where CoreJam's Rust interfaces are replaced with Solidity/EVM Contracts + OP Stack's Golang. In this adaptation of "Collect-Refine-Join-Accumulate", a Solidity/EVM Smart Contract has two primary entry-points: `refine` and `accumulate`. Work packages are processed "in-core" (and thus parallelized) via `refine`, and the refined outputs of this processing are gathered together and an on-chain state-machine progressed according to `accumulate`. -For pedagogical reasons, the write up below retains most of the original language, with small adaptation made for the OP Stack. +For pedagogical reasons, the write up below retains much of the original language. ## Motivation -* CoreJam impact outside Polkadot ecosystem -* Provide On-ramp of new developers familiar with Solidity, new users with ETH liquidity and other EVM Chains -* Reduced Time-to-market of CoreJam +* CoreJam impact in Ethereum and other EVM L1s +* Provide an on-ramp to CoreJam developers already familiar with Solidity +* Reduced Time-to-market of CoreJam in OP Stack relative to Polkadot 2.0 Engineering -From a usability and impact perspective, this expands the set of CoreJam developers from a small but growing number Substrate+Rust developers solely in the Polkadot ecosystem to a large but growing number of Solidity+EVM Developers in a larger number of EVM ecosystems beyond Polkadot alone. It is believed Polkadot may benefit from an increasing number of CoreJam developers onboarded from Ethereum and other EVM Chains through this adaptation. +From a usability and impact perspective, having CoreJam's map-reduce in Ethereum expands the set of CoreJam developers from a small but growing number Substrate+Rust developers solely in the Polkadot ecosystem to a large but growing number of Solidity developers in a larger number of EVM ecosystems beyond Polkadot alone. Provided that Polkadot executes on its high throughput DA (proportional to number of cores), minimal relay chain scalability, and genuine L1 decentralization, it is believed Polkadot may benefit from an increasing number of CoreJam developers onboarded from Ethereum and other EVM Chains through this adaptation. ## Requirements In order of importance: -1. This proposal must be compatible with Solidity+Ethereum to have developer + user impact -2. Implementation should be practical, needing minimal changes to OP Stack. -3. Utilization of CoreJam must be permissionless. -4. Support decentralization of nodes. +1. This proposal must be compatible with Solidity+Ethereum to have developer and ultimately, user impact. +2. The CoreJam implementation should be practical, needing minimal changes to OP Stack. +3. Utilization of CoreJam must be permissionless and maintain the same security model of Optimistic Rollups. +4. There should be a clear path for CoreJam System Chain, despite its ORU L2 setting, to support decentralization of nodes. ## Stakeholders 1. Solidity/EVM Smart Contract developers already building on EVM L1+L2 Chains needing the scalability of map-reduce. 2. Anyone wanting to create decentralised/unstoppable/resilient applications. 3. Anyone with exposure to the token economies of EVM Chains, specifically Ethereum. +4. If funded by the Polkadot Treasury, DOT Stakeholders supporting this adaptation ## Explanation @@ -62,103 +63,93 @@ The following two groups of participants are coordinated with OP Stack nodes: - *ScBG*: System-chain Backing Group, a grouping of System-chain validators who act as the initial guarantors over the result of some computation done off-chain. These are OP Stack Nodes runing the "Map" Collect-Refine using Solidity/EVM Contracts. - *ScBA*: System-chain Block Author, the author of some particular block on the CoreJam System-chain, an EVM L2 ORU also using OP Stack, and run "Reduce" Join-Accumulate using Solidity/EVM Contracts -Unlike Polkadot and Kusama, the CoreJam System Chain (contemplated on Ethereum at L2, and potentially other EVM Chains) is not a relay chain: it does not offer parachain security or have a messaging protocol between parachains. For this reason, CoreJam's RcBG and RcBA are just ScBG and ScBA. In addition, EVM ORUs are centralized. +Unlike Polkadot and Kusama, the CoreJam System Chain (contemplated on Ethereum at L2, and potentially other EVM Chains) is not a relay chain: it does not offer parachain security or a messaging protocol (UMP/DMP/HRMP) between parachains. For this reason, CoreJam's RcBG and RcBA are just ScBG and ScBA. + +Optimistic rollups are called "optimistic" because they assume that transactions are usually valid and only require on-chain settlement on Ethereum L1 in the event of a dispute. Fault proofs have been developed for OP Stack in 2023, described [here](https://github.com/ethereum-optimism/optimism/blob/develop/specs/fault-proof.md) which has been brought to Goerli testnet. ORUs currently store the L2 data in L1 using Call Data, but it is widely expected that EIP-4844's blob transactions will require small adaptations to this. This Ethereum DA capability is expected to have a short window of around 30 days, long enough for a fault proof to be submitted. + +CoreJam makes significant use of Data Availability (DA) resources in both the map/refine and reduce/accumulate steps. In a Polkadot setting, CoreJam on Polkadot would use Polkadot's own DA on the Relay Chain. However, with the CoreJam System as an Ethereum L2 ORU, it would be necessary to rely on Ethereum's blob transactions for DA instead for the critical `accumulate` "on-chain" step. (For the "refine" step, this is not necessary.) It is expected that OP Stack's fault proof system, in accommodating EIP-4844 would be used CoreJam ETH L2's use of blob transactions will mirror the fault proof adjustments. + +As Polkadot DA is believed to be significantly higher throughput and linear in the number of cores, this sets up the Ethereum as a place as an "on-ramp" for CoreJam application developers. ### Overview -The essence of CoreJam is to run a map-reduce process with `refine`: - * map Work Items into Work Results using `refine` - * Work Results using `accumulate` +The essence of CoreJam is to run a map-reduce-like process with: + * `refine` mapping Work Items into Work Results + * `accumulate` mapping Work Results into actual state mutations This can be done permissionlessly in Solidity/EVM Smart Contracts with the following interface: ```solidity -pragma solidity ^0.8.0; - interface CoreJamService { - struct Tuple { - CoreJam.Authorization authorization; - CoreJam.WorkResult[] data; + struct Context { + uint256 blockNumber; + bytes32 prerequisite; } - function refine(bytes memory payload, CoreJam.PackageInfo memory packageInfo) external pure returns (CoreJam.BoundedVecOutput memory); + struct WorkItem { + uint32 service; + bytes payload; + } + + struct WorkResult { + uint32 service; + bytes32 itemHash; + bytes result; + uint256 gas; + } - function applyRefine(CoreJam.WorkItem memory item) external pure returns (CoreJam.WorkResult memory); + struct WorkPackage { + Authorization authorization; + Context context; + WorkItem[] items; + } - function accumulate(CoreJam.Tuple[] memory results) external; + // ********* IN-CORE: Any ScBG can call this -- key design question is how work packages get assigned to cores! + function isAuthorized(WorkPackage package, uint32 coreIndex) external view returns (bool); + // applyRefine maps a WorkItem into a WorkResult + function applyRefine(WorkItem item) external pure returns (WorkResult memory); + // refine maps a package containing work items into work results using applyRefine + function refine(WorkPackage package) external pure returns (WorkResult[] memory); - // TODO: prune + // ********* ON-CHAIN: accumulate is called by the system chain block author (ScBA) given work report attestation + function accumulate(WorkResult[] results) external; + // TODO: prune - removes conflicting items } contract MyCoreJamService is CoreJamService { - using CoreJam for *; - - // ************* Any ScBG can call this - function refine(bytes memory payload, CoreJam.PackageInfo memory packageInfo) external pure returns (CoreJam.BoundedVecOutput memory) { - // Implement refine function logic here based on the provided parameters - // Return refined output as BoundedVecOutput - // Make sure to handle the logic according to your specific use case - CoreJam.BoundedVecOutput memory output; - // Implement the refining logic here - return output; + modifier onlySCBA() { + // restrict accumulate to SCBA only + _; } - function applyRefine(CoreJam.WorkItem memory item) external pure returns (CoreJam.WorkResult memory) { - // Implement applyRefine function logic here based on the provided WorkItem - // Return WorkResult containing service, item_hash, result, and gas - // Make sure to handle the logic according to your specific use case - CoreJam.WorkResult memory result; - // Implement the applyRefine logic here - return result; + function applyRefine(WorkItem calldata item) external pure override returns (WorkResult memory) { } - // Define modifier to restrict access to authorized users - modifier onlyRCBA(uint32 authId, CoreJam.WorkPackage memory package, uint32 coreIndex) { - require(CoreJam.isAuthorized(authorizers[authId].param, package, coreIndex), "Unauthorized"); - _; + function refine(WorkPackage calldata package) external pure override returns (WorkResult[] memory) { } - // ************* only ScBA can call this, and it can mutate the state of others - function accumulate(CoreJam.Tuple[] memory results) external onlyRCBA(authId, package, coreIndex) { - // Implement accumulate function logic here based on the provided parameters - // Iterate through the input results and process each tuple as needed - for (uint256 i = 0; i < results.length; i++) { - CoreJam.Authorization memory authorization = results[i].authorization; - CoreJam.WorkResult[] memory workResults = results[i].data; - - // Process authorization and workResults as needed - // ... - // CoreJam's transfer / on transfer is accomplished with a internal contract to another accumulate - } + function accumulate(Tuple[] calldata results) external override onlySCBA() { + // Process authorization and workResults as needed... + // Note: CoreJam's transfer / on transfer is accomplished with a internal contract to another accumulate } } ``` -Under ordinary circumstances, a CoreJam developer will deploy a CoreJamService along with a mechanism to add Work Items. -The core insights of CoreJam is: -* the "in-core" Collect-Refine (with Smart Contract entry point`refine` ) supports mass parallelization and can be done with a _scalable_ distributed nodes (ScBG) organized by "core index", -* the "on-chain" `accumulate` is done by CoreJam System Chain ScBA, fundamentally bottlenecked by the system chain +The core insight of CoreJam is that the following map-reduce refactorization efficiently addresses the underlying dependency graph of what can be computed in what order better than smart contracts alone, separating what can be parallelised in the `refine` step from what cannot be parallelised in the `accumulate` step +* the "in-core" Collect-Refine (with Smart Contract entry point`refine`) supports mass parallelization and can be done with a _scalable_ distributed nodes (ScBG) organized by "core index", +* the "on-chain" `accumulate` done by CoreJam System Chain ScBA, fundamentally bottlenecked by the system chain -It is thought that CoreJams's map-reduce refactorization efficiently addresses the underlying dependency graph of what can be computed in what order better than smart contracts alone, separating what can be parallelised in the `refine` step. +The above is a first-pass adaptation matching RFC #31, which aims for full generality of Work Package. Work is organized in Work Packages, and in this adaptation, Service is EVM Contract on the ETH L2 whose work items are first preprocessed with cheap decentralized compute power. We anticipate revising the above to support streamlined Service construction. Work Items are a pair where the first item, `service`, itself identifies a pairing of code and state known as a *Service*; and the second item, `payload`, is a block of data which, through the aforementioned code, mutates said state in some presumably useful way. Under ordinary circumstances, a CoreJam developer will deploy a CoreJamService along with a mechanism to add Work Items by users. -A CoreJam Work Package are placed in EVM Optimistic Rollup (ORU) context by mapping the 3 components: +A *Work Package* is an *Authorization* together with a series of *Work Items* and a context, limited in plurality, versioned and with a maximum encoded size. The Context includes an optional reference to a Work Package (`WorkPackageHash`) which limits the relative order of the Work Package (see **Work Package Ordering**, later). In EVM Optimistic Rollup (ORU) context by mapping the 3 components: * Authorization - this just a contract address with a `isAuthorized` interface * Context - this is simplified in an EVM ORU to be just a block number and block hash. This is possible because EVM L2 ORUs in OP Stack are fundamentally centralized sequencers, at least for now. -* Prerequisite - TBD - -A *Work Package* is an *Authorization* together with a series of *Work Items* and a context, limited in plurality, versioned and with a maximum encoded size. The Context includes an optional reference to a Work Package (`WorkPackageHash`) which limits the relative order of the Work Package (see **Work Package Ordering**, later). - -(The number of prerequisites of a Work Package is limited to at most one. However, we cannot trivially control the number of dependents in the same way, nor would we necessarily wish to since it would open up a griefing vector for misbehaving Work Package Builders who interrupt a sequence by introducing their own Work Packages with a prerequisite which is within another's sequence.) +* Prerequisite - TBD. (The number of prerequisites of a Work Package is limited to at most one. However, we cannot trivially control the number of dependents in the same way, nor would we necessarily wish to since it would open up a griefing vector for misbehaving Work Package Builders who interrupt a sequence by introducing their own Work Packages with a prerequisite which is within another's sequence.) -Work Items are a pair where the first item, `service`, itself identifies a pairing of code and state known as a *Service*; and the second item, `payload`, is a block of data which, through the aforementioned code, mutates said state in some presumably useful way. +A Service IS a smart contract stored on-chain and transitioned only using on-chain logic, strictly and deterministically constrained, holding funds and call into each other synchronously. However, the Service's `accumulate` function cannot be transacted with and is only called by either the ScBA or a inter-service calling. Otherwise, all input data (and state progression) must come as the result of a Work Item. A Work Item is a blob of data meant for a particular Service and crafted by some source external to consensus, which can be a user or another Service. It may be thought of as akin to a transaction or internal contract call. -A Service IS a smart contract stored on-chain and transitioned only using on-chain logic, strictly and deterministically constrained, holding funds and call into each other synchronously. -However, the on-chain `accumulate` function cannot be transacted with and is only called by either the ScBA or a inter-service calling. Otherwise, all input data (and state progression) must come as the result of a Work Item. A Work Item is a blob of data meant for a particular Service and crafted by some source external to consensus, which can be a user or another Service. It may be thought of as akin to a transaction or internal contract call. - -The Work Item is first processed *in-core* through `refine`, which is to say an OP Stack standalone node, yielding a distillate known as a *Work Result*. It is this Work Result which is collated together with others of the same service and Accumulated into the Service on-chain. - -In short, a Service is EVM Contract on the ETH L2 whose work items are first preprocessed with cheap decentralized compute power. +The Work Item is first processed *in-core* through `refine`, which is to say an OP Stack node chosen in a System Chain Backing Group (ScBG), yielding a *Work Result*. It is this Work Result which is collated together with others of the same service and Accumulated into the Service on-chain. Though this process happens entirely in consensus, there are two main consensus environments at play, _in-core_ and _on-chain_. We therefore partition the progress into two pairs of stages: Collect & Refine and Join & Accumulate. @@ -178,38 +169,34 @@ Finally, at some point later still `T+r+i+a+o`, the Results of the Work Package ### Collect-Refine -The first two stages of the CoreJam process are *Collect* and *Refine*. *Collect* refers to the collection and authorization of Work Packages (collections of items together with an authorization) to utilize a Polkadot Core. *Refine* refers to the performance of computation according to the Work Packages in order to yield *Work Results*. Finally, each Backing Group member attests to a Work Package yielding a series of Work Results and these Attestations form the basis for bringing the Results on-chain and integrating them into the Service's state which happens in the following stages. +The first two stages of the CoreJam process are *Collect* and *Refine*. *Collect* refers to the collection and authorization of Work Packages (collections of items together with an authorization) to utilize a ScBG Core. *Refine* refers to the performance of computation according to the Work Packages in order to yield *Work Results*. Finally, each Backing Group member attests to a Work Package yielding a series of Work Results and these Attestations form the basis for bringing the Results on-chain and integrating them into the Service's state which happens in the following stages. -#### Collection and `is_authorized` +#### Collection and `isAuthorized` Collection is the means of a Backing Group member attaining a Work Package which is authorized to be performed on their assigned Core at the current time. Authorization is a prerequisite for a Work Package to be included on-chain. Computation of Work Packages which are not Authorized is not rewarded. Incorrectly attesting that a Work Package is authorized is a disputable offence and can result in substantial punishment. On arrival of a Work Package, after the initial decoding, a first check is that the `context` field is valid. This must reference a header hash of a known block which may yet be finalized and the additional fields must correspond to the data of that block. -We introduce the concept of an *Authorizer* procedure, which is a smart contract. Assigning computational resources to an Authorizer implies allowing any Work Package which passes that authorization process to utilize computational resources in order to be submitted on-chain. It controls the circumstances under which ScBGs (OP Stack standalone nodes capable of watching L2 OP Stack) may be rewarded for evaluation and submission of Work Packages (and thus what Work Packages become valid to submit onto CoreJam System Chain). Authorization logic is entirely arbitrary and need not be restricted to authorizing a single collator, Work Package builder, or even a single Service. - -An *Authorizer* is a EVM Smart Contract that is executed and is expressed in this code (which must be capable of in-core VM execution). Its entry-point prototype is: +The *Authorizer* entry point of: ```solidity -interface Authorizer { - function isAuthorized(WorkPackage calldata package, uint32 coreIndex) external view returns (bool); -} +function isAuthorized(WorkPackage calldata package, uint32 coreIndex) external view returns (bool); ``` -This function is executed by a ScBG node in a _metered VM_ and subject to a modest system-wide limitation on execution time. If it overruns this limit or panics on some input, it is considered equivalent to returning `false`. While it is mostly stateless (e.g. isolated from any System-chain state) it is provided with the package's `context` field in order to give information about a recent System-chain block. This allows it to be provided with a concise proof over some recent System-chain state. +is executed by a ScBG node in a _metered VM_ and subject to a modest system-wide limitation on execution time. If it overruns this limit or panics on some input, it is considered equivalent to returning `false`. While it is mostly stateless (e.g. isolated from any System-chain state) it is provided with the package's `context` field in order to give information about a recent System-chain block. This allows it to be provided with a concise proof over some recent System-chain state. A single `Authorizer` value is associated with the index of the Core at a particular System-chain block and limits in some way what Work Packages may be legally processed by that Core. -The need of validators to be rewarded for doing work they might reasonably expect to be useful competes with that of the procurers of work to be certain to get work done which is useful to them. With CoreJam, validators have little ability to identify a high-quality Work Package builder and the permissionless design means a greater expectation of flawed code executing in-core. Because of this, we make a slightly modified approach: Work Packages must have a valid Authorization, i.e. `is_authorized` returns `true` when provided with the Work Package. However, Validators get rewarded for *any* such authorized Work Package, even one which ultimately panics or overruns on its evaluation. +The need of ScBG nodes to be rewarded for doing work competes with that of the procurers of work to be certain to get work done which is useful to them. With CoreJam, ScBG nodes have little ability to identify a high-quality Work Package builder and the permissionless design means a greater expectation of flawed code executing in-core. Because of this, we make a slightly modified approach: Work Packages must have a valid Authorization, i.e. `isAuthorized` returns `true` when provided with the Work Package. However, Validators get rewarded for *any* such authorized Work Package, even one which ultimately panics or overruns on its evaluation. -This ensures that Validators do a strictly limited amount of work before knowing whether they will be rewarded and are able to discontinue and attempt other candidates earlier than would otherwise be the case. There is the possibility of wasting computational resources by processing Work Packages which result in error, but well-written authorization procedures can mitigate this risk by making a prior validation of the Work Items. +This ensures that ScBG nodes do a strictly limited amount of work before knowing whether they will be rewarded and are able to discontinue and attempt other candidates earlier than would otherwise be the case. There is the possibility of wasting computational resources by processing Work Packages which result in error, but well-written authorization procedures can mitigate this risk by making a prior validation of the Work Items. ### Refine The `refine` function is implemented as an entry-point inside a Service Contract Address: ```solidity -function refine(bytes memory payload, CoreJam.PackageInfo memory packageInfo) external pure returns (CoreJam.BoundedVecOutput memory); +function refine(PackageInfo packageInfo) external pure returns (WorkResult[] memory) ``` Both `refine` and `isAuthorized` are only ever executed in-core. Within this environment, we need to ensure that we can interrupt computation not long after some well-specified limit and deterministically determine when an invocation of the VM exhausts this limit. Since the exact point at which interruption of computation need not be deterministic, it is expected to be executed by a streaming JIT transpiler with a means of approximate and overshooting interruption coupled with deterministic metering. @@ -275,6 +262,8 @@ Each System-chain block, every Backing Group representing a Core which is assign WorkReports are gossiped among ScBG nodes (OP Stack nodes) to form a sufficient number attestations, which arrive at the System-chain Block Author. +In an OP Stack setting, a set of OP Stack nodes all can validate the activity of the ScBA, which are chosen via some process to be part of a ScBG. See RFC #3 and discussion of various processes, one of which should be adapted for this purpose and placed in a OP Stack setting. + ### Join-Accumulate Join-Accumulate is the second major stage of computation and is independent from Collect-Refine. Unlike with the computation in Collect-Refine which happens contemporaneously within one of many isolated cores, the consensus computation of Join-Accumulate is both entirely synchronous with all other computation of its stage and operates within (and has access to) the same shared state-machine. @@ -340,9 +329,10 @@ While it will generally be the case that ScBGs know precisely which Work Reports #### Availability -Once the Work Report of a Work Package is Reported on-chain, the Work Package itself must be made *Available* through the off-chain Availability Protocol, which ensures that any dispute over the correctness of the Work Report can be easily objectively judged by all validators. Being off-chain this is _not_ block-synchronized and any given Work Package may take one or more blocks to be made Available or may even fail. +Once the Work Report of a Work Package is Reported on-chain, the Work Package itself must be made *Available* through the off-chain Availability Protocol, which ensures that any dispute over the correctness of the Work Report can be easily objectively judged by all validators. Being off-chain this is _not_ block-synchronized and any given Work Package may take one or more blocks to be made Available or may even fail. -Only once a Work Report's Work Package is made Available the processing continue with the next steps of Joining and Accumulation. Ordering requirements of Work Packages may also affect this variable latency and this is discussed later in the section **Work Package Ordering**. +Only once a Work Report's Work Package is made Available the processing continue with the next steps of Joining and Accumulation. +We will follow RFC #31's soft-ordering. #### Gas Provisioning @@ -406,19 +396,13 @@ Since `accumulate` is permissionless and untrusted code, we must ensure that its Host functions are provided allowing any state changes to be committed at fail-safe checkpoints to provide resilience in case of gas overrun (or even buggy code which panics). The amount of gas remaining may also be queried without setting a checkpoint. `Gas` is expressed in a regular fashion for a solo-chain (i.e. one-dimensional). -`accumulate` of one service may call `accumulate` of another service, as is common with smart contracts This is an entirely synchronous function which transfers the execution over to a `destination` Service as well as the provided `amount` into their account. - - +The `accumulate` of one service may call `accumulate` of another service via internal contract call, which may also transfer a `value` into their account. -### CoreJam Storage API - -To reference large, immutable and long-term data payloads both in-core (`refine`) and on-chain (`accumulate`), we expose a *CoreJam Storage API*, accessible to untrusted code through Solidity library and to trusted System-chain code via a Golang interface. - -Internally, data is stored with a reference count so that two separate usages of `store` need not be concerned about the other. +### CoreJam Storage API + EIP-4844 +To reference large, immutable and long-term data payloads both in-core (`refine`) and on-chain (`accumulate`), we expose a *CoreJam Storage API*, accessible to untrusted code through Solidity library and to trusted System-chain code via a Golang interface. Internally, data is stored with a reference count so that two separate usages of `store` need not be concerned about the other. Every piece of data stored for an untrusted caller requires a sizeable deposit. When used by untrusted code via a host function, the `depositor` would be set to an account controlled by the executing code. - ```golang type Storage interface { /// Immutable function to attempt to determine the preimage for the given `hash`. @@ -480,9 +464,9 @@ New OVM opcodes for each function are as follows: | `APEXP` | `ExpungeUntrusted(depositor AccountID, hash common.Hash) error` | Removes the specified hash and its corresponding data from the storage, based on a request made by the specified depositor. Returns Ok(()) if the operation is successful, an error otherwise. | | `APSTORE` | `Store(data []byte) common.Hash` | Stores the provided data in the storage and returns the hash of the stored data. | - Removing data happens in a two-phase procedure; first the data is unrequested, signalling that calling `lookup` on its hash may no longer work (it may still work if there are other requests active). 24 hours following this, the data is expunged with a second call which, actually removes the data from the chain assuming no other requests for it are active. Only once expunge is called successfuly is the deposit returned. If the data was never provided, or is additional requests are still active, then expunge may be called immediately after a successful unrequest. +The above Storage API is NOT necessary to connect to Ethereum L1 via call data or the future [EIP-4844](https://www.eip4844.com/) by submitting Blob Transactions on Ethereum L1, as `accumulate` calls would be included in OP Stack Blob transactions with a future upgrade that accommodates EIP-4844 in OP Stack. Ethereum's solution of 32 kB/sec of throughput post Dencun (part 2) and 1.3 MB/sec full danksharding should be relevant for `accumulate`. Note that this is significantly lower than Polkadot DA's 66-133MiB/s (assuming 100-200 cores), which presumably would be used for both `refine` and `accumulate`. ### Work Package Ordering @@ -492,22 +476,30 @@ Initial Validation are made on-chain prior to the Availability Protocol begins f ### Fees -Fees are ETH on L2, bridged from Ethereum L1. This allows a succinct *Authorization* (i.e. a small blob of data) to be included in the Work Package which, when fed into the relevant Authorizer function could verify that some Work Package is indeed allowed to utilize that Core at (roughly) that time. A simple proof system would be a regular PKI signature. More complex proof systems could include more exotic cryptography (e.g. multisignatures or zk-SNARKs). - -In this model, we would expect any authorized Work Packages which panic or overrun to result in a punishment to the specific author by the logic of the Service. +Fees are ETH on L2, bridged from Ethereum L1. OP Stack models the L1 fees explicitly. While both the ScBG `refine` and ScBA `accumulate` are both metered in gas usage, the relatively scarce nature of the `accumulate` stage demands a massive premium over the `refine`. ## Performance, Ergonomics and Compatibility, Notes on Implementation in OP Stack -We envision an initial version of this proposal with modifications to the OP Stack: +Ethereum L2 Users of a Service on the CoreJam System Chain should be able to submit transactions that result in Work Items causing Work Packages. CoreJam's map-reduce architecture is most valuable when significant portions of gas consumption would have been expensive to do in `accumulate` may be done at lower gas with fewer compute resources in `refine`. -1. We require a mapping between Work Packages and Core Index, with an equally easy to implement ScBG VRF, following [Dynamic Backing Groups: Preparing Cores for Agile Scheduling](https://forum.polkadot.network/t/dynamic-backing-groups-preparing-cores-for-agile-scheduling/3629/10) -2. We require extending the [Optimistic Virtual Machine] Availability Protocol in OP Stack, using KAGOME's erasure coding, using Go C++ bindings. It is not clear how the CoreJam Storage API relates to Optimism fraud proofs. -3. It is not claimed that the CoreJam L2 Chain is +We envision an initial version of this proposal with modifications to the OP Stack: +1. We require a Work Package scheduling system that can orchestrate a distributed array of "cores". This in turn requires a mapping between Work Packages/Services and Core Index, with implementable ScBG strategy, following [Dynamic Backing Groups: Preparing Cores for Agile Scheduling](https://forum.polkadot.network/t/dynamic-backing-groups-preparing-cores-for-agile-scheduling/3629/10) and RFC #3. +2. We require metering ScBG and ScBA and charging the user in "Instantaneous" ETH in a manner consistent with the relative scarcity of each resource, and compensating the cores for their contributions. It may be essential to adopt the Bulk CoreTime model for services instead for efficiency. +3. We require extending the [Optimistic Virtual Machine] Availability Protocol in OP Stack, using KAGOME's erasure coding or similar, using Go C++ bindings. +4. We require OP Stack fault proofs extended to use EIP-4844 blobs to accommodate the above ## Testing, Security and Privacy -Standard testing as per OP Stack and Solidity security auditing applies. +ORUs derive their security largely from fault proofs, where we expect to rely on OP Stack's Fault Proof / Dispute game, developed by Optimism. + +The fault proof mechanism must be extended to accommodate EIP-4844, not just for blocks but for the `refine` work result output. It is necessary to extend the OP Stack Fault proof game. + +The precise details of Work Reports / Attestation from "in core" nodes where Attestation / Work Reports can be run by a large number of nodes participating in ScBGs needs a precise signature scheme. + +The fundamental work product is not Solidity contracts, but representative pedagogical service contracts merit expert attention. + +Testing norms have been established in OP Stack, which must be matched OP Stack implementation. The proposal introduces no new privacy concerns. @@ -525,4 +517,4 @@ It is highly desirable to: ## Prior Art and References -This is an adaptation of RFC #31. +This is an adaptation of RFC #31. We are grateful for useful feedback from both the Polkadot and Ethereum community. From cfd9b75c815ad06bed3f0c41237a9939719eb921 Mon Sep 17 00:00:00 2001 From: Sourabh Niyogi Date: Tue, 7 Nov 2023 00:38:31 -0800 Subject: [PATCH 5/8] adjustments --- text/0033-corejam-system-chain-evm-l2s.md | 48 ++++++----------------- 1 file changed, 12 insertions(+), 36 deletions(-) diff --git a/text/0033-corejam-system-chain-evm-l2s.md b/text/0033-corejam-system-chain-evm-l2s.md index d53f7fe05..45d65c858 100644 --- a/text/0033-corejam-system-chain-evm-l2s.md +++ b/text/0033-corejam-system-chain-evm-l2s.md @@ -1,9 +1,9 @@ -# RFC-0033: CoreJam System Chain on Ethereum L2 +# RFC-0033: CoreJam System Chain on EVM L2s with OP Stack | | | | --------------- | ------------------------------------------------------------------------------------------- | | **Start Date** | 5 November 2023 | -| **Description** | CoreJam System Chain on Ethereum L2 with OP Stack | +| **Description** | CoreJam System Chain on EVM L2s with OP Stack | | **Authors** | Sourabh Niyogi | ## Summary @@ -29,7 +29,7 @@ In order of importance: 1. This proposal must be compatible with Solidity+Ethereum to have developer and ultimately, user impact. 2. The CoreJam implementation should be practical, needing minimal changes to OP Stack. 3. Utilization of CoreJam must be permissionless and maintain the same security model of Optimistic Rollups. -4. There should be a clear path for CoreJam System Chain, despite its ORU L2 setting, to support decentralization of nodes. +4. There should be a path for the CoreJam System Chain, despite its ORU L2 setting, to support permissionless ScBG nodes and ScBA nodes. ## Stakeholders @@ -67,22 +67,23 @@ Unlike Polkadot and Kusama, the CoreJam System Chain (contemplated on Ethereum a Optimistic rollups are called "optimistic" because they assume that transactions are usually valid and only require on-chain settlement on Ethereum L1 in the event of a dispute. Fault proofs have been developed for OP Stack in 2023, described [here](https://github.com/ethereum-optimism/optimism/blob/develop/specs/fault-proof.md) which has been brought to Goerli testnet. ORUs currently store the L2 data in L1 using Call Data, but it is widely expected that EIP-4844's blob transactions will require small adaptations to this. This Ethereum DA capability is expected to have a short window of around 30 days, long enough for a fault proof to be submitted. -CoreJam makes significant use of Data Availability (DA) resources in both the map/refine and reduce/accumulate steps. In a Polkadot setting, CoreJam on Polkadot would use Polkadot's own DA on the Relay Chain. However, with the CoreJam System as an Ethereum L2 ORU, it would be necessary to rely on Ethereum's blob transactions for DA instead for the critical `accumulate` "on-chain" step. (For the "refine" step, this is not necessary.) It is expected that OP Stack's fault proof system, in accommodating EIP-4844 would be used CoreJam ETH L2's use of blob transactions will mirror the fault proof adjustments. +CoreJam makes significant use of Data Availability (DA) resources in both the map/refine and reduce/accumulate steps. In a Polkadot setting, CoreJam on Polkadot would use Polkadot's own DA on the Relay Chain. However, with the CoreJam System as an Ethereum L2 ORU, it would be necessary to rely on Ethereum's blob transactions for DA instead for the critical `accumulate` "on-chain" step. (For the "refine" step, this is not necessary.) It is expected that OP Stack's fault proof system would be extended to support EIP-4844. As Polkadot DA is believed to be significantly higher throughput and linear in the number of cores, this sets up the Ethereum as a place as an "on-ramp" for CoreJam application developers. ### Overview The essence of CoreJam is to run a map-reduce-like process with: - * `refine` mapping Work Items into Work Results - * `accumulate` mapping Work Results into actual state mutations + * `refine` mapping Work Items into Work Results "in-core" by ScBG nodes + * `accumulate` mapping Work Results into "on-chain" by the ScBA, given This can be done permissionlessly in Solidity/EVM Smart Contracts with the following interface: ```solidity interface CoreJamService { struct Context { - uint256 blockNumber; + uint256 block_number; + bytes32 header_hash; bytes32 prerequisite; } @@ -260,9 +261,7 @@ type Attestation struct { Each System-chain block, every Backing Group representing a Core which is assigned work provides a series of Work Results coherent with an authorized Work Package. Validators are rewarded when they take part in their Group and process such a Work Package. Thus, together with some information concerning their execution context, they sign a *Report* concerning the work done and the results of it. This is also known as a *Candidate*. This signed Report is called an *Attestation*, and is provided to the System-chain block author. If no such Attestation is provided (or if the System-chain block author refuses to introduce it for Reporting), then that Backing Group is not rewarded for that block. -WorkReports are gossiped among ScBG nodes (OP Stack nodes) to form a sufficient number attestations, which arrive at the System-chain Block Author. - -In an OP Stack setting, a set of OP Stack nodes all can validate the activity of the ScBA, which are chosen via some process to be part of a ScBG. See RFC #3 and discussion of various processes, one of which should be adapted for this purpose and placed in a OP Stack setting. +WorkReports are gossiped among ScBG nodes (OP Stack nodes, using op-geth's libp2p-based messaging) to form a sufficient number attestations, which arrive at the ScBa. In an OP Stack setting, a set of OP Stack nodes all can validate the activity of the ScBA, which are chosen via some process to be part of a ScBG. See RFC #3 and discussion of design alternatives, one of which should be adapted for this purpose and placed in a OP Stack setting. ### Join-Accumulate @@ -286,38 +285,15 @@ There are a number of Initial Validation requirements which the ScBA must do in Firstly, any given Work Report must have enough signatures in the Attestation to be considered for Reporting on-chain. Only one Work Report may be considered for Reporting from each ScBG per block. -Secondly, any Work Reports introduced by the ScBA must be *Recent*, defined as having a `context.header_hash` which is an ancestor of the ScBA head and whose height is less than `RECENT_BLOCKS` from the block which the ScBA is now authoring. - -```golang -const RECENT_BLOCKS uint32 = 16 -``` +Secondly, any Work Reports introduced by the ScBA must be *Recent*, at a height is less than `RECENT_BLOCKS` (eg 16) from the block which the ScBA is now authoring. -Thirdly, dependent elements of the Context (`context.block_number`) must correctly correspond to those on-chain for the block corresponding to the provided `context.header_hash`. For this to be possible, the System-chain is expected to track Recent state roots in a queue. +Thirdly, dependent elements of the Context (`context.block_number`) must correctly correspond to those on-chain for the block corresponding to the provided `context.header_hash`. For this to be possible, the System-chain is expected to track Recent blocks in a queue. Fourthly, the ScBA may not attempt to report multiple Work Reports for the same Work Package. Since Work Reports become inherently invalid once they are no longer *Recent*, then this check may be simplified to ensuring that there are no Work Reports of the same Work Package within any *Recent* blocks. Finally, the ScBA may not register Work Reports whose prerequisite is not itself Reported in *Recent* blocks. -In order to ensure all of the above tests are honoured by the ScBA, a block which contains Work Reports which fail any of these tests shall panic on import. The System-chain's on-chain logic will thus include these checks in order to ensure that they are honoured by the ScBA. We therefore introduce the *Recent Reports* storage item, which retaining all Work Package hashes which were Reported in the *Recent* blocks: - -```golang -const MAX_CORES = 512 -/// Must be ordered. -type ReportSet = Items [MAX_CORES]WorkPackageHash -type WorkPackageHash common.Hash - -type BoundedVec struct { - Items []WorkPackageHash -} - -type ReportSet struct { - BoundedVec -} - -type RecentReports struct { - StorageValue BoundedVec -} -``` +In order to ensure all of the above tests are honored by the ScBA, a block which contains Work Reports which fail any of these tests shall panic on import. The System-chain's on-chain logic will thus include these checks in order to ensure that they are honoured by the ScBA. The ScBA should track *Recent Reports*, and retain all Work Package hashes which were Reported in the *Recent* blocks. The ScBA must keep an up to date set of which Work Packages have already been Reported in order to avoid accidentally attempting to introduce a duplicate Work Package or one whose prerequisite has not been fulfilled. Since the currently authored block is considered *Recent*, Work Reports introduced earlier in the same block do satisfy the prerequisite of Work Packages introduced later. From 1d468101f88511c5767c5827cb2010fc3c332dc0 Mon Sep 17 00:00:00 2001 From: Sourabh Niyogi Date: Wed, 8 Nov 2023 02:18:41 -0800 Subject: [PATCH 6/8] refine --- text/0033-corejam-system-chain-evm-l2s.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/text/0033-corejam-system-chain-evm-l2s.md b/text/0033-corejam-system-chain-evm-l2s.md index 45d65c858..d579dd9dd 100644 --- a/text/0033-corejam-system-chain-evm-l2s.md +++ b/text/0033-corejam-system-chain-evm-l2s.md @@ -442,7 +442,9 @@ New OVM opcodes for each function are as follows: Removing data happens in a two-phase procedure; first the data is unrequested, signalling that calling `lookup` on its hash may no longer work (it may still work if there are other requests active). 24 hours following this, the data is expunged with a second call which, actually removes the data from the chain assuming no other requests for it are active. Only once expunge is called successfuly is the deposit returned. If the data was never provided, or is additional requests are still active, then expunge may be called immediately after a successful unrequest. -The above Storage API is NOT necessary to connect to Ethereum L1 via call data or the future [EIP-4844](https://www.eip4844.com/) by submitting Blob Transactions on Ethereum L1, as `accumulate` calls would be included in OP Stack Blob transactions with a future upgrade that accommodates EIP-4844 in OP Stack. Ethereum's solution of 32 kB/sec of throughput post Dencun (part 2) and 1.3 MB/sec full danksharding should be relevant for `accumulate`. Note that this is significantly lower than Polkadot DA's 66-133MiB/s (assuming 100-200 cores), which presumably would be used for both `refine` and `accumulate`. +The Storage API should be connectable to Ethereum [EIP-4844](https://www.eip4844.com/) with Blob Transactions. While `accumulate` calls would be included in OP Stack Blob transactions, the veracity of the `refine` requires that the WorkItems (which may be large in number) must also be executable on Ethereum L1, requiring the `refine` inputs to be included. Fault proofs for both `refine` "in-core" and `accumulate` "on-chain" calls are necessary. + +Ethereum's solution of 32 kB/sec of throughput post Dencun (part 2) and 1.3 MB/sec full danksharding should be relevant. Note that this is significantly lower than Polkadot DA's 66-133MiB/s (assuming 100-200 cores), which presumably would be used for both `refine` and `accumulate`. ### Work Package Ordering From c28b080f7f3f330a9da6557bc1a754c006fea5e3 Mon Sep 17 00:00:00 2001 From: Sourabh Niyogi Date: Sat, 26 Oct 2024 07:14:11 -0700 Subject: [PATCH 7/8] Update 0033-JAM-Service-for-Validating-Ethereum-Optimistic-Rollups.md JAM's *rollup host* function can be extended from Polkadot rollups to non-Polkadot rollups. We outline the design of a JAM service capable of securing Ethereum *optimistic* rollups, such as OP Stack and ArbOS. This service transforms optimistic rollups to cynical rollups, allowing users to benefit from the finality and high throughput of JAM, alongside JAM's anticipated messaging service. This JAM service's competitive advantage enables rollup operators to choose JAM/Polkadot over Ethereum or other rollup providers. A precise design for the service's refine-accumulate function is outlined, using tools already available. --- ...-Validating-Ethereum-Optimistic-Rollups.md | 200 +++++++ text/0033-corejam-system-chain-evm-l2s.md | 498 ------------------ 2 files changed, 200 insertions(+), 498 deletions(-) create mode 100644 text/0033-JAM-Service-for-Validating-Ethereum-Optimistic-Rollups.md delete mode 100644 text/0033-corejam-system-chain-evm-l2s.md diff --git a/text/0033-JAM-Service-for-Validating-Ethereum-Optimistic-Rollups.md b/text/0033-JAM-Service-for-Validating-Ethereum-Optimistic-Rollups.md new file mode 100644 index 000000000..a00b49d62 --- /dev/null +++ b/text/0033-JAM-Service-for-Validating-Ethereum-Optimistic-Rollups.md @@ -0,0 +1,200 @@ + +# RFC-0033: JAM Service for Validating Ethereum Optimistic Rollups + +| | | +| --------------- | ------------------------------------------------------------------------------------------- | +| **Start Date** | 26 October 2024 | +| **Description** | JAM Service for Securing Ethereum Optimistic Rollups | +| **Authors** | Sourabh Niyogi | +| **Abstract** | JAM's *rollup host* function can be extended from Polkadot rollups to non-Polkadot rollups. We outline the design of a JAM service capable of securing Ethereum *optimistic* rollups, such as OP Stack and ArbOS. This service transforms optimistic rollups to cynical rollups, allowing users to benefit from the finality and high throughput of JAM, alongside JAM's anticipated messaging service. This JAM service's competitive advantage enables rollup operators to choose JAM/Polkadot over Ethereum or other rollup providers. A precise design for the service's refine-accumulate function is outlined, using tools already available. | + +## Background + +The Gray Paper suggests a design for applying the same audit protocol from Polkadot's parachain validation service to ETH rollups: "_Smart-contract state may be held in a coherent format on the JAM chain so long as any updates are made through the 15kb/core/sec work results, which would need to contain **only the hashes of the altered contracts’ state roots**._" This proposal concretely addresses two top non-Polkadot optimistic rollup platforms: [OP Stack](https://stack.optimism.io/) and [ArbOS](https://docs.arbitrum.io/how-arbitrum-works/arbos/introduction). + +Optimistic rollups use centralized sequencers and have no forks, creating an illusion of fast finality while actually relying on delayed fraud proofs. Optimistic rollups are termed "optimistic" because they assume transactions are valid by default, requiring fraud proofs on Ethereum L1 if a dispute arises. Currently, ORUs store L2 data on ETH L1, using EIP-4844's blob transactions or similar DA alternatives, just long enough to allow for fraud proof submission. + +This approach, however, incurs a cost: a 7-day exit window to accommodate fraud proofs. JAM Service can eliminate this long exit window by validating optimistic rollups. + +## Motivation + +JAM is intended to host rollups rather than serve end users directly. + +A secure JAM service for Optimistic Rollups will expand JAM's service scope to include optimistic rollups and enhance their appeal with JAM's finality. + +This expanded focus on rollups will increase CoreTime demand, making JAM attractive to both existing and new optimistic rollups by reducing exit windows. + +This JAM Service converts optimistic rollups into cynical rollups, delivering a user experience advantage that is marketable. With popular CEXes (Coinbase, Kraken) adopting OP Stack, this improvement is highly visible to retail users, turning ORUs into CRUs "Secured by JAM" for a competitive edge. + +## Requirements + +1. Securing optimistic rollups with a JAM Service should be practical and require minimal changes by the ORU operator. +2. Securing Polkadot rollups with a JAM Service should **not** be affected. + +## Stakeholders + +1. Optimistic Rollup Operators seeking shorter exit windows, lower costs, higher scalability, and DA. +2. Web3 developers wanting to create applications on cynical rollups secured by JAM. +3. DOT Holders aiming to increase CoreTime demand. + +# JAM Service Design + +## Overview + +Rollups produce a sequence of blocks ${\bf B}$ with headers ${\bf H}$. The header ${\bf H}$ contains a parent header hash ${\bf H}_p$ and a state root ${\bf H}_{r}$, which represents the global state after applying all block transactions. The transactions trie root is unnecessary here; validating the state trie root alone is sufficient for securing rollups. + +This JAM Service strategy, as hinted in the JAM Gray Paper, aggregates storage proofs of altered smart account/contract states into a chain of work packages. These validate an uninterrupted chain of an optimistic rollup's blocks against the state root ${\bf H}_r$. + +Instead of relying on a *promise* that the state root is correct unless proven otherwise, JAM validates it using storage proofs for each contract update across $N$ blocks. + +The rollup operator submits headers and Merkle proofs in a chain of work packages $..., p_{n-1}, p_{n}, p_{n+1}, ...$ corresponding to the rollup chain, grouping $N$ blocks of data in each package. Typically, each work package $p_n$ requires a prerequisite work package $p_{n-1}$. + +## Refine: + +#### Key Input: Work Packages + +In JAM's refine code, work package content is as follows: + +| JAM Refine | Content | +| --- | --- | +| *Work Package* $p_n \in \mathbb{P}$ | Data submitted by ORU operator for validation of blocks $i$ through $j$ | +| *Payload* ${\bf y}$ | Chain ID (e.g., 10, 42161, etc.), start and end block numbers expected in extrinsics | +| *Imported Data Segment* ${\bf i}$ | Last ${\bf H}$ validated by the service in the _previous_ work package $p_{n-1}$ | +| *Extrinsics* $\bar{{\bf x}}$ | Header ${\bf H}$, block hash $H({\bf B})$, and Merkle proofs $\Pi$ detailed below | +| *Work Items* ${\bf w} \in \mathbb{I}$ | Summary of the first and last blocks ($i$ to $j$) | +| *Exported Data Segment* ${\bf e}$ | Last validated ${\bf H}$, used in the _next_ work package $p_{n+1}$ | +| *Work Result* ${\bf r}$ | $N$ header hashes $H({\bf H})$ and $N$ block hashes $H({\bf B})$ | + +Refine's operations: + +1. Check for authorization ${\bf o}$. +2. Perform `historical_lookup` to fetch the last on-chain state in the anchor block. +3. `Import` the last header ${\bf H}$ and confirm that the imported data segment ${\bf i}$ matches the `historical_lookup`. +4. For each extrinsic header in ${\bf H}$: + - Ensure that each header forms a chain, with ${\bf H}_p$ matching the previous extrinsic $H({\bf H})$, except for the first header, which must match the previous 2. + - Using the state root ${\bf H}_r$, validate all Merkle proofs. +5. `Export` the header of the last extrinsic successfully validated. +6. Output the work result, up to $2(j-i+1)$ hashes: two hashes per extrinsic, $H({\bf H})$ and $H({\bf B})$. + +The refine process uses 3 host functions, with the main computation focused on Merkle proof validation. + +As detailed in GP, work results in guaranteed work reports are assured, audited, and judged following the ELVES protocol. JAM finalizes its audited blocks via GRANDPA voting, ensuring the rollup’s state can be trusted for subsequent operations. + +### Key Validation Process: Storage Proofs + +Optimistic rollups, including OP Stack and ArbOS, use Ethereum's Merkle Patricia Trie to commit to state data on each block. See [this architecture diagram](https://i.sstatic.net/OdfkP.jpg) for an overview. The state root ${\bf H}_r$ is a cryptographic commitment to the entire rollup state at that block height. Notably, validation does not calculate the state root ${\bf H_r}$ based on block _transactions_, which would necessitate the entire state of the rollup. Instead, storage proofs (or "witness data") enable the ORU to prove that specific storage values exist within a particular state root. Only the last witness per contract address is needed in a work package of $N$ blocks. + +The central assumption is that if the work package proves all account/contract storage items over $N$ of blocks, which are guaranteed/assured/audited by a subset of JAM validators, then the sequence of state transitions in these blocks is valid. + + +This chaining of state roots ensures continuity and trust in the rollup secured by this JAM Service: +* Each block’s state root can be verified against the previous state, allowing nodes to validate state changes trustlessly. + +* Inclusion in Parent Block Hash: Since each block header ${\bf H}$ includes the previous header hash ${\bf H}_p$ and indirectly references the previous state root ${\bf H}_r$, each block and header also serves as a cryptographic reference to the prior state. + +The core operation verifies proofs against the state root, enabling efficient, trustless block sequence validation based solely on headers, which are much smaller than full transaction blocks. + +For now, we consider _100%_ of storage keys and values output by the trace of a block. A more sophisticated approach to sample the set of storage keys based on JAM's public entropy $\eta_3$ may be warranted to limit work packages to within JAM's limits, made available through `historical_lookup` operations. + + +## Accumulate + +The `accumulate` process ensures that all rollup data necessary to validate the rollup is available in JAM DA. Accumulate is an entirely synchronous *on-chain* operation performed by all JAM Validators (rather than *in-core* as with Refine). Computation is carried out first by the JAM Block Author and the newly authored block sent to all Validators. + +#### On-chain Storage of Rollup State + +The service maintains a summary of fully validated blocks of each chain in 3 service storage keys: +* the first block number $a$ validated with data fully available +* the last block number $b$ validated with data fully available +* a window parameter $w$, modeling the maximum $b-a$ + +For every block number in this range, the block number has its own key to store the header hash and the block hash. + +The function of this data storage represents **both** of the following: +* the refine validation has been completed **and** +* the block and header preimages are available in JAM's DA + +#### Key Process + +| JAM Accumulate | Content | +| --- | --- | +| *Wrangled Work Results* | Chain ID (e.g., 10, 42161, etc.), block number start $i$ and end $j$ followed by pairs of $H({\bf H})$ and $H({\bf B})$ | + +1. `read` fetch $a, b, w$ based on Chain ID. Validate that $i=b+1$. +2. `solicit` is called for both hashes in the wrangled work results. +3. Advance $b$ by 1 if: + - both ${\bf a}_l(H({\bf B}))$ and ${\bf a}_l(H({\bf H}))$ are "available" + - the header ${\bf H}$ is contained with ${\bf B}$ +5. If $b-a$ exceeds the window $w$, advance $a$ by 1 if both ${\bf a}_l(H({\bf B}))$ and ${\bf a}_l(H({\bf H}))$ are "forget". +6. `write` $a$ and $b$ back to storage. + +Necessarily, the ORU operator must provide both preimages for the chain to be considered valid. If any block or header is not made available by the ORU operator, $b$ will not advance. + +Under normal operations, the above process will result in a full set of blocks and header preimages being in JAM's DA layer. + +Moreover, the `forget` operation ensures that storage is bounded by $w$. + +### Transfer + +Transfer details concerning fees of storage balance are not considered at this time. A full model appears to necessitate a clear interface between JAM and CoreTime as well as a model of how this service can support a single vs. multiple Chain IDs. Naturally, all service fees would be in DOT rather than ETH on L2. + +Multiple service instantiations under different CoreTime leases may be envisioned for each rollup, but a set of homogenous ORU rollups to support sync/async models by the same ORU operator could also be supported. + +### Beefy + +Following the BEEFY protocol, JAM's own state trie aggregates work packages' accumulate roots into its own recent history $\beta$, signed by JAM validators. BEEFY’s model allows JAM's state to be verified on Ethereum and other external chains without needing all JAM Chain data on the external chain. +All rollups, whether built on Substrate or optimistic turned cynical, are committed into this state, supporting cross-ecosystem proofs using Merkle trees as today. + +To support this goal, the accumulate result is the last block hash, last header hash, or last state root as suitable to match Polkadot rollup behavior. + +### Service PoC Implementation + +This JAM Service can be implemented in Rust using `polkatool` and tested on a JAM Testnet by JAM implementers. A Keccak256 host function is used for expedience, and the preimage hash function will be adjusted to this for PoC. + +Real-world work packages can be developed for OP Stack chains like Base and Optimism (currently with 30 blocks/minute) and Arbitrum, which operates at 150 blocks/minute. The results of "trace_block" JSON-RPC calls enable 3 x 1440 work packages per day for these top 3 ETH rollups. This is ample for PoC. + +### Gas + Storage Considerations + +JAM's gas parameters for PVM instructions and host calls have not been fully developed yet. Combining Polkadot and non-Polkadot rollup services will provide valuable data for determining these parameters. + +The gas required for `refine` is proportional to the number of storage proofs submitted in a work package and the average size of each proof. To upper-bound storage proofs, sampling using an entropy state variable $\eta_3$ may be necessary, although accessing it remains unclear. + +The gas required for `accumulate` is proportional to the number of solicit and forget operations. It is believed that accumulate's gas cost is nominal and poses no significant issue. However, the storage rent model common to all blockchain storage applies to the preimage, which is explicitly tallied. Fortunately, this is upper-bounded by $w$. + +## Compatibility + +#### CoreTime + +Instead of ETH, rollups would require DOT for CoreTime to secure their rollup. However, rollups are not locked into JAM and may freely enter and exit the JAM ecosystem since work packages do not need to start at genesis. + +Different rollups may need to scale their core usage based on rollup activity. JAM's connectivity to CoreTime is expected to handle this effectively. + +#### Hashing + +Currently, preimages are specified to use the Blake2b hash, while Ethereum rollup block hashes utilize Keccak256. + +## Testing, Security, and Privacy + +The described service requires expert attention from security experts familiar with JAM, ELVES, and Ethereum. + +The ELVES and JAM protocols are expected to undergo audit with the 1.0 ratification of JAM. + +The proposal introduces no new privacy concerns. + +Raw PVM assembly may be preferred in a production implementation, along with a host function for Keccak256 to minimize issues related to compiler bugs. + +## Future Directions and Related Material + +Solutions to model other elements in the state trie (account balances and nonces) are not covered in this service design yet. + +As JAM implementers move towards conformant implementations in 2025, which support gas modeling and justify performance improvements, a proper model of storage costs and fees will be necessary. + +JAM enables a semi-coherent model for non-Polkadot rollups, starting with optimistic rollups as described here. A similar service may be envisioned for mature ZK rollup ecosystems. + +An interoperable JAM messaging service between both kinds of non-Polkadot rollups and Polkadot rollups would be highly desirable. + +The growth of optimistic rollup platforms, led by OP Stack with CEXes (Coinbase and Kraken) and ArbOS, threatens JAM's viability as a rollup host. Achieving JAM's rollup host goals may require urgency to match the speed at which these network effects emerge. + +## Drawbacks, Alternatives, and Unknowns + +Alongside Ethereum DA (via EIP-4844), numerous DA alternatives for rollups exist: Avail, Celestia, Eigenlayer. ORUs rely primarily on fraud proofs, but they require a lengthy 7-day window for these "fraud proofs." The cynical rollup model offers significant UX improvements by eliminating this "exit window," commonly 7 days. None of these alternatives provide the same UX enhancement enabled by "cynical" rollups. diff --git a/text/0033-corejam-system-chain-evm-l2s.md b/text/0033-corejam-system-chain-evm-l2s.md deleted file mode 100644 index d579dd9dd..000000000 --- a/text/0033-corejam-system-chain-evm-l2s.md +++ /dev/null @@ -1,498 +0,0 @@ -# RFC-0033: CoreJam System Chain on EVM L2s with OP Stack - -| | | -| --------------- | ------------------------------------------------------------------------------------------- | -| **Start Date** | 5 November 2023 | -| **Description** | CoreJam System Chain on EVM L2s with OP Stack | -| **Authors** | Sourabh Niyogi | - -## Summary - -CoreJam (RFC [#31](https://github.com/polkadot-fellows/RFCs/pull/31)) offers a significant refactorization of the blockchain state transition function with its map-reduce architecture, starting with a Polkadot/Substrate 2.0 foundation. This proposal adapts the CoreJam architecture to EVM L2s, specifically utilizing [OP Stack](https://stack.optimism.io/) + Solidity instead of Polkadot/Substrate. The endpoint being proposed is a _CoreJam System Chain on Ethereum L2_. Almost all CoreJam concepts from #31 are retained (but none of Polkadot 1.0's primary "product" of parachains), where CoreJam's Rust interfaces are replaced with Solidity/EVM Contracts + OP Stack's Golang. - -In this adaptation of "Collect-Refine-Join-Accumulate", a Solidity/EVM Smart Contract has two primary entry-points: `refine` and `accumulate`. Work packages are processed "in-core" (and thus parallelized) via `refine`, and the refined outputs of this processing are gathered together and an on-chain state-machine progressed according to `accumulate`. - -For pedagogical reasons, the write up below retains much of the original language. - -## Motivation - -* CoreJam impact in Ethereum and other EVM L1s -* Provide an on-ramp to CoreJam developers already familiar with Solidity -* Reduced Time-to-market of CoreJam in OP Stack relative to Polkadot 2.0 Engineering - -From a usability and impact perspective, having CoreJam's map-reduce in Ethereum expands the set of CoreJam developers from a small but growing number Substrate+Rust developers solely in the Polkadot ecosystem to a large but growing number of Solidity developers in a larger number of EVM ecosystems beyond Polkadot alone. Provided that Polkadot executes on its high throughput DA (proportional to number of cores), minimal relay chain scalability, and genuine L1 decentralization, it is believed Polkadot may benefit from an increasing number of CoreJam developers onboarded from Ethereum and other EVM Chains through this adaptation. - -## Requirements - -In order of importance: - -1. This proposal must be compatible with Solidity+Ethereum to have developer and ultimately, user impact. -2. The CoreJam implementation should be practical, needing minimal changes to OP Stack. -3. Utilization of CoreJam must be permissionless and maintain the same security model of Optimistic Rollups. -4. There should be a path for the CoreJam System Chain, despite its ORU L2 setting, to support permissionless ScBG nodes and ScBA nodes. - -## Stakeholders - -1. Solidity/EVM Smart Contract developers already building on EVM L1+L2 Chains needing the scalability of map-reduce. -2. Anyone wanting to create decentralised/unstoppable/resilient applications. -3. Anyone with exposure to the token economies of EVM Chains, specifically Ethereum. -4. If funded by the Polkadot Treasury, DOT Stakeholders supporting this adaptation - -## Explanation - -**CoreJam is a general model for utilization of Cores. It is a mechanism by which Work Packages are communicated, authorized, computed and verified, and their results gathered, combined and accumulated into particular parts of the CoreJam chain's state.** - -### CoreJam in Polkadot Relay Chain vs CoreJam System Chain on ETH/.. L2 - -All of CoreJams interrelated concepts: *Work Package*, *Service*, *Work Item*, *Work Output*, *Work Result*, *Work Report*, *Guarantee* and *Service Trie* are situated within OP Stack. - -| CoreJam model | Context | -| --- | --- | -| *CoreJam EVM L2 Chain* | Primary block-chain | -| *Work Package* | Untrusted data provided to ScBG | -| *Work Item* | State-transition inputs and witness | -| *Work Output* | State-transition consequence | -| *Work Report* | Target of attestation | -| *(Work Package) Attestation* | Output signed in attestation | -| *Reporting* | Placement of Attestation on-chain | -| *Integration* | Irreversible transition of state | -| *Builder* | Creator of data worthy of Attestation | - -The following two groups of participants are coordinated with OP Stack nodes: - -- *ScBG*: System-chain Backing Group, a grouping of System-chain validators who act as the initial guarantors over the result of some computation done off-chain. These are OP Stack Nodes runing the "Map" Collect-Refine using Solidity/EVM Contracts. -- *ScBA*: System-chain Block Author, the author of some particular block on the CoreJam System-chain, an EVM L2 ORU also using OP Stack, and run "Reduce" Join-Accumulate using Solidity/EVM Contracts - -Unlike Polkadot and Kusama, the CoreJam System Chain (contemplated on Ethereum at L2, and potentially other EVM Chains) is not a relay chain: it does not offer parachain security or a messaging protocol (UMP/DMP/HRMP) between parachains. For this reason, CoreJam's RcBG and RcBA are just ScBG and ScBA. - -Optimistic rollups are called "optimistic" because they assume that transactions are usually valid and only require on-chain settlement on Ethereum L1 in the event of a dispute. Fault proofs have been developed for OP Stack in 2023, described [here](https://github.com/ethereum-optimism/optimism/blob/develop/specs/fault-proof.md) which has been brought to Goerli testnet. ORUs currently store the L2 data in L1 using Call Data, but it is widely expected that EIP-4844's blob transactions will require small adaptations to this. This Ethereum DA capability is expected to have a short window of around 30 days, long enough for a fault proof to be submitted. - -CoreJam makes significant use of Data Availability (DA) resources in both the map/refine and reduce/accumulate steps. In a Polkadot setting, CoreJam on Polkadot would use Polkadot's own DA on the Relay Chain. However, with the CoreJam System as an Ethereum L2 ORU, it would be necessary to rely on Ethereum's blob transactions for DA instead for the critical `accumulate` "on-chain" step. (For the "refine" step, this is not necessary.) It is expected that OP Stack's fault proof system would be extended to support EIP-4844. - -As Polkadot DA is believed to be significantly higher throughput and linear in the number of cores, this sets up the Ethereum as a place as an "on-ramp" for CoreJam application developers. - -### Overview - -The essence of CoreJam is to run a map-reduce-like process with: - * `refine` mapping Work Items into Work Results "in-core" by ScBG nodes - * `accumulate` mapping Work Results into "on-chain" by the ScBA, given - -This can be done permissionlessly in Solidity/EVM Smart Contracts with the following interface: - -```solidity -interface CoreJamService { - struct Context { - uint256 block_number; - bytes32 header_hash; - bytes32 prerequisite; - } - - struct WorkItem { - uint32 service; - bytes payload; - } - - struct WorkResult { - uint32 service; - bytes32 itemHash; - bytes result; - uint256 gas; - } - - struct WorkPackage { - Authorization authorization; - Context context; - WorkItem[] items; - } - - // ********* IN-CORE: Any ScBG can call this -- key design question is how work packages get assigned to cores! - function isAuthorized(WorkPackage package, uint32 coreIndex) external view returns (bool); - // applyRefine maps a WorkItem into a WorkResult - function applyRefine(WorkItem item) external pure returns (WorkResult memory); - // refine maps a package containing work items into work results using applyRefine - function refine(WorkPackage package) external pure returns (WorkResult[] memory); - - // ********* ON-CHAIN: accumulate is called by the system chain block author (ScBA) given work report attestation - function accumulate(WorkResult[] results) external; - // TODO: prune - removes conflicting items -} - -contract MyCoreJamService is CoreJamService { - modifier onlySCBA() { - // restrict accumulate to SCBA only - _; - } - - function applyRefine(WorkItem calldata item) external pure override returns (WorkResult memory) { - } - - function refine(WorkPackage calldata package) external pure override returns (WorkResult[] memory) { - } - - function accumulate(Tuple[] calldata results) external override onlySCBA() { - // Process authorization and workResults as needed... - // Note: CoreJam's transfer / on transfer is accomplished with a internal contract to another accumulate - } -} -``` - - -The core insight of CoreJam is that the following map-reduce refactorization efficiently addresses the underlying dependency graph of what can be computed in what order better than smart contracts alone, separating what can be parallelised in the `refine` step from what cannot be parallelised in the `accumulate` step -* the "in-core" Collect-Refine (with Smart Contract entry point`refine`) supports mass parallelization and can be done with a _scalable_ distributed nodes (ScBG) organized by "core index", -* the "on-chain" `accumulate` done by CoreJam System Chain ScBA, fundamentally bottlenecked by the system chain - -The above is a first-pass adaptation matching RFC #31, which aims for full generality of Work Package. Work is organized in Work Packages, and in this adaptation, Service is EVM Contract on the ETH L2 whose work items are first preprocessed with cheap decentralized compute power. We anticipate revising the above to support streamlined Service construction. Work Items are a pair where the first item, `service`, itself identifies a pairing of code and state known as a *Service*; and the second item, `payload`, is a block of data which, through the aforementioned code, mutates said state in some presumably useful way. Under ordinary circumstances, a CoreJam developer will deploy a CoreJamService along with a mechanism to add Work Items by users. - -A *Work Package* is an *Authorization* together with a series of *Work Items* and a context, limited in plurality, versioned and with a maximum encoded size. The Context includes an optional reference to a Work Package (`WorkPackageHash`) which limits the relative order of the Work Package (see **Work Package Ordering**, later). In EVM Optimistic Rollup (ORU) context by mapping the 3 components: -* Authorization - this just a contract address with a `isAuthorized` interface -* Context - this is simplified in an EVM ORU to be just a block number and block hash. This is possible because EVM L2 ORUs in OP Stack are fundamentally centralized sequencers, at least for now. -* Prerequisite - TBD. (The number of prerequisites of a Work Package is limited to at most one. However, we cannot trivially control the number of dependents in the same way, nor would we necessarily wish to since it would open up a griefing vector for misbehaving Work Package Builders who interrupt a sequence by introducing their own Work Packages with a prerequisite which is within another's sequence.) - -A Service IS a smart contract stored on-chain and transitioned only using on-chain logic, strictly and deterministically constrained, holding funds and call into each other synchronously. However, the Service's `accumulate` function cannot be transacted with and is only called by either the ScBA or a inter-service calling. Otherwise, all input data (and state progression) must come as the result of a Work Item. A Work Item is a blob of data meant for a particular Service and crafted by some source external to consensus, which can be a user or another Service. It may be thought of as akin to a transaction or internal contract call. - -The Work Item is first processed *in-core* through `refine`, which is to say an OP Stack node chosen in a System Chain Backing Group (ScBG), yielding a *Work Result*. It is this Work Result which is collated together with others of the same service and Accumulated into the Service on-chain. - -Though this process happens entirely in consensus, there are two main consensus environments at play, _in-core_ and _on-chain_. We therefore partition the progress into two pairs of stages: Collect & Refine and Join & Accumulate. - -### Processing stages of a Work Package - -A Work Package has several stages of consensus computation associated with its processing, which happen as the system becomes more certain that it represents a correct and useful transition of its Service. - -While a Work Package is being built, the *ScBG* must have access to the CoreJam System-chain state in order to supply a specific *Context*. The Context dictates a certain *Scope* for the Work Package which is used by the Initial Validation to limit which System-chain blocks it may be processed on to a small sequence of a specific fork (which is yet to be built, presumably). We define the System-chain height at this point to be `T`. - -The first consensus computation to be done is the Work Package having its Authorization checked in-core, hosted by the System-chain Backing Group. If it is determined to be authorized, then the same environment hosts the Refinement of the Work Package into a series of Work Results. This concludes the bulk of the computation that the Work Package represents. We would assume that the System-chain's height at this point is shortly after the authoring time, `T+r` where `r` could be as low as zero. - -The second consensus computation happens on-chain at the behest of the System-chain Block Author of the time `T+r+i`, where `i` is generally zero or one, the time taken for the Work Results to be transported from within the Core to get to the gateway of being on-chain. The computation done essentially just ensures that the Work Package is still in scope and that the prerequisite it relies upon (if any) has been submitted ahead of it. This is called the on-chain *Reporting* and initiates the *Availability Protocol* for this Work Package once System-chain Validators synchronize to the block. This protocol guarantees that the Work Package will be made available for as long as we allow disputes over its validity to be made. - -At some point later `T+r+i+a` (where `a` is the time to distribute the fragments of the Work Package and report their archival to the next System-chain Block Author) the Availability Protocol has concluded and the System-chain Block Author of the time brings this information on-chain in the form of a bitfield in which an entry flips from zero to one. At this point we can say that the Work Report's Package is *Available*. - -Finally, at some point later still `T+r+i+a+o`, the Results of the Work Package are aggregated into groups of Services, and then *Pruned* and *Accumulated* into the common state of the System-chain. This process is known as *Integration* (in the fixed-function parachains model, this is known as "inclusion") and is irreversible within any given fork. Additional latency from being made *Available* to being *Integrated* (i.e. the `o` component) may be incurred due to ordering requirements, though it is expected to be zero in the variant of this proposal to be implemented initially. - -### Collect-Refine - -The first two stages of the CoreJam process are *Collect* and *Refine*. *Collect* refers to the collection and authorization of Work Packages (collections of items together with an authorization) to utilize a ScBG Core. *Refine* refers to the performance of computation according to the Work Packages in order to yield *Work Results*. Finally, each Backing Group member attests to a Work Package yielding a series of Work Results and these Attestations form the basis for bringing the Results on-chain and integrating them into the Service's state which happens in the following stages. - -#### Collection and `isAuthorized` - -Collection is the means of a Backing Group member attaining a Work Package which is authorized to be performed on their assigned Core at the current time. Authorization is a prerequisite for a Work Package to be included on-chain. Computation of Work Packages which are not Authorized is not rewarded. Incorrectly attesting that a Work Package is authorized is a disputable offence and can result in substantial punishment. - -On arrival of a Work Package, after the initial decoding, a first check is that the `context` field is valid. This must reference a header hash of a known block which may yet be finalized and the additional fields must correspond to the data of that block. - -The *Authorizer* entry point of: - -```solidity -function isAuthorized(WorkPackage calldata package, uint32 coreIndex) external view returns (bool); -``` - -is executed by a ScBG node in a _metered VM_ and subject to a modest system-wide limitation on execution time. If it overruns this limit or panics on some input, it is considered equivalent to returning `false`. While it is mostly stateless (e.g. isolated from any System-chain state) it is provided with the package's `context` field in order to give information about a recent System-chain block. This allows it to be provided with a concise proof over some recent System-chain state. - -A single `Authorizer` value is associated with the index of the Core at a particular System-chain block and limits in some way what Work Packages may be legally processed by that Core. - -The need of ScBG nodes to be rewarded for doing work competes with that of the procurers of work to be certain to get work done which is useful to them. With CoreJam, ScBG nodes have little ability to identify a high-quality Work Package builder and the permissionless design means a greater expectation of flawed code executing in-core. Because of this, we make a slightly modified approach: Work Packages must have a valid Authorization, i.e. `isAuthorized` returns `true` when provided with the Work Package. However, Validators get rewarded for *any* such authorized Work Package, even one which ultimately panics or overruns on its evaluation. - -This ensures that ScBG nodes do a strictly limited amount of work before knowing whether they will be rewarded and are able to discontinue and attempt other candidates earlier than would otherwise be the case. There is the possibility of wasting computational resources by processing Work Packages which result in error, but well-written authorization procedures can mitigate this risk by making a prior validation of the Work Items. - -### Refine - -The `refine` function is implemented as an entry-point inside a Service Contract Address: - -```solidity -function refine(PackageInfo packageInfo) external pure returns (WorkResult[] memory) -``` - -Both `refine` and `isAuthorized` are only ever executed in-core. Within this environment, we need to ensure that we can interrupt computation not long after some well-specified limit and deterministically determine when an invocation of the VM exhausts this limit. Since the exact point at which interruption of computation need not be deterministic, it is expected to be executed by a streaming JIT transpiler with a means of approximate and overshooting interruption coupled with deterministic metering. - -When applying `refine` from the client code, we must allow for the possibility that the VM exits unexpectedly or does not end. Validators are always rewarded for computing properly authorized Work Packages, including those which include such broken Work Items. But they must be able to report their broken state into the System-chain in order to collect their reward. Thus we define a type `WorkResult`: - -```solidity -enum WorkError { - None, - Timeout, - Panic -} - -struct WorkResult { - uint32 service; - bytes32 item_hash; - WorkError error; - bytes result; - uint256 gas_used; -} - -function applyRefine(CoreJam.WorkItem memory item) external pure returns (CoreJam.WorkResult memory); -``` - -The amount of gas used in executing the `refine` function is noted in the `WorkResult` value, and this is used later in order to help apportion remaining gas in the Join-Accumulate process to the Services whose items appear in the Work Packages. - -```golang -/// Secure reference to a Work Package. -type WorkPackageSpec struct { - /// The hash of the SCALE encoded `EncodedWorkPackage`. - Hash WorkPackageHash - /// The erasure root of the SCALE encoded `EncodedWorkPackage`. - Root ErasureRoot - /// The length in bytes of SCALE encoded `EncodedWorkPackage`. - Len uint32 -} - -/// The specification of the underlying Work Package. -type WorkReport struct { - PackageId WorkPackageId - /// The context of the underlying Work Package. - Context Context - /// The Core index under which the Work Package was Refined to generate the Report. - CoreIndex CoreIndex - /// The results of the evaluation of the Items in the underlying Work Package. - Results []WorkResult // MaxWorkItemsInPackage -} - -/// Multiple signatures are consolidated into a single Attestation in a space-efficient -/// manner using a `Bitmap` to succinctly express which validators have attested. -type Attestation struct { - /// The Work Report which is being attested. - Report WorkReport - /// Which validators from the group have a signature in `attestations`. - Validators *bitmap.Bitmap - /// The signatures of the ScBG members set out in `validators` whose message is the - /// hash of the `report`. The order of the signatures is the same order as the validators appear in `validators`. - Attestations []Signature -} -``` - -Each System-chain block, every Backing Group representing a Core which is assigned work provides a series of Work Results coherent with an authorized Work Package. Validators are rewarded when they take part in their Group and process such a Work Package. Thus, together with some information concerning their execution context, they sign a *Report* concerning the work done and the results of it. This is also known as a *Candidate*. This signed Report is called an *Attestation*, and is provided to the System-chain block author. If no such Attestation is provided (or if the System-chain block author refuses to introduce it for Reporting), then that Backing Group is not rewarded for that block. - -WorkReports are gossiped among ScBG nodes (OP Stack nodes, using op-geth's libp2p-based messaging) to form a sufficient number attestations, which arrive at the ScBa. In an OP Stack setting, a set of OP Stack nodes all can validate the activity of the ScBA, which are chosen via some process to be part of a ScBG. See RFC #3 and discussion of design alternatives, one of which should be adapted for this purpose and placed in a OP Stack setting. - -### Join-Accumulate - -Join-Accumulate is the second major stage of computation and is independent from Collect-Refine. Unlike with the computation in Collect-Refine which happens contemporaneously within one of many isolated cores, the consensus computation of Join-Accumulate is both entirely synchronous with all other computation of its stage and operates within (and has access to) the same shared state-machine. - -Being *on-chain* (rather than *in-core* as with Collect-Refine), information and computation done in the Join-Accumulate stage is carried out (initially) by the Block Author and the resultant block evaluated by all Validators and full-nodes. Because of this, and unlike in-core computation, it has full access to the System-chain's state. - -The Join-Accumulate stage may be seen as a synchronized counterpart to the parallelised Collect-Refine stage. It may be used to integrate the work done from the context of an isolated VM into a self-consistent singleton world model. In concrete terms this means ensuring that the independent work components, which cannot have been aware of each other during the Collect-Refine stage, do not conflict in some way. Less dramatically, this stage may be used to enforce ordering or provide a synchronisation point. Finally, this stage may be a sensible place to manage asynchronous interactions between subcomponents of a Service or even different Services and oversee message queue transitions. - -#### Reporting and Integration - -There are two main phases of on-chain logic before a Work Package's ramifications are irreversibly assimilated into the state of the (current fork of the) System-chain. The first is where the Work Package is *Reported* on-chain. This is proposed through an extrinsic introduced by the ScBA and implies the successful outcome of some *Initial Validation* (described next). This kicks-off an off-chain process of *Availability* which, if successful, culminates in a second extrinsic being introduced on-chain shortly afterwards specifying that the Availability requirements of the Work Report are met. - -Since this is an asynchronous process, there are no ordering guarantees on Work Reports' Availability requirements being fulfilled. There may or may not be provision for adding further delays at this point to ensure that Work Reports are processed according to strict ordering. See *Work Package Ordering*, later, for more discussion here. - -Once both Availability and any additional requirements are met (including ordering and dependencies, but possibly also including reevaluation of some of the Initial Validation checks), then the second phase is executed which is known as *Integration*. This is the irreversible application of Work Report consequences into the Service's State Trie and (via certain permissionless host functions) the wider state of the System-chain. Work Results are segregated into groups based on their Service, joined into a `Vec` and passed through the immutable Prune function and into the mutable Accumulate function. - -#### Initial Validation - -There are a number of Initial Validation requirements which the ScBA must do in order to ensure no time is wasted on further, possibly costly, computation. Since the same tests are done on-chain, then for a Block Author to expect to make a valid block these tests must be done prior to actually placing the Attestations in the System-chain Block Body. - -Firstly, any given Work Report must have enough signatures in the Attestation to be considered for Reporting on-chain. Only one Work Report may be considered for Reporting from each ScBG per block. - -Secondly, any Work Reports introduced by the ScBA must be *Recent*, at a height is less than `RECENT_BLOCKS` (eg 16) from the block which the ScBA is now authoring. - -Thirdly, dependent elements of the Context (`context.block_number`) must correctly correspond to those on-chain for the block corresponding to the provided `context.header_hash`. For this to be possible, the System-chain is expected to track Recent blocks in a queue. - -Fourthly, the ScBA may not attempt to report multiple Work Reports for the same Work Package. Since Work Reports become inherently invalid once they are no longer *Recent*, then this check may be simplified to ensuring that there are no Work Reports of the same Work Package within any *Recent* blocks. - -Finally, the ScBA may not register Work Reports whose prerequisite is not itself Reported in *Recent* blocks. - -In order to ensure all of the above tests are honored by the ScBA, a block which contains Work Reports which fail any of these tests shall panic on import. The System-chain's on-chain logic will thus include these checks in order to ensure that they are honoured by the ScBA. The ScBA should track *Recent Reports*, and retain all Work Package hashes which were Reported in the *Recent* blocks. - -The ScBA must keep an up to date set of which Work Packages have already been Reported in order to avoid accidentally attempting to introduce a duplicate Work Package or one whose prerequisite has not been fulfilled. Since the currently authored block is considered *Recent*, Work Reports introduced earlier in the same block do satisfy the prerequisite of Work Packages introduced later. - -While it will generally be the case that ScBGs know precisely which Work Reports will have been introduced at the point that their Attestation arrives with the ScBA by keeping the head of the System-chain in sync, it will not always be possible. Therefore, ScBGs will never be punished for providing an Attestation which fails any of these tests; the Attestation will simply be kept until either: - -1. it stops being *Recent*; -2. it becomes Reported on-chain; or -3. some other Attestation of the same Work Package becomes Reported on-chain. - -#### Availability - -Once the Work Report of a Work Package is Reported on-chain, the Work Package itself must be made *Available* through the off-chain Availability Protocol, which ensures that any dispute over the correctness of the Work Report can be easily objectively judged by all validators. Being off-chain this is _not_ block-synchronized and any given Work Package may take one or more blocks to be made Available or may even fail. - -Only once a Work Report's Work Package is made Available the processing continue with the next steps of Joining and Accumulation. -We will follow RFC #31's soft-ordering. - -#### Gas Provisioning - -Join-Accumulate is, as the name suggests, comprised of two subordinate stages. Both stages involve executing code inside a VM on-chain. Thus code must be executed in a *metered* format, meaning it must be able to be executed in a sandboxed and deterministic fashion but also with a means of providing an upper limit on the amount of gas it may consume and a guarantee that this limit will never be breached. - -Practically speaking, we may allow a similar VM execution metering system similar to that for the `refine` execution, whereby we do not require a strictly deterministic means of interrupting, but do require deterministic metering and only approximate interruption. This would mean that full-nodes and System-chain validators could be made to execute some additional margin worth of computation without payment, though any attack could easily be mitigated by attaching a fixed cost (either economically or in gas terms) to an VM invocation. - -Each Service defines some requirements it has regarding the provision of on-chain gas. Since all on-chain gas requirements must be respected of all processed Work Packages, it is important that each Work Report does not imply using more gas than its fair portion of the total available, and in doing so provides enough gas to its constituent items to meet their requirements. - -```rust -struct WorkItemGasRequirements { - prune: uint256, - accumulate: uint256, -} -type GasRequirements = StorageMap; -``` - -Each Service has two gas requirements associated with it corresponding to the two pieces of permissionless on-chain Service logic and represent the amount of gas allotted for each Work Item of this service within in a Work Package assigned to a Core. - -The total amount of gas utilizable by each Work Package (`gas_per_package`) is specified as: - -```rust -gas_per_package = system_block_gas * safety_margin / max_cores -``` - -`safety_margin` ensures that other System-chain system processes can happen and important transactions can be processed and is likely to be around 75%. - -A Work Report is only valid if all gas liabilities of all Work Items to be Accumulated fit within this limit: - -```rust -let total_gas_requirement = work_statement - .items - .map(|item| gas_requirements[item.service]) - .sum(|requirements| requirements.prune + requirements.accumulate); -total_gas_requirement <= gas_per_package -``` - -Because of this, Work Report builders must be aware of any upcoming alterations to `max_cores` and build Statements which are in accordance with it not at present but also in the near future when it may have changed. - -### Accumulate - -The next phase, which happens on-chain, is Accumulate. This governs the amalgamation of the Work Package Outputs calculated during the Refinement stage into the System-chain's overall state and in particular into the various Child Tries of the Services whose Items were refined. Crucially, since the Refinement happened in-core, and since all in-core logic must be disputable and therefore its inputs made *Available* for all future disputers, Accumulation of a Work Package may only take place *after* the Availability process for it has completed. - -The function signature to the `accumulate` entry-point in the Service's code blob is: - -```solidity -function accumulate(CoreJam.Tuple[] memory results) external; -``` - -The logic in `accumulate` may need to know how the various Work Items arrived into a processed Work Package. Since a Work Package could have multiple Work Items of the same Service, it makes sense to have a separate inner `Vec` for Work Items sharing the Authorization (by virtue of being in the same Work Package). - -Work Items are identified by their Keccak hash, known at the *Item Hash* (`ItemHash`). We provide both the Authorization of the Package and the constituent Work Item Hashes and their Results in order to allow the `refine` logic to take appropriate action in the case that an invalid Work Item was submitted (i.e. one which caused its Refine operation to panic or time-out). - -There is an amount of gas which it is allowed to use before being forcibly terminated and any non-committed state changes lost. The lowest amount of gas provided to `accumulate` is defined as the number of `WorkResult` values passed in `results` to `accumulate` multiplied by the `accumulate` field of the Service's gas requirements. - -However, the actual amount of gas may be substantially more. Each Work Package is allotted a specific amount of gas for all on-chain activity (`gas_per_package` above) and has a gas liability defined by the gas requirements of all Work Items it contains (`total_gas_requirement` above). Any gas remaining after the liability (i.e. `gas_per_package - total_gas_requirement`) may be apportioned to the Services of Items within the Report on a pro-rata basis according to the amount of gas they utilized during `refine`. Any gas unutilized by Classes within one Package may be carried over to the next Package and utilized there. - -Read-access to the System-chain state is allowed with EVM Precompiles. No direct write access may be provided since `accumulate` is untrusted code. - -Since `accumulate` is permissionless and untrusted code, we must ensure that its child trie does not grow to degrade the System-chain's overall performance or place untenable requirements on the storage of full-nodes. To this goal, we require an account sovereign to the Service to be holding an amount of funds proportional to the overall storage footprint of its Child Trie. `set_work_storage` may return an error should the balance requirement not be met. - -Host functions are provided allowing any state changes to be committed at fail-safe checkpoints to provide resilience in case of gas overrun (or even buggy code which panics). The amount of gas remaining may also be queried without setting a checkpoint. `Gas` is expressed in a regular fashion for a solo-chain (i.e. one-dimensional). - -The `accumulate` of one service may call `accumulate` of another service via internal contract call, which may also transfer a `value` into their account. - -### CoreJam Storage API + EIP-4844 - -To reference large, immutable and long-term data payloads both in-core (`refine`) and on-chain (`accumulate`), we expose a *CoreJam Storage API*, accessible to untrusted code through Solidity library and to trusted System-chain code via a Golang interface. Internally, data is stored with a reference count so that two separate usages of `store` need not be concerned about the other. -Every piece of data stored for an untrusted caller requires a sizeable deposit. When used by untrusted code via a host function, the `depositor` would be set to an account controlled by the executing code. - -```golang -type Storage interface { - /// Immutable function to attempt to determine the preimage for the given `hash`. - Lookup(hash common.Hash) []byte - /// Allow a particular preimage to be `provide`d. Once provided, this will be available through `lookup` until `unrequest` is called. - Request(hash common.Hash, len int) bool - /// Remove request that some data be made available. If the data was never - /// available or the data will remain available due to another request, - /// then `false` is returned and `expunge` may be called immediately. - /// Otherwise, `true` is returned and `expunge` may be called in - /// 24 hours. - Unrequest(hash common.Hash) bool - // Functions used by implementations of untrusted functions; such as - // extrinsics or host functions. - /// Place a deposit in order to allow a particular preimage to be `provide`d. - /// Once provided, this will be available through `lookup` until - /// `unrequest_untrusted` is called. - RequestUntrusted(depositor AccountID, hash common.Hash, len int) - /// Remove request that some data be made available. If the data was never - /// available or the data will remain available due to another request, - /// then `false` is returned and `expunge_untrusted` may be called immediately. - /// Otherwise, `true` is returned and `expunge_untrusted` may be called in - /// 24 hours. - UnrequestUntrusted(depositor AccountID, hash common.Hash) bool - // Permissionless items utilizable directly by an extrinsic or task. - - /// Provide the preimage of some requested hash. Returns `Some` if its hash - /// was requested; `None` otherwise. - /// - /// Usually utilized by an extrinsic and is free if `Some` is returned. - Provide(preimage []byte) (common.Hash, bool) - - /// Potentially remove the preimage of `hash` from the chain when it was - /// unrequested using `unrequest`. `Ok` is returned iff the operation is - /// valid. - /// - /// Usually utilized by a task and is free if it returns `Ok`. - Expunge(hash common.Hash) error - /// Return the deposit associated with the removal of the request by - /// `depositor` using `unrequest_untrusted`. Potentially - /// remove the preimage of `hash` from the chain also. `Ok` is returned - /// iff the operation is valid. - /// - /// Usually utilized by a task and is free if it returns `Ok`. - ExpungeUntrusted(depositor AccountID, hash common.Hash) error - /// Equivalent to `request` followed immediately by `provide`. - Store(data []byte) common.Hash -} -``` - -New OVM opcodes for each function are as follows: - -| OVM Opcode | Go Interface | Description | -|------------|--------------|-------------| -| `APLOOKUP` | `Lookup(hash common.Hash) []byte` | Looks up the provided hash in the storage and returns the corresponding value as a byte array. If the hash is not found, returns None. | -| `APREQ` | `RequestUntrusted(depositor AccountID, hash common.Hash, len int)` | Places a deposit to request the specified hash with a given length to be made available in the storage. This function is used by untrusted sources. -| `APUNREQ` | `UnrequestUntrusted(depositor AccountID, hash common.Hash) bool` | Removes the request for the specified hash made by the specified depositor. If the request is successfully removed, returns true. Otherwise, returns false. | -| `APPROVIDE` | `Provide(preimage []byte) (common.Hash, bool)` | Provides the preimage data. If the preimage corresponds to a previously requested hash, returns the hash. Otherwise, returns None. | -| `APEXP` | `ExpungeUntrusted(depositor AccountID, hash common.Hash) error` | Removes the specified hash and its corresponding data from the storage, based on a request made by the specified depositor. Returns Ok(()) if the operation is successful, an error otherwise. | -| `APSTORE` | `Store(data []byte) common.Hash` | Stores the provided data in the storage and returns the hash of the stored data. | - -Removing data happens in a two-phase procedure; first the data is unrequested, signalling that calling `lookup` on its hash may no longer work (it may still work if there are other requests active). 24 hours following this, the data is expunged with a second call which, actually removes the data from the chain assuming no other requests for it are active. Only once expunge is called successfuly is the deposit returned. If the data was never provided, or is additional requests are still active, then expunge may be called immediately after a successful unrequest. - -The Storage API should be connectable to Ethereum [EIP-4844](https://www.eip4844.com/) with Blob Transactions. While `accumulate` calls would be included in OP Stack Blob transactions, the veracity of the `refine` requires that the WorkItems (which may be large in number) must also be executable on Ethereum L1, requiring the `refine` inputs to be included. Fault proofs for both `refine` "in-core" and `accumulate` "on-chain" calls are necessary. - -Ethereum's solution of 32 kB/sec of throughput post Dencun (part 2) and 1.3 MB/sec full danksharding should be relevant. Note that this is significantly lower than Polkadot DA's 66-133MiB/s (assuming 100-200 cores), which presumably would be used for both `refine` and `accumulate`. - -### Work Package Ordering - -We target the soft-ordering variant: providing ordering only on a *best-effort* basis, whereby Work Reports respect the ordering requested in their Work Packages as much as possible, but it is not guaranteed. Work Reports may be Accumulated before, or even entirely without, their prerequisites. We refer to this *Soft-Ordering*. The alternative is to provide a guarantee that the Results of Work Packages will always be Accumulated no earlier than the Result of any prerequisite Work Package. As we are unable to alter the Availability Protocol, this is achieved through on-chain queuing and deferred Accumulation. - -Initial Validation are made on-chain prior to the Availability Protocol begins for any given Work Package. This ensures that the Work Package is still in scope. However, this does not ensure that the Work Package is still in scope at the point that the Work Results are actually Accumulated. It is as yet unclear whether this is especially problematic. - -### Fees - -Fees are ETH on L2, bridged from Ethereum L1. OP Stack models the L1 fees explicitly. While both the ScBG `refine` and ScBA `accumulate` are both metered in gas usage, the relatively scarce nature of the `accumulate` stage demands a massive premium over the `refine`. - -## Performance, Ergonomics and Compatibility, Notes on Implementation in OP Stack - -Ethereum L2 Users of a Service on the CoreJam System Chain should be able to submit transactions that result in Work Items causing Work Packages. CoreJam's map-reduce architecture is most valuable when significant portions of gas consumption would have been expensive to do in `accumulate` may be done at lower gas with fewer compute resources in `refine`. - -We envision an initial version of this proposal with modifications to the OP Stack: - -1. We require a Work Package scheduling system that can orchestrate a distributed array of "cores". This in turn requires a mapping between Work Packages/Services and Core Index, with implementable ScBG strategy, following [Dynamic Backing Groups: Preparing Cores for Agile Scheduling](https://forum.polkadot.network/t/dynamic-backing-groups-preparing-cores-for-agile-scheduling/3629/10) and RFC #3. -2. We require metering ScBG and ScBA and charging the user in "Instantaneous" ETH in a manner consistent with the relative scarcity of each resource, and compensating the cores for their contributions. It may be essential to adopt the Bulk CoreTime model for services instead for efficiency. -3. We require extending the [Optimistic Virtual Machine] Availability Protocol in OP Stack, using KAGOME's erasure coding or similar, using Go C++ bindings. -4. We require OP Stack fault proofs extended to use EIP-4844 blobs to accommodate the above - -## Testing, Security and Privacy - -ORUs derive their security largely from fault proofs, where we expect to rely on OP Stack's Fault Proof / Dispute game, developed by Optimism. - -The fault proof mechanism must be extended to accommodate EIP-4844, not just for blocks but for the `refine` work result output. It is necessary to extend the OP Stack Fault proof game. - -The precise details of Work Reports / Attestation from "in core" nodes where Attestation / Work Reports can be run by a large number of nodes participating in ScBGs needs a precise signature scheme. - -The fundamental work product is not Solidity contracts, but representative pedagogical service contracts merit expert attention. - -Testing norms have been established in OP Stack, which must be matched OP Stack implementation. - -The proposal introduces no new privacy concerns. - -## Future Directions and Related Material - -It is highly desirable to: -1. build multiple services demonstrating scaling performance of the CoreJam architecture in Services -2. measure performance of Availability Protocol as a function of # of cores in the CoreJam System Chain - -## Drawbacks, Alternatives and Unknowns - -1. In the case of composite Work Packages, allowing synchronous (and therefore causal) interactions between the Work Items. If this were to be the case, then some sort of synchronisation sentinel would be needed to ensure that should one subpackage result without the expected effects on its Service State (by virtue of the `accumulate` outcome for that subpackage), that the `accumulate` of any causally entangled subpackages takes appropriate account for this (i.e. by dropping it and not effecting any changes from it). - -2. Work Items may need some degree of coordination to be useful by the `accumulate` function of their Service. To a large extent this is outside of the scope of this proposal's computation model by design. Through the authorization framework we assert that it is the concern of the Service and not of the System-chain validators themselves. - -## Prior Art and References - -This is an adaptation of RFC #31. We are grateful for useful feedback from both the Polkadot and Ethereum community. From 2c1a217844f176a05a5e9dd30fcbde9a978b43d0 Mon Sep 17 00:00:00 2001 From: Sourabh Niyogi Date: Sat, 26 Oct 2024 15:00:02 -0700 Subject: [PATCH 8/8] RFC-0127 --- ...7-JAM-Service-for-Validating-Ethereum-Optimistic-Rollups.md} | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename text/{0033-JAM-Service-for-Validating-Ethereum-Optimistic-Rollups.md => 0127-JAM-Service-for-Validating-Ethereum-Optimistic-Rollups.md} (99%) diff --git a/text/0033-JAM-Service-for-Validating-Ethereum-Optimistic-Rollups.md b/text/0127-JAM-Service-for-Validating-Ethereum-Optimistic-Rollups.md similarity index 99% rename from text/0033-JAM-Service-for-Validating-Ethereum-Optimistic-Rollups.md rename to text/0127-JAM-Service-for-Validating-Ethereum-Optimistic-Rollups.md index a00b49d62..5f8b8ebd1 100644 --- a/text/0033-JAM-Service-for-Validating-Ethereum-Optimistic-Rollups.md +++ b/text/0127-JAM-Service-for-Validating-Ethereum-Optimistic-Rollups.md @@ -1,5 +1,5 @@ -# RFC-0033: JAM Service for Validating Ethereum Optimistic Rollups +# RFC-0127: JAM Service for Validating Ethereum Optimistic Rollups | | | | --------------- | ------------------------------------------------------------------------------------------- |