Faster multitree bench #238
Conversation
admin/src/multitree_bench/mod.rs
Outdated
| /// be empty and non direct children will be NodeSpec::UnresolvedPath. | ||
| fn generate_node( | ||
| fn generate_node_data(&self, rng: &mut rand::rngs::SmallRng, leaf: bool) -> Vec<u8> { | ||
| // Polkadot doesn't store actual values in branch nodes, only in leaf nodes. Hence the value |
There was a problem hiding this comment.
Guess it is a result of the content being well addressed in polkadot, not a rule in the absolute. Maybe:
| // Polkadot doesn't store actual values in branch nodes, only in leaf nodes. Hence the value | |
| // Polkadot avoid storing actual values in branch nodes, mainly in leaf nodes. Hence the value |
There was a problem hiding this comment.
Good point. I'll change the comment.
| if let Some(existing_index) = existing_index { | ||
| let previous_child_node = &previous_node.unwrap().children[*existing_index]; | ||
| let previous_child = previous_child_node.get_child_as_ref(); | ||
| if *removals > 0 || *additions > 0 { |
There was a problem hiding this comment.
I am not too sure about the logic of these removals/additions counters.
The way it should work is : on changed node:
- if removed (eg replacement but there can be removal with no new node in some trie related corner case) then only do Rc minus one, if rc is < 0 then do rc minus one on all children.
- if new add new node.
- if updated, do removed as in first line, and for all children do rc + 1.
The logic I want to assert here is that we got for every changed node this rc + 1 of every of its children and for every pruned node we do the rc - 1 of every of its children.
(seen some io issue running this on a binary trie, if we end up running in similar io issue there is an alternate design, but no use looking at it unless we see such issue)
There was a problem hiding this comment.
Those values aren’t refcounts, they’re targets for how many values (basically leaf nodes) to add and remove.
This is in the part where it changes the tree for each new commit. It starts with a target number of additions and removals to make for the whole tree then distributes them through child nodes as best it can. Sometimes it adds/removes leaf nodes otherwise it passes the numbers down to the children for them to deal with. They also aren’t guaranteed to be able to make the changes but on average each commit should end up with an appropriate number of changes.
The actual node refcounting happens within the Db and it’ll deal with whether nodes need to be removed.
There was a problem hiding this comment.
ok, I was wondering abot the number of NodeRef::Existing(NodeAddress) in the commit set. I feel like it may have been underestimated (not directly related with this pr though).
Much improved multitree stress test. It can now generate and submit over 1000 tree commits a second.
When in archive mode the database can keep up with this. When doing full reference counting etc it can't and commit processing is the bottleneck. In this situation it gets around 130 tree commits a second.
I fixed an issue where tree commits weren't added to the commit queue size. This caused huge numbers of commits to get queued up without the database keeping up. Now it throttles commits so it doesn't get left behind.