Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge partial processing #5

Merged

Conversation

realbigsean
Copy link
Collaborator

Issue Addressed

Which issue # does this PR address?

Proposed Changes

Please list or describe the changes introduced by this PR.

Additional Info

Please provide any additional information. For example, future considerations
or information useful for reviewers.

ethDreamer and others added 30 commits March 19, 2023 23:15
## Issue Addressed

* sigp#3895 

Co-authored-by: ethDreamer <[email protected]>
Co-authored-by: Michael Sproul <[email protected]>
If a node is also a bootnode it can try to add itself to its own local routing table which will emit an error. 

The error is entirely harmless but we would prefer to avoid emitting the error. 

This PR does not attempt to add a boot node ENR if that ENR corresponds to our local peer-id/node-id.
Currently Lighthouse will remain uncontactable if users port forward a port that is not the same as the one they are listening on. 

For example, if Lighthouse runs with port 9000 TCP/UDP locally but a router is configured to pass 9010 externally to the lighthouse node on 9000, other nodes on the network will not be able to reach the lighthouse node. 

This occurs because Lighthouse does not update its ENR TCP port on external socket discovery. The intention was always that users should use `--enr-tcp-port` to customise this, but this is non-intuitive. 

The difficulty arises because we have no discovery mechanism to find our external TCP port. If we discovery a new external UDP port, we must guess what our external TCP port might be. This PR assumes the external TCP port is the same as the external UDP port (which may not be the case) and thus updates the TCP port along with the UDP port if the `--enr-tcp-port` flag is not set. 

Along with this PR, will be added documentation to the Lighthouse book so users can correctly understand and configure their ENR to maximize Lighthouse's connectivity. 

This relies on sigp/discv5#166 and we should wait for a new release in discv5 before adding this PR.
This PR enables the user to adjust the shuffling cache size.

This is useful for some HTTP API requests which require re-computing old shufflings. This PR currently optimizes the
beacon/states/{state_id}/committees HTTP API by first checking the cache before re-building shuffling.

If the shuffling is set to a non-default value, then the HTTP API request will also fill the cache when as it constructs new shufflings.

If the CLI flag is not present or the value is set to the default of 16 the default behaviour is observed.


Co-authored-by: Michael Sproul <[email protected]>
## Issue Addressed

NA

## Proposed Changes

Replaces sigp#4058 to attempt to reduce `ERRO Failed to send scheduled attestation` spam and provide more information for diagnosis. With this PR we achieve:

- When dequeuing attestations after a block is received, send only one log which reports `n` failures (rather than `n` logs reporting `n` failures).
- Make a distinction in logs between two separate attestation dequeuing events.
- Add more information to both log events to help assist with troubleshooting.

## Additional Info

NA
## Issue Addressed

NA

## Proposed Changes

Sets the mainnet Capella fork epoch as per ethereum/consensus-specs#3300

## Additional Info

I expect the `ef_tests` to fail until we get a compatible consensus spec tests release.
## Issue Addressed

NA

## Proposed Changes

- Implements ethereum/consensus-specs#3290
- Bumps `ef-tests` to [v1.3.0-rc.4](https://github.com/ethereum/consensus-spec-tests/releases/tag/v1.3.0-rc.4).

The `CountRealizedFull` concept has been removed and the `--count-unrealized-full` and `--count-unrealized` BN flags now do nothing but log a `WARN` when used.

## Database Migration Debt

This PR removes the `best_justified_checkpoint` from fork choice. This field is persisted on-disk and the correct way to go about this would be to make a DB migration to remove the field. However, in this PR I've simply stubbed out the value with a junk value. I've taken this approach because if we're going to do a DB migration I'd love to remove the `Option`s around the justified and finalized checkpoints on `ProtoNode` whilst we're at it. Those options were added in sigp#2822 which was included in Lighthouse v2.1.0. The options were only put there to handle the migration and they've been set to `Some` ever since v2.1.0. There's no reason to keep them as options anymore.

I started adding the DB migration to this branch but I started to feel like I was bloating this rather critical PR with nice-to-haves. I've kept the partially-complete migration [over in my repo](https://github.com/paulhauner/lighthouse/tree/fc-pr-18-migration) so we can pick it up after this PR is merged.
## Issue Addressed

NA

## Proposed Changes

Bump versions to `v4.0.0`

## Additional Info

NA
## Issue Addressed

NA

## Proposed Changes

Ensures that we log the values of the *head* block rather than the *justified* block.

## Additional Info

NA
@realbigsean realbigsean merged commit 71f8222 into pawanjay176:partial-processing Mar 24, 2023
pawanjay176 pushed a commit that referenced this pull request Jan 23, 2024
* add metrics layer

* add metrics

* simplify getting the target

* make clippy happy

* fix typos

* unify deps under workspace

* make import statement shorter, fix typos

* enable warn by default, mark flag as deprecated

* do not exit on error when initializing logging fails

* revert exit on error

* adjust bootnode logging

* use target as is by default

* make libp2p events register correctly

* adjust repilcated cli help

* turn on debug logs by default, remove deprecation warning

* suppress output (#5)

---------

Co-authored-by: Age Manning <[email protected]>
pawanjay176 pushed a commit that referenced this pull request Jan 23, 2024
* add metrics layer

* add metrics

* simplify getting the target

* make clippy happy

* fix typos

* unify deps under workspace

* make import statement shorter, fix typos

* enable warn by default, mark flag as deprecated

* do not exit on error when initializing logging fails

* revert exit on error

* adjust bootnode logging

* add logging layer

* non blocking file writer

* non blocking file writer

* add tracing visitor

* use target as is by default

* make libp2p events register correctly

* adjust repilcated cli help

* refactor tracing layer

* linting

* filesize

* log gossipsub, dont filter by log level

* turn on debug logs by default, remove deprecation warning

* truncate file, add timestamp, add unit test

* suppress output (#5)

* use tracing appender

* cleanup

* Add a task to remove old log files and upgrade to warn level

* Add the time feature for tokio

* Udeps and fmt

---------

Co-authored-by: Diva M <[email protected]>
Co-authored-by: Divma <[email protected]>
Co-authored-by: Age Manning <[email protected]>
pawanjay176 pushed a commit that referenced this pull request Jul 12, 2024
* move gossipsub into a separate crate

* Merge branch 'unstable' of github.com:sigp/lighthouse into separate-gossipsub

* update rpc.proto and generate rust bindings

* gossipsub: implement IDONTWANT messages

* address review

* move GossipPromises out of PeerScore

* impl PeerKind::is_gossipsub

that returns true if peer speaks any version of gossipsub

* address review 2

* Merge branch 'separate-gossipsub' of github.com:sigp/lighthouse into impl-gossipsub-idontwant

* Merge branch 'unstable' of github.com:sigp/lighthouse into impl-gossipsub-idontwant

* add metrics

* add tests

* make 1.2 beta before spec is merged

* Merge branch 'unstable' of github.com:sigp/lighthouse into impl-gossipsub-idontwant

* cargo clippy

* Collect decoded IDONTWANT messages

* Use the beta tag in most places to simplify the transition

* Fix failed test by using fresh message-ids

* Gossipsub v1.2-beta

* Merge latest unstable

* Cargo update

* Merge pull request #5 from ackintosh/impl-gossipsub-idontwant-ackintosh-fix-test

Fix `test_ignore_too_many_messages_in_ihave` test

* Merge branch 'unstable' of github.com:sigp/lighthouse into impl-gossipsub-idontwant

* update CHANGELOG.md

* remove beta for 1.2 IDONTWANT spec has been merged

* Merge branch 'unstable' of github.com:sigp/lighthouse into impl-gossipsub-idontwant

* Merge branch 'impl-gossipsub-idontwant' of github.com:jxs/lighthouse into impl-gossipsub-idontwant

* Merge branch 'unstable' of github.com:sigp/lighthouse into impl-gossipsub-idontwant

* improve comments wording

* Merge branch 'impl-gossipsub-idontwant' of github.com:jxs/lighthouse into impl-gossipsub-idontwant
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants