Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix shuffling testgen when multiprocessing #3383

Merged
merged 1 commit into from
May 24, 2023
Merged

Fix shuffling testgen when multiprocessing #3383

merged 1 commit into from
May 24, 2023

Conversation

hwwhww
Copy link
Contributor

@hwwhww hwwhww commented May 24, 2023

Issue

Errors when we run testgen with MODE_MULTIPROCESSING

With CPython3.9:

Traceback (most recent call last):
  File "/Users/hww/dev/consensus-specs/tests/generators/shuffling/main.py", line 57, in <module>
    gen_runner.run_generator("shuffling", [create_provider(MINIMAL), create_provider(MAINNET)])
  File "/Users/hww/dev/consensus-specs/tests/generators/shuffling/venv/lib/python3.9/site-packages/eth2spec/gen_helpers/gen_base/gen_runner.py", line 244, in run_generator
    results = pool.map(worker_function, iter(all_test_case_params))
  File "/Users/hww/dev/consensus-specs/tests/generators/shuffling/venv/lib/python3.9/site-packages/pathos/multiprocessing.py", line 135, in map
    return _pool.map(star(f), zip(*args)) # chunksize
  File "/Users/hww/dev/consensus-specs/tests/generators/shuffling/venv/lib/python3.9/site-packages/multiprocess/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/Users/hww/dev/consensus-specs/tests/generators/shuffling/venv/lib/python3.9/site-packages/multiprocess/pool.py", line 771, in get
    raise self._value
  File "/Users/hww/dev/consensus-specs/tests/generators/shuffling/venv/lib/python3.9/site-packages/multiprocess/pool.py", line 537, in _handle_tasks
    put(task)
  File "/Users/hww/dev/consensus-specs/tests/generators/shuffling/venv/lib/python3.9/site-packages/multiprocess/connection.py", line 214, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/Users/hww/dev/consensus-specs/tests/generators/shuffling/venv/lib/python3.9/site-packages/multiprocess/reduction.py", line 54, in dumps
    cls(buf, protocol, *args, **kwds).dump(obj)
  File "/Users/hww/dev/consensus-specs/tests/generators/shuffling/venv/lib/python3.9/site-packages/dill/_dill.py", line 394, in dump
    StockPickler.dump(self, obj)
  File "/Users/hww/.pyenv/versions/3.9.7/lib/python3.9/pickle.py", line 487, in dump
    self.save(obj)
  File "/Users/hww/dev/consensus-specs/tests/generators/shuffling/venv/lib/python3.9/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/hww/.pyenv/versions/3.9.7/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/hww/.pyenv/versions/3.9.7/lib/python3.9/pickle.py", line 901, in save_tuple
    save(element)
  File "/Users/hww/dev/consensus-specs/tests/generators/shuffling/venv/lib/python3.9/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/hww/.pyenv/versions/3.9.7/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/hww/.pyenv/versions/3.9.7/lib/python3.9/pickle.py", line 886, in save_tuple
    save(element)
  File "/Users/hww/dev/consensus-specs/tests/generators/shuffling/venv/lib/python3.9/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/hww/.pyenv/versions/3.9.7/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/hww/.pyenv/versions/3.9.7/lib/python3.9/pickle.py", line 886, in save_tuple
    save(element)
  File "/Users/hww/dev/consensus-specs/tests/generators/shuffling/venv/lib/python3.9/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/hww/.pyenv/versions/3.9.7/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/hww/.pyenv/versions/3.9.7/lib/python3.9/pickle.py", line 901, in save_tuple
    save(element)
  File "/Users/hww/dev/consensus-specs/tests/generators/shuffling/venv/lib/python3.9/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/hww/.pyenv/versions/3.9.7/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/hww/.pyenv/versions/3.9.7/lib/python3.9/pickle.py", line 886, in save_tuple
    save(element)
  File "/Users/hww/dev/consensus-specs/tests/generators/shuffling/venv/lib/python3.9/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/hww/.pyenv/versions/3.9.7/lib/python3.9/pickle.py", line 603, in save
    self.save_reduce(obj=obj, *rv)
  File "/Users/hww/.pyenv/versions/3.9.7/lib/python3.9/pickle.py", line 688, in save_reduce
    save(args)
  File "/Users/hww/dev/consensus-specs/tests/generators/shuffling/venv/lib/python3.9/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/hww/.pyenv/versions/3.9.7/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/hww/.pyenv/versions/3.9.7/lib/python3.9/pickle.py", line 901, in save_tuple
    save(element)
  File "/Users/hww/dev/consensus-specs/tests/generators/shuffling/venv/lib/python3.9/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/hww/.pyenv/versions/3.9.7/lib/python3.9/pickle.py", line 603, in save
    self.save_reduce(obj=obj, *rv)
  File "/Users/hww/.pyenv/versions/3.9.7/lib/python3.9/pickle.py", line 717, in save_reduce
    save(state)
  File "/Users/hww/dev/consensus-specs/tests/generators/shuffling/venv/lib/python3.9/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/hww/.pyenv/versions/3.9.7/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/hww/dev/consensus-specs/tests/generators/shuffling/venv/lib/python3.9/site-packages/dill/_dill.py", line 1186, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/Users/hww/.pyenv/versions/3.9.7/lib/python3.9/pickle.py", line 971, in save_dict
    self._batch_setitems(obj.items())
  File "/Users/hww/.pyenv/versions/3.9.7/lib/python3.9/pickle.py", line 997, in _batch_setitems
    save(v)
  File "/Users/hww/dev/consensus-specs/tests/generators/shuffling/venv/lib/python3.9/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/hww/.pyenv/versions/3.9.7/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/hww/dev/consensus-specs/tests/generators/shuffling/venv/lib/python3.9/site-packages/dill/_dill.py", line 1824, in save_function
    _save_with_postproc(pickler, (_create_function, (
  File "/Users/hww/dev/consensus-specs/tests/generators/shuffling/venv/lib/python3.9/site-packages/dill/_dill.py", line 1089, in _save_with_postproc
    pickler.save_reduce(*reduction)
  File "/Users/hww/.pyenv/versions/3.9.7/lib/python3.9/pickle.py", line 692, in save_reduce
    save(args)
  File "/Users/hww/dev/consensus-specs/tests/generators/shuffling/venv/lib/python3.9/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/hww/.pyenv/versions/3.9.7/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/hww/.pyenv/versions/3.9.7/lib/python3.9/pickle.py", line 886, in save_tuple
    save(element)
  File "/Users/hww/dev/consensus-specs/tests/generators/shuffling/venv/lib/python3.9/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/hww/.pyenv/versions/3.9.7/lib/python3.9/pickle.py", line 603, in save
    self.save_reduce(obj=obj, *rv)
  File "/Users/hww/.pyenv/versions/3.9.7/lib/python3.9/pickle.py", line 687, in save_reduce
    save(cls)
  File "/Users/hww/dev/consensus-specs/tests/generators/shuffling/venv/lib/python3.9/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/hww/.pyenv/versions/3.9.7/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/hww/dev/consensus-specs/tests/generators/shuffling/venv/lib/python3.9/site-packages/dill/_dill.py", line 1698, in save_type
    _save_with_postproc(pickler, (_create_type, (
  File "/Users/hww/dev/consensus-specs/tests/generators/shuffling/venv/lib/python3.9/site-packages/dill/_dill.py", line 1070, in _save_with_postproc
    pickler.save_reduce(*reduction, obj=obj)
  File "/Users/hww/.pyenv/versions/3.9.7/lib/python3.9/pickle.py", line 692, in save_reduce
    save(args)
  File "/Users/hww/dev/consensus-specs/tests/generators/shuffling/venv/lib/python3.9/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/hww/.pyenv/versions/3.9.7/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/hww/.pyenv/versions/3.9.7/lib/python3.9/pickle.py", line 901, in save_tuple
    save(element)
  File "/Users/hww/dev/consensus-specs/tests/generators/shuffling/venv/lib/python3.9/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/hww/.pyenv/versions/3.9.7/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/hww/dev/consensus-specs/tests/generators/shuffling/venv/lib/python3.9/site-packages/dill/_dill.py", line 1186, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/Users/hww/.pyenv/versions/3.9.7/lib/python3.9/pickle.py", line 971, in save_dict
    self._batch_setitems(obj.items())
  File "/Users/hww/.pyenv/versions/3.9.7/lib/python3.9/pickle.py", line 997, in _batch_setitems
    save(v)
  File "/Users/hww/dev/consensus-specs/tests/generators/shuffling/venv/lib/python3.9/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/hww/.pyenv/versions/3.9.7/lib/python3.9/pickle.py", line 578, in save
    rv = reduce(self.proto)
TypeError: cannot pickle '_abc._abc_data' object
generator shuffling finished

How did I fix it

I found the issue is in for seed in [spec.hash(seed_init_value.to_bytes(length=4, byteorder='little')) for seed_init_value in range(30)]. If we change the seeds to something hardcoded, like [b’\x10', b'\x11’], it's good.

My workaround is to use random.Random to generate the seeds instead of the hash-generated seed. Still not sure why it raises such an error, though.

djrtwo
djrtwo approved these changes May 24, 2023
View reviewed changes
Copy link
Contributor

@djrtwo djrtwo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

weird

@hwwhww hwwhww merged commit ce5a9fa into dev May 24, 2023
@hwwhww hwwhww deleted the fix-pickle branch May 24, 2023 14:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@djrtwo djrtwo djrtwo approved these changes

Assignees
No one assigned
Labels
scope:CI/tests/pyspec
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@hwwhww @djrtwo