Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve discoverability in the presence of conflicting requirements #79

Closed
areitz opened this issue Apr 15, 2015 · 1 comment
Closed

Improve discoverability in the presence of conflicting requirements #79

areitz opened this issue Apr 15, 2015 · 1 comment

Comments

@areitz
Copy link

areitz commented Apr 15, 2015

In PEX 0.8.6, it's easy to get into a situation like this where two requirements have transitive dependencies that conflict. Here is an example, generated from pants inside of Twitter:

...
16:44:59 00:11     [pytest]
16:44:59 00:11       [run]
                     Building chroot for [PythonTests(BuildFileAddress(/Users/areitz/workspace/source/science/tests/python/twitter/source_eng/snapshot/BUILD, snapshot))]:
  Dumping library: PythonLibrary(BuildFileAddress(/Users/areitz/workspace/source/science/src/python/twitter/common/contextutil/BUILD, contextutil))
  Dumping library: PythonLibrary(BuildFileAddress(/Users/areitz/workspace/source/science/src/python/twitter/common/dirutil/BUILD, dirutil))
  Dumping library: PythonLibrary(BuildFileAddress(/Users/areitz/workspace/source/science/src/python/twitter/git/BUILD, git))
  Dumping library: PythonLibrary(BuildFileAddress(/Users/areitz/workspace/source/science/src/python/twitter/source_eng/snapshot/BUILD, lib))
  Dumping library: PythonLibrary(BuildFileAddress(/Users/areitz/workspace/source/science/src/python/twitter/common/app/BUILD, app))
  Dumping library: PythonLibrary(BuildFileAddress(/Users/areitz/workspace/source/science/src/python/twitter/common/collections/BUILD, collections))
  Dumping library: PythonLibrary(BuildFileAddress(/Users/areitz/workspace/source/science/src/python/twitter/common/lang/BUILD, lang))
  Dumping library: PythonLibrary(BuildFileAddress(/Users/areitz/workspace/source/science/src/python/twitter/common/log/BUILD, log))
  Dumping library: PythonLibrary(BuildFileAddress(/Users/areitz/workspace/source/science/src/python/twitter/common/options/BUILD, options))
  Dumping library: PythonLibrary(BuildFileAddress(/Users/areitz/workspace/source/science/src/python/twitter/common/process/BUILD, process))
  Dumping library: PythonLibrary(BuildFileAddress(/Users/areitz/workspace/source/science/src/python/twitter/common/util/BUILD, util))
  Dumping library: PythonLibrary(BuildFileAddress(/Users/areitz/workspace/source/science/src/python/twitter/common/app/modules/BUILD, modules))
  Dumping library: PythonLibrary(BuildFileAddress(/Users/areitz/workspace/source/science/src/python/twitter/common/fs/BUILD, fs))
  Dumping library: PythonLibrary(BuildFileAddress(/Users/areitz/workspace/source/science/src/python/twitter/common/quantity/BUILD, quantity))
  Dumping requirement: twitter.common.contextutil==0.3.2
  Dumping requirement: twitter.common.dirutil==0.3.2
  Dumping requirement: twitter.common.collections==0.3.2
  Dumping requirement: twitter.common.lang==0.3.2
  Dumping requirement: twitter.common.log==0.3.2
  Dumping requirement: twitter.common.options==0.3.2
  Dumping requirement: twitter.common.process==0.3.2
  Dumping requirement: twitter.common.util==0.3.2
  Dumping requirement: twitter.common.fs==0.3.2
  Dumping requirement: twitter.common.quantity==0.3.2
  Dumping requirement: arrow==0.5.4
  Dumping requirement: six==1.4.1
  Dumping requirement: mock==1.0.1
  Dumping requirement: pytest-timeout==0.3
  Dumping requirement: setuptools==1.1.7
  Dumping requirement: pytest
  Dumping requirement: pytest-timeout
  Dumping requirement: pytest-cov
  Dumping requirement: unittest2
Skipping PythonRequirement(unittest2py3k) based upon version filter

               FAILURE

Exception caught:
  File "/Users/areitz/.pants.d/bin/pants.pex/pants-1.1.0-testing-py27.pex/.bootstrap/_pex/pex.py", line 271, in execute
    self.execute_entry(entry_point, args)
  File "/Users/areitz/.pants.d/bin/pants.pex/pants-1.1.0-testing-py27.pex/.bootstrap/_pex/pex.py", line 320, in execute_entry
    runner(entry_point)
  File "/Users/areitz/.pants.d/bin/pants.pex/pants-1.1.0-testing-py27.pex/.bootstrap/_pex/pex.py", line 343, in execute_pkg_resources
    runner()
  File "/Users/areitz/workspace/source/science/.pex/install/pantsbuild.pants-0.0.32-py2-none-any.whl.69fb72b6af1d39fa7822b427eed13767627a68b9/pantsbuild.pants-0.0.32-py2-none-any.whl/pants/bin/pants_exe.py", line 81, in main
    _run(exiter)
  File "/Users/areitz/workspace/source/science/.pex/install/pantsbuild.pants-0.0.32-py2-none-any.whl.69fb72b6af1d39fa7822b427eed13767627a68b9/pantsbuild.pants-0.0.32-py2-none-any.whl/pants/bin/pants_exe.py", line 74, in _run
    result = goal_runner.run()
  File "/Users/areitz/workspace/source/science/.pex/install/pantsbuild.pants-0.0.32-py2-none-any.whl.69fb72b6af1d39fa7822b427eed13767627a68b9/pantsbuild.pants-0.0.32-py2-none-any.whl/pants/bin/goal_runner.py", line 174, in run
    result = self._do_run()
  File "/Users/areitz/workspace/source/science/.pex/install/pantsbuild.pants-0.0.32-py2-none-any.whl.69fb72b6af1d39fa7822b427eed13767627a68b9/pantsbuild.pants-0.0.32-py2-none-any.whl/pants/bin/goal_runner.py", line 230, in _do_run
    return engine.execute(context, self.goals)
  File "/Users/areitz/workspace/source/science/.pex/install/pantsbuild.pants-0.0.32-py2-none-any.whl.69fb72b6af1d39fa7822b427eed13767627a68b9/pantsbuild.pants-0.0.32-py2-none-any.whl/pants/engine/engine.py", line 26, in execute
    self.attempt(context, goals)
  File "/Users/areitz/workspace/source/science/.pex/install/pantsbuild.pants-0.0.32-py2-none-any.whl.69fb72b6af1d39fa7822b427eed13767627a68b9/pantsbuild.pants-0.0.32-py2-none-any.whl/pants/engine/round_engine.py", line 212, in attempt
    goal_executor.attempt(explain)
  File "/Users/areitz/workspace/source/science/.pex/install/pantsbuild.pants-0.0.32-py2-none-any.whl.69fb72b6af1d39fa7822b427eed13767627a68b9/pantsbuild.pants-0.0.32-py2-none-any.whl/pants/engine/round_engine.py", line 45, in attempt
    task.execute()
  File "/Users/areitz/workspace/source/science/.pex/install/pantsbuild.pants-0.0.32-py2-none-any.whl.69fb72b6af1d39fa7822b427eed13767627a68b9/pantsbuild.pants-0.0.32-py2-none-any.whl/pants/backend/python/tasks/pytest_run.py", line 108, in execute
    if self.run_tests(test_targets, workunit):
  File "/Users/areitz/workspace/source/science/.pex/install/pantsbuild.pants-0.0.32-py2-none-any.whl.69fb72b6af1d39fa7822b427eed13767627a68b9/pantsbuild.pants-0.0.32-py2-none-any.whl/pants/backend/python/tasks/pytest_run.py", line 113, in run_tests
    return 0 if self._do_run_tests(targets, workunit).success else 1
  File "/Users/areitz/workspace/source/science/.pex/install/pantsbuild.pants-0.0.32-py2-none-any.whl.69fb72b6af1d39fa7822b427eed13767627a68b9/pantsbuild.pants-0.0.32-py2-none-any.whl/pants/backend/python/tasks/pytest_run.py", line 356, in _do_run_tests
    with self._test_runner(targets, workunit) as (pex, test_args):
  File "/opt/twitter/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/Users/areitz/workspace/source/science/.pex/install/pantsbuild.pants-0.0.32-py2-none-any.whl.69fb72b6af1d39fa7822b427eed13767627a68b9/pantsbuild.pants-0.0.32-py2-none-any.whl/pants/backend/python/tasks/pytest_run.py", line 336, in _test_runner
    builder = chroot.dump()
  File "/Users/areitz/workspace/source/science/.pex/install/pantsbuild.pants-0.0.32-py2-none-any.whl.69fb72b6af1d39fa7822b427eed13767627a68b9/pantsbuild.pants-0.0.32-py2-none-any.whl/pants/backend/python/python_chroot.py", line 216, in dump
    find_links=find_links)
  File "/Users/areitz/workspace/source/science/.pex/install/pantsbuild.pants-0.0.32-py2-none-any.whl.69fb72b6af1d39fa7822b427eed13767627a68b9/pantsbuild.pants-0.0.32-py2-none-any.whl/pants/backend/python/resolver.py", line 68, in resolve_multi
    cache_ttl=ttl)
  File "/Users/areitz/workspace/source/science/.pex/install/pex-0.8.6-py2.py3-none-any.whl.3cbdb5d909a2f0a8f8d3e8a8dc123e44a80b45a6/pex-0.8.6-py2.py3-none-any.whl/pex/resolver.py", line 220, in resolve
    raise Unsatisfiable('Cannot satisfy requirements: %s' % requirement_set[requirement.key])

Exception message: Cannot satisfy requirements: [PythonRequirement(six==1.4.1), Requirement.parse('six>=1.5')]

In this case, the target has a direct dependency on six 1.4.1, but some other dependency is bringing in a conflicting requirement transitively. It's really hard to find this, even with debugging turned on, because PEX doesn't print out what additional requirements it's adding to the set, and where it's getting them from. I added this small patch:

diff --git a/.deps/pex-0.8.6-py2.py3-none-any.whl/pex/resolver.py b/.deps/pex-0.8.6-py2.py3-none-any.whl/pex/resolver.py
index 98aa7b8..4ca92c4 100644
--- a/.deps/pex-0.8.6-py2.py3-none-any.whl/pex/resolver.py
+++ b/.deps/pex-0.8.6-py2.py3-none-any.whl/pex/resolver.py
@@ -228,6 +228,8 @@ def resolve(
           continue
         new_requirements.update(requires(highest_package, requirement))
         processed_requirements.add(requirement)
+      if new_requirements:
+        TRACER.log('Requirement "{}" brought in extra requirements: {}'.format(requirement_key, new_requirements))
       requirements.extend(list(new_requirements))

     if not requirements:

Which adds the following output:

pex: Requirement "twitter.common.contextutil" brought in extra requirements: OrderedSet([Requirement.parse('twitter.common.dirutil==0.3.2')])
pex: Requirement "pytest-timeout" brought in extra requirements: OrderedSet([Requirement.parse('pytest>=2.0')])
pex: Requirement "twitter.common.collections" brought in extra requirements: OrderedSet([Requirement.parse('twitter.common.lang==0.3.2')])
pex: Requirement "twitter.common.quantity" brought in extra requirements: OrderedSet([Requirement.parse('twitter.common.lang==0.3.2')])
pex: Requirement "twitter.common.process" brought in extra requirements: OrderedSet([Requirement.parse('twitter.common.string==0.3.2')])
pex: Requirement "twitter.common.util" brought in extra requirements: OrderedSet([Requirement.parse('twitter.common.contextutil==0.3.2')])
pex: Requirement "arrow" brought in extra requirements: OrderedSet([Requirement.parse('python-dateutil')])
pex: Requirement "pytest" brought in extra requirements: OrderedSet([Requirement.parse('py>=1.4.25')])
pex: Requirement "twitter.common.log" brought in extra requirements: OrderedSet([Requirement.parse('twitter.common.dirutil==0.3.2'), Requirement.parse('twitter.common.options==0.3.2')])
pex: Requirement "twitter.common.dirutil" brought in extra requirements: OrderedSet([Requirement.parse('twitter.common.lang==0.3.2')])
pex: Requirement "twitter.common.fs" brought in extra requirements: OrderedSet([Requirement.parse('twitter.common.util==0.3.2'), Requirement.parse('twitter.common.quantity==0.3.2'), Requirement.parse('twitter.common.string==0.3.2')])
pex: Requirement "pytest-cov" brought in extra requirements: OrderedSet([Requirement.parse('pytest>=2.2.3'), Requirement.parse('cov-core>=1.6')])
pex: Requirement "twitter.common.contextutil" brought in extra requirements: OrderedSet([Requirement.parse('twitter.common.dirutil==0.3.2')])
pex: Requirement "pytest" brought in extra requirements: OrderedSet([Requirement.parse('py>=1.4.25')])
pex: Requirement "twitter.common.string" brought in extra requirements: OrderedSet([Requirement.parse('twitter.common.lang==0.3.2')])
pex: Requirement "twitter.common.util" brought in extra requirements: OrderedSet([Requirement.parse('twitter.common.contextutil==0.3.2')])
pex: Requirement "python-dateutil" brought in extra requirements: OrderedSet([Requirement.parse('six>=1.5')])
pex: Requirement "twitter.common.quantity" brought in extra requirements: OrderedSet([Requirement.parse('twitter.common.lang==0.3.2')])
pex: Requirement "cov-core" brought in extra requirements: OrderedSet([Requirement.parse('coverage>=3.4')])
pex: Requirement "twitter.common.dirutil" brought in extra requirements: OrderedSet([Requirement.parse('twitter.common.lang==0.3.2')])

This makes it trivially easy to see that the floating dep on python-dateutil is causing the conflict.

I went to submit this patch upstream, but this code has totally changed. I'll wait until we're consuming PEX 1.0 inside Twitter, and once I have something that runs in our enviornment, I'll see if this patch can be re-applied.
@wickman
Copy link
Contributor

wickman commented Apr 17, 2015

This now works

mba=pex=; dist/pex twitter.common.{contextutil,dirutil,collections,lang,log,options,process,util,fs,quantity}==0.3.2 arrow==0.5.4 six==1.4.1 mock==1.0.1 pytest-timeout==0.3 setuptools==1.1.7 pytest pytest-timeout pytest-cov unittest2 -v
Could not satisfy all requirements for six==1.4.1:
    six==1.4.1, six>=1.4(from: unittest2), six>=1.5(from: arrow==0.5.4->python-dateutil)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@wickman @areitz