Exclude package?

[ariscn]

I'm running pip-compile under OS X and have a requirements/dev.in file which includes IPython. This pulls in gnureadline. When I build a pex (https://github.com/pantsbuild/pex) from this requirements file for a Linux system, it's not usable, because gnureadline is only available for OS X.

I can work around this by removing or commenting out the gnureadline dependency, but this is a bit hacky. It would be great to be able to do: pip-compile requirements/base.in && pip-compile requirements/dev.in && pip-compile --remove gnureadline requirements/stage.in.

[jeffwidman]

FYI--latest version of ipython removed the gnureadline dependency in favor of the cross-platform prompt-toolkit.

The feature request is still valid in the abstract though...

[nvie]

If someone can make a PR for a --exclude <pkg> flag, I'd be happy to merge the PR. Please make sure to allow multiple occurrances of this flag, i.e. --exclude foo --exclude bar ...

[decentral1se]

@nvie @ariscn

I've started working on this but have questions.

Should this exclusion happen only for packages described in a requirements.in or should also exclude dependencies?

If we start to exclude dependencies, then we might break some packages. The user might not know at which level some package they want to exclude will be excluded for some other package.

[Groxx]

Yeah, this would be great. We've had to add a lint step to prevent enum from sneaking in because enum34 collides with it (why the heck did they think that was a good idea?!), ideally we could --exclude enum or similar right in the .in file so it's unmissable.

[decentral1se]

@Groxx could you check out the branch (add-exclude-flag) of https://github.com/lwm/pip-tools/commit/f5771a3aa855887f22dad2ba4a58eef1f158f34f and give it a check if it works for you? I am afraid #388 is going to rot without review or progress.

[milin]

@lwm Would this fix issues like this #366? I was thinking along with exclude, one could also make pip-compile flexible, so if issues like that happen, we could fall back to a backup version.

[decentral1se]

@milin I am not sure (new to the code here). Try it and let's find out? Not sure what you mean by a 'backup version'? How would the interface look like?

[milin]

@lwm I ended up fixing it since I needed it for a project. #394 as more info on the problem and the proposed solution.

[AndydeCleyre]

Now that pip-sync happily ignores requirements with markers that don't match the current environment, we may have the beginnings of another approach to the original problem here. We can generate "fully-marked" output files for each relevant platform, and have a master txt which includes them all of them (with -r).

[ssbarnea]

I think that this feature could be implemented easier by making the UNSAFE_PACKAGES list extensible as these are already removed from the dependency list. Basically by allowing other packages to be added to unsafe list, we close this feature request.

I am unable to work on this myself but I would be able to find time to review the PR and also test it manually.

[hmc-cs-mdrissi]

I've made a pr to close this task here, #1509. The main pr change is small with adding unit test cases being bulk of line changes.

[IanTayler]

@hmc-cs-mdrissi @ssbarnea @atugushev

Any news regarding #1509 ? Seems to have been deprioritized several times in a row. Are we expecting to see this anytime in the near future? Is there any way I can help out with this? I've been hacking this with a grep, but it's far from ideal.

@skieffer Where did #1645 come from? Assuming PR #1509 does get merged, what would the plan be for your PR? In my opinion there's too much overlap between both options.

[skieffer]

Where did #1645 come from?

It addresses problems like this one and this one.

As I see it, both #1645 and #1509 are important, because they are complementary. #1509 deletes nodes out of the dependency graph after the whole graph has been computed. #1645 deletes edges out of the graph, thereby preventing parts of it from ever being reached in the first place.

As a result, these two options serve two different use cases:

• (1509): "There's a package I don't want, under any circumstances."

• (1645): "There's a package I don't need right now, but might need later, as I add new dependencies."

I've been hacking this with a grep, but it's far from ideal.

Yeah, I too have some hacky solutions in place right now, and I think it would be great to add these features!

[skieffer]

Actually it would be very easy to add a third usage pattern to #1645, like this:

pip-compile --cut-deps :C

(note the leading :), meaning, "Cut all incoming edges to package C."
This would then allow you to exclude C, no matter which packages depended on it.

So altogether there would be three patterns:

1. Cut just the edge from A to C:

pip-compile --cut-deps A:C

2. Cut all outgoing edges from package A:

pip-compile --cut-deps A

3. Cut all incoming edges to package C:

pip-compile --cut-deps :C

With this, I wouldn't see much need for #1509.

[hmc-cs-mdrissi]

I'm happy to resolve conflicts by this weekend with pr (may do now if they look small enough). As a non-maintainer, I'm unaware when it'll move forward though.

As a note on pip-compile --cut-deps :C I'm a little unsure what this means. My primary use case is I want all of C's dependencies but I don't want C itself in compiled requirements. I do not want to exclude C's dependencies. That's why my pr has both exclude-packages + do you want to keep dependencies of excluded packages. Are both of those options possible with cut-deps? Cutting only C vs C + it's dependencies?

The primary use case I had was top level requirements that I needed to install separately. As an example if you have multiple editable requirements, those are unhashable and incompatible with --require-hashes. But if you want pip compile to make a file of all there dependencies you need to exclude them and then install them separately. Similarly if you have requirements that are specified by vcs then they are unhashable and need exclusion, but there dependencies are still desired. So I actually do need "C", but C needs to be treated separately. My issue is partly hashes are all or nothing today and you can't have some packages unhashed in a requirements file.

[IanTayler]

My use case matches hmc-cs-mdrissi's exactly. Currently I'm doing that by sed -i-ing the .txt file after the fact, but in my humble opinion I think it's a common enough use case that it's worth having in the CLI.

The --exclude option would save me from a lot of ugly hackery with sed in several projects.

[ssbarnea]

This is already implemented on main branch now and will be included in next release.

[AndydeCleyre]

Related: #1656 -- Someone's expecting the unpinned excluded package to be included (without a version) in pip-compile's output.

[jamesbraza]

The --exclude option would save me from a lot of ugly hackery with sed in several projects.

Yeah that ticket has to do with what @IanTayler is saying on --exclude. And I am familiar with these sort of sed hacks, lol.

One question for @hmc-cs-mdrissi (or whomever knows the answer), if one wants to mark several packages with --unsafe-package, how can one do it?

Currently I just include the flag multiple times, but it feels sub-optimal. Is there a way to have several unsafe packages marked at once, without having --unsafe-package repeated multiple times?

# Can this be done with --unsafe-package only being written once?
pip-compile --unsafe-package foo --unsafe-package bar --unsafe-package baz

[jamesbraza]

Working with the current master (pip-compile==6.8.1.dev5+geed2719), I discovered a potential bug with this.

requirements.in

setuptools
unsafe-package

---

Case 1

pip-compile --no-header --no-emit-index-url --unsafe-package=unsafe-package requirements.in:

# The following packages are considered to be unsafe in a requirements file:
# unsafe-package

In the generated requirements.txt: setuptools isn't reflected in the file anywhere (not desirable).

---

Case 2

pip-compile --allow-unsafe --no-header --no-emit-index-url --unsafe-package=unsafe-package requirements.in:

unsafe-package==1.1.1
    # via -r requirements.in

# The following packages are considered to be unsafe in a requirements file:
setuptools==63.2.0
    # via -r requirements.in

In the generated requirements.txt: setuptools is pinned, but now we also pinned the unsafe-package (accidental pinning of unsafe-package, and no warning of this --> not desirable).

---

Case 3

pip-compile --no-header --no-emit-index-url --unsafe-package=setuptools --unsafe-package=unsafe-package requirements.in:

# The following packages are considered to be unsafe in a requirements file:
# setuptools
# unsafe-package

In the generated requirements.txt: I would say this is now the expected behavior. I guess setuptools is now automatically considered "unsafe"?

---

In other words: since setuptools is now silently considered unsafe, and we have no possibility of having setuptools in a generated requirements.txt. This seems like an issue to me, at least pip-compile should emit a warning if it's silencing a setuptools dependency.

@hmc-cs-mdrissi (cc @AndydeCleyre) can you read through this? Imo there's still some outstanding work before this feature should be released.

[AndydeCleyre]

@jamesbraza Thanks!

Case 1 looks like a bug, with the option not fulfilling its promise of replacing the default unsafe package list. I agree it should be fixed before a release.

In case 2 I would indeed expect all packages to be pinned, since --allow-unsafe is passed. But I would also expect the "unsafe" packages to be grouped together, if grouped at all.

Case 3 looks good to me.

[hmc-cs-mdrissi]

I agree with @AndydeCleyre view on all 3 cases. I’m not really sure what use case for allow unsafe and custom unsafe packages is besides grouping.

I’ll try to take a look later this week especially on case 1.

[AndydeCleyre]

Currently I just include the flag multiple times, but it feels sub-optimal. Is there a way to have several unsafe packages marked at once, without having --unsafe-package repeated multiple times?

# Can this be done with --unsafe-package only being written once?
pip-compile --unsafe-package foo --unsafe-package bar --unsafe-package baz

We could add some processing sugar for potential CSVs, like we do for --extra:

extras = tuple(itertools.chain.from_iterable(ex.split(",") for ex in extras))

[atugushev]

Closed as completed via #1509.