Project seems dead due to Constantly being Incompatible with newer versions of PIP and its own dependencies.

[duaneking]

We cant use Chalice if we are following python best practices and using the most up to date version of pip.

Python packages shouldn't be dependent on a specific version of pip anyway, that in itself is a serous defect that creates more work consistently as new versions of pip are released, and pip gets updated all the time, so Chalice is broken all the time since its constantly hitting a new version it thinks it cant use.

This creates more work for everybody.

Every time this happens, somebody tries to be helpful and files a defect, that defect is ignored for weeks while people rage about arbitrary version conflicts, and then that defect is ultimately closed after being marked as a feature request when in reality, its not a feature request, its a basic functional requirement. Then a day later somebody else updates pip and the cycle repeats.

The dev team has a history of not being community focused.

Reading the closed bugs that are still reproducing - because the Chalice dev team seems to really enjoy closing open and reproducible defects as "feature requests" without fixing them and i have no idea why this is - the version limit in place for PIP is in place for the absolute wrong reasons, out of fear instead of metrics that can be acted on, yet other projects dont have this issue and are perfectly working fine on AWS without the same pip version restrictions.

The old dependencies are getting harder to maintain

Add to this the fact that its using an out of date version of click and other dependencies, and we now have multiple defects spread across the project created due to the increasingly difficult maintenance right now, as this project actively looks unmaintained and dead due to lack of support from AWS on these issues, and as time goes on these issues are getting bigger and bigger as the package versions getting used are getting older and older and start to conflict more and more with a modern python dependency map using up to date versions of basic things like pip and click. Its a ticking time bomb.

People cant trust Chalice if it wont even install in pip.

Due to this, people are not investing in Chalice or using it for projects like they could be, because they can not trust that it will be around and get the correct and proper maintenance support that it requires from AWS in the future, since it looks like AWS has already given up on it, while projects like Zappa, etc are still going strong and don't have these issues with newer versions of PIP like Chalice does, so it doesn't make any sense to us as developers to see other projects not having these issues and Chalice for bad or worse, looks like the black sheep that doesn't work as well.

Are the developers eating their own dogfood?

I'm worried they are not actually using Chalice themselves as many of its sharp edges would go away if a passionate developer that actively worked on the code was actually using it. Many of the issues reported by the community are easy to find if the dev team working on Chalice actually used Chalice themselves, and they honestly would have been fixed easier as part of that initial discovery.. but that is not the case based on the defect history.

We shouldn't be forced to keep old versions of libs with possible breaking bugs around.

People who just set up their devbox today but also do other python projects like with django or flask that require modern up to date versions of pip without conflict are having to manage the system pip verses the venvs pip and just cant use Chalice unless they create a virtual machine or manage pip installs manually and add to their workload, making sure to always keep an out of date version of pip on their systems so they can update only venvs that need it. That is not at all in the spirit of python, where the newest version of pip is somewhat mandatory since with it comes the packages it supports, and in some cases the new version has security fixes as packages are removed and issues found, etc. So in many cases, either for compliance or security, not upgrading is not an option.

Other libs that do what Chalice does do not have the problems Chalice does.

Yet other libraries like Zappa, Serverless, etc all do fine and don't have this pip version restriction or the issues with old dependencies, because they are actively maintained.. yet they are in many cases open source projects with significantly less resources than AWS as a company and yet they are out performing AWS and the Chalice team. Why is that?

Valid Bugs are getting closed without getting fixed.

Right now, even bugs that people file in an attempt to help get traction on issues are just getting wrongly closed after being set as "feature requests" when the bugs themselves that are filed are not even fixed before the defects describing them are closed. Why is that?

Question

So is this project dead?

[duaneking]

Examples:

#991

#981

[Sytten]

This project has a lot of potential, but is lacking proper support I agree. Basically it's only @jamesls doing what he can as a side task I believe. We started by using chalice, but now I am thinking of moving to serverless because of the missing features in chalice and the fact that the community is much bigger.

[tedivm]

While I don't think "dead" is the right term, I definitely agree that this project is moving very slowly compared to similar projects. I don't fault the devs for it, but do think that AWS as a whole needs to either give this framework the support and resources it needs, or just give up and admit that this isn't a priority for them.

That being said I really hope they commit to it and give some more resources to the project. The Flask like API makes this project really appealing, and it seems to be one of the only projects that lets you define the resources right alongside the code (as opposed to requiring a stack of configuration files).

[jamesls]

Hi, thanks for the feedback on Chalice.

There's a lot of points being raised here, so I wanted to try and address each
of them individually.

I wanted to separate the discussion of Chalice's use of pip from the status of
the project, I think they're separate from each other that it's worth diving
into each of them on their own.

First, let's talk about Chalice and pip, and clear up a few points.

Python packages shouldn't be dependent on a specific version of pip anyway,
that in itself is a serous defect that creates more work consistently as new
versions of pip are released, and pip gets updated all the time, so Chalice is
broken all the time since its constantly hitting a new version it thinks it
cant use.

You have a valid concern, but I'd like to explain why we have an upper version
restriction of pip. Unlike most other projects, we are actually using internal
parts of pip to download the appropriate python packages for lambda. This
happens during a chalice deployment and a chalice package. We've run into
issues in the past where new versions of pip have broken our packaging code,
and we already have branching to handle
different versions of pip (https://github.com/aws/chalice/blob/master/chalice/compat.py#L10-L20).

the version limit in place for PIP is in place for the absolute wrong
reasons, out of fear instead of metrics that can be acted on

Just to reiterate: we are using internal parts of pip, and this code has
broken chalice in the past. It would be great if there was a guaranteed
backwards compatible way to do what we need, but AFAIK that doesn't exist. So
to guard against this, we have an upper bound of pip. If we don't do this,
then we risk running into exactly the problem you're talking about: a new
version of pip could completely break Chalice's packager. This has already
happened. Even worse, suppose you had a deployment pipeline that installed a
specific chalice version and then ran chalice deploy or chalice package.
That could start failing if a new version of pip changed its internals, even if
you change nothing in your code or deployment process. That's just as bad, if
not worse.

Now, that being said, I don't think the current situation is ideal either. I'm
open to suggestions on how to make this better, but I don't think we should
go with a solution that has potential to start breaking chalice deploy. As
far as I can tell, having unbounded package dependencies means that you will
always run this risk, regardless of whether or not it's pip or some other
python package.

I'd want to make sure we can keep this discussion productive. Does anyone have
any suggestions on how to fix this issue? Or if people are really set on not
having an upper bound on pip, what are your thoughts on this suddenly breaking
your deployments (for example, see: #808).

I'm worried they are not actually using Chalice themselves as many of its
sharp edges would go away if a passionate developer that actively worked on the
code was actually using it.

Yes, we use Chalice internally in a number of places. In fact, Chalice started
because I was working on some internal serverless apps and wanted to streamline
the development of it. We are very aware of some of the missing features
that we would really like to add. The issue is not lack of awareness.

So is this project dead?

No, this project is not dead.

Our team works on several open source python projects including aws/aws-cli,
boto/boto3, boto/botocore, boto/s3transfer, and of course aws/chalice. Our
time is split between these projects and we shift resources to these various
projects based on priorities. I completely understand @tedivm's concerns
that it's not moving as quickly as other projects. I am sharing these
concerns internally and starting a discussion on making sure Chalice has
the resources it needs.

Please keep the feedback coming. A lot of what we do is driven by
feedback we receive from customers. It helps make a much stronger case
when we can point to specific feedback from everyone. Thanks again everyone.

[duaneking]

Thank you for the detailed response, I deeply appreciate the extra data points.

First big issue I see is that in python, using internals of pip is at best frowned on, in polite circles considered verboten or actively considered to be evil, and actively goes against multiple PEP's and community driven standards. If you need an interface, the correct and community supported thing to do is to contribute to pip development directly to provide that public hook, then use that public hook, not lock your module to be so restrictive and brittle/buggy. Please do that instead!

The second is that when something does break, it takes a long time to get a resolution, leaving the users effectively SOL until a fix happens. If the chalice team could define what they need in terms of a PIP public interface, I'm sure the community would do its best to assist.

[dstufft]

Hi there!

So let me first provide some context, where I have two hats that I wear in this discussion, one is my hat as a pip core developer and longtime member of the Python packaging community (several of those PEPs are written be me ;)), and the other hat is as an AWS employee on the team tasked with producing/maintaining Chalice (among other libraries as @jamesls mentioned).

It is true that pip does not have a supported public interface besides it's CLI, and that we generally recommend people not depend on the internals of pip (going so far as to have pretty much all of pip live inside of pip._internal). However, like Python itself we generally take a "consenting adults" stance, and say that if you're going to reach in, that's fine we're not going to stop you, but what you're doing is unsupported by pip itself, and thus you should take precautions to ensure that our lack of a backwards compatibility promise on these interfaces does not break you (or if it does, that you can live with that breakage). Projects such as pipenv took the route of simply bundling a copy of pip inside of itself, whereas other projects like Chalice depend on pip, but pin an upper version so that we can test/bump before release, or other projects like pip-tools don't pin a version, and just break periodically when pip releases a version that happens to break them.

None of these approaches are wrong, and they all just reflect a different set of tradeoffs, given the need to use an internal part of pip. In this particular project the decision was made to cap the upper bounds on pip, to try and prevent breaking users when a new version of pip is released. It is possible that a different trade off would be better, and we can discuss the relative merits of the different strategies if folks feel strongly one way or another.

The remaining strategy is the one mentioned here about adding interfaces to pip (or somewhere else) that makes what Chalice is attempting to do here a supported, public interface so that these work arounds are not required. Unfortunately what Chalice needs to do here is special case enough that I think it's unlikely to get implemented in pip, as I'm struggling to think of a use case where it's useful outside of Chalice itself. The right longer term solution is likely to continue the effort that pip has already been undergoing to break out its internals into dedicated libraries, so that pip itself becomes just a front end over that collection of libraries. At that point Chalice could then just also use those dedicated libraries, but use them in a way that implements the behavior chalice needs, rather than the more general purpose behavior that pip supports.

Unfortunately that effort is non trivial and unlikely to be "finished" in the near term future, but it is ongoing, and hopefully we will eventually get to a point where projects like Chalice, pipenv, pip-tools, etc no longer are needing to reach into pip's internals.

[jamesls]

Thanks @dstufft for chiming in.

Just want to close the loop here. Again if anyone has specific proposals they'd prefer to see in Chalice for improving pip integration, feel free to chime in. I do think it's worth investigating the pipenv approach of bundling pip as an interim solution.

cc @stealthycoin

[no-response]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.