'chalice package' fails

[hgibsonqb]

Hi,
I'm trying to run the command chalice package --merge-template resource-policies/resource-policy-name.json --stage name lz-dev dist/name

But it's been failing with the traceback

Creating deployment package.
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/chalice/cli/__init__.py", line 485, in main
    return cli(obj={})
  File "/usr/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/chalice/cli/__init__.py", line 414, in package
    packager.package_app(config, out, stage)
  File "/usr/lib/python3.6/site-packages/chalice/package.py", line 919, in package_app
    config, chalice_stage_name)
  File "/usr/lib/python3.6/site-packages/chalice/package.py", line 82, in construct_resources
    self._build_stage.execute(config, resources)
  File "/usr/lib/python3.6/site-packages/chalice/deploy/deployer.py", line 995, in execute
    step.handle(config, resource)
  File "/usr/lib/python3.6/site-packages/chalice/deploy/deployer.py", line 851, in handle
    handler(config, resource)
  File "/usr/lib/python3.6/site-packages/chalice/deploy/deployer.py", line 878, in handle_deploymentpackage
    config.project_dir, config.lambda_python_version)
  File "/usr/lib/python3.6/site-packages/chalice/deploy/packager.py", line 94, in create_deployment_package
    abi, requirements_filepath, site_packages_dir)
  File "/usr/lib/python3.6/site-packages/chalice/deploy/packager.py", line 506, in build_site_packages
    abi, tempdir, requirements_filepath)
  File "/usr/lib/python3.6/site-packages/chalice/deploy/packager.py", line 369, in _download_dependencies
    requirements_filename, directory)
  File "/usr/lib/python3.6/site-packages/chalice/deploy/packager.py", line 327, in _download_all_dependencies
    self._pip.download_all_dependencies(requirements_filename, directory)
  File "/usr/lib/python3.6/site-packages/chalice/deploy/packager.py", line 745, in download_all_dependencies
    raise PackageDownloadError(error)
chalice.deploy.packager.PackageDownloadError: Traceback (most recent call last):
  File "<string>", line 1, in <module>
TypeError: 'module' object is not callable

Any idea what's happening? I haven't changed the configuration or code

[stealthycoin]

Interesting just a cursory attempt to repro doesn't fail. I ran

chalice package --merge-template resource-policy-name.json --stage lz-dev dist/name

There was one extra argument so I choose to remove name from the command above. Can you give me more details such as the content of that template that is getting merged. Or can you try running it with

chalice --debug package ...

Could you also give me the output of pip freeze, that may be useful.

Edit: oh and does chalice deploy succeed? Is this isolated to the package command or a shared failure when using deploy or package?

[hgibsonqb]

Hi,
"name" was the output file.

Output of chalice --version

Loading .env environment variables…
chalice 1.11.1, python 3.6.8, linux 4.9.184-linuxkit

Output of pip freeze

asn1crypto==1.0.1
atomicwrites==1.3.0
attrs==19.2.0
autopep8==1.4.4
aws-sam-translator==1.15.0
aws-xray-sdk==2.4.2
awscli==1.16.255
bleach==3.1.0
boto==2.49.0
boto3==1.9.245
botocore==1.12.245
certifi==2019.9.11
cffi==1.12.3
cfn-lint==0.24.4
chalice==1.11.1
chardet==3.0.4
Click==7.0
colorama==0.4.1
coverage==4.5.4
cryptography==2.7
DateTime==4.3
docker==4.1.0
docopt==0.6.2
docutils==0.15.2
dpath==1.4.2
ecdsa==0.13.3
entrypoints==0.3
enum-compat==0.0.2
flake8==3.7.8
future==0.17.1
headerparser==0.3.0
humanfriendly==4.18
idna==2.8
importlib-metadata==0.23
Jinja2==2.10.3
jmespath==0.9.4
js-regex==1.0.0
jsondiff==1.1.2
jsonpatch==1.24
jsonpickle==1.2
jsonpointer==2.0
jsonschema==3.1.0
jump-consistent-hash==3.0.2
MarkupSafe==1.1.1
mccabe==0.6.1
mock==3.0.5
more-itertools==7.2.0
moto==1.3.13
packaging==19.2
pluggy==0.13.0
property-manager==2.3.1
py==1.8.0
py-cpuinfo==5.0.0
pyasn1==0.4.7
pycodestyle==2.5.0
pycparser==2.19
pyflakes==2.1.1
Pygments==2.4.2
pyparsing==2.4.2
pyrsistent==0.15.4
pytest==5.2.1
pytest-benchmark==3.2.2
pytest-cov==2.8.1
pytest-html==2.0.0
pytest-metadata==1.8.0
pytest-mock==1.11.1
python-dateutil==2.8.0
python-jose==3.0.1
pytz==2019.3
PyYAML==5.1.2
readme-renderer==24.0
requests==2.22.0
responses==0.10.6
rsa==3.4.2
s3transfer==0.2.1
six==1.12.0
sshpubkeys==3.1.0
sure==1.4.11
urllib3==1.25.6
verboselogs==1.7
wcwidth==0.1.7
webencodings==0.5.1
websocket-client==0.56.0
Werkzeug==0.16.0
wheel-inspect==1.3.0
wrapt==1.11.2
xmltodict==0.12.0
xxhash==1.4.1
zipp==0.6.0

Output of chalice --debug package

chalice --debug package --merge-template resource-policies/stage-name.json --stage stage-name dist/stage-name
Loading .env environment variables…
Creating deployment package.
2019-10-15 12:21:26,239 chalice.deploy.packager [DEBUG] Creating deployment package.
2019-10-15 12:21:26,245 chalice.deploy.packager [DEBUG] calling pip download -r /kitchen/requirements.txt --dest /tmp/tmpm2ct8x00
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/chalice/cli/__init__.py", line 485, in main
    return cli(obj={})
  File "/usr/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/chalice/cli/__init__.py", line 414, in package
    packager.package_app(config, out, stage)
  File "/usr/lib/python3.6/site-packages/chalice/package.py", line 919, in package_app
    config, chalice_stage_name)
  File "/usr/lib/python3.6/site-packages/chalice/package.py", line 82, in construct_resources
    self._build_stage.execute(config, resources)
  File "/usr/lib/python3.6/site-packages/chalice/deploy/deployer.py", line 995, in execute
    step.handle(config, resource)
  File "/usr/lib/python3.6/site-packages/chalice/deploy/deployer.py", line 851, in handle
    handler(config, resource)
  File "/usr/lib/python3.6/site-packages/chalice/deploy/deployer.py", line 878, in handle_deploymentpackage
    config.project_dir, config.lambda_python_version)
  File "/usr/lib/python3.6/site-packages/chalice/deploy/packager.py", line 94, in create_deployment_package
    abi, requirements_filepath, site_packages_dir)
  File "/usr/lib/python3.6/site-packages/chalice/deploy/packager.py", line 506, in build_site_packages
    abi, tempdir, requirements_filepath)
  File "/usr/lib/python3.6/site-packages/chalice/deploy/packager.py", line 369, in _download_dependencies
    requirements_filename, directory)
  File "/usr/lib/python3.6/site-packages/chalice/deploy/packager.py", line 327, in _download_all_dependencies
    self._pip.download_all_dependencies(requirements_filename, directory)
  File "/usr/lib/python3.6/site-packages/chalice/deploy/packager.py", line 745, in download_all_dependencies
    raise PackageDownloadError(error)
chalice.deploy.packager.PackageDownloadError: Traceback (most recent call last):
  File "<string>", line 1, in <module>
TypeError: 'module' object is not callable

make: *** [Makefile:142: package-lz-playground] Error 2

We have multiple stages and it's failing for every stage I try to package. Also, it's failing for deploy.
Outputs of chalice --debug deploy --stage stage-name --api-gateway-stage stage-name

2019-10-15 12:27:46,774 botocore.hooks [DEBUG] Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2019-10-15 12:27:46,780 botocore.hooks [DEBUG] Changing event name from before-call.apigateway to before-call.api-gateway
2019-10-15 12:27:46,781 botocore.hooks [DEBUG] Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2019-10-15 12:27:46,783 botocore.hooks [DEBUG] Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2019-10-15 12:27:46,784 botocore.hooks [DEBUG] Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2019-10-15 12:27:46,785 botocore.hooks [DEBUG] Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2019-10-15 12:27:46,786 botocore.hooks [DEBUG] Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2019-10-15 12:27:46,789 botocore.hooks [DEBUG] Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2019-10-15 12:27:46,790 botocore.hooks [DEBUG] Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2019-10-15 12:27:46,790 botocore.hooks [DEBUG] Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2019-10-15 12:27:46,791 botocore.hooks [DEBUG] Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
Creating deployment package.
2019-10-15 12:27:46,915 chalice.deploy.packager [DEBUG] Creating deployment package.
2019-10-15 12:27:46,922 chalice.deploy.packager [DEBUG] calling pip download -r /kitchen/requirements.txt --dest /tmp/tmp6om9kma8
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/chalice/cli/__init__.py", line 485, in main
    return cli(obj={})
  File "/usr/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/chalice/cli/__init__.py", line 202, in deploy
    deployed_values = d.deploy(config, chalice_stage_name=stage)
  File "/usr/lib/python3.6/site-packages/chalice/deploy/deployer.py", line 342, in deploy
    return self._deploy(config, chalice_stage_name)
  File "/usr/lib/python3.6/site-packages/chalice/deploy/deployer.py", line 352, in _deploy
    self._build_stage.execute(config, resources)
  File "/usr/lib/python3.6/site-packages/chalice/deploy/deployer.py", line 995, in execute
    step.handle(config, resource)
  File "/usr/lib/python3.6/site-packages/chalice/deploy/deployer.py", line 851, in handle
    handler(config, resource)
  File "/usr/lib/python3.6/site-packages/chalice/deploy/deployer.py", line 878, in handle_deploymentpackage
    config.project_dir, config.lambda_python_version)
  File "/usr/lib/python3.6/site-packages/chalice/deploy/packager.py", line 94, in create_deployment_package
    abi, requirements_filepath, site_packages_dir)
  File "/usr/lib/python3.6/site-packages/chalice/deploy/packager.py", line 506, in build_site_packages
    abi, tempdir, requirements_filepath)
  File "/usr/lib/python3.6/site-packages/chalice/deploy/packager.py", line 369, in _download_dependencies
    requirements_filename, directory)
  File "/usr/lib/python3.6/site-packages/chalice/deploy/packager.py", line 327, in _download_all_dependencies
    self._pip.download_all_dependencies(requirements_filename, directory)
  File "/usr/lib/python3.6/site-packages/chalice/deploy/packager.py", line 745, in download_all_dependencies
    raise PackageDownloadError(error)
chalice.deploy.packager.PackageDownloadError: Traceback (most recent call last):
  File "<string>", line 1, in <module>
TypeError: 'module' object is not callable

[hgibsonqb]

We think it's a pip issue actually. We're using pipenv to manage dependencies and they recently released pip v 19.3. Running chalice package with pip==19.1.1 worked just fine.

I traced the bug to pypa/pip@09fd200 and made this change locally to chalice/compat.py for it to work with the new version

def pip_import_string():
    # type: () -> str
    import pip
    pip_major_version = pip.__version__.split('.')[0]
    pip_minor_version = pip.__version__.split('.')[1]
    # Pip moved its internals to an _internal module in version 10.
    # In order to be compatible with version 9 which has it at at the
    # top level we need to figure out the correct import path here.
    if pip_major_version == '9':
        return 'from pip import main'
    elif pip_major_version == '19' and pip_minor_version == '3':
        return 'from pip._internal.main import main'
    else:
        return 'from pip._internal import main'

[jamesls]

Awesome, thanks for digging in to this. We'll get this updated.

[hgibsonqb]

I've opened a corresponding ticket with pip here pypa/pip#7220

[jamesls]

To be fair, we are importing from pip._internal so I wouldn't expect them to maintain backwards compatibility for this. However, we don't have a lot of great options for the functionality we need (see #1085 (comment))

[stealthycoin]

This should be fixed in Chalice 1.12 now. Thanks for the patch.