'Connection aborted.', error(32, 'Broken pipe' when uploading ImageMagick

[baileyspace]

I'm using Moviepy to make captioned videos on the python Lambda. ImageMagick is a dependency of Moveipy and comes preinstalled on Lambda, which would be great; however, when I try to create a frame using moviepy.TextClip, I get the following error:

MoviePy Error: creation of None failed because of the following error:
 
convert: not authorized `@/tmp/tmpafOCVo.txt' @ error/constitute.c/ReadImage/454.
convert: no images defined `PNG32:/tmp/tmphhE8gT.png' @ error/convert.c/ConvertImageCommand/3046.

This error can be due to the fact that ImageMagick is not installed on your computer, or (for Windows users) that you didn't specify the path to the ImageMagick binary in file conf.py, or.that the path you specified is incorrect: IOError

I googled forever and it looks like the problem is fallout from https://alas.aws.amazon.com/ALAS-2016-699.html. "This update contains an updated /etc/ImageMagick/policy.xml file that disables the EPHEMERAL, HTTPS, HTTP, URL, FTP, MVG, MSL, TEXT, and LABEL coders."

It also says, "if you experience any problems after the update, it may be necessary to manually adjust the policy.xml file to match your requirements." But I can't do that on Lambda.

StackOverflow folks suggest repackaging lambda and uploading it to Lambda yourself (http://stackoverflow.com/questions/37046586/aws-lambda-not-working-along-with-the-gm-module) but whenever I add ImageMagick to the vendor folder and try to deploy I get a broken pipe error from Botocore

  File "/usr/local/bin/chalice", line 9, in <module>
    load_entry_point('chalice==0.5.1', 'console_scripts', 'chalice')()
  File "/usr/local/lib/python2.7/site-packages/chalice/cli/__init__.py", line 298, in main
    return cli(obj={})
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/chalice/cli/__init__.py", line 177, in deploy
    d.deploy(config)
  File "/usr/local/lib/python2.7/site-packages/chalice/deployer.py", line 272, in deploy
    self._lambda_deploy.deploy(config)
  File "/usr/local/lib/python2.7/site-packages/chalice/deployer.py", line 665, in deploy
    self._update_lambda_function(config)
  File "/usr/local/lib/python2.7/site-packages/chalice/deployer.py", line 748, in _update_lambda_function
    zip_contents)
  File "/usr/local/lib/python2.7/site-packages/chalice/awsclient.py", line 86, in update_function_code
    FunctionName=function_name, ZipFile=zip_contents)
  File "/usr/local/lib/python2.7/site-packages/botocore/client.py", line 159, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python2.7/site-packages/botocore/client.py", line 483, in _make_api_call
    operation_model, request_dict)
  File "/usr/local/lib/python2.7/site-packages/botocore/endpoint.py", line 141, in make_request
    return self._send_request(request_dict, operation_model)
  File "/usr/local/lib/python2.7/site-packages/botocore/endpoint.py", line 170, in _send_request
    success_response, exception):
  File "/usr/local/lib/python2.7/site-packages/botocore/endpoint.py", line 249, in _needs_retry
    caught_exception=caught_exception, request_dict=request_dict)
  File "/usr/local/lib/python2.7/site-packages/botocore/hooks.py", line 227, in emit
    return self._emit(event_name, kwargs)
  File "/usr/local/lib/python2.7/site-packages/botocore/hooks.py", line 210, in _emit
    response = handler(**kwargs)
  File "/usr/local/lib/python2.7/site-packages/botocore/retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "/usr/local/lib/python2.7/site-packages/botocore/retryhandler.py", line 251, in __call__
    caught_exception)
  File "/usr/local/lib/python2.7/site-packages/botocore/retryhandler.py", line 277, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/usr/local/lib/python2.7/site-packages/botocore/retryhandler.py", line 317, in __call__
    caught_exception)
  File "/usr/local/lib/python2.7/site-packages/botocore/retryhandler.py", line 223, in __call__
    attempt_number, caught_exception)
  File "/usr/local/lib/python2.7/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
botocore.vendored.requests.exceptions.ConnectionError: ('Connection aborted.', error(32, 'Broken pipe'))

[jamesls]

The vendor/ changes are available in 0.6.0, which was just released today (though it's been in the master branch for a while). Would you mind upgrading to 0.6.0 and seeing if you're still having these issues?

[baileyspace]

Still getting the same error.

[jamesls]

Ok, I'll take a look and see if I can repro.

[baileyspace]

Any update on this?

[kyleknap]

I am able to reproduce this issue. What is happening is Lambda only allows for a 50 MB deployment package. So if the deployment package is over 50 MB, the service will try to send an error message back about the deployment package and abort the connection. For this case though, the aborted connection happens as chalice is still sending data before the error message is returned resulting in a general connection error.

Unfortunately, there is not really much we can do to make the deployment actually go through given Lambda controls the deployment size limit. While there may be optimizations chalice could make to decrease the deployment size, I think it is important at the very least to detect this error and raise a more informative error related to the deployment package being too large.