4

Running AWS lambda service packaged using Zappa.io

The service is running however, its not able to reach the S3 file due to ssl error Getting the below error while trying to access remote_env from an s3 bucket

[1592935276008] [DEBUG] 2020-06-23T18:01:16.8Z b8374974-f820-484a-bcc3-64a530712769 Exception received when sending HTTP request.
Traceback (most recent call last):
  File "/var/task/urllib3/util/ssl_.py", line 336, in ssl_wrap_socket
  context.load_verify_locations(ca_certs, ca_cert_dir)
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/runtime/botocore/httpsession.py", line 254, in send
  urllib_response = conn.urlopen(
  File "/var/task/urllib3/connectionpool.py", line 719, in urlopen
  retries = retries.increment(
  File "/var/task/urllib3/util/retry.py", line 376, in increment
  raise six.reraise(type(error), error, _stacktrace)
  File "/var/task/six.py", line 703, in reraise
  raise value
  File "/var/task/urllib3/connectionpool.py", line 665, in urlopen
  httplib_response = self._make_request(
  File "/var/task/urllib3/connectionpool.py", line 376, in _make_request
  self._validate_conn(conn)
  File "/var/task/urllib3/connectionpool.py", line 996, in _validate_conn
  conn.connect()
  File "/var/task/urllib3/connection.py", line 352, in connect
  self.sock = ssl_wrap_socket(
  File "/var/task/urllib3/util/ssl_.py", line 338, in ssl_wrap_socket
  raise SSLError(e)
urllib3.exceptions.SSLError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/runtime/botocore/endpoint.py", line 200, in _do_get_response
  http_response = self._send(request)
  File "/var/runtime/botocore/endpoint.py", line 244, in _send
  return self.http_session.send(request)
  File "/var/runtime/botocore/httpsession.py", line 281, in send
  raise SSLError(endpoint_url=request.url, error=e)
botocore.exceptions.SSLError: SSL validation failed for .......  [Errno 2] No such file or directory

My Environment

Zappa version used: 0.51.0 Operating System and Python version: Ubuntu , Python 3.8 Output of pip freeze

appdirs==1.4.3
argcomplete==1.11.1
boto3==1.14.8
botocore==1.17.8
CacheControl==0.12.6
certifi==2019.11.28
cffi==1.14.0
cfn-flip==1.2.3
chardet==3.0.4
click==7.1.2
colorama==0.4.3
contextlib2==0.6.0
cryptography==2.9.2
distlib==0.3.0
distro==1.4.0
docutils==0.15.2
durationpy==0.5
Flask==1.1.2
Flask-Cors==3.0.8
future==0.18.2
h11==0.9.0
hjson==3.0.1
html5lib==1.0.1
httptools==0.1.1
idna==2.8
ipaddr==2.2.0
itsdangerous==1.1.0
Jinja2==2.11.2
jmespath==0.10.0
kappa==0.6.0
lockfile==0.12.2
mangum==0.9.2
MarkupSafe==1.1.1
msgpack==0.6.2
packaging==20.3
pep517==0.8.2
pip-tools==5.2.1
placebo==0.9.0
progress==1.5
pycparser==2.20
pydantic==1.5.1
PyMySQL==0.9.3
pyOpenSSL==19.1.0
pyparsing==2.4.6
python-dateutil==2.6.1
python-slugify==4.0.0
pytoml==0.1.21
PyYAML==5.3.1
requests==2.22.0
retrying==1.3.3
s3transfer==0.3.3
six==1.14.0
starlette==0.13.4
text-unidecode==1.3
toml==0.10.1
tqdm==4.46.1
troposphere==2.6.1
typing-extensions==3.7.4.2
urllib3==1.25.8
uvloop==0.14.0
webencodings==0.5.1
websockets==8.1
Werkzeug==0.16.1
wsgi-request-logger==0.4.6
zappa==0.51.0

My zappa_settings.json:

{
    "dev": {
        "app_function": "main.app",
        "aws_region": "us-west-2",
        "profile_name": "default",
        "project_name": "d3c",
        "runtime": "python3.8",
        "keep_warm":false,
        "cors": true,
        "s3_bucket": "my-lambda-deployables",
        "remote_env":"<my remote s3 file>"
    }
}

I have confirmed that my S3 file is accessible from my local ubuntu machine however does not work on aws

virtuvious
  • 1,954
  • 2
  • 15
  • 15

2 Answers2

3

This seems to be related to an open issue open issue on Zappa

I had the same issue my Zappa deployment, I tried all possible options but nothing was working, But after trying different suggestions the following steps worked for me

  1. I copied python3.8/site-packages/botocore/cacert.pem to my lambda folder
  2. I Set the "REQUESTS_CA_BUNDLE" environment variable to /var/task/cacert.pem

/var/task is where AWS Lambda extracts your zipped up code to.

How to set environment variables in Zappa

  1. I updated my Zappa function and everything worked fine
jaywonder20
  • 31
  • 2
  • 7
  • Welcome and thanks for your answer, that really solved (as a workaround at least) an issue I was having. Could you please add a link to where you found out that the Lambda code is deployed to `/var/task`? – elactic Sep 04 '20 at 09:03
  • 1
    @elactic I found out where lambda code is deployed when I read an article on thetestlabs.io, Even though the article was on perceived security flaws of AWS The article can be found here https://thetestlabs.io/code/exploiting-common-serverless-security-flaws-in-aws/ – jaywonder20 Sep 04 '20 at 23:32
  • @jaywonder20 I investigated a bit more, and it looks like its not necessary to copy the cacert.pem file. The following zappa envron setting worked for me without moving the cert (by inspecting where the botocore lib is installed on lambda): `"REQUESTS_CA_BUNDLE": "/var/task/botocore/cacert.pem"` – yellowcap Feb 18 '21 at 01:35
-1

fixed this by adding the cert path to environment (python)

os.environ['REQUESTS_CA_BUNDLE'] = os.path.join('/etc/ssl/certs/','ca-certificates.crt')

Edit: Sorry the issue was not really fixed with the above code, but found a hack work around by adding verify=False for all ssl requests

boto3.client('s3', verify=False)
virtuvious
  • 1,954
  • 2
  • 15
  • 15
  • 1
    This only works, though, if you yourself use the boto client, doesn't it? If you are using Zappa's `@task`, for instance, you have no control over how it makes that call. In general, however, suggesting to disable SSL verification is not ideal IMO. – elactic Sep 04 '20 at 09:16
  • @elactic you are right....thats why i called it a hack work around. – virtuvious Sep 09 '20 at 15:28
  • 1
    Please don't recommend that people stop verifying certs. These security measures exist for a reason. Instead, please try to solve the problem. – james.garriss Mar 02 '21 at 13:33