5

I've been trying to get OpenCV into an S3 bucket and then assign it to a lambda layer.

Theres very little about this online and what I have seen hasn't worked.

I've managed to use docker with the amazon linux environment, and followed this tutorial. https://aws.amazon.com/premiumsupport/knowledge-center/lambda-layer-simulated-docker/

I've added setuptools, wheel and opencv-python==4.4.0.42 to the requirements.txt file.

setuptools and wheel because of an earlier error where the recommendation was to include these as they need updating, even though I have updated them. But it works with them, so who knows.

Created the docker image which I've zipped and put in an S3 bucket.

I keep getting { "errorMessage": "Unable to import module 'lambda_function': libGL.so.1: cannot open shared object file: No such file or directory", "errorType": "Runtime.ImportModuleError" } when I run it though.

I can't seem to figure out what is wrong.

Any ideas?

Ryan
  • 59
  • 6

2 Answers2

11

You will need to add a bunch of dependencies to your layer. Below are the steps that I've used for opencv_python on lambda.

1. On local workstation (terminal window 1)

mkdir /tmp/mylayer && cd /tmp/mylayer

echo opencv-python==4.4.0.42 > ./requirements.txt

2. On local workstation (terminal window 2)


docker run -it -v /tmp/mylayer:/mylayer  lambci/lambda:build-python3.8 bash

The above command will put you into the docker container.

Inside the container:

cd /mylayer

pip install --no-deps -t python/lib/python3.8/site-packages/ -r requirements.txt

yum install -y mesa-libGL

cp -v /usr/lib64/libGL.so.1 /mylayer/python/lib/python3.8/site-packages/opencv_python.libs/
cp -v /usr/lib64/libGL.so.1.7.0 /mylayer/python/lib/python3.8/site-packages/opencv_python.libs/
cp -v /usr/lib64/libgthread-2.0.so.0 /mylayer/python/lib/python3.8/site-packages/opencv_python.libs/
cp -v /usr/lib64/libgthread-2.0.so.0 /mylayer/python/lib/python3.8/site-packages/opencv_python.libs/
cp -v /usr/lib64/libglib-2.0.so.0 /mylayer/python/lib/python3.8/site-packages/opencv_python.libs/
cp -v /usr/lib64/libGLX.so.0 /mylayer/python/lib/python3.8/site-packages/opencv_python.libs/
cp -v /usr/lib64/libX11.so.6 /mylayer/python/lib/python3.8/site-packages/opencv_python.libs/
cp -v /usr/lib64/libXext.so.6 /mylayer/python/lib/python3.8/site-packages/opencv_python.libs/
cp -v /usr/lib64/libGLdispatch.so.0 /mylayer/python/lib/python3.8/site-packages/opencv_python.libs/
cp -v /usr/lib64/libGLESv1_CM.so.1.2.0 /mylayer/python/lib/python3.8/site-packages/opencv_python.libs/
cp -v /usr/lib64/libGLX_mesa.so.0.0.0 /mylayer/python/lib/python3.8/site-packages/opencv_python.libs/
cp -v /usr/lib64/libGLESv2.so.2.1.0 /mylayer/python/lib/python3.8/site-packages/opencv_python.libs/
cp -v /usr/lib64/libxcb.so.1 /mylayer/python/lib/python3.8/site-packages/opencv_python.libs/
cp -v /usr/lib64/libXau.so.6 /mylayer/python/lib/python3.8/site-packages/opencv_python.libs/
cp -v /usr/lib64/libXau.so.6 /mylayer/python/lib/python3.8/site-packages/opencv_python.libs/
cp -v /lib64/libGLdispatch.so.0.0.0 /mylayer/python/lib/python3.8/site-packages/opencv_python.libs/

3. On local workstation again (terminal window 1)

Pack the python folder into mylayer.zip.

zip -r -9 mylayer.zip python

In AWS console

  1. Create lambda layer based on mylayer.zip in the AWS Console. Don't forget to specify Compatible runtimes to python3.8.

  2. Add AWS provide SciPy layer AWSLambda-Python38-SciPy1x and your own layer with cv2 into your function.

So you will have two layers in your function.

  1. Perform basic test of the layer in lambda using the following lambda function:
import cv2

def lambda_handler(event, context):    
    print(dir(csv))

The function executes correctly (partial printout shown).

slation3D', 'exp', 'extractChannel', 'fastAtan2', 'fastNlMeansDenoising', 'fastNlMeansDenoisingColored', 'fastNlMeansDenoisingColoredMulti', 'fastNlMeansDenoisingMulti', 'fillConvexPoly', 'fillPoly', 'filter2D', 'filterHomographyDecompByVisibleRefpoints', 'filterSpeckles', 'find4QuadCornerSubpix', 'findChessboardCorners', 'findChessboardCornersSB', 'findChessboardCornersSBWithMeta', 'findCirclesGrid', 'findContours', 'findEssentialMat', 'findFundamentalMat', 'findHomography', 'findNonZero', 'findTransformECC', 'fisheye', 'fitEllipse', 'fitEllipseAMS', 'fitEllipseDirect', 'fitLine', 'flann', 'flann_Index', 'flip', 'floodFill', 'gemm', 'getAffineTransform', 'getBuildInformation', 'getCPUFeaturesLine', 'getCPUTickCount', 'getDefaultNewCameraMatrix', 'getDerivKernels', 'getFontScaleFromHeight', 'getGaborKernel', 'getGaussianKernel', 'getHardwareFeatureName', 'getNumThreads', 'g
Marcin
  • 108,294
  • 7
  • 83
  • 138
  • 1
    Marcin, I love you!! You have no idea how many days I've been struggling with this. Just used this and it worked without any errors. Thank you! – Ryan Sep 23 '20 at 10:39
  • 1
    OMG you are my new best friend! – Paul Raupach Oct 20 '20 at 19:20
  • @PaulRaupach Glad my answer was helpful:-) – Marcin Oct 20 '20 at 21:35
  • 1
    The `libGLES*.so.*` files didn't get installed for me, but it seems to work without them. – Leopd Nov 03 '20 at 22:53
  • 1
    Thank you so much!! I spent days trying to figure this out!!! – An0n1m1ty Dec 08 '20 at 03:59
  • @An0n1m1ty Glad it worked out:-) – Marcin Dec 08 '20 at 04:00
  • 1
    I spent three hours on the three garbage tutorials at the top of the Google search rankings and this worked instantly. Thanks so much! You should turn this into a blog post so that it drowns out the other nonsense out there wasting countless hours of people's lives. – James Shapiro Dec 19 '20 at 03:54
  • @Marcin you're killing it lately! I have a similar issue and your solution wasn't able to solve it. Take a look if you have time! https://stackoverflow.com/questions/66448962/aws-lambda-unable-to-find-libgl-so-package-while-using-moderngl-glcontext – transposeglobal Mar 04 '21 at 00:11
  • I've followed these steps to create the layer which works fine, but when I use the layer in a sam-build the libGL.so.1 is still missing (i checked the zip and its definately there..) Also the sam build is copying most of the other.so files from opencv_pyhon.libs. Any ideas? – Litrico Mar 12 '21 at 13:59
  • @Litrico Difficult to say. You can make new question for that. Or can cook at lambda containers. They make installing dependencies for lambda easier and give you 10GB of space. – Marcin Mar 13 '21 at 08:51
  • Tbh I was planning on looking into the containers, but haven't read into it yet though, I'll create a new post as well on Monday, tnx for the response. – Litrico Mar 13 '21 at 09:40
  • 1
    Found out that the problem was that i still had opencv-python==4.4.0.42 in my lambda, this caused the layer dependencies to be overwritten and thus not used! – Litrico Mar 15 '21 at 13:24
1

Although this solution does work (adding dependency layers), this doesn't get around the hard size limit of 250MB. Luckily AWS have added support for Lamda's to run container images - see my answer here: How to increase the maximum size of the AWS lambda deployment package (RequestEntityTooLargeException)?

jonnocraig
  • 161
  • 1
  • 4