Building a Lambda Python Package with Docker

Not too long ago I was working on a Lambda function that added watermarks to images uploaded to S3. For the life of me, I couldn’t get my function to work. Time and time again of troubleshooting my python script, I would run into the same error, Lambda wouldn’t recognize the Pillow package installed. I couldn’t figure out why. Eventually, I found out why…

ZIP FILES NEED TO BE ZIPED IN A LINUX ENVIRONMENT

Note: If you’re unfamiliar with using Lambda or how to import your python script with dependencies, check this out for a refresher.

I am working from a MacOS machine, so I could create a zip from a number of different options:
1. Vagrant
2. Linux VM in VirtualBox
3. Docker
I rather stick with Docker in this case since it’s a lighter footprint on my machine and I can deploy it whenever I want.

Before we begin, we need to know which versions and runtimes of Python Lambda will support. Click on the image for the source link.

From the image above, we now know that Python runtime uses Amazon Linux and Amazon Linux 2 which was created from CentOS (RedHat/RHEL).

This matters because with Lambda, we might be using an external package in our Python Application, and if we are, it HAS to be compatible with the Amazon Linux distro. Take another look at the official container image with Docker.

Creating Your Docker Container

We now need to setup our Python 3.7 environment using the AWS Docker Image from above. Start out by creating a Dockerfile in your terminal or editor of preference.

FROM amazonlinux:2018.03
RUN yum update -y
RUN yum install -y \
    gcc \
    openssl-devel \
    zlib-devel \
    libffi-devel \
    wget && \
    yum -y clean all

WORKDIR usr/src
# Install Python 3.7
RUN wget https://www.python.org/ftp/python/3.7.4/Python-3.7.4.tgz
RUN tar xzf Python-3.7.4.tgz
RUN cd Python-3.7.4 ; ./configure --enable-optimizations; make altinstall
RUN rm Python-3.7.4.tgz
RUN rm -rf Python-3.7.4
RUN python3.7 -V
# Install pip
RUN wget https://bootstrap.pypa.io/get-pip.py
RUN python3.7 get-pip.py
RUN rm get-pip.py
RUN pip -V

Now we can build our container image from this Dockerfile. You can give it any name, but I’m gonna keep it simple. Run the command below in Terminal.
Note: It’s going to take a minute to finish

$ docker build -t lambda-linux-3.7 .

For more ease of use and organization, I’d recommend creating a new directory to house your packages and script.

$ mkdir my_lambda
$ cd my_lambda

From here add the desired python script into your directory then run the Docker image. The way AWS Lambda works, you’ll need to have the packages associated with the script packaged together in a zip file. In order to have everything together, let’s run the image and ssh into bash.

docker run -v $(pwd):/my_project -ti lambda-linux-3.7

The above command has you sshing into the now running container image. From inside the container, begin your installation of packages:

bash-4.2# pip install paramiko

At this point I’m just installing an SSH tool for Python called Paramiko. Great tool for some automation, but that’s for a different post.

When you’re done installing your desired packages, you can exit the container and find that all of those associated packages have been installed in your my_lambda directory.

Once you’re done, ZIP it.

zip -r my_lambda.zip *

From here, log into your AWS console and navigate to Lambda. Upload your new zip file and enjoy your serverless application.