til: awslambda_asgi-mangum.md
This data as json
| path | topic | title | url | body | html | shot | created | created_utc | updated | updated_utc | shot_hash | slug |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| awslambda_asgi-mangum.md | awslambda | Deploying Python web apps as AWS Lambda functions | https://github.com/simonw/til/blob/main/awslambda/asgi-mangum.md | I've been wanting to figure out how to do this for years. Today I finally put all of the pieces together for it. [AWS Lambda](https://aws.amazon.com/lambda/) can host functions written in Python. These are "scale to zero" - my favourite definition of serverless! - which means you only pay for the traffic that they serve. A project with no traffic costs nothing to run. You used to have to jump through a whole bunch of extra hoops to get a working URL that triggered those functions, but in April 2022 they [released Lambda Function URLs](https://aws.amazon.com/blogs/aws/announcing-aws-lambda-function-urls-built-in-https-endpoints-for-single-function-microservices/) and dramatically simplified that process. There are still a lot of steps involved though. Here's how to deploy a Python web application as a Lambda function. ## Set up the AWS CLI tool I did this so long ago I can't remember how. You need an AWS account and you need to have the [AWS CLI tool](https://aws.amazon.com/cli/) installed and configured. The `aws --version` should return a version number of `1.22.90` or higher, as [that's when they added function URL support](https://github.com/simonw/help-scraper/commit/d217b9d7f44a1200d0582d02aeccf27e006b8b78). I found I had too old a version of the tool. I ended up figuring out this as the way to upgrade it: ```bash head -n 1 $(which aws) ``` Output: ``` #!/usr/local/opt/python@3.9/bin/python3.9 ``` This showed me the location of the Python environment that contained the tool. I could then edit that path to upgrade it like so: ```bash /usr/local/opt/python@3.9/bin/pip3 install -U awscli ``` ## Create a Python handler function This is "hello world" as a Python handler function. Put it in `lambda_function.py`: ```python def lambda_handler(event, context): return { "statusCode": 200, "headers": { "Content-Type": "text/html" }, "body": "<h1>Hello World from Python</h1>" } ``` ## Add that to a zip file This is the part of the process that I found most unintuitive. Lambda functions are deployed as zip files. The zip file needs to contain both the Python code AND all of its dependencies - more on that to come. Our first function doesn't have any dependencies, which makes things a lot easier. Here's how to turn it into a zip file ready to be deployed: ```bash zip function.zip lambda_function.py ``` ## Create a role with a policy You only have to do this the first time you deploy a Lambda function. You need an IAM role that you can use for the other steps. This command creates a role called `lambda-ex`: ``` aws iam create-role \ --role-name lambda-ex \ --assume-role-policy-document '{ "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": { "Service": "lambda.amazonaws.com" }, "Action": "sts:AssumeRole"} ]}' ``` Then you have to run this. I don't know why this can't be handled as part of the `create-role` command, but it's necessary: ``` aws iam attach-role-policy \ --role-name lambda-ex \ --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole ``` ## Find your AWS account ID You need to know your AWS account ID for the next step. You can find it by running this command: ```bash aws sts get-caller-identity \ --query "Account" --output text ``` I assigned it to an environment variable so I could use it later like this: ```bash export AWS_ACCOUNT_ID=$( aws sts get-caller-identity \ --query "Account" --output text ) ``` Run this to confirm that worked: ```bash echo $AWS_ACCOUNT_ID ``` ## Deploy that function Now we can deploy the zip file as a new Lambda function! Pick a unique function name - I picked `lambda-python-hello-world`. Then run the following: ```bash aws lambda create-function \ --function-name lambda-python-hello-world \ --zip-file fileb://function.zip \ --runtime python3.9 \ --handler "lambda_function.lambda_handler" \ --role "arn:aws:iam::${AWS_ACCOUNT_ID}:role/lambda-ex" ``` We're telling it to deploy our `function.zip` file using the `python3.9` runtime. We list `lambda_function.lambda_handler` as the handler because our Python file was called `lambda_function.py` and the function was called `lambda_handler`. If all goes well you should get back a response from that command that looks something like this: ```json { "FunctionName": "lambda-python-hello-world", "FunctionArn": "arn:aws:lambda:us-east-1:462092780466:function:lambda-python-hello-world", "Runtime": "python3.9", "Role": "arn:aws:iam::462092780466:role/lambda-ex", "Handler": "lambda_function.lambda_handler", "CodeSize": 332, "Description": "", "Timeout": 3, "MemorySize": 128, "LastModified": "2022-09-19T02:27:18.213+0000", "CodeSha256": "Y1nCZLN6KvU9vUmhHAgcAkYfvgu6uBhmdGVprq8c97Y=", "Version": "$LATEST", "TracingConfig": { "Mode": "PassThrough" }, "RevisionId": "316481f5-7934-4e54-914f-6b075bb7d9dd", "State": "Pending", "StateReason": "The function is being created.", "StateReasonCode": "Creating", "PackageType": "Zip", "Architectures": [ "x86_64" ], "EphemeralStorage": { "Size": 512 } } ``` ## Grant permission for it to be executed This magic command is also necessary for everything to work: ```bash aws lambda add-permission \ --function-name lambda-python-hello-world \ --action lambda:InvokeFunctionUrl \ --principal "*" \ --function-url-auth-type "NONE" \ --statement-id url ``` ## Give it a Function URL We need a URL that we can access in a browser to trigger our function. Here's how to add a new Function URL to our deployed function: ``` aws lambda create-function-url-config \ --function-name lambda-python-hello-world \ --auth-type NONE ``` That `--auth-type NONE` means anyone on the internet will be able to trigger the function by visiting the URL. This should return something like the following: ```json { "FunctionUrl": "https://m2jatdfy4bulhvsfcrfc6sfw2i0bjfpx.lambda-url.us-east-1.on.aws/", "FunctionArn": "arn:aws:lambda:us-east-1:462092780466:function:lambda-python-hello-world", "AuthType": "NONE", "CreationTime": "2022-09-19T02:27:48.356967Z" } ``` And sure enough, https://m2jatdfy4bulhvsfcrfc6sfw2i0bjfpx.lambda-url.us-east-1.on.aws/ now returns "Hello World from Python". ## Updating the function Having deployed the function, updating it is pleasantly easy. You create a new `function.zip` file - which I do like this: ```bash rm -f function.zip # Delete if it exists zip function.zip lambda_function.py ``` And then deploy the update like so: ```bash aws lambda update-function-code \ --function-name lambda-python-hello-world \ --zip-file fileb://function.zip ``` ## Adding pure Python dependencies Adding dependencies to the project was by far the most confusing aspect of this whole process. Eventually I found a [good way to do it](https://github.com/pixegami/fastapi-tutorial/blob/4ec9247faf53e4c399ea18a4ac27c0e85a137955/README.md#deploying-fastapi-to-aws-lambda) thanks to the example code published to accompany [this YouTube video](https://www.youtube.com/watch?v=RGIM4JfsSk0) by Pixegami. The trick is to include ALL of your dependencies _in the root of your zip file_. Forget about `requirements.txt` and suchlike - you need to install copies of the actual dependencies themselves. Here's the recipe that works for me. First, create a `requirements.txt` file listing your dependencies: ``` cowsay ``` Now use the `pip install -t` command to install those requirements into a specific directory - I use `lib`: ```bash python3 -m pip install -t lib -r requirements.txt ``` Run `ls -lah lib` to confirm that the files are in there. ``` ls lib | cat ``` ``` bin cowsay cowsay-5.0-py3.10.egg-info ``` Now use this recipe to add everything in `lib` to the root of your zip file: ```bash (cd lib; zip ../function.zip -r .) ``` You can run this command to see the list of files in the zip: ```bash unzip -l function.zip ``` Let's update `lambda_function.py` to demonstrate the `cowsay` library: ```python import cowsay def lambda_handler(event, context): return { "statusCode": 200, "headers": { "Content-Type": "text/plain" }, "body": cowsay.get_output_string("pig", "Hello world, I am a pig") } ``` Add that updated `lambda_function.py` to the zip file again: ```bash zip function.zip lambda_function.py ``` Deploy the update: ``` aws lambda update-function-code \ --function-name lambda-python-hello-world \ --zip-file fileb://function.zip ``` Hit refresh on the URL from earlier and you should see: ``` _______________________ | Hello world, I am a pig | ======================= \ \ \ \ ,. (_|,. ,' /, )_______ _ __j o``-' `.'-)' (") \' `-j | `-._( / |_\ |--^. / /_]'|_| /_)_/ /_]' /_]' ``` ## Advanced Python dependencies The above recipe works fine for dependencies that are written only in Python. Where things get more complicated is when you want to use a dependency that includes native code. I use a Mac. If I run `pip install -t lib -r requirements.txt` I'll get the Mac versions of those dependencies. But AWS Lambda functions run on Amazon Linux. So we need to include version of our packages that are built for that platform in our zip file. I first had to do this because I realized the `python3.9` Lambda runtime includes a truly ancient version of SQLite - version 3.7.17 [from 2013-05-20](http://www.sqlite.org/releaselog/3_7_17.html). The [pysqlite3-binary](https://pypi.org/project/pysqlite3-binary/) package provides a much more recent SQLite, and [Datasette](https://datasette.io/) uses that automatically if it's installed. I figured the best way to do this would be to run the `pip install` command inside an Amazon Linux Docker container. After much head scratching, I came up with this recipe for doing that: ```bash docker run -t -v $(pwd):/mnt \ public.ecr.aws/sam/build-python3.9:latest \ /bin/sh -c "pip install -r /mnt/requirements.txt -t /mnt/lib" ``` - The `-v $(pwd):/mnt` flag mounts the current directory as `/mnt` inside the container. - The `public.ecr.aws/sam/build-python3.9:latest` image is the official AWS Lambda Python 3.9 image. - `/bin/sh -c "pip install -r /mnt/requirements.txt -t /mnt/lib"` runs `pip install` inside the container, but ensures the files are written to `/mnt/lib` which is the `lib` folder in our current directory. This recipe works! The result is a `lib/` folder full of Amazon Linux Python packages, ready to be zipped up and deployed. ## Running an ASGI application I want to deploy [Datasette](https://datasette.io/). Datasette is an [ASGI application](https://simonwillison.net/2019/Jun/23/datasette-asgi/). But... AWS Lambda functions have their own weird interface to HTTP - the `event` and `context` parameters shown above. [Mangum](https://github.com/jordaneremieff/mangum) is a well regarded library that bridges the gap between the two. Here's how I got Datasette and Mangum working. It was surprisingly straight-forward! I added the following to my `requirements.txt` file: ``` datasette pysqlite3-binary mangum ``` I deleted my `lib` folder: ``` rm -rf lib ``` Then I ran the magic incantation from above: ```bash docker run -t -v $(pwd):/mnt \ public.ecr.aws/sam/build-python3.9:latest \ /bin/sh -c "pip install -r /mnt/requirements.txt -t /mnt/lib" ``` I added the dependencies to a new `function.zip` file: ```bash rm -rf function.zip (cd lib; zip ../function.zip -r .) ``` Then I added the following to `lambda_function.py`: ```python import asyncio from datasette.app import Datasette import mangum ds = Datasette(["fixtures.db"]) # Handler wraps the Datasette ASGI app with Mangum: lambda_handler = mangum.Mangum(ds.app()) ``` I added that to the zip: ``` zip function.zip lambda_function.py ``` Finally, I grabbed a copy of the standard Datasette `fixtures.db` database file and added that to the zip as well: ```bash wget https://latest.datasette.io/fixtures.db zip function.zip fixtures.db ``` The finished `function.zip` is 7.1MB. Time to deploy it: ``` aws lambda update-function-code \ --function-name lambda-python-hello-world \ --zip-file fileb://function.zip ``` This did the trick! I now had a Datasette instance running on Lambda: https://fnkvspusjrl5dxytaxnuwidxem0hverw.lambda-url.us-east-1.on.aws/ The default Lambda configuration only provides 128MB of RAM, and I was getting occasional timeout errors. Bumping that up to 256MB fixed the problem: ```bash aws lambda update-function-configuration \ --function-name lambda-python-hello-world \ --memory-size 256 ``` ## This should work for Starlette and FastAPI too Mangum works with ASGI apps, so any app built using [Starlette](https://www.starlette.io/) or [FastAPI](https://fastapi.tiangolo.com/) should work exactly the same way. ## Pretty URLs One thing I haven't figured out yet is how to assign a custom domain name to a Lambda function. I understand doing that involves several other AWS services, potentially API Gateway, CloudFront and Route53. I'll update this once I figure those out. | <p>I've been wanting to figure out how to do this for years. Today I finally put all of the pieces together for it.</p> <p><a href="https://aws.amazon.com/lambda/" rel="nofollow">AWS Lambda</a> can host functions written in Python. These are "scale to zero" - my favourite definition of serverless! - which means you only pay for the traffic that they serve. A project with no traffic costs nothing to run.</p> <p>You used to have to jump through a whole bunch of extra hoops to get a working URL that triggered those functions, but in April 2022 they <a href="https://aws.amazon.com/blogs/aws/announcing-aws-lambda-function-urls-built-in-https-endpoints-for-single-function-microservices/" rel="nofollow">released Lambda Function URLs</a> and dramatically simplified that process.</p> <p>There are still a lot of steps involved though. Here's how to deploy a Python web application as a Lambda function.</p> <h2><a id="user-content-set-up-the-aws-cli-tool" class="anchor" aria-hidden="true" href="#set-up-the-aws-cli-tool"><span aria-hidden="true" class="octicon octicon-link"></span></a>Set up the AWS CLI tool</h2> <p>I did this so long ago I can't remember how. You need an AWS account and you need to have the <a href="https://aws.amazon.com/cli/" rel="nofollow">AWS CLI tool</a> installed and configured.</p> <p>The <code>aws --version</code> should return a version number of <code>1.22.90</code> or higher, as <a href="https://github.com/simonw/help-scraper/commit/d217b9d7f44a1200d0582d02aeccf27e006b8b78">that's when they added function URL support</a>.</p> <p>I found I had too old a version of the tool. I ended up figuring out this as the way to upgrade it:</p> <div class="highlight highlight-source-shell"><pre>head -n 1 <span class="pl-s"><span class="pl-pds">$(</span>which aws<span class="pl-pds">)</span></span></pre></div> <p>Output:</p> <pre><code>#!/usr/local/opt/python@3.9/bin/python3.9 </code></pre> <p>This showed me the location of the Python environment that contained the tool. I could then edit that path to upgrade it like so:</p> <div class="highlight highlight-source-shell"><pre>/usr/local/opt/python@3.9/bin/pip3 install -U awscli</pre></div> <h2><a id="user-content-create-a-python-handler-function" class="anchor" aria-hidden="true" href="#create-a-python-handler-function"><span aria-hidden="true" class="octicon octicon-link"></span></a>Create a Python handler function</h2> <p>This is "hello world" as a Python handler function. Put it in <code>lambda_function.py</code>:</p> <div class="highlight highlight-source-python"><pre><span class="pl-k">def</span> <span class="pl-en">lambda_handler</span>(<span class="pl-s1">event</span>, <span class="pl-s1">context</span>): <span class="pl-k">return</span> { <span class="pl-s">"statusCode"</span>: <span class="pl-c1">200</span>, <span class="pl-s">"headers"</span>: { <span class="pl-s">"Content-Type"</span>: <span class="pl-s">"text/html"</span> }, <span class="pl-s">"body"</span>: <span class="pl-s">"<h1>Hello World from Python</h1>"</span> }</pre></div> <h2><a id="user-content-add-that-to-a-zip-file" class="anchor" aria-hidden="true" href="#add-that-to-a-zip-file"><span aria-hidden="true" class="octicon octicon-link"></span></a>Add that to a zip file</h2> <p>This is the part of the process that I found most unintuitive. Lambda functions are deployed as zip files. The zip file needs to contain both the Python code AND all of its dependencies - more on that to come.</p> <p>Our first function doesn't have any dependencies, which makes things a lot easier. Here's how to turn it into a zip file ready to be deployed:</p> <div class="highlight highlight-source-shell"><pre>zip function.zip lambda_function.py</pre></div> <h2><a id="user-content-create-a-role-with-a-policy" class="anchor" aria-hidden="true" href="#create-a-role-with-a-policy"><span aria-hidden="true" class="octicon octicon-link"></span></a>Create a role with a policy</h2> <p>You only have to do this the first time you deploy a Lambda function. You need an IAM role that you can use for the other steps.</p> <p>This command creates a role called <code>lambda-ex</code>:</p> <pre><code>aws iam create-role \ --role-name lambda-ex \ --assume-role-policy-document '{ "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": { "Service": "lambda.amazonaws.com" }, "Action": "sts:AssumeRole"} ]}' </code></pre> <p>Then you have to run this. I don't know why this can't be handled as part of the <code>create-role</code> command, but it's necessary:</p> <pre><code>aws iam attach-role-policy \ --role-name lambda-ex \ --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole </code></pre> <h2><a id="user-content-find-your-aws-account-id" class="anchor" aria-hidden="true" href="#find-your-aws-account-id"><span aria-hidden="true" class="octicon octicon-link"></span></a>Find your AWS account ID</h2> <p>You need to know your AWS account ID for the next step.</p> <p>You can find it by running this command:</p> <div class="highlight highlight-source-shell"><pre>aws sts get-caller-identity \ --query <span class="pl-s"><span class="pl-pds">"</span>Account<span class="pl-pds">"</span></span> --output text</pre></div> <p>I assigned it to an environment variable so I could use it later like this:</p> <div class="highlight highlight-source-shell"><pre><span class="pl-k">export</span> AWS_ACCOUNT_ID=<span class="pl-s"><span class="pl-pds">$(</span></span> <span class="pl-s"> aws sts get-caller-identity \</span> <span class="pl-s"> --query <span class="pl-s"><span class="pl-pds">"</span>Account<span class="pl-pds">"</span></span> --output text</span> <span class="pl-s"><span class="pl-pds">)</span></span></pre></div> <p>Run this to confirm that worked:</p> <div class="highlight highlight-source-shell"><pre><span class="pl-c1">echo</span> <span class="pl-smi">$AWS_ACCOUNT_ID</span></pre></div> <h2><a id="user-content-deploy-that-function" class="anchor" aria-hidden="true" href="#deploy-that-function"><span aria-hidden="true" class="octicon octicon-link"></span></a>Deploy that function</h2> <p>Now we can deploy the zip file as a new Lambda function!</p> <p>Pick a unique function name - I picked <code>lambda-python-hello-world</code>.</p> <p>Then run the following:</p> <div class="highlight highlight-source-shell"><pre>aws lambda create-function \ --function-name lambda-python-hello-world \ --zip-file fileb://function.zip \ --runtime python3.9 \ --handler <span class="pl-s"><span class="pl-pds">"</span>lambda_function.lambda_handler<span class="pl-pds">"</span></span> \ --role <span class="pl-s"><span class="pl-pds">"</span>arn:aws:iam::<span class="pl-smi">${AWS_ACCOUNT_ID}</span>:role/lambda-ex<span class="pl-pds">"</span></span></pre></div> <p>We're telling it to deploy our <code>function.zip</code> file using the <code>python3.9</code> runtime.</p> <p>We list <code>lambda_function.lambda_handler</code> as the handler because our Python file was called <code>lambda_function.py</code> and the function was called <code>lambda_handler</code>.</p> <p>If all goes well you should get back a response from that command that looks something like this:</p> <div class="highlight highlight-source-json"><pre>{ <span class="pl-ent">"FunctionName"</span>: <span class="pl-s"><span class="pl-pds">"</span>lambda-python-hello-world<span class="pl-pds">"</span></span>, <span class="pl-ent">"FunctionArn"</span>: <span class="pl-s"><span class="pl-pds">"</span>arn:aws:lambda:us-east-1:462092780466:function:lambda-python-hello-world<span class="pl-pds">"</span></span>, <span class="pl-ent">"Runtime"</span>: <span class="pl-s"><span class="pl-pds">"</span>python3.9<span class="pl-pds">"</span></span>, <span class="pl-ent">"Role"</span>: <span class="pl-s"><span class="pl-pds">"</span>arn:aws:iam::462092780466:role/lambda-ex<span class="pl-pds">"</span></span>, <span class="pl-ent">"Handler"</span>: <span class="pl-s"><span class="pl-pds">"</span>lambda_function.lambda_handler<span class="pl-pds">"</span></span>, <span class="pl-ent">"CodeSize"</span>: <span class="pl-c1">332</span>, <span class="pl-ent">"Description"</span>: <span class="pl-s"><span class="pl-pds">"</span><span class="pl-pds">"</span></span>, <span class="pl-ent">"Timeout"</span>: <span class="pl-c1">3</span>, <span class="pl-ent">"MemorySize"</span>: <span class="pl-c1">128</span>, <span class="pl-ent">"LastModified"</span>: <span class="pl-s"><span class="pl-pds">"</span>2022-09-19T02:27:18.213+0000<span class="pl-pds">"</span></span>, <span class="pl-ent">"CodeSha256"</span>: <span class="pl-s"><span class="pl-pds">"</span>Y1nCZLN6KvU9vUmhHAgcAkYfvgu6uBhmdGVprq8c97Y=<span class="pl-pds">"</span></span>, <span class="pl-ent">"Version"</span>: <span class="pl-s"><span class="pl-pds">"</span>$LATEST<span class="pl-pds">"</span></span>, <span class="pl-ent">"TracingConfig"</span>: { <span class="pl-ent">"Mode"</span>: <span class="pl-s"><span class="pl-pds">"</span>PassThrough<span class="pl-pds">"</span></span> }, <span class="pl-ent">"RevisionId"</span>: <span class="pl-s"><span class="pl-pds">"</span>316481f5-7934-4e54-914f-6b075bb7d9dd<span class="pl-pds">"</span></span>, <span class="pl-ent">"State"</span>: <span class="pl-s"><span class="pl-pds">"</span>Pending<span class="pl-pds">"</span></span>, <span class="pl-ent">"StateReason"</span>: <span class="pl-s"><span class="pl-pds">"</span>The function is being created.<span class="pl-pds">"</span></span>, <span class="pl-ent">"StateReasonCode"</span>: <span class="pl-s"><span class="pl-pds">"</span>Creating<span class="pl-pds">"</span></span>, <span class="pl-ent">"PackageType"</span>: <span class="pl-s"><span class="pl-pds">"</span>Zip<span class="pl-pds">"</span></span>, <span class="pl-ent">"Architectures"</span>: [ <span class="pl-s"><span class="pl-pds">"</span>x86_64<span class="pl-pds">"</span></span> ], <span class="pl-ent">"EphemeralStorage"</span>: { <span class="pl-ent">"Size"</span>: <span class="pl-c1">512</span> } }</pre></div> <h2><a id="user-content-grant-permission-for-it-to-be-executed" class="anchor" aria-hidden="true" href="#grant-permission-for-it-to-be-executed"><span aria-hidden="true" class="octicon octicon-link"></span></a>Grant permission for it to be executed</h2> <p>This magic command is also necessary for everything to work:</p> <div class="highlight highlight-source-shell"><pre>aws lambda add-permission \ --function-name lambda-python-hello-world \ --action lambda:InvokeFunctionUrl \ --principal <span class="pl-s"><span class="pl-pds">"</span>*<span class="pl-pds">"</span></span> \ --function-url-auth-type <span class="pl-s"><span class="pl-pds">"</span>NONE<span class="pl-pds">"</span></span> \ --statement-id url</pre></div> <h2><a id="user-content-give-it-a-function-url" class="anchor" aria-hidden="true" href="#give-it-a-function-url"><span aria-hidden="true" class="octicon octicon-link"></span></a>Give it a Function URL</h2> <p>We need a URL that we can access in a browser to trigger our function.</p> <p>Here's how to add a new Function URL to our deployed function:</p> <pre><code>aws lambda create-function-url-config \ --function-name lambda-python-hello-world \ --auth-type NONE </code></pre> <p>That <code>--auth-type NONE</code> means anyone on the internet will be able to trigger the function by visiting the URL.</p> <p>This should return something like the following:</p> <div class="highlight highlight-source-json"><pre>{ <span class="pl-ent">"FunctionUrl"</span>: <span class="pl-s"><span class="pl-pds">"</span>https://m2jatdfy4bulhvsfcrfc6sfw2i0bjfpx.lambda-url.us-east-1.on.aws/<span class="pl-pds">"</span></span>, <span class="pl-ent">"FunctionArn"</span>: <span class="pl-s"><span class="pl-pds">"</span>arn:aws:lambda:us-east-1:462092780466:function:lambda-python-hello-world<span class="pl-pds">"</span></span>, <span class="pl-ent">"AuthType"</span>: <span class="pl-s"><span class="pl-pds">"</span>NONE<span class="pl-pds">"</span></span>, <span class="pl-ent">"CreationTime"</span>: <span class="pl-s"><span class="pl-pds">"</span>2022-09-19T02:27:48.356967Z<span class="pl-pds">"</span></span> }</pre></div> <p>And sure enough, <a href="https://m2jatdfy4bulhvsfcrfc6sfw2i0bjfpx.lambda-url.us-east-1.on.aws/" rel="nofollow">https://m2jatdfy4bulhvsfcrfc6sfw2i0bjfpx.lambda-url.us-east-1.on.aws/</a> now returns "Hello World from Python".</p> <h2><a id="user-content-updating-the-function" class="anchor" aria-hidden="true" href="#updating-the-function"><span aria-hidden="true" class="octicon octicon-link"></span></a>Updating the function</h2> <p>Having deployed the function, updating it is pleasantly easy.</p> <p>You create a new <code>function.zip</code> file - which I do like this:</p> <div class="highlight highlight-source-shell"><pre>rm -f function.zip <span class="pl-c"><span class="pl-c">#</span> Delete if it exists</span> zip function.zip lambda_function.py </pre></div> <p>And then deploy the update like so:</p> <div class="highlight highlight-source-shell"><pre>aws lambda update-function-code \ --function-name lambda-python-hello-world \ --zip-file fileb://function.zip</pre></div> <h2><a id="user-content-adding-pure-python-dependencies" class="anchor" aria-hidden="true" href="#adding-pure-python-dependencies"><span aria-hidden="true" class="octicon octicon-link"></span></a>Adding pure Python dependencies</h2> <p>Adding dependencies to the project was by far the most confusing aspect of this whole process.</p> <p>Eventually I found a <a href="https://github.com/pixegami/fastapi-tutorial/blob/4ec9247faf53e4c399ea18a4ac27c0e85a137955/README.md#deploying-fastapi-to-aws-lambda">good way to do it</a> thanks to the example code published to accompany <a href="https://www.youtube.com/watch?v=RGIM4JfsSk0" rel="nofollow">this YouTube video</a> by Pixegami.</p> <p>The trick is to include ALL of your dependencies <em>in the root of your zip file</em>.</p> <p>Forget about <code>requirements.txt</code> and suchlike - you need to install copies of the actual dependencies themselves.</p> <p>Here's the recipe that works for me. First, create a <code>requirements.txt</code> file listing your dependencies:</p> <pre><code>cowsay </code></pre> <p>Now use the <code>pip install -t</code> command to install those requirements into a specific directory - I use <code>lib</code>:</p> <div class="highlight highlight-source-shell"><pre>python3 -m pip install -t lib -r requirements.txt</pre></div> <p>Run <code>ls -lah lib</code> to confirm that the files are in there.</p> <pre><code>ls lib | cat </code></pre> <pre><code>bin cowsay cowsay-5.0-py3.10.egg-info </code></pre> <p>Now use this recipe to add everything in <code>lib</code> to the root of your zip file:</p> <div class="highlight highlight-source-shell"><pre>(cd lib<span class="pl-k">;</span> zip ../function.zip -r .)</pre></div> <p>You can run this command to see the list of files in the zip:</p> <div class="highlight highlight-source-shell"><pre>unzip -l function.zip</pre></div> <p>Let's update <code>lambda_function.py</code> to demonstrate the <code>cowsay</code> library:</p> <div class="highlight highlight-source-python"><pre><span class="pl-k">import</span> <span class="pl-s1">cowsay</span> <span class="pl-k">def</span> <span class="pl-en">lambda_handler</span>(<span class="pl-s1">event</span>, <span class="pl-s1">context</span>): <span class="pl-k">return</span> { <span class="pl-s">"statusCode"</span>: <span class="pl-c1">200</span>, <span class="pl-s">"headers"</span>: { <span class="pl-s">"Content-Type"</span>: <span class="pl-s">"text/plain"</span> }, <span class="pl-s">"body"</span>: <span class="pl-s1">cowsay</span>.<span class="pl-en">get_output_string</span>(<span class="pl-s">"pig"</span>, <span class="pl-s">"Hello world, I am a pig"</span>) }</pre></div> <p>Add that updated <code>lambda_function.py</code> to the zip file again:</p> <div class="highlight highlight-source-shell"><pre>zip function.zip lambda_function.py</pre></div> <p>Deploy the update:</p> <pre><code>aws lambda update-function-code \ --function-name lambda-python-hello-world \ --zip-file fileb://function.zip </code></pre> <p>Hit refresh on the URL from earlier and you should see:</p> <pre><code> _______________________ | Hello world, I am a pig | ======================= \ \ \ \ ,. (_|,. ,' /, )_______ _ __j o``-' `.'-)' (") \' `-j | `-._( / |_\ |--^. / /_]'|_| /_)_/ /_]' /_]' </code></pre> <h2><a id="user-content-advanced-python-dependencies" class="anchor" aria-hidden="true" href="#advanced-python-dependencies"><span aria-hidden="true" class="octicon octicon-link"></span></a>Advanced Python dependencies</h2> <p>The above recipe works fine for dependencies that are written only in Python.</p> <p>Where things get more complicated is when you want to use a dependency that includes native code.</p> <p>I use a Mac. If I run <code>pip install -t lib -r requirements.txt</code> I'll get the Mac versions of those dependencies.</p> <p>But AWS Lambda functions run on Amazon Linux. So we need to include version of our packages that are built for that platform in our zip file.</p> <p>I first had to do this because I realized the <code>python3.9</code> Lambda runtime includes a truly ancient version of SQLite - version 3.7.17 <a href="http://www.sqlite.org/releaselog/3_7_17.html" rel="nofollow">from 2013-05-20</a>.</p> <p>The <a href="https://pypi.org/project/pysqlite3-binary/" rel="nofollow">pysqlite3-binary</a> package provides a much more recent SQLite, and <a href="https://datasette.io/" rel="nofollow">Datasette</a> uses that automatically if it's installed.</p> <p>I figured the best way to do this would be to run the <code>pip install</code> command inside an Amazon Linux Docker container. After much head scratching, I came up with this recipe for doing that:</p> <div class="highlight highlight-source-shell"><pre>docker run -t -v <span class="pl-s"><span class="pl-pds">$(</span>pwd<span class="pl-pds">)</span></span>:/mnt \ public.ecr.aws/sam/build-python3.9:latest \ /bin/sh -c <span class="pl-s"><span class="pl-pds">"</span>pip install -r /mnt/requirements.txt -t /mnt/lib<span class="pl-pds">"</span></span></pre></div> <ul> <li>The <code>-v $(pwd):/mnt</code> flag mounts the current directory as <code>/mnt</code> inside the container.</li> <li>The <code>public.ecr.aws/sam/build-python3.9:latest</code> image is the official AWS Lambda Python 3.9 image.</li> <li> <code>/bin/sh -c "pip install -r /mnt/requirements.txt -t /mnt/lib"</code> runs <code>pip install</code> inside the container, but ensures the files are written to <code>/mnt/lib</code> which is the <code>lib</code> folder in our current directory.</li> </ul> <p>This recipe works! The result is a <code>lib/</code> folder full of Amazon Linux Python packages, ready to be zipped up and deployed.</p> <h2><a id="user-content-running-an-asgi-application" class="anchor" aria-hidden="true" href="#running-an-asgi-application"><span aria-hidden="true" class="octicon octicon-link"></span></a>Running an ASGI application</h2> <p>I want to deploy <a href="https://datasette.io/" rel="nofollow">Datasette</a>.</p> <p>Datasette is an <a href="https://simonwillison.net/2019/Jun/23/datasette-asgi/" rel="nofollow">ASGI application</a>.</p> <p>But... AWS Lambda functions have their own weird interface to HTTP - the <code>event</code> and <code>context</code> parameters shown above.</p> <p><a href="https://github.com/jordaneremieff/mangum">Mangum</a> is a well regarded library that bridges the gap between the two.</p> <p>Here's how I got Datasette and Mangum working. It was surprisingly straight-forward!</p> <p>I added the following to my <code>requirements.txt</code> file:</p> <pre><code>datasette pysqlite3-binary mangum </code></pre> <p>I deleted my <code>lib</code> folder:</p> <pre><code>rm -rf lib </code></pre> <p>Then I ran the magic incantation from above:</p> <div class="highlight highlight-source-shell"><pre>docker run -t -v <span class="pl-s"><span class="pl-pds">$(</span>pwd<span class="pl-pds">)</span></span>:/mnt \ public.ecr.aws/sam/build-python3.9:latest \ /bin/sh -c <span class="pl-s"><span class="pl-pds">"</span>pip install -r /mnt/requirements.txt -t /mnt/lib<span class="pl-pds">"</span></span></pre></div> <p>I added the dependencies to a new <code>function.zip</code> file:</p> <div class="highlight highlight-source-shell"><pre>rm -rf function.zip (cd lib<span class="pl-k">;</span> zip ../function.zip -r .)</pre></div> <p>Then I added the following to <code>lambda_function.py</code>:</p> <div class="highlight highlight-source-python"><pre><span class="pl-k">import</span> <span class="pl-s1">asyncio</span> <span class="pl-k">from</span> <span class="pl-s1">datasette</span>.<span class="pl-s1">app</span> <span class="pl-k">import</span> <span class="pl-v">Datasette</span> <span class="pl-k">import</span> <span class="pl-s1">mangum</span> <span class="pl-s1">ds</span> <span class="pl-c1">=</span> <span class="pl-v">Datasette</span>([<span class="pl-s">"fixtures.db"</span>]) <span class="pl-c"># Handler wraps the Datasette ASGI app with Mangum:</span> <span class="pl-s1">lambda_handler</span> <span class="pl-c1">=</span> <span class="pl-s1">mangum</span>.<span class="pl-v">Mangum</span>(<span class="pl-s1">ds</span>.<span class="pl-en">app</span>())</pre></div> <p>I added that to the zip:</p> <pre><code>zip function.zip lambda_function.py </code></pre> <p>Finally, I grabbed a copy of the standard Datasette <code>fixtures.db</code> database file and added that to the zip as well:</p> <div class="highlight highlight-source-shell"><pre>wget https://latest.datasette.io/fixtures.db zip function.zip fixtures.db</pre></div> <p>The finished <code>function.zip</code> is 7.1MB. Time to deploy it:</p> <pre><code>aws lambda update-function-code \ --function-name lambda-python-hello-world \ --zip-file fileb://function.zip </code></pre> <p>This did the trick! I now had a Datasette instance running on Lambda: <a href="https://fnkvspusjrl5dxytaxnuwidxem0hverw.lambda-url.us-east-1.on.aws/" rel="nofollow">https://fnkvspusjrl5dxytaxnuwidxem0hverw.lambda-url.us-east-1.on.aws/</a></p> <p>The default Lambda configuration only provides 128MB of RAM, and I was getting occasional timeout errors. Bumping that up to 256MB fixed the problem:</p> <div class="highlight highlight-source-shell"><pre>aws lambda update-function-configuration \ --function-name lambda-python-hello-world \ --memory-size 256</pre></div> <h2><a id="user-content-this-should-work-for-starlette-and-fastapi-too" class="anchor" aria-hidden="true" href="#this-should-work-for-starlette-and-fastapi-too"><span aria-hidden="true" class="octicon octicon-link"></span></a>This should work for Starlette and FastAPI too</h2> <p>Mangum works with ASGI apps, so any app built using <a href="https://www.starlette.io/" rel="nofollow">Starlette</a> or <a href="https://fastapi.tiangolo.com/" rel="nofollow">FastAPI</a> should work exactly the same way.</p> <h2><a id="user-content-pretty-urls" class="anchor" aria-hidden="true" href="#pretty-urls"><span aria-hidden="true" class="octicon octicon-link"></span></a>Pretty URLs</h2> <p>One thing I haven't figured out yet is how to assign a custom domain name to a Lambda function.</p> <p>I understand doing that involves several other AWS services, potentially API Gateway, CloudFront and Route53. I'll update this once I figure those out.</p> | <Binary: 75,022 bytes> | 2022-09-18T20:08:14-07:00 | 2022-09-19T03:08:14+00:00 | 2022-09-19T11:39:51-07:00 | 2022-09-19T18:39:51+00:00 | d6285f78938f820ca587824fc6eba035 | asgi-mangum |