Build & Deploy a Serverless Express API to AWS
Tags: javascript tutorial serverless sam
Introduction
Released in 2015, AWS API Gateway allows developers to build and publish APIs that can be consumed by clients over public internet and virtual private cloud networks. At Re:Invent 2019, AWS announced a significant addition to the service that they have called “HTTP APIs”. The HTTP API is a direct response to customers who wanted to build simple HTTP backed API’s, who did not need the complete feature set of API Gateway. The new HTTP API service is perfect for hosting Lamba backend applications that are written in popular HTTP Frameworks like Express, Flask, .NET etc. This wouldn’t be my first choice when building a new application, and isn’t how we typically build applications at Mechanical Rock - but it’s certainly going to be useful for a particular set of developers, so why not take a look?
How is this different from the existing API Gateway
This existing API Gateway has A LOT of features. Someone who wants to build an application that proxies requests to a lambda-hosted web framework probably won’t need most of those features. By removing those features, we get a simpler service to work with, at improved performance and reduced cost. AWS estimates those migrating to HTTP API’s from API Gateway can expect a 70% cost reduction and faster response times.
Let’s create a hello world app with SAM!
Let’s build a very simple guestbook API using Express. The guestbook will be used to record a comment, the name of the person who made the comment, and the time the comment was made. We will add an additional endpoint that can retrieve all the comments that have been made, starting with the latest comment. We will use S3 to store the comments. Note that while I could use an RDBMS or NoSQL database for this, as I only have a requirement for a pageable list this is overkill. If I needed to retrieve comments by an ID or some other attribute, then I would start looking at storage solutions with flexible options for retrieving data. (note: this makes deleting the API a bit of a pain - because unless the comments bucket is empty the stack will fail to delete. Keep in mind that you will need to delete the bucket independently of the stack when you remove this application from your account).
We will build this all using the AWS Serverless Application Model (SAM).
The complete (finished) example is available here.
Setting up your environment for AWS development
Before we get started we will need to install a few tools to do this. We will need to install the aws-cli, aws-sam-cli, and configure our AWS profile. If you have already done this you can skip this section
!!! install docker for local invoke
-
Install the aws-cli tools if you have not already.
Documentation on how to install the aws-cli is available (here)[https://docs.aws.amazon.com/cli/latest/userguide/install-cliv1.html]. Details instructions for particular platforms are provided at the end of the page.
-
Then install the aws-sam-cli.
You can find instructions on how to do this for all major platforms (here)[https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html]
-
Configure your profile to point to your AWS account.
You will need to do this to ensure that the SAM tooling can make API calls on your behalf to your AWS environment. For test purposes using your personal profile is usually fine.
(This)[https://medium.com/blechatech/how-to-setup-aws-credentials-for-new-code-cc80c44cc67] article does a reasonable job of explaining what is required, as does the official documentation for the AWS CLI.
The way I typically test to see whether things are working is to create an S3 bucket in my account - I will then issue an
aws s3 lscall. If the bucket I made is returned in the call, I know that everything is set up correctly. -
Ensure you have nodejs 12 installed.
Let’s scaffold out a SAM Application
-
First create a directory to that will contain the project.
mkdir -p projects && cd projects -
Now we can initialize the project using the AWS SAM CLI.
sam init --no-interactive \ --runtime nodejs12.x \ --dependency-manager npm \ --output-dir . \ --name simple-node-api \ --app-template hello-world
This will generate the following structure:
.
├── README.md
├── events
│ └── event.json
├── hello-world
│ ├── app.js
│ ├── package.json
│ └── tests
│ └── unit
│ └── test-handler.js
└── template.yaml
Now we can start working on our implementation!
Get Express JS working inside the handler
Our code for the application is stored inside hello-world/app.js. At the moment, it is pretty slim, so so change it to the below.
const serverless = require('serverless-http');
const express = require('express');
const app = new express();
app.get('/', (req, res) => {
res.send('Hello World')
});
module.exports.lambdaHandler = serverless(app);
This is about as simple as it can get. We’ll add the logic for our guestbook application in a minute - we’ll first get this deployed and ensure it works as is first with our ‘hello world’ response.
Make a Simple SAM Template
A sample is provided under template.yaml. It is mostly fine, but we will need to make a few changes. Use the following modified template instead.
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
simple-node-api
Sample SAM Template for simple-node-api
Globals:
Function:
Timeout: 3
Resources:
ExpressBackend:
Type: AWS::Serverless::Function
Properties:
CodeUri: hello-world/
Handler: app.lambdaHandler
Runtime: nodejs12.x
Events:
HelloWorld:
Type: HttpApi
Properties:
Path: /{proxy+}
Method: get
Outputs:
HelloWorldApi:
Description: API Gateway endpoint URL for Prod stage for Hello World function
Value:
Fn::Sub: https://${ServerlessHttpApi}.execute-api.${AWS::Region}.amazonaws.com/
ExpressBackendFunction:
Description: Express Backend Lambda Function ARN
Value: !Sub ExpressBackend.Arn
ExpressBackendIamRole:
Description: Implicit IAM Role created for Hello World function
Value: !Sub ExpressBackendFunctionRole.Arn
We don’t really need to change much. All that needed to be done was
- modifying the event type to
HttpApi - change the path to
/{proxy}+. This will ensure all get requests are passed to the lambda request, regardless of their path. - Change the output to reference
${ServerlessHttpApi}instead of${ServerlessRestApi}
Build, Test, & Deploy the App
Firstly, we execute sam build from our base directory. This should result in the following output from the sam-cli.
Building resource 'ExpressBackend'
Running NodejsNpmBuilder:NpmPack
Running NodejsNpmBuilder:CopyNpmrc
Running NodejsNpmBuilder:CopySource
Running NodejsNpmBuilder:NpmInstall
Running NodejsNpmBuilder:CleanUpNpmrc
Build Succeeded
Built Artifacts : .aws-sam/build
Built Template : .aws-sam/build/template.yaml
Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Deploy: sam deploy --guided
Assuming that you have docker installed, you can use sam local invoke as a quick test. If you do, your output will look like this;
▶ sam local invoke
Invoking app.lambdaHandler (nodejs12.x)
Fetching lambci/lambda:nodejs12.x Docker container image......
Mounting /Users/matt.tyler/projects/simple-node-api/simple-node-api/.aws-sam/build/ExpressBackend as /var/task:ro,delegated inside runtime container
START RequestId: 6bb44d66-e096-124b-5ce9-5f1f1fea88f9 Version: $LATEST
2020-01-02T06:00:30.213Z 6bb44d66-e096-124b-5ce9-5f1f1fea88f9 ERROR (node:17) [DEP0066] DeprecationWarning: OutgoingMessage.prototype._headers is deprecated
END RequestId: 6bb44d66-e096-124b-5ce9-5f1f1fea88f9
REPORT RequestId: 6bb44d66-e096-124b-5ce9-5f1f1fea88f9 Init Duration: 473.40 ms Duration: 10.32 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 50 MB
{"statusCode":200,"headers":{"x-powered-by":"Express","content-type":"text/html; charset=utf-8","content-length":"11","etag":"W/\"b-Ck1VqNd45QIvq3AZd8XYQLvEhtA\""},"isBase64Encoded":false,"body":"Hello World"}
We can see the ‘hello world’ response inside the JSON payload that was returned from our lambda function.
Now let’s deploy the application - to do this we will perform a guided deploy. Upon completion of a guided deploy, a configuration file will be created that allows future deploys to use our previous responses.
sam deploy --guided
Answer the questions as appropriate - the command will output the following -
▶ sam deploy --guided
Configuring SAM deploy
======================
Looking for samconfig.toml : Found
Reading default arguments : Success
Setting default arguments for 'sam deploy'
=========================================
Stack Name [simple-node-api]:
AWS Region [ap-southeast-2]:
#Shows you resources changes to be deployed and require a 'Y' to initiate deploy
Confirm changes before deploy [Y/n]:
#SAM needs permission to be able to create roles to connect to the resources in your template
Allow SAM CLI IAM role creation [Y/n]:
Save arguments to samconfig.toml [Y/n]:
Looking for resources needed for deployment: Found!
Managed S3 bucket: <aws-sam-cli-managed-default-samclisourcebucket-HASH>
A different default S3 bucket can be set in samconfig.toml
Saved arguments to config file
Running 'sam deploy' for future deployments will use the parameters saved above.
The above parameters can be changed by modifying samconfig.toml
Learn more about samconfig.toml syntax at
https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-config.html
Deploying with following values
===============================
Stack name : simple-node-api
Region : ap-southeast-2
Confirm changeset : True
Deployment s3 bucket : <aws-sam-cli-managed-default-samclisourcebucket-HASH>
Capabilities : ["CAPABILITY_IAM"]
Parameter overrides : {}
Initiating deployment
=====================
Uploading to simple-node-api/0948bb837790c8b67731569145c3b9f1 860364 / 860364.0 (100.00%)
Uploading to simple-node-api/6dd8e36f41145e6820661afcc04594a4.template 1068 / 1068.0 (100.00%)
Waiting for changeset to be created..
CloudFormation stack changeset
---------------------------------------------------------------------------------------------------------
Operation LogicalResourceId ResourceType
---------------------------------------------------------------------------------------------------------
+ Add ExpressBackendHelloWorldPermissio AWS::Lambda::Permission
n
+ Add ExpressBackendRole AWS::IAM::Role
+ Add ExpressBackend AWS::Lambda::Function
+ Add ServerlessHttpApiApiGatewayDefaul AWS::ApiGatewayV2::Stage
tStage
+ Add ServerlessHttpApi AWS::ApiGatewayV2::Api
---------------------------------------------------------------------------------------------------------
Changeset created successfully. arn:aws:cloudformation:ap-southeast-2:<ACCOUNT_ID>:changeSet/samcli-deploy1577946076/01b8938e-9205-4489-b1a2-0599a8ebfc41
Previewing CloudFormation changeset before deployment
======================================================
Deploy this changeset? [y/N]: y
2020-01-02 14:21:49 - Waiting for stack create/update to complete
CloudFormation events from changeset
---------------------------------------------------------------------------------------------------------
ResourceStatus ResourceType LogicalResourceId ResourceStatusReason
---------------------------------------------------------------------------------------------------------
CREATE_IN_PROGRESS AWS::IAM::Role ExpressBackendRole -
CREATE_IN_PROGRESS AWS::IAM::Role ExpressBackendRole Resource creation
Initiated
CREATE_COMPLETE AWS::IAM::Role ExpressBackendRole -
CREATE_IN_PROGRESS AWS::Lambda::Function ExpressBackend -
CREATE_IN_PROGRESS AWS::Lambda::Function ExpressBackend Resource creation
Initiated
CREATE_COMPLETE AWS::Lambda::Function ExpressBackend -
CREATE_IN_PROGRESS AWS::ApiGatewayV2::Api ServerlessHttpApi -
CREATE_COMPLETE AWS::ApiGatewayV2::Api ServerlessHttpApi -
CREATE_IN_PROGRESS AWS::ApiGatewayV2::Api ServerlessHttpApi Resource creation
Initiated
CREATE_IN_PROGRESS AWS::Lambda::Permission ExpressBackendHelloWorld Resource creation
Permission Initiated
CREATE_IN_PROGRESS AWS::ApiGatewayV2::Stage ServerlessHttpApiApiGate -
wayDefaultStage
CREATE_IN_PROGRESS AWS::Lambda::Permission ExpressBackendHelloWorld -
Permission
CREATE_IN_PROGRESS AWS::ApiGatewayV2::Stage ServerlessHttpApiApiGate Resource creation
wayDefaultStage Initiated
CREATE_COMPLETE AWS::ApiGatewayV2::Stage ServerlessHttpApiApiGate -
wayDefaultStage
CREATE_COMPLETE AWS::Lambda::Permission ExpressBackendHelloWorld -
Permission
CREATE_COMPLETE AWS::CloudFormation::Sta simple-node-api -
ck
---------------------------------------------------------------------------------------------------------
Stack simple-node-api outputs:
---------------------------------------------------------------------------------------------------------
OutputKey-Description OutputValue
---------------------------------------------------------------------------------------------------------
ExpressBackendFunction - Express Backend Lambda ExpressBackend.Arn
Function ARN
ExpressBackendIamRole - Implicit IAM Role created ExpressBackendFunctionRole.Arn
for Hello World function
HelloWorldApi - API Gateway endpoint URL for Prod https://cukqdke1jj.execute-api.ap-
stage for Hello World function southeast-2.amazonaws.com/
---------------------------------------------------------------------------------------------------------
Successfully created/updated stack - simple-node-api in ap-southeast-2
The command will additionally create a samconfig.toml which remembers the settings you applied during the guided deploy.
Initiating a curl command will at the API Gateway endpoint in your outputs section should retrieve the ‘hello world’ message from your cloud deployed endpoint!
Note: You can use the aws-cli to tear it down with the following command at any time.
aws cloudformation delete-stack --stack-name simple-node-api
Implementing the API Methods
Now let’s start adding some API endpoints. The first thing we need to do is ensure we have an S3 bucket to store the comments in. Let’s start by adding an S3 bucket resource to our template, and adding a policy to enable our lambda function to read and write from the bucket. Lastly, we need to expand the events section to include a post event. Our Resources section will now include the following.
Resources:
CommentsBucket:
Type: AWS::S3::Bucket
ExpressBackend:
Type: AWS::Serverless::Function
Properties:
CodeUri: hello-world/
Handler: app.lambdaHandler
Runtime: nodejs12.x
Environment:
Variables:
BUCKET: !Ref CommentsBucket
Policies:
- S3CrudPolicy:
BucketName: !Ref CommentsBucket
Events:
Get:
Type: HttpApi
Properties:
Path: /{proxy+}
Method: get
Post:
Type: HttpApi
Properties:
Path: /{proxy+}
Method: post
Now we need to implement the logic for our application.
I’m importing the following libraries:
const serverless = require('serverless-http');
const express = require('express');
const s3 = require('aws-sdk/clients/s3');
const v5 = require('uuid/v5');
const env = require('env-var');
const bodyParser = require('body-parser');
First I’ll add some basic middleware to handle errors thrown from our handlers. Normally I’d be a little more specific, but for now I’ll log the error and return a 500 response.
app.use('/', (err, _, res, _) => {
console.log(err);
res.status(500).json({ message: 'Internal Server Error'});
});
We’ll create a function to return a new s3 client. If the bucket isn’t present as an environment variable, we will throw an error.
function newS3Client() {
// You can use the 'params' argument to the client to set default variables
// for requests. In this example, all API calls will default the value of
// the 'Bucket' parameter to the value stored in the BUCKET environment
// variable.
return new s3({ params: { Bucket: env.get('BUCKET').required() } });
}
I’ve also created a function to return an author. At the moment it will just return ‘anonymous’, but we will modify this to add identity in a later post.
function getAuthor() {
return 'anonymous';
}
Writing a message is fairly straight-forward.
- I generate a namespace UUID using the author and the URL UUID namespace.
- I generate a new UUID based on the message content and namespace UUID I just generated. This is to help reduce the odds of a key collision.
- I generate an ISO date string, which will (obviously) be used as the date and time the message was submitted.
- I generate the 9’s complement from the datetime. This is an implementation detail used to ensure that when I try to retrieve the messages later, they will be returned in order from newest-to-oldest.
By default, any calls made to list objects in an S3 bucket will are returned in lexographic order. In practical terms, this will return earlier dates first. By converting the date to 9’s complement, the order will be reversed, and newer comments will be earlier in the order.
async function writeMessage(s3, message, author) {
const namespace = v5(author, v5.URL);
const id = v5(message, namespace);
const date = new Date();
const Key = `${ninesComplement(date)}/${id}`;
const body = { message, date: date.toISOString(), author };
await s3.put_object({ Key, Body: JSON.stringify(body) });
return body;
}
Reading messages out is also relatively simple. This code will list out ‘maxItems’ worth of objects, continuing to iterate based on the continuation token. The contents of each object is then retrieved and returned along with the next continuation token if there are more results available. The tokens are used to paginate the results. Note that I did have to base64 encode/decode the token to ensure query arguments were not mangled by express’ query argument parsing (though this is not an unusual thing to do).
async function getMessages(client, maxItems, token) {
const { Contents, NextContinuationToken } = await client.listObjectsV2({
MaxKeys: maxItems,
ContinuationToken: token ||
new Buffer(token, 'base64').toString('ascii')
}).promise();
const res = await Promise.all(Contents
.map(({ Key }) => client.getObject({ Key }).promise()));
return {
Items: res.map(({ Body }) => JSON.parse(Body)),
NextToken: NextContinuationToken ||
new Buffer(NextContinuationToken, 'ascii').toString('base64')
}
}
You can learn more about pagination in serverless applications from Serverless Hero Yan Cui, who wrote this excellent post: “Guys, we’re doing pagination wrong…”
After doing another round of sam build && sam deploy, let’s curl our new API.
# Your API endpoint address is available from the output of your deployment
ENDPOINT=<ENDPOINT>
# this should return nothing e.g. {"Items":[]}
curl $ENDPOINT
# now send some data
for i in {1..10}; do
curl -XPOST -H "Content-Type: text/plain" -d "Message: $i" $ENDPOINT
done
If we curl the endpoint we should recieve the following
{
"Items": [
{
"message": "Message: 10",
"date": "2020-01-06T01:17:05.691Z",
"author": "anonymous"
},
{
"message": "Message: 9",
"date": "2020-01-06T01:17:05.288Z",
"author": "anonymous"
},
{
"message": "Message: 8",
"date": "2020-01-06T01:17:04.876Z",
"author": "anonymous"
},
{
"message": "Message: 7",
"date": "2020-01-06T01:17:04.475Z",
"author": "anonymous"
},
{
"message": "Message: 6",
"date": "2020-01-06T01:17:04.070Z",
"author": "anonymous"
},
{
"message": "Message: 5",
"date": "2020-01-06T01:17:03.656Z",
"author": "anonymous"
},
{
"message": "Message: 4",
"date": "2020-01-06T01:17:02.156Z",
"author": "anonymous"
},
{
"message": "Message: 3",
"date": "2020-01-06T01:17:01.679Z",
"author": "anonymous"
},
{
"message": "Message: 2",
"date": "2020-01-06T01:17:00.576Z",
"author": "anonymous"
},
{
"message": "Message: 1",
"date": "2020-01-06T01:16:59.034Z",
"author": "anonymous"
}
]
}
Paging through the result set is possible by using the maxItems query parameter. If we set it to 1 e.g. curl "$ENDPOINT?maxItems=1", we will receive the first item and a token to retrieve more data.
{
"Items": [
{
"message": "Message: 10",
"date": "2020-01-06T01:17:05.691Z",
"author": "anonymous"
}
],
"NextToken": "1mSTs9j4zPH1Dw7LkIZOXfUOpfd/vijAKI5gVyyeW0KjJ2BQqoxKgH3c2cyYQd74GYuEn0xQyKapxfmXVCd6yzT7cDUfA2pdExAWszRdL8EmEATzr4WMxeZQ5QtHJHpz7rN7q+8wIuE4mbwyYHCsBXf8ELJVmeRut"
}
Now using the value of NextToken, we can retrieve the next value using `curl “$ENDPOINT?maxItems=1&token=MU5ZVjBnR0Nza2g1cXF4Nm5HSDZoUU5IaFg4bjk4R0Z1Uzc2TkFlQWY3blI0S0xDak5DQVZ6cG5aZy92aEQxMHFUeUJJd1A5cC8xRnNFME9Hakw2VnJlczBRWVdnaWVpTm8vTnhLREhvMUlZQ2UwSCtVVHd6SXVCL0NFSlJ5OE15bktHUjNFa0QwNnNybUlqeW9RekdrMUwvTDR0NHUyTlQ=”
{
"Items": [
{
"message": "Message: 9",
"date": "2020-01-06T01:17:05.288Z",
"author": "anonymous"
},
],
"NextToken": "1XqRKzDWBlYIFrJLHMoTCTIHAjURZIAOz/Rq6EDIAihbhVcrXxV6Wzi5/WsNUvPeN1fCMVLUXgERX3w6zgQ7554S97HcGWZ+/iO2lkPj1JvGKTOd48u1qTdcywMCcmflcBR3oqd+aNROdH9nMa8KBIQHPSAfFy/SK"
}
What next?
We now have a basic API working but there are still issues that remain.
-
Our API is public, and all users are anonymous.
We haven’t put in any authorization/authentication controls, which we would probably want to do in order to tie comments to a particular user. Adding an extra step to authenticate before-hand will likely reduce the number of potential calls to the API.
-
We are not doing any validation of the incoming message.
There is a potential for the API to be used in an injection attacked. For example, a malicious javascript payload could be sent to the service and this could be executed by a browser upon retrieving a message. Whilst the client should ultimately be responsible for protecting against this, it would not be a bad idea to add some level of detection/sanitisation on the server side to make this less likely.
-
The API is not particularly usable.
Sure, we’ve curl’ed some endpoints to show everything kinda works, but we would typically call the API from a frontend. This isn’t very easy to do at the moment - we should generate some code that we can call from the client to consume the API.
-
We have not accounted for observability
We are currently not logging anything, or tracing calls to other services, or keeping track of metrics. We should add structured logging, trace calls with X-Ray, and create CloudWatch metrics, in order to make this a production-ready service.
I intend to address these points in future articles.
In closing, I hope this gives you a good starting point to building serverless applications. The new HTTP features in API Gateway are a good way to get start if you are already familiar with an in existing framework like Express and Flask. Hopefully this has also given you insight into the great SAM tooling that is available from AWS as well.
Starting out with Serverless? Contact Mechanical Rock to Get Started!