NGINX reverse proxy sidecar for a web container hosted with Amazon ECS and AWS Fargate
About
NGINX is a high performance HTTP server and reverse proxy which has achieved significant adoption because of its asynchronous event driven architecture which allows it to serve thousands of concurrent requests with very low memory footprint.
Amazon Elastic Container Service (ECS) is a highly scalable, high performance container management service that runs containers on a managed cluster of Amazon EC2 instances, or using serverless AWS Fargate capacity.
This pattern will show how to deploy open source NGINX as a reverse proxy container in front of your application container.
Architecture
This pattern will deploy the following architecture:
- An ECS task runs in AWS Fargate. The task has two containers: an NGINX sidecar, and a simple JavaScript webserver
- The task only accepts inbound traffic on the NGINX traffic port. The NGINX server filters out bad traffic, and forwards good traffic to the backend task on it's local port.
- The NGINX server responds back to clients, returning the response from the application server. The NGINX server can also transform responses, such as doing compression of plaintext responses. This offloads work from the application itself.
Why use a reverse proxy?
A reverse proxy fetches resources from another server on behalf of a client. One of the challenges of running a web server that serves requests from the public is that you can expect to receive quite a lot of unwanted traffic every day. Some of this traffic is relatively benign scans by researchers and tools such as Shodan or nmap:
[18/May/2017:15:10:10 +0000] "GET /YesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScann HTTP/1.1" 404 1389 - Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36
[18/May/2017:18:19:51 +0000] "GET /clientaccesspolicy.xml HTTP/1.1" 404 322 - Cloud mapping experiment. Contact research@pdrlabs.net
But other traffic is much more malicious. For example here is what a web server sees while being scanned by the hacking tool ZmEu which scans web servers trying to find PHPMyAdmin installations to exploit:
[18/May/2017:16:27:39 +0000] "GET /mysqladmin/scripts/setup.php HTTP/1.1" 404 391 - ZmEu
[18/May/2017:16:27:39 +0000] "GET /web/phpMyAdmin/scripts/setup.php HTTP/1.1" 404 394 - ZmEu
[18/May/2017:16:27:39 +0000] "GET /xampp/phpmyadmin/scripts/setup.php HTTP/1.1" 404 396 - ZmEu
[18/May/2017:16:27:40 +0000] "GET /apache-default/phpmyadmin/scripts/setup.php HTTP/1.1" 404 405 - ZmEu
[18/May/2017:16:27:40 +0000] "GET /phpMyAdmin-2.10.0.0/scripts/setup.php HTTP/1.1" 404 397 - ZmEu
[18/May/2017:16:27:40 +0000] "GET /mysql/scripts/setup.php HTTP/1.1" 404 386 - ZmEu
[18/May/2017:16:27:41 +0000] "GET /admin/scripts/setup.php HTTP/1.1" 404 386 - ZmEu
[18/May/2017:16:27:41 +0000] "GET /forum/phpmyadmin/scripts/setup.php HTTP/1.1" 404 396 - ZmEu
[18/May/2017:16:27:41 +0000] "GET /typo3/phpmyadmin/scripts/setup.php HTTP/1.1" 404 396 - ZmEu
[18/May/2017:16:27:42 +0000] "GET /phpMyAdmin-2.10.0.1/scripts/setup.php HTTP/1.1" 404 399 - ZmEu
[18/May/2017:16:27:44 +0000] "GET /administrator/components/com_joommyadmin/phpmyadmin/scripts/setup.php HTTP/1.1" 404 418 - ZmEu
[18/May/2017:18:34:45 +0000] "GET /phpmyadmin/scripts/setup.php HTTP/1.1" 404 390 - ZmEu
[18/May/2017:16:27:45 +0000] "GET /w00tw00t.at.blackhats.romanian.anti-sec:) HTTP/1.1" 404 401 - ZmEu
In addition to hacking tools scanning your servers you can also end up receiving unwanted web traffic that was intended for another server. In a dynamic cloud environment your application may end up taking over a public IP address that was formerly connected to another service. When this happens its not uncommon for misconfigured or misbehaving DNS servers to result in traffic that was intended to go to a completely different host to continue to be directed to an IP address which is now connected to your own server.
Anyone running a web server connected to the internet has to assume the responsibility of handling and rejecting potentially malicious traffic or unwanted traffic. Ideally the web server is capable of rejecting this traffic as early as possible, before it actually reaches your core application code. A reverse proxy is one way to provide an extra layer of protection for your application server. It can be configured to reject these requests before they reach your application server.
Another potential benefit of using a reverse proxy is that you can offload some static responses from the application itself. In this pattern you will notice that the healthcheck requests that the Application Load Balancer sends to the task are also being offloaded onto NGINX instead of going all the way to the application code. You could use a similar approach to host your own static HTML webpage, or other static content that you wish to serve to the internet.
Dependencies
This pattern requires that you have an AWS account and the following tools locally:
- Docker or similar OCI compatible container image builder
- AWS Serverless Application Model (SAM) CLI - If not already installed then please install SAM CLI for your system.
Build the application
Create a folder app
and put the following files into the folder. These three files define a basic JavaScript application that implements an API:
- index.js
- package.json
- Dockerfile
// Express web server for an API
const express = require('express');
const app = express();
const port = 3000;
// Healthcheck endpoint
app.get('/', (req, res) => {
res.send('Hello World!');
})
// Get user by ID endpoint
app.get('/api/users/:id', (req, res) => {
const userId = req.params.id;
res.send(`User ${userId} found!`);
})
// Get user by username endpoint
app.get('/api/users/by/username/:username', (req, res) => {
const username = req.params.username;
res.send(`User ${username} found!`);
})
app.listen(port, () => {
console.log(`Running on port ${port}...\n`);
})
You should now have an app
directory with three files in it:
index.js
- The application code that the web server will runpackage.json
- Describes dependencies for the application codeDockerfile
- Describes how to package up the application code and it's dependencies into a container image.
You can now build and push the container image to AWS by running the following commands in the app
directory.
APP_URI=$(aws ecr create-repository --repository-name app --query 'repository.repositoryUri' --output text)
docker build -t $APP_URI .
docker push $APP_URI
Build the NGINX proxy
Next we need to build a customized NGINX reverse proxy with the configuration that will forward requests to the application, while rejecting unwanted requests.
Create a folder nginx
and put the following files in the folder:
- nginx.conf
- Dockerfile
- index.html
events {
worker_connections 1024;
}
http {
# NGINX will handle gzip compression of responses from the app server
gzip on;
gzip_proxied any;
gzip_types text/plain application/json;
gzip_min_length 1000;
server {
listen 80;
# NGINX will reject anything not matching /api
location /api {
# Reject requests with unsupported HTTP method
if ($request_method !~ ^(GET|POST|HEAD|OPTIONS|PUT|DELETE)$) {
return 405;
}
# Only requests matching the expectations will
# get sent to the application server
proxy_pass http://localhost:3000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_cache_bypass $http_upgrade;
}
}
}
You should now have an nginx
directory with two files in it:
nginx.conf
- Defines the proxy configurationindex.html
- Basic file that we will use as a healthcheck responseDockerfile
- Defines how to apply the proxy configuration on top of a generic NGINX image from the Elastic Conatiner Registry Public Gallery.
Some things to note in the nginx.conf
:
- Lines 3-6 tell NGINX to compress outgoing content. This allows your application to just return plaintext responses, and offload compression onto the NGINX sidecar.
- Line 12 limits what traffic paths will be forwarded to the application. The application will only receive requests that match
/api*
. All other requests will be rejected and the response will be returned directly from the NGINX container, without your application ever being touched. - Lines 14-15 will reject a variety of malformed requests that don't match known HTTP methods
- Lines 20-26 control how traffic is forwarded to the application container.
You can now build and push the container image to AWS by running the following commands in the nginx
directory.
NGINX_URI=$(aws ecr create-repository --repository-name nginx --query 'repository.repositoryUri' --output text)
docker build -t $NGINX_URI .
docker push $NGINX_URI
Define the network
This pattern can be deployed on top of either of the following VPC patterns:
Which one you choose depends on your goals for this deployment. You can choose the low cost VPC to start with and upgrade to the large sized VPC later on if you have additional private services, or private database servers you wish to deploy in the VPC.
If you have any doubts as to which VPC to choose, then go with the "Low cost VPC" option.
Download the vpc.yml
file from your chosen pattern, but do not deploy it yet. Deployment will be done later in the process
Define the cluster
Next we need to define an ECS cluster. For this pattern we will deploy the workload to AWS Fargate:
AWSTemplateFormatVersion: '2010-09-09'
Description: Empty ECS cluster that has no EC2 instances. It is designed
to be used with AWS Fargate serverless capacity
Resources:
# Cluster that keeps track of container deployments
ECSCluster:
Type: AWS::ECS::Cluster
Properties:
ClusterSettings:
- Name: containerInsights
Value: enabled
# This is a role which is used within Fargate to allow the Fargate agent
# to download images, and upload logs.
ECSTaskExecutionRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service: [ecs-tasks.amazonaws.com]
Action: ['sts:AssumeRole']
Condition:
ArnLike:
aws:SourceArn: !Sub arn:aws:ecs:${AWS::Region}:${AWS::AccountId}:*
StringEquals:
aws:SourceAccount: !Ref AWS::AccountId
Path: /
# This role enables basic features of ECS. See reference:
# https://docs.aws.amazon.com/AmazonECS/latest/developerguide/security-iam-awsmanpol.html#security-iam-awsmanpol-AmazonECSTaskExecutionRolePolicy
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
Outputs:
ClusterName:
Description: The ECS cluster into which to launch resources
Value: !Ref ECSCluster
ECSTaskExecutionRole:
Description: The role used to start up a task
Value: !Ref ECSTaskExecutionRole
Define the service
Next we need to define the service that will run:
AWSTemplateFormatVersion: '2010-09-09'
Description: An example service that deploys in AWS VPC networking mode
on AWS Fargate. Service runs with networking in public
subnets and public IP addresses
Parameters:
VpcId:
Type: String
Description: The VPC that the service is running inside of
PublicSubnetIds:
Type: List<AWS::EC2::Subnet::Id>
Description: List of public subnet ID's to put the load balancer and tasks in
ClusterName:
Type: String
Description: The name of the ECS cluster into which to launch capacity.
ECSTaskExecutionRole:
Type: String
Description: The role used to start up an ECS task
ServiceName:
Type: String
Default: web
Description: A name for the service
AppImageUrl:
Type: String
Description: The url of a docker image that contains the application process that
will handle the traffic for this service
NginxImageUrl:
Type: String
Description: The url of a docker image that provides the NGINX reverse proxy
ContainerCpu:
Type: Number
Default: 1024
Description: How much CPU to give the container. 1024 is 1 CPU
ContainerMemory:
Type: Number
Default: 2048
Description: How much memory in megabytes to give the container
ContainerPort:
Type: Number
Default: 80
Description: What port that the application expects traffic on
DesiredCount:
Type: Number
Default: 2
Description: How many copies of the service task to run
Resources:
# The task definition. This is a simple metadata description of what
# container(s) to run, and what resource requirements the task has
TaskDefinition:
Type: AWS::ECS::TaskDefinition
Properties:
Family: !Ref ServiceName
Cpu: !Ref ContainerCpu
Memory: !Ref ContainerMemory
NetworkMode: awsvpc
RequiresCompatibilities:
- FARGATE
ExecutionRoleArn: !Ref ECSTaskExecutionRole
ContainerDefinitions:
# NGINX sidecar for the task
- Name: nginx
Image: !Ref NginxImageUrl
PortMappings:
- ContainerPort: !Ref ContainerPort
HostPort: !Ref ContainerPort
LogConfiguration:
LogDriver: 'awslogs'
Options:
mode: non-blocking
max-buffer-size: 25m
awslogs-group: !Ref NginxLogGroup
awslogs-region: !Ref AWS::Region
awslogs-stream-prefix: !Sub "${ServiceName}/nginx"
# Application container
- Name: app
Image: !Ref AppImageUrl
LogConfiguration:
LogDriver: 'awslogs'
Options:
mode: non-blocking
max-buffer-size: 25m
awslogs-group: !Ref AppLogGroup
awslogs-region: !Ref AWS::Region
awslogs-stream-prefix: !Sub "${ServiceName}/app"
# The service. The service is a resource which allows you to run multiple
# copies of a type of task, and gather up their logs and metrics, as well
# as monitor the number of running tasks and replace any that have crashed
Service:
Type: AWS::ECS::Service
# Avoid race condition between ECS service creation and associating
# the target group with the LB
DependsOn: PublicLoadBalancerListener
Properties:
ServiceName: !Ref ServiceName
Cluster: !Ref ClusterName
LaunchType: FARGATE
NetworkConfiguration:
AwsvpcConfiguration:
AssignPublicIp: ENABLED
SecurityGroups:
- !Ref ServiceSecurityGroup
Subnets: !Ref PublicSubnetIds
DeploymentConfiguration:
MaximumPercent: 200
MinimumHealthyPercent: 75
DesiredCount: !Ref DesiredCount
TaskDefinition: !Ref TaskDefinition
LoadBalancers:
- ContainerName: nginx
ContainerPort: !Ref ContainerPort
TargetGroupArn: !Ref ServiceTargetGroup
# Security group that limits network access
# to the task
ServiceSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: Security group for service
VpcId: !Ref VpcId
# Keeps track of the list of tasks for the service
ServiceTargetGroup:
Type: AWS::ElasticLoadBalancingV2::TargetGroup
Properties:
HealthCheckIntervalSeconds: 6
HealthCheckPath: /
HealthCheckProtocol: HTTP
HealthCheckTimeoutSeconds: 5
HealthyThresholdCount: 2
TargetType: ip
Port: !Ref ContainerPort
Protocol: HTTP
UnhealthyThresholdCount: 10
VpcId: !Ref VpcId
TargetGroupAttributes:
- Key: deregistration_delay.timeout_seconds
Value: 0
# A public facing load balancer, this is used as ingress for
# public facing internet traffic.
PublicLoadBalancerSG:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: Access to the public facing load balancer
VpcId: !Ref VpcId
SecurityGroupIngress:
# Allow access to public facing ALB from any IP address
- CidrIp: 0.0.0.0/0
IpProtocol: -1
PublicLoadBalancer:
Type: AWS::ElasticLoadBalancingV2::LoadBalancer
Properties:
Scheme: internet-facing
LoadBalancerAttributes:
- Key: idle_timeout.timeout_seconds
Value: '30'
Subnets: !Ref PublicSubnetIds
SecurityGroups:
- !Ref PublicLoadBalancerSG
PublicLoadBalancerListener:
Type: AWS::ElasticLoadBalancingV2::Listener
Properties:
DefaultActions:
- Type: 'forward'
ForwardConfig:
TargetGroups:
- TargetGroupArn: !Ref ServiceTargetGroup
Weight: 100
LoadBalancerArn: !Ref 'PublicLoadBalancer'
Port: 80
Protocol: HTTP
# Open up the service's security group to traffic originating
# from the security group of the load balancer.
ServiceIngressfromLoadBalancer:
Type: AWS::EC2::SecurityGroupIngress
Properties:
Description: Ingress from the public ALB
GroupId: !Ref ServiceSecurityGroup
IpProtocol: -1
SourceSecurityGroupId: !Ref 'PublicLoadBalancerSG'
# This log group stores the stdout logs from the app
AppLogGroup:
Type: AWS::Logs::LogGroup
# This log group stores the access logs from NGINX
NginxLogGroup:
Type: AWS::Logs::LogGroup
Things to note in this template:
- The
AWS::ECS::TaskDefinition
uses networking modeawsvpc
. This means that every task gets it's own private networking interface. Containers that share a task can communicate with each other usinglocalhost
. This allows thenginx.conf
proxy configuration to work, as it expects the application container to be available athttp://localhost:3000
. - The task has two container definitions. Each container has it's own log group. This allows us to keep the application logs separately from the NGINX access logs.
Deploy it all
In order to deploy everything, we will use a parent stack that defines each of the children stacks and what values they expect.
AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Description: Parent stack that deploys a serverless AWS Fargate web service.
The service tasks use an NGINX reverse proxy in front of the application.
Parameters:
AppImageUrl:
Type: String
Description: The url of a docker image that contains the application process that
will handle the traffic for this service
NginxImageUrl:
Type: String
Description: The url of a docker image that provides the NGINX reverse proxy
Resources:
# The networking configuration. This creates an isolated
# network specific to this particular environment
VpcStack:
Type: AWS::Serverless::Application
Properties:
Location: vpc.yml
# This stack contains the Amazon ECS cluster itself
ClusterStack:
Type: AWS::Serverless::Application
Properties:
Location: cluster.yml
# This stack contains the container deployment
ServiceStack:
Type: AWS::Serverless::Application
Properties:
Location: service.yml
Parameters:
VpcId: !GetAtt VpcStack.Outputs.VpcId
PublicSubnetIds: !GetAtt VpcStack.Outputs.PublicSubnetIds
ClusterName: !GetAtt ClusterStack.Outputs.ClusterName
ECSTaskExecutionRole: !GetAtt ClusterStack.Outputs.ECSTaskExecutionRole
AppImageUrl: !Ref AppImageUrl
NginxImageUrl: !Ref NginxImageUrl
You should have four files:
vpc.yml
- Defines the networking setupcluster.yml
- Defines the Amazon ECS clusterservice.yml
- Defines the service and task to runparent.yml
- Top level stack that deploys the three child stacks.
You can deploy everything with the following command:
sam deploy \
--template-file parent.yml \
--stack-name nginx-reverse-proxy \
--resolve-s3 \
--capabilities CAPABILITY_IAM \
--parameter-overrides AppImageUrl=$APP_URI NginxImageUrl=$NGINX_URI
Test it out
Open your Amazon ECS console, and locate the ECS service that you just deployed. You can find the public address of the service on the "Networking" tab, under the DNS names section. Click "open address" to open this URI in your browser.
You will see a message:
Service is healthy
This response is coming from NGINX, which is serving the contents of the index.html
file.
Now try sending a request to the same URI but add /api/users/1
to the end of the URI. You will see a response like:
User 1 found!
This response is coming from the application container, via NGINX. The NGINX reverse proxy has forwarded the request to the app container since it matched the pattern /api
, and then returned the application container's response to the client.
Try sending a request to a URL like /web/phpmyadmin
. You will see a 404 Not Found
message coming back from NGINX. The reverse proxy has answered the request without burdening the application container at all.
Tear it down
When you are done experimenting with this stack you can run the following command to tear everything done:
sam delete --stack-name nginx-reverse-proxy
See Also
- Consider using a serverless API Gateway Ingress instead of Application Load Balancer.
- Add target tracking auto scaling to your service, so that it can handle bursts of traffic better.