NGINX reverse proxy sidecar for a web container hosted with Amazon ECS and AWS Fargate

Nathan Peck profile picture
Nathan Peck
Senior Developer Advocate at AWS

About

NGINX is a high performance HTTP server and reverse proxy which has achieved significant adoption because of its asynchronous event driven architecture which allows it to serve thousands of concurrent requests with very low memory footprint.

Amazon Elastic Container Service (ECS) is a highly scalable, high performance container management service that runs containers on a managed cluster of Amazon EC2 instances, or using serverless AWS Fargate capacity.

This pattern will show how to deploy open source NGINX as a reverse proxy container in front of your application container.

Architecture

This pattern will deploy the following architecture:

Public subnetPublic subnetVPCAvailability Zone 1Availability Zone 2NGINXInternet gatewayApplicationLoad BalancerLoad Balancer has nodes in each AZTrafficAWS FargatePort 80Port 80AppPort 3000NGINXPort 80AppPort 3000

  1. An ECS task runs in AWS Fargate. The task has two containers: an NGINX sidecar, and a simple JavaScript webserver
  2. The task only accepts inbound traffic on the NGINX traffic port. The NGINX server filters out bad traffic, and forwards good traffic to the backend task on it's local port.
  3. The NGINX server responds back to clients, returning the response from the application server. The NGINX server can also transform responses, such as doing compression of plaintext responses. This offloads work from the application itself.

Why use a reverse proxy?

A reverse proxy fetches resources from another server on behalf of a client. One of the challenges of running a web server that serves requests from the public is that you can expect to receive quite a lot of unwanted traffic every day. Some of this traffic is relatively benign scans by researchers and tools such as Shodan or nmap:

Language: txt
[18/May/2017:15:10:10 +0000] "GET /YesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScann HTTP/1.1" 404 1389 - Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36
[18/May/2017:18:19:51 +0000] "GET /clientaccesspolicy.xml HTTP/1.1" 404 322 - Cloud mapping experiment. Contact research@pdrlabs.net

But other traffic is much more malicious. For example here is what a web server sees while being scanned by the hacking tool ZmEu which scans web servers trying to find PHPMyAdmin installations to exploit:

Language: txt
[18/May/2017:16:27:39 +0000] "GET /mysqladmin/scripts/setup.php HTTP/1.1" 404 391 - ZmEu
[18/May/2017:16:27:39 +0000] "GET /web/phpMyAdmin/scripts/setup.php HTTP/1.1" 404 394 - ZmEu
[18/May/2017:16:27:39 +0000] "GET /xampp/phpmyadmin/scripts/setup.php HTTP/1.1" 404 396 - ZmEu
[18/May/2017:16:27:40 +0000] "GET /apache-default/phpmyadmin/scripts/setup.php HTTP/1.1" 404 405 - ZmEu
[18/May/2017:16:27:40 +0000] "GET /phpMyAdmin-2.10.0.0/scripts/setup.php HTTP/1.1" 404 397 - ZmEu
[18/May/2017:16:27:40 +0000] "GET /mysql/scripts/setup.php HTTP/1.1" 404 386 - ZmEu
[18/May/2017:16:27:41 +0000] "GET /admin/scripts/setup.php HTTP/1.1" 404 386 - ZmEu
[18/May/2017:16:27:41 +0000] "GET /forum/phpmyadmin/scripts/setup.php HTTP/1.1" 404 396 - ZmEu
[18/May/2017:16:27:41 +0000] "GET /typo3/phpmyadmin/scripts/setup.php HTTP/1.1" 404 396 - ZmEu
[18/May/2017:16:27:42 +0000] "GET /phpMyAdmin-2.10.0.1/scripts/setup.php HTTP/1.1" 404 399 - ZmEu
[18/May/2017:16:27:44 +0000] "GET /administrator/components/com_joommyadmin/phpmyadmin/scripts/setup.php HTTP/1.1" 404 418 - ZmEu
[18/May/2017:18:34:45 +0000] "GET /phpmyadmin/scripts/setup.php HTTP/1.1" 404 390 - ZmEu
[18/May/2017:16:27:45 +0000] "GET /w00tw00t.at.blackhats.romanian.anti-sec:) HTTP/1.1" 404 401 - ZmEu

In addition to hacking tools scanning your servers you can also end up receiving unwanted web traffic that was intended for another server. In a dynamic cloud environment your application may end up taking over a public IP address that was formerly connected to another service. When this happens its not uncommon for misconfigured or misbehaving DNS servers to result in traffic that was intended to go to a completely different host to continue to be directed to an IP address which is now connected to your own server.

Anyone running a web server connected to the internet has to assume the responsibility of handling and rejecting potentially malicious traffic or unwanted traffic. Ideally the web server is capable of rejecting this traffic as early as possible, before it actually reaches your core application code. A reverse proxy is one way to provide an extra layer of protection for your application server. It can be configured to reject these requests before they reach your application server.

Another potential benefit of using a reverse proxy is that you can offload some static responses from the application itself. In this pattern you will notice that the healthcheck requests that the Application Load Balancer sends to the task are also being offloaded onto NGINX instead of going all the way to the application code. You could use a similar approach to host your own static HTML webpage, or other static content that you wish to serve to the internet.

Dependencies

This pattern requires that you have an AWS account and the following tools locally:

  • Docker or similar OCI compatible container image builder
  • AWS Serverless Application Model (SAM) CLI - If not already installed then please install SAM CLI for your system.

Build the application

Create a folder app and put the following files into the folder. These three files define a basic JavaScript application that implements an API:

  • index.js
  • package.json
  • Dockerfile
File: index.jsLanguage: js
// Express web server for an API
const express = require('express');
const app = express();
const port = 3000;

// Healthcheck endpoint
app.get('/', (req, res) => {
  res.send('Hello World!');
})

// Get user by ID endpoint
app.get('/api/users/:id', (req, res) => {
  const userId = req.params.id;
  res.send(`User ${userId} found!`);
})

// Get user by username endpoint
app.get('/api/users/by/username/:username', (req, res) => {
  const username = req.params.username;
  res.send(`User ${username} found!`);
})

app.listen(port, () => {
  console.log(`Running on port ${port}...\n`);
})

You should now have an app directory with three files in it:

  • index.js - The application code that the web server will run
  • package.json - Describes dependencies for the application code
  • Dockerfile - Describes how to package up the application code and it's dependencies into a container image.

You can now build and push the container image to AWS by running the following commands in the app directory.

Language: sh
APP_URI=$(aws ecr create-repository --repository-name app --query 'repository.repositoryUri' --output text)
docker build -t $APP_URI .
docker push $APP_URI

Build the NGINX proxy

Next we need to build a customized NGINX reverse proxy with the configuration that will forward requests to the application, while rejecting unwanted requests.

Create a folder nginx and put the following files in the folder:

  • nginx.conf
  • Dockerfile
  • index.html
File: nginx.confLanguage: nginx
events {
  worker_connections 1024;
}

http {
  # NGINX will handle gzip compression of responses from the app server
  gzip on;
  gzip_proxied any;
  gzip_types text/plain application/json;
  gzip_min_length 1000;

  server {
    listen 80;

    # NGINX will reject anything not matching /api
    location /api {
      # Reject requests with unsupported HTTP method
      if ($request_method !~ ^(GET|POST|HEAD|OPTIONS|PUT|DELETE)$) {
        return 405;
      }

      # Only requests matching the expectations will
      # get sent to the application server
      proxy_pass http://localhost:3000;
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection 'upgrade';
      proxy_set_header Host $host;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_cache_bypass $http_upgrade;
    }
  }
}

You should now have an nginx directory with two files in it:

  • nginx.conf - Defines the proxy configuration
  • index.html - Basic file that we will use as a healthcheck response
  • Dockerfile - Defines how to apply the proxy configuration on top of a generic NGINX image from the Elastic Conatiner Registry Public Gallery.

Some things to note in the nginx.conf:

  1. Lines 3-6 tell NGINX to compress outgoing content. This allows your application to just return plaintext responses, and offload compression onto the NGINX sidecar.
  2. Line 12 limits what traffic paths will be forwarded to the application. The application will only receive requests that match /api*. All other requests will be rejected and the response will be returned directly from the NGINX container, without your application ever being touched.
  3. Lines 14-15 will reject a variety of malformed requests that don't match known HTTP methods
  4. Lines 20-26 control how traffic is forwarded to the application container.

You can now build and push the container image to AWS by running the following commands in the nginx directory.

Language: sh
NGINX_URI=$(aws ecr create-repository --repository-name nginx --query 'repository.repositoryUri' --output text)
docker build -t $NGINX_URI .
docker push $NGINX_URI

Define the network

This pattern can be deployed on top of either of the following VPC patterns:

Which one you choose depends on your goals for this deployment. You can choose the low cost VPC to start with and upgrade to the large sized VPC later on if you have additional private services, or private database servers you wish to deploy in the VPC.

If you have any doubts as to which VPC to choose, then go with the "Low cost VPC" option.

Download the vpc.yml file from your chosen pattern, but do not deploy it yet. Deployment will be done later in the process

Define the cluster

Next we need to define an ECS cluster. For this pattern we will deploy the workload to AWS Fargate:

File: cluster.ymlLanguage: yml
AWSTemplateFormatVersion: '2010-09-09'
Description: Empty ECS cluster that has no EC2 instances. It is designed
             to be used with AWS Fargate serverless capacity

Resources:
  # Cluster that keeps track of container deployments
  ECSCluster:
    Type: AWS::ECS::Cluster
    Properties:
      ClusterSettings:
        - Name: containerInsights
          Value: enabled

  # This is a role which is used within Fargate to allow the Fargate agent
  # to download images, and upload logs.
  ECSTaskExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Principal:
              Service: [ecs-tasks.amazonaws.com]
            Action: ['sts:AssumeRole']
            Condition:
              ArnLike:
                aws:SourceArn: !Sub arn:aws:ecs:${AWS::Region}:${AWS::AccountId}:*
              StringEquals:
                aws:SourceAccount: !Ref AWS::AccountId
      Path: /

      # This role enables basic features of ECS. See reference:
      # https://docs.aws.amazon.com/AmazonECS/latest/developerguide/security-iam-awsmanpol.html#security-iam-awsmanpol-AmazonECSTaskExecutionRolePolicy
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy

Outputs:
  ClusterName:
    Description: The ECS cluster into which to launch resources
    Value: !Ref ECSCluster
  ECSTaskExecutionRole:
    Description: The role used to start up a task
    Value: !Ref ECSTaskExecutionRole

Define the service

Next we need to define the service that will run:

File: service.ymlLanguage: yml
AWSTemplateFormatVersion: '2010-09-09'
Description: An example service that deploys in AWS VPC networking mode
             on AWS Fargate. Service runs with networking in public
             subnets and public IP addresses

Parameters:
  VpcId:
    Type: String
    Description: The VPC that the service is running inside of
  PublicSubnetIds:
    Type: List<AWS::EC2::Subnet::Id>
    Description: List of public subnet ID's to put the load balancer and tasks in
  ClusterName:
    Type: String
    Description: The name of the ECS cluster into which to launch capacity.
  ECSTaskExecutionRole:
    Type: String
    Description: The role used to start up an ECS task
  ServiceName:
    Type: String
    Default: web
    Description: A name for the service
  AppImageUrl:
    Type: String
    Description: The url of a docker image that contains the application process that
                 will handle the traffic for this service
  NginxImageUrl:
    Type: String
    Description: The url of a docker image that provides the NGINX reverse proxy
  ContainerCpu:
    Type: Number
    Default: 1024
    Description: How much CPU to give the container. 1024 is 1 CPU
  ContainerMemory:
    Type: Number
    Default: 2048
    Description: How much memory in megabytes to give the container
  ContainerPort:
    Type: Number
    Default: 80
    Description: What port that the application expects traffic on
  DesiredCount:
    Type: Number
    Default: 2
    Description: How many copies of the service task to run

Resources:

  # The task definition. This is a simple metadata description of what
  # container(s) to run, and what resource requirements the task has
  TaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      Family: !Ref ServiceName
      Cpu: !Ref ContainerCpu
      Memory: !Ref ContainerMemory
      NetworkMode: awsvpc
      RequiresCompatibilities:
        - FARGATE
      ExecutionRoleArn: !Ref ECSTaskExecutionRole
      ContainerDefinitions:
        # NGINX sidecar for the task
        - Name: nginx
          Image: !Ref NginxImageUrl
          PortMappings:
            - ContainerPort: !Ref ContainerPort
              HostPort: !Ref ContainerPort
          LogConfiguration:
            LogDriver: 'awslogs'
            Options:
              mode: non-blocking
              max-buffer-size: 25m
              awslogs-group: !Ref NginxLogGroup
              awslogs-region: !Ref AWS::Region
              awslogs-stream-prefix: !Sub "${ServiceName}/nginx"
        # Application container
        - Name: app
          Image: !Ref AppImageUrl
          LogConfiguration:
            LogDriver: 'awslogs'
            Options:
              mode: non-blocking
              max-buffer-size: 25m
              awslogs-group: !Ref AppLogGroup
              awslogs-region: !Ref AWS::Region
              awslogs-stream-prefix: !Sub "${ServiceName}/app"

  # The service. The service is a resource which allows you to run multiple
  # copies of a type of task, and gather up their logs and metrics, as well
  # as monitor the number of running tasks and replace any that have crashed
  Service:
    Type: AWS::ECS::Service
    # Avoid race condition between ECS service creation and associating
    # the target group with the LB
    DependsOn: PublicLoadBalancerListener
    Properties:
      ServiceName: !Ref ServiceName
      Cluster: !Ref ClusterName
      LaunchType: FARGATE
      NetworkConfiguration:
        AwsvpcConfiguration:
          AssignPublicIp: ENABLED
          SecurityGroups:
            - !Ref ServiceSecurityGroup
          Subnets: !Ref PublicSubnetIds
      DeploymentConfiguration:
        MaximumPercent: 200
        MinimumHealthyPercent: 75
      DesiredCount: !Ref DesiredCount
      TaskDefinition: !Ref TaskDefinition
      LoadBalancers:
        - ContainerName: nginx
          ContainerPort: !Ref ContainerPort
          TargetGroupArn: !Ref ServiceTargetGroup

  # Security group that limits network access
  # to the task
  ServiceSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Security group for service
      VpcId: !Ref VpcId

  # Keeps track of the list of tasks for the service
  ServiceTargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      HealthCheckIntervalSeconds: 6
      HealthCheckPath: /
      HealthCheckProtocol: HTTP
      HealthCheckTimeoutSeconds: 5
      HealthyThresholdCount: 2
      TargetType: ip
      Port: !Ref ContainerPort
      Protocol: HTTP
      UnhealthyThresholdCount: 10
      VpcId: !Ref VpcId
      TargetGroupAttributes:
        - Key: deregistration_delay.timeout_seconds
          Value: 0

  # A public facing load balancer, this is used as ingress for
  # public facing internet traffic.
  PublicLoadBalancerSG:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Access to the public facing load balancer
      VpcId: !Ref VpcId
      SecurityGroupIngress:
        # Allow access to public facing ALB from any IP address
        - CidrIp: 0.0.0.0/0
          IpProtocol: -1
  PublicLoadBalancer:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties:
      Scheme: internet-facing
      LoadBalancerAttributes:
      - Key: idle_timeout.timeout_seconds
        Value: '30'
      Subnets: !Ref PublicSubnetIds
      SecurityGroups:
        - !Ref PublicLoadBalancerSG
  PublicLoadBalancerListener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    Properties:
      DefaultActions:
        - Type: 'forward'
          ForwardConfig:
            TargetGroups:
              - TargetGroupArn: !Ref ServiceTargetGroup
                Weight: 100
      LoadBalancerArn: !Ref 'PublicLoadBalancer'
      Port: 80
      Protocol: HTTP

  # Open up the service's security group to traffic originating
  # from the security group of the load balancer.
  ServiceIngressfromLoadBalancer:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      Description: Ingress from the public ALB
      GroupId: !Ref ServiceSecurityGroup
      IpProtocol: -1
      SourceSecurityGroupId: !Ref 'PublicLoadBalancerSG'

  # This log group stores the stdout logs from the app
  AppLogGroup:
    Type: AWS::Logs::LogGroup

  # This log group stores the access logs from NGINX
  NginxLogGroup:
    Type: AWS::Logs::LogGroup

Things to note in this template:

  • The AWS::ECS::TaskDefinition uses networking mode awsvpc. This means that every task gets it's own private networking interface. Containers that share a task can communicate with each other using localhost. This allows the nginx.conf proxy configuration to work, as it expects the application container to be available at http://localhost:3000.
  • The task has two container definitions. Each container has it's own log group. This allows us to keep the application logs separately from the NGINX access logs.

Deploy it all

In order to deploy everything, we will use a parent stack that defines each of the children stacks and what values they expect.

File: parent.ymlLanguage: yml
AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Description: Parent stack that deploys a serverless AWS Fargate web service.
             The service tasks use an NGINX reverse proxy in front of the application.
Parameters:
  AppImageUrl:
    Type: String
    Description: The url of a docker image that contains the application process that
                 will handle the traffic for this service
  NginxImageUrl:
    Type: String
    Description: The url of a docker image that provides the NGINX reverse proxy

Resources:

  # The networking configuration. This creates an isolated
  # network specific to this particular environment
  VpcStack:
    Type: AWS::Serverless::Application
    Properties:
      Location: vpc.yml

  # This stack contains the Amazon ECS cluster itself
  ClusterStack:
    Type: AWS::Serverless::Application
    Properties:
      Location: cluster.yml

  # This stack contains the container deployment
  ServiceStack:
    Type: AWS::Serverless::Application
    Properties:
      Location: service.yml
      Parameters:
        VpcId: !GetAtt VpcStack.Outputs.VpcId
        PublicSubnetIds: !GetAtt VpcStack.Outputs.PublicSubnetIds
        ClusterName: !GetAtt ClusterStack.Outputs.ClusterName
        ECSTaskExecutionRole: !GetAtt ClusterStack.Outputs.ECSTaskExecutionRole
        AppImageUrl: !Ref AppImageUrl
        NginxImageUrl: !Ref NginxImageUrl

You should have four files:

  • vpc.yml - Defines the networking setup
  • cluster.yml - Defines the Amazon ECS cluster
  • service.yml - Defines the service and task to run
  • parent.yml - Top level stack that deploys the three child stacks.

You can deploy everything with the following command:

Language: sh
sam deploy \
  --template-file parent.yml \
  --stack-name nginx-reverse-proxy \
  --resolve-s3 \
  --capabilities CAPABILITY_IAM \
  --parameter-overrides AppImageUrl=$APP_URI NginxImageUrl=$NGINX_URI

Test it out

Open your Amazon ECS console, and locate the ECS service that you just deployed. You can find the public address of the service on the "Networking" tab, under the DNS names section. Click "open address" to open this URI in your browser.

You will see a message:

Language: txt
Service is healthy

This response is coming from NGINX, which is serving the contents of the index.html file.

Now try sending a request to the same URI but add /api/users/1 to the end of the URI. You will see a response like:

Language: txt
User 1 found!

This response is coming from the application container, via NGINX. The NGINX reverse proxy has forwarded the request to the app container since it matched the pattern /api, and then returned the application container's response to the client.

Try sending a request to a URL like /web/phpmyadmin. You will see a 404 Not Found message coming back from NGINX. The reverse proxy has answered the request without burdening the application container at all.

Tear it down

When you are done experimenting with this stack you can run the following command to tear everything done:

Language: shell
sam delete --stack-name nginx-reverse-proxy

See Also