Back to all patterns
About
Development Tool
AWS CloudFormation
Type
Infrastructure Pattern
Feature
Amazon Elastic File System (EFS)
Feature
ECS Exec
License
MIT No Attribution

Add durable storage to an ECS task, with Amazon Elastic File System

Nathan Peck profile picture
Nathan Peck
Senior Developer Advocate at AWS

About ​

In this example you will deploy two NGINX web server tasks that have a shared durable web content folder stored on an Elastic File System. You will also use Amazon ECS Exec to access the containers and verify that data is synced across tasks and persisted across task restarts.

Setup your environment ​

Deploy the CloudFormation ​

File: ecs-mount-efs-storage.ymlLanguage: yml
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::LanguageExtensions
Description: An example of how to provision an Elastic File System and
             mount it to an ECS task running in AWS Fargate

Parameters:
  VpcId:
    Type: AWS::EC2::VPC::Id
    Description: The virtual private network into which to launch all resources
  SubnetOne:
    Type: AWS::EC2::Subnet::Id
    Description: The first subnet across which to distribute the application and
                 EFS access
  SubnetTwo:
    Type: AWS::EC2::Subnet::Id
    Description: The second subnet across which to distribute the application and
                 EFS access

Resources:

  # The ECS cluster that will be controlling the tasks in AWS Fargate
  Cluster:
    Type: AWS::ECS::Cluster

  # The filesystem itself
  EFSFileSystem:
    Type: AWS::EFS::FileSystem
    Properties:
      Encrypted: true
      PerformanceMode: generalPurpose
      ThroughputMode: bursting

  # Mount target allows usage of the EFS inside of subnet one
  EFSMountTargetOne:
    Type: AWS::EFS::MountTarget
    Properties:
      FileSystemId: !Ref EFSFileSystem
      SubnetId: !Ref SubnetOne
      SecurityGroups:
        - !Ref EFSFileSystemSecurityGroup

  # Mount target allows usage of the EFS inside of subnet two
  EFSMountTargetTwo:
    Type: AWS::EFS::MountTarget
    Properties:
      FileSystemId: !Ref EFSFileSystem
      SubnetId: !Ref SubnetTwo
      SecurityGroups:
        - !Ref EFSFileSystemSecurityGroup

  # This security group is used by the mount targets so
  # that they will allow inbound NFS connections from
  # the AWS Fargate tasks that we launch
  EFSFileSystemSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Security group for EFS file system
      VpcId: !Ref VpcId
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 2049
          ToPort: 2049
          SourceSecurityGroupId: !Ref ServiceSecurityGroup

  # This role is used to setup the execution environment for
  # the task, in this case to connect to the Elastic File System
  TaskExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Principal:
              Service: [ecs-tasks.amazonaws.com]
            Action: ['sts:AssumeRole']
            Condition:
              ArnLike:
                aws:SourceArn: !Sub arn:aws:ecs:${AWS::Region}:${AWS::AccountId}:*
              StringEquals:
                aws:SourceAccount: !Ref AWS::AccountId
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
      Policies:
        - PolicyName: EFSAccess
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - elasticfilesystem:ClientMount
                  - elasticfilesystem:ClientWrite
                  - elasticfilesystem:DescribeMountTargets
                  - elasticfilesystem:DescribeFileSystems
                Resource: !GetAtt EFSFileSystem.Arn

  # This role is used at runtime.
  TaskRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Principal:
              Service: [ecs-tasks.amazonaws.com]
            Action: ['sts:AssumeRole']
            Condition:
              ArnLike:
                aws:SourceArn: !Sub arn:aws:ecs:${AWS::Region}:${AWS::AccountId}:*
              StringEquals:
                aws:SourceAccount: !Ref AWS::AccountId
      Policies:
        - PolicyName: ExecAccess
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - ssmmessages:CreateControlChannel
                  - ssmmessages:CreateDataChannel
                  - ssmmessages:OpenControlChannel
                  - ssmmessages:OpenDataChannel
                Resource: '*'

  # Store the logs from the task for inspection and review
  # for up to 7 days
  EfsTaskLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      RetentionInDays: 7

  # This task definition describes how to launch the application, and how
  # to mount the Elastic File System into the container
  TaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      Family: my-ecs-task
      TaskRoleArn: !GetAtt TaskRole.Arn
      ExecutionRoleArn: !GetAtt TaskExecutionRole.Arn
      NetworkMode: awsvpc
      ContainerDefinitions:
        - Name: nginx
          Image: public.ecr.aws/nginx/nginx:latest
          Essential: true
          LinuxParameters:
            InitProcessEnabled: true
          MountPoints:
            - SourceVolume: efs-volume
              ContainerPath: /usr/share/nginx/html
          LogConfiguration:
            LogDriver: awslogs
            Options:
              mode: non-blocking
              max-buffer-size: 25m
              awslogs-group: !Ref EfsTaskLogGroup
              awslogs-region: !Ref AWS::Region
              awslogs-stream-prefix: efs-task
          PortMappings:
            - ContainerPort: 80
              Protocol: tcp
      Volumes:
        - Name: efs-volume
          EFSVolumeConfiguration:
            FilesystemId: !Ref EFSFileSystem
            RootDirectory: /
            TransitEncryption: ENABLED
      RequiresCompatibilities:
        - FARGATE
      Cpu: '256'
      Memory: '512'

  # Security group that the task will use to run
  ServiceSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Security group for service
      VpcId: !Ref VpcId

  # Launch multiple copies of the task as a service
  Service:
    Type: AWS::ECS::Service
    DependsOn:
      # This ensures that the service doesn't
      # try to start tasks before the EFS filesystem is
      # actually available in the VPC
      - EFSMountTargetOne
      - EFSMountTargetTwo
    Properties:
      ServiceName: 'ecs-efs-demo'
      Cluster: !Ref Cluster
      LaunchType: FARGATE
      DeploymentConfiguration:
        MaximumPercent: 200
        MinimumHealthyPercent: 75
      DesiredCount: 2
      EnableExecuteCommand: true
      NetworkConfiguration:
        AwsvpcConfiguration:
          AssignPublicIp: ENABLED
          SecurityGroups:
            - !Ref ServiceSecurityGroup
          Subnets:
            - !Ref SubnetOne
            - !Ref SubnetTwo
      TaskDefinition: !Ref 'TaskDefinition'

This CloudFormation template provisions an Elastic File System (EFS) and mounts it to an ECS task running in AWS Fargate.

The template begins with defining parameters that will be passed into the CloudFormation stack. The stack requires the following three parameters:

  • VpcId - A virtual private cloud ID. This can be the default VPC that comes with your AWS account. Example value: vpc-79508710
  • SubnetOne - A public subnet inside of that VPC. Example value: subnet-b4676dfe
  • SubnetTwo - Another public subnet inside of that VPC. Example value: subnet-c71ebfae

TIP

When using the CloudFormation web console it will suggest appropriate parameter values in the drop down on each parameter field, if you are not certain what to enter for each parameter.

The CloudFormation stack deploys the following resources:

  • Cluster: an ECS cluster that will be controlling the tasks in AWS Fargate.
  • EFSFileSystem: the EFS filesystem itself with properties including encryption, performance mode, and throughput mode.
  • EFSMountTargetOne and EFSMountTargetTwo: mount targets that allow usage of the EFS inside subnet one and subnet two, respectively.
  • EFSFileSystemSecurityGroup: a security group used by the mount targets that allows inbound NFS connections from the AWS Fargate tasks launched.
  • TaskExecutionRole: a role used to setup the execution environment for the task, including connecting to the EFS.
  • TaskRole: a role used by the containerized task at runtime.
  • EfsTaskLogGroup: a log group to store the logs from the task for up to 7 days.
  • TaskDefinition: describes how to launch the application and how to mount the Elastic File System into the container, with properties including the task role ARN, execution role ARN, network mode, container definitions, volumes, and requires compatibilities.

This CloudFormation template creates a complete infrastructure to run an application on AWS Fargate, which requires shared, durable file persistence.

Deploy the CloudFormation template by using the web console, or with the following AWS CLI command. (Substitute your own VPC details.)

Language: sh
aws cloudformation deploy \
  --template-file ecs-mount-efs-storage.yml \
  --stack-name test-stack \
  --capabilities CAPABILITY_IAM \
  --parameter-overrides VpcId=vpc-79508710 SubnetOne=subnet-b4676dfe SubnetTwo=subnet-c71ebfae

Test out Elastic File System ​

First open an ECS Exec session to each running container using a command like this:

Language: sh
aws ecs execute-command \
  --cluster <insert cluster name here> \
  --task <insert task ID here> \
  --container nginx \
  --interactive \
  --command "/bin/sh"

Run the above command twice in two different terminals, and with two different task ID's so that you have one session open to each task.

In each session type:

Language: sh
cd /usr/share/nginx/html
ls

This will verify that both directories are empty.

Try running the following command in both containers:

Language: sh
curl localhost:80

You will see a 403 Forbidden error from NGINX because there is no content in the shared HTML directory.

Run the following command in only one of the open sessions:

Language: sh
echo "Hello world" > index.html

In the other container's session run ls to see that an index.html file has appeared.

Run the curl command again in both containers:

Language: sh
curl localhost:80

Observe that both NGINX web servers are seeing the same web content over the Elastic File System.

Test data durability ​

Terminate both tasks using the Amazon ECS web console, or API.

When the ECS service starts a replacement task, open a new ECS Exec session to the replacement task.

Run the following commands again in this replacement task:

Language: sh
cd /usr/share/nginx/html
ls
curl localhost:80

You will see that the index.html file you wrote to the Elastic File System in the previous instance of the container has been persisted and is still there in the new replacement task.

See Also ​

Edit this page on Github
Last Updated

Alternative Patterns

Not quite right for you? Try another way to do this:

AWS Cloud Development Kit (CDK)  Durable storage volume for AWS Fargate, using Cloud Development Kit (CDK)

AWS Cloud Development Kit is better if you don't like writing CloudFormation YAML by hand, and would like to have a higher level SDK that writes CloudFormation for you automatically.

AWS Copilot CLI  Launch a task with durable storage, using AWS Copilot

AWS Copilot is a command line tool for developers that deploys prebuilt, production-ready CloudFormation templates that were written by AWS engineers.