Using ECS Service Extensions to attach a file system volume to a task

About

The ecs-service-extensions package is an extendable plugin system for defining Amazon ECS service deployments in AWS Cloud Development Kit (CDK).

Amazon ECS has a large configuration area, and many different features that can be configured. The goal of ECS Service Extensions is to make smaller, reusable chunks of declarative CDK configuration that can be applied to your service in layers.

This pattern shows a service extension that attaches an Amazon Elastic File System to a task. It configures the volume on both the container as well as the task, and also provisions the appropriate IAM permissions and security group rules to allow communication between the file system and the container.

Architecture

The following diagram shows the architecture of this pattern:

1. The application deployment consists of two NGINX web server containers that run as tasks in AWS Fargate. Traffic can be sent to the containers using an Application Load Balancer.
2. Amazon Elastic Container Service orchestrates attaching a durable storage volume to both containers, at the path /usr/share/nginx/html.
3. Both containers now see and share the same index.html file. Changes to the file are automatically propagated to both containers.
4. We can use the Amazon ECS Exec feature to open a secure shell to a running container and change the contents of index.html, then see the changes propagate to all tasks.

Dependencies

To use this pattern you will need:

• Node.js and NPM (TypeScript will be automatically installed via package.json)
• AWS CLI and the Session Manager plugin for AWS CLI.

Setup Cloud Development Kit

To use this pattern you need Node.js installed. First, ensure that you have Node.js installed on your development machine. Then create the following files:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

{
  "name": "ecs-service-extensions-cdk-efs",
  "version": "1.0.0",
  "description": "A container application with Elastic File System attached",
  "private": true,
  "scripts": {
    "build": "tsc",
    "watch": "tsc -w",
    "cdk": "cdk"
  },
  "author": {
    "name": "Amazon Web Services",
    "url": "https://aws.amazon.com",
    "organization": true
  },
  "license": "Apache-2.0",
  "devDependencies": {
    "@types/node": "^8.10.38",
    "aws-cdk": "2.102.0",
    "typescript": "~4.6.0",
    "ts-node": "^10.9.1"
  },
  "dependencies": {
    "aws-cdk-lib": "2.102.0",
    "constructs": "^10.0.0",
    "@aws-cdk-containers/ecs-service-extensions": "2.0.1-alpha.183"
  }
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

{
  "compilerOptions": {
    "target": "ES2018",
    "module": "commonjs",
    "lib": [
      "es2016",
      "es2017.object",
      "es2017.string"
    ],
    "strict": true,
    "noImplicitAny": true,
    "strictNullChecks": true,
    "noImplicitThis": true,
    "alwaysStrict": true,
    "noUnusedLocals": true,
    "noUnusedParameters": true,
    "noImplicitReturns": true,
    "noFallthroughCasesInSwitch": false,
    "inlineSourceMap": true,
    "inlineSources": true,
    "experimentalDecorators": true,
    "strictPropertyInitialization": false
  }
}

1
2
3

{
  "app": "npx ts-node --prefer-ts-exts index.ts"
}

The files above serve the following purpose:

• package.json - This file is used by NPM or Yarn to identify and install all the required dependencies:
• tsconfig.json - Configures the TypeScript settings for the project:
• cdk.json - Tells CDK what command to run, and provides a place to pass other contextual settings to CDK.

Run the following commands to install dependencies and setup your AWS account for the deployment:

npm install
npm run-script cdk bootstrap

Setup the Service Extensions

We will use the following three files to define three service extensions that will be applied to our service.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75

import ecs = require('aws-cdk-lib/aws-ecs');
import iam = require('aws-cdk-lib/aws-iam');
import efs = require('aws-cdk-lib/aws-efs');
import { Service, ServiceExtension } from '@aws-cdk-containers/ecs-service-extensions';
import { Construct } from 'constructs';

export interface VolumeProperties {
  path: string,
  readonly: boolean
}

// Attach a durable volume to a task, with the IAM permissions and
// security group rules that allow the filesystem to be used
export class DurableVolume extends ServiceExtension {
  private filesystem: efs.FileSystem;
  private path: string;
  private readonly: boolean;

  constructor(props: VolumeProperties) {
    super('durable-volume');

    this.path = props.path;
    this.readonly = props.readonly;
  }

  public prehook(parentService: Service, scope: Construct) {
    this.filesystem = new efs.FileSystem(scope, `${parentService.id}-file-system`, {
      vpc: parentService.vpc,
      lifecyclePolicy: efs.LifecyclePolicy.AFTER_14_DAYS, // files are not transitioned to infrequent access (IA) storage by default
      performanceMode: efs.PerformanceMode.GENERAL_PURPOSE, // default
      outOfInfrequentAccessPolicy: efs.OutOfInfrequentAccessPolicy.AFTER_1_ACCESS, // files are not transitioned back from (infrequent access) IA to primary storage by default
    });
  }

  public useTaskDefinition(taskDefinition: ecs.TaskDefinition): void {
    taskDefinition.addVolume({
      name: 'durable-volume',
      efsVolumeConfiguration: {
        fileSystemId: this.filesystem.fileSystemId,
        rootDirectory: '/',
        transitEncryption: 'ENABLED'
      }
    })

    // Add a policy to the task definition allowing it to point the Elastic File System
    const efsMountPolicy = (new iam.PolicyStatement({
      actions: [
        'elasticfilesystem:ClientMount',
        'elasticfilesystem:ClientWrite',
        'elasticfilesystem:ClientRootAccess'
      ],
      resources: [
        this.filesystem.fileSystemArn
      ]
    }))
    taskDefinition.addToTaskRolePolicy(efsMountPolicy)

    const appContainer = taskDefinition.findContainer('app');

    if (!appContainer) {
      throw new Error('Can not add a volume to a task before adding the application container');
    }

    appContainer.addMountPoints({
      containerPath: this.path,
      readOnly: this.readonly,
      sourceVolume: 'durable-volume'
    })
  }

  public useService(service: ecs.Ec2Service | ecs.FargateService): void {
    // Ensure that the service has access to communicate to the filesystem.
    this.filesystem.connections.allowDefaultPortFrom(service);
  }
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

import { ServiceExtension, ServiceBuild } from '@aws-cdk-containers/ecs-service-extensions';

export interface ScalingProperties {
  desiredCount: number
}

// Scales out a service to a static desired count
export class StaticScaleOut extends ServiceExtension {
  private desiredCount: number;

  constructor(props: ScalingProperties) {
    super('static-scale');

    this.desiredCount = props.desiredCount;
  }

  public modifyServiceProps(props: ServiceBuild): ServiceBuild {
    return {
      ...props,

      desiredCount: this.desiredCount
    } as ServiceBuild;
  }
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

import { ServiceExtension, ServiceBuild } from '@aws-cdk-containers/ecs-service-extensions';

// Enables the ECS Exec feature on an ECS Service
export class Exec extends ServiceExtension {
  constructor() {
    super('ecs-exec');
  }

  public modifyServiceProps(props: ServiceBuild): ServiceBuild {
    return {
      ...props,
      enableExecuteCommand: true
    } as ServiceBuild;
  }
}

These extensions serve the following purpose:

• DurableVolume - This extension configures an Elastic File System and attaches it to the ECS task
• StaticScale - This extension scales the service to a static size of two deployed tasks
• Exec - This extension enables ECS Exec so that we can open an interactive shell to a container in a task

Create the CDK App

Now create the following file to define the basic CDK application:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

import ecs = require('aws-cdk-lib/aws-ecs');
import cdk = require('aws-cdk-lib');
import {
  Container,
  Environment,
  HttpLoadBalancerExtension,
  Service,
  ServiceDescription
} from '@aws-cdk-containers/ecs-service-extensions';
import { DurableVolume } from './efs-volume';
import { StaticScaleOut } from './static-scale';
import { Exec } from './ecs-exec';

const app = new cdk.App();
const stack = new cdk.Stack(app, 'efs-sample');

// Create an environment to deploy a service in.
const environment = new Environment(stack, 'production');

// Build out the service description
const description = new ServiceDescription();

// Define the container for the service.
description.add(new Container({
  cpu: 1024,
  memoryMiB: 2048,
  trafficPort: 80,
  image: ecs.ContainerImage.fromRegistry("public.ecr.aws/ecs-sample-image/amazon-ecs-sample"),
}));

// Create a load balancer and attach it to the
// container's traffic port.
description.add(new HttpLoadBalancerExtension());
description.add(new DurableVolume({
  path: '/srv',
  readonly: false
}));
description.add(new StaticScaleOut({
  desiredCount: 2
}))
description.add(new Exec())

// Use the service description to make a service
// inside of the environment.
new Service(stack, 'ecs-sample', {
  environment: environment,
  serviceDescription: description,
});

app.synth();

This file attaches the service extensions to a ServiceDescription and launches it into a Service running inside of an Environment.

Deploy it all

Once you have all the files setup, it is time to deploy them. Use the following commands to preview and then deploy the CDK application:

npx cdk diff
npx cdk deploy

Once the CDK deploy completes you will see CDK output a URL that looks similar to this:

Outputs:
efs-sample.ecssampleloadbalancerdnsoutput = efs-s-ecssa-FM5O3208L1OY-1442904617.us-east-2.elb.amazonaws.com

You can load this URL up in your browser to verify that the application is running. All you should see at this point is an Amazon ECS logo though.

Hydrate the durable storage volume

Right now the durable storage volume is attached to /srv inside of the container and it is empty. Let’s fix that.

Run the following command locally to open a shell to an running instance of the container. Note that you will need to open up the AWS ECS console to locate the cluster name and the ID of a running task from the service:

aws ecs execute-command \
  --cluster <insert cluster name here> \
  --task <insert task ID here> \
  --container app \
  --interactive \
  --command "/bin/sh"

💡 Tip: If you get an error that says SessionManagerPlugin is not found, please install the Session Manager plugin for AWS CLI.

Once the shell opens you can start to run commands inside of the remote container. Use the following commands to create an index.html file inside of the /srv folder that is the durable filesystem volume:

cd /srv
echo "Hello world" > index.html

If you would like to, you can open a second shell to the second container now and verify that the content has been synced over to the second container.

Move the volume into place

Now that the durable volume is hydrated with some content, we can move it into place inside of the image so that you see that content when you load up the URL of the service. Update index.ts by changing the DurableStorage extension configuration to look like this:

description.add(new DurableVolume({
  path: '/usr/share/nginx/html',
  readonly: false
}));

Then rerun the following commands:

npx cdk diff
npx cdk deploy

This is redeploying the service, with the persistent storage volume attached to the web server hosting path. When the deployment completes you can load up the URL of the service in your browser, refresh once or twice to clear the browser cache, and you will see a “Hello world” message instead of the Amazon ECS logo.

At this point you can do live edits to the contents EFS volume and those changes will sync to the web server instances automatically. You can also stop and restart the web server instances and your changes to the durable volume will be persisted.

Tear it down

When you are done experimenting you can destroy the created resources with the following command:

npx cdk destroy

⚠️ Warning: After the CDK destroy has completed you must go to the Amazon EFS console to delete the elastic filesystem. By default, CDK does not destroy filesystems or other stores of durable state, in order to avoid accidental data loss.