Enable ENI trunking for Amazon ECS, using a CloudFormation custom resource

CloudFormation custom resource that adjusts the ENI trunking setting for the EC2 role of Amazon ECS hosts.

Nathan Peck

Senior Developer Advocate at AWS

Terminology and Background

Amazon Elastic Container Service (ECS) is an orchestrator that launches and manages application containers on your behalf. It deploys fleets of application containers as tasks across a wide range of compute capacity types, including Amazon EC2.

Amazon Virtual Private Cloud (VPC) is how you launch AWS resources in a logically isolated virtual network.

An elastic network interface (ENI) is a logical networking component in a VPC that represents a virtual network card. Multiple ENI’s can be attached to a single Amazon EC2 instance, so that the EC2 instance can have multiple IP addresses in a VPC.

ECS integrates with VPC so that you can give each ECS task it’s own IP address in a VPC. When you enable awsvpc networking mode ECS automatically provisions ENI’s for your containers, and attaches these ENI’s to EC2 instances that are hosting your application containers.

This pattern will demonstrate how to extend the capabilities of awsvpc networking mode by additionally enabling ENI trunking for awsvpc networking mode.

To make the solution repeatable and reusable, you will deploy the ENI trunking setting using an AWS CloudFormation custom resource.

Why?

In awsvpc networking mode, each of your application containers gets it’s own unique ENI. However, there is a limit on how many ENI’s can be attached to an EC2 instance. This means that there is a limit on how many container tasks ECS can launch on an EC2 instance. Depending on the size of your ECS task this may prevent you from achieving enough task density to efficiently make use of the available CPU and memory capacity of the EC2 instance.

ENI trunking dramatically increases the number of container tasks ECS can launch per EC2 instance. In ENI trunking mode only two ENI’s are consumed per EC2 host. One ENI is used by the EC2 host itself, and one trunk ENI is shared by all containers on the host. You can find a comparison table of task density per EC2 host with and without ENI trunking, in the ENI trunking documentation.

The ENI trunking setting can be a bit challenging to turn on, because it must be set by either the root account, or by the IAM role of the EC2 instances that will have the ENI trunking enabled. Therefore, in order to control the setting through CloudFormation it is necessary to use a CloudFormation custom resource.

Architecture

The following diagram illustrates how this custom resource solution works:

An AWS Lambda function is used to power a CloudFormation custom resource that can update the ECS settings for an IAM role.
Two IAM roles are created:
1. EC2Role - A role for EC2 instances to use when joining an ECS cluster
2. CustomEniTrunkingRole - A role for the Lambda function to use
CustomEniTrunkingRole is granted the ability to assume EC2Role. This allows the Lambda function to assume the EC2 instance’s role.
EC2Role is granted the ability to call the ECS PutAccountSetting API.
When the CloudFormation stack is deployed, the Lambda function that powers the custom resource uses it’s CustomEniTrunkingRole to assume EC2Role. Then it uses the EC2Role to call the ECS PutAccountSetting API to turn on ENI trunking.
Later when an EC2 instance bearing the EC2Role gets launched, it will automatically enable ENI trunking on itself, and begin assigning awsvpc container ENI’s on a shared trunk ENI.

Custom Resource

The following CloudFormation template deploys the ENI trunking setting for an IAM role that could be used by EC2 instances joining an ECS cluster:

File: eni-trunking.yml Language: yml

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147


AWSTemplateFormatVersion: '2010-09-09'
Description: CloudFormation custom resource example, which turns on ENI trunking for the
             EC2 Role that is used by EC2 instances that join the ECS cluster

Resources:
  # Turn on ENI trunking for the EC2 instances. This setting is not on by default,
  # but it is highly important for increasing the density of AWS VPC networking mode
  # tasks per instance. Additionally, it is not controllable by default in CloudFormation
  # because it has some complexity of needing to be turned on by a bearer of the role
  # of the EC2 instances themselves. With this custom function we can assume the EC2 role,
  # then use that role to call the ecs:PutAccountSetting API in order to enable
  # ENI trunking
  CustomEniTrunkingFunction:
    Type: AWS::Lambda::Function
    Properties:
      Code:
        ZipFile: |
          const { ECSClient, PutAccountSettingCommand } = require("@aws-sdk/client-ecs");
          const { STSClient, AssumeRoleCommand } = require("@aws-sdk/client-sts");

          const response = require('cfn-response');

          exports.handler = async function(event, context) {
            console.log(event);

            if (event.RequestType == "Delete") {
              await response.send(event, context, response.SUCCESS);
              return;
            }

            const sts = new STSClient({ region: event.ResourceProperties.Region });

            const assumeRoleResponse = await sts.send(new AssumeRoleCommand({
              RoleArn: event.ResourceProperties.EC2Role,
              RoleSessionName: "eni-trunking-enable-session",
              DurationSeconds: 900
            }));

            // Instantiate an ECS client using the credentials of the EC2 role
            const ecs = new ECSClient({
              region: event.ResourceProperties.Region,
              credentials: {
                accessKeyId: assumeRoleResponse.Credentials.AccessKeyId,
                secretAccessKey: assumeRoleResponse.Credentials.SecretAccessKey,
                sessionToken: assumeRoleResponse.Credentials.SessionToken
              }
            });

            const putAccountResponse = await ecs.send(new PutAccountSettingCommand({
              name: 'awsvpcTrunking',
              value: 'enabled'
            }));
            console.log(putAccountResponse);

            await response.send(event, context, response.SUCCESS);
          };
      Handler: index.handler
      Runtime: nodejs20.x
      Timeout: 30
      Role: !GetAtt CustomEniTrunkingRole.Arn

  # The role used by the Lambda function that turns on ENI trunking
  CustomEniTrunkingRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
            Action:
              - sts:AssumeRole
      ManagedPolicyArns:
        # https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AWSLambdaBasicExecutionRole.html
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

  # This allows the Lambda function that backs the CloudFormation custom resources
  # to assume the role that is used by the EC2 instances. The Lambda function must
  # assume this role because the ecs:PutAccountSetting must be called either
  # by the role that the setting is for, or by the root account, and we aren't
  # using the root account for CloudFormation.
  AllowEniTrunkingRoleToAssumeEc2Role:
    Type: AWS::IAM::Policy
    Properties:
      Roles:
        - !Ref CustomEniTrunkingRole
      PolicyName: allow-to-assume-ec2-role
      PolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action: sts:AssumeRole
            Resource: !GetAtt EC2Role.Arn

  # Role for the EC2 hosts. This is the role that allows the ECS agent on the
  # EC2 hosts to communciate with the ECS control plane, as well as download the
  # container images from ECR to run on your host.
  EC2Role:
    Type: AWS::IAM::Role
    Properties:
      Path: /
      AssumeRolePolicyDocument:
        Statement:
          # Allow the EC2 instances to assume this role
          - Effect: Allow
            Principal:
              Service: [ec2.amazonaws.com]
            Action: ['sts:AssumeRole']
          # Allow the ENI trunking function to assume this role in order to enable
          # ENI trunking while operating under the identity of this role
          - Effect: Allow
            Principal:
              AWS: !GetAtt CustomEniTrunkingRole.Arn
            Action: ['sts:AssumeRole']

      # See reference: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/security-iam-awsmanpol.html#security-iam-awsmanpol-AmazonEC2ContainerServiceforEC2Role
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role

      # The ENI trunking function will assume this role and then use
      # the ecs:PutAccountSetting to set ENI trunking on for this role
      Policies:
        - PolicyName: allow-to-modify-ecs-settings
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action: ecs:PutAccountSetting
                Resource: '*'

  # This is the actual custom resource, which triggers the invocation
  # of the Lambda function that enables ENI trunking during the stack deploy
  CustomEniTrunking:
    Type: Custom::CustomEniTrunking
    DependsOn:
      - AllowEniTrunkingRoleToAssumeEc2Role
    Properties:
      ServiceToken: !GetAtt CustomEniTrunkingFunction.Arn
      Region: !Ref "AWS::Region"
      EC2Role: !GetAtt EC2Role.Arn

Outputs:
  EC2Role:
    Description: The role used by EC2 instances in the cluster
    Value: !Ref EC2Role

Usage Instructions

Deploy the given CloudFormation template using a command similar to this:

aws cloudformation deploy \
  --stack-name eni-trunking \
  --template-file eni-trunking.yml \
  --capabilities CAPABILITY_IAM

Retrieve the name of the EC2 role that was created:

EC2_ROLE=$(aws cloudformation describe-stacks --stack-name eni-trunking --query "Stacks[0].Outputs[?OutputKey=='EC2Role'].OutputValue" --output text) && echo $EC2_ROLE

You can now update your ECS cluster instances to use this IAM role. On boot, the ENI trunking feature will be enabled for any EC2 instances that assigned this IAM role.

⚠️ Warning: You will need to launch new EC2 instances for this setting to take effect. Any previously existing EC2 instances registered to ECS will remain as they were. Also see the list of ENI trunking considerations in the official docs.

For an example of ENI trunking in action as part of an end to end ECS deployment, see the pattern: “Amazon ECS Capacity Providers for EC2 instances”

Table of Contents

About

Development Tool

Type