Enable ENI trunking for Amazon ECS, using a CloudFormation custom resource
Terminology and Background
Amazon Elastic Container Service (ECS) is an orchestrator that launches and manages application containers on your behalf. It deploys fleets of application containers as tasks across a wide range of compute capacity types, including Amazon EC2.
Amazon Virtual Private Cloud (VPC) is how you launch AWS resources in a logically isolated virtual network.
An elastic network interface (ENI) is a logical networking component in a VPC that represents a virtual network card. Multiple ENI's can be attached to a single Amazon EC2 instance, so that the EC2 instance can have multiple IP addresses in a VPC.
ECS integrates with VPC so that you can give each ECS task it's own IP address in a VPC. When you enable awsvpc
networking mode ECS automatically provisions ENI's for your containers, and attaches these ENI's to EC2 instances that are hosting your application containers.
This pattern will demonstrate how to extend the capabilities of awsvpc
networking mode by additionally enabling ENI trunking for awsvpc
networking mode.
To make the solution repeatable and reusable, you will deploy the ENI trunking setting using an AWS CloudFormation custom resource.
Why?
In awsvpc
networking mode, each of your application containers gets it's own unique ENI. However, there is a limit on how many ENI's can be attached to an EC2 instance. This means that there is a limit on how many container tasks ECS can launch on an EC2 instance. Depending on the size of your ECS task this may prevent you from acheiving enough task density to efficiently make use of the available CPU and memory capacity of the EC2 instance.
ENI trunking dramatically increases the number of container tasks ECS can launch per EC2 instance. In ENI trunking mode only two ENI's are consumed per EC2 host. One ENI is used by the EC2 host itself, and one trunk ENI is shared by all containers on the host. You can find a comparison table of task density per EC2 host with and without ENI trunking, in the ENI trunking documentation.
The ENI trunking setting can be a bit challenging to turn on, because it must be set by either the root account, or by the IAM role of the EC2 instances that will have the ENI trunking enabled. Therefore, in order to control the setting through CloudFormation it is necessary to use a CloudFormation custom resource.
Architecture
The following diagram illustrates how this custom resource solution works:
- An AWS Lambda function is used to power a CloudFormation custom resource that can update the ECS settings for an IAM role.
- Two IAM roles are created:
EC2Role
- A role for EC2 instances to use when joining an ECS clusterCustomEniTrunkingRole
- A role for the Lambda function to use
CustomEniTrunkingRole
is granted the ability to assumeEC2Role
. This allows the Lambda function to assume the EC2 instance's role.EC2Role
is granted the ability to call the ECSPutAccountSetting
API.- When the CloudFormation stack is deployed, the Lambda function that powers the custom resource uses it's
CustomEniTrunkingRole
to assumeEC2Role
. Then it uses theEC2Role
to call the ECSPutAccountSetting
API to turn on ENI trunking. - Later when an EC2 instance bearing the
EC2Role
gets launched, it will automatically enable ENI trunking on itself, and begin assigningawsvpc
container ENI's on a shared trunk ENI.
Custom Resource
The following CloudFormation template deploys the ENI trunking setting for an IAM role that could be used by EC2 instances joining an ECS cluster:
AWSTemplateFormatVersion: '2010-09-09'
Description: CloudFormation custom resource example, which turns on ENI trunking for the
EC2 Role that is used by EC2 instances that join the ECS cluster
Resources:
# Turn on ENI trunking for the EC2 instances. This setting is not on by default,
# but it is highly important for increasing the density of AWS VPC networking mode
# tasks per instance. Additionally, it is not controllable by default in CloudFormation
# because it has some complexity of needing to be turned on by a bearer of the role
# of the EC2 instances themselves. With this custom function we can assume the EC2 role,
# then use that role to call the ecs:PutAccountSetting API in order to enable
# ENI trunking
CustomEniTrunkingFunction:
Type: AWS::Lambda::Function
Properties:
Code:
ZipFile: |
const { ECSClient, PutAccountSettingCommand } = require("@aws-sdk/client-ecs");
const { STSClient, AssumeRoleCommand } = require("@aws-sdk/client-sts");
const response = require('cfn-response');
exports.handler = async function(event, context) {
console.log(event);
if (event.RequestType == "Delete") {
await response.send(event, context, response.SUCCESS);
return;
}
const sts = new STSClient({ region: event.ResourceProperties.Region });
const assumeRoleResponse = await sts.send(new AssumeRoleCommand({
RoleArn: event.ResourceProperties.EC2Role,
RoleSessionName: "eni-trunking-enable-session",
DurationSeconds: 900
}));
// Instantiate an ECS client using the credentials of the EC2 role
const ecs = new ECSClient({
region: event.ResourceProperties.Region,
credentials: {
accessKeyId: assumeRoleResponse.Credentials.AccessKeyId,
secretAccessKey: assumeRoleResponse.Credentials.SecretAccessKey,
sessionToken: assumeRoleResponse.Credentials.SessionToken
}
});
const putAccountResponse = await ecs.send(new PutAccountSettingCommand({
name: 'awsvpcTrunking',
value: 'enabled'
}));
console.log(putAccountResponse);
await response.send(event, context, response.SUCCESS);
};
Handler: index.handler
Runtime: nodejs20.x
Timeout: 30
Role: !GetAtt CustomEniTrunkingRole.Arn
# The role used by the Lambda function that turns on ENI trunking
CustomEniTrunkingRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action:
- sts:AssumeRole
ManagedPolicyArns:
# https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AWSLambdaBasicExecutionRole.html
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
# This allows the Lambda function that backs the CloudFormation custom resources
# to assume the role that is used by the EC2 instances. The Lambda function must
# assume this role because the ecs:PutAccountSetting must be called either
# by the role that the setting is for, or by the root account, and we aren't
# using the root account for CloudFormation.
AllowEniTrunkingRoleToAssumeEc2Role:
Type: AWS::IAM::Policy
Properties:
Roles:
- !Ref CustomEniTrunkingRole
PolicyName: allow-to-assume-ec2-role
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action: sts:AssumeRole
Resource: !GetAtt EC2Role.Arn
# Role for the EC2 hosts. This is the role that allows the ECS agent on the
# EC2 hosts to communciate with the ECS control plane, as well as download the
# container images from ECR to run on your host.
EC2Role:
Type: AWS::IAM::Role
Properties:
Path: /
AssumeRolePolicyDocument:
Statement:
# Allow the EC2 instances to assume this role
- Effect: Allow
Principal:
Service: [ec2.amazonaws.com]
Action: ['sts:AssumeRole']
# Allow the ENI trunking function to assume this role in order to enable
# ENI trunking while operating under the identity of this role
- Effect: Allow
Principal:
AWS: !GetAtt CustomEniTrunkingRole.Arn
Action: ['sts:AssumeRole']
# See reference: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/security-iam-awsmanpol.html#security-iam-awsmanpol-AmazonEC2ContainerServiceforEC2Role
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role
# The ENI trunking function will assume this role and then use
# the ecs:PutAccountSetting to set ENI trunking on for this role
Policies:
- PolicyName: allow-to-modify-ecs-settings
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action: ecs:PutAccountSetting
Resource: '*'
# This is the actual custom resource, which triggers the invocation
# of the Lambda function that enables ENI trunking during the stack deploy
CustomEniTrunking:
Type: Custom::CustomEniTrunking
DependsOn:
- AllowEniTrunkingRoleToAssumeEc2Role
Properties:
ServiceToken: !GetAtt CustomEniTrunkingFunction.Arn
Region: !Ref "AWS::Region"
EC2Role: !GetAtt EC2Role.Arn
Outputs:
EC2Role:
Description: The role used by EC2 instances in the cluster
Value: !Ref EC2Role
Usage Instructions
Deploy the given CloudFormation template using a command similar to this:
aws cloudformation deploy \
--stack-name eni-trunking \
--template-file eni-trunking.yml \
--capabilities CAPABILITY_IAM
Retrieve the name of the EC2 role that was created:
EC2_ROLE=$(aws cloudformation describe-stacks --stack-name eni-trunking --query "Stacks[0].Outputs[?OutputKey=='EC2Role'].OutputValue" --output text) && echo $EC2_ROLE
You can now update your ECS cluster instances to use this IAM role. On boot, the ENI trunking feature will be enabled for any EC2 instances that assigned this IAM role.
WARNING
You will need to launch new EC2 instances for this setting to take effect. Any previously existing EC2 instances registered to ECS will remain as they were. Also see the list of ENI trunking considerations in the official docs.
For an example of ENI trunking in action as part of an end to end ECS deployment, see the pattern: "Amazon ECS Capacity Providers for EC2 instances"