AWS FireLens with ECS Service Extensions

About ​

The ecs-service-extensions package is an extendable plugin system for defining Amazon ECS service deployments in AWS Cloud Development Kit. To learn the basics of ecs-service-extensions refer to "Load balanced container service with ECS Service Extensions".

AWS FireLens is built-in to Amazon ECS to help you get high performance log routing, filtering, and splitting via Fluent Bit. FireLens also enhances your logs with ECS specific metadata such as which task definition version, which cluster, and which task ARN the log originated from. For more info on FireLens read "Under the hood: FireLens for Amazon ECS Tasks", which goes into depth about FireLens features, with performance benchmark comparisons.

This pattern shows how to use ecs-service-extensions to add AWS FireLens to a service, then verify that the logs are being routed with FireLens.

Setup Cloud Development Kit ​

To use this pattern you need TypeScript and Node. First, ensure that you have Node.js installed on your development machine. Then create the following files:

{
  "name": "ecs-service-extensions-firelens-cdk",
  "version": "1.0.0",
  "description": "A container application with logs routed via AWS FireLens",
  "private": true,
  "scripts": {
    "build": "tsc",
    "watch": "tsc -w",
    "cdk": "cdk"
  },
  "author": {
    "name": "Amazon Web Services",
    "url": "https://aws.amazon.com",
    "organization": true
  },
  "license": "Apache-2.0",
  "devDependencies": {
    "@types/node": "^8.10.38",
    "aws-cdk": "2.102.0",
    "typescript": "~4.6.0",
    "ts-node": "^10.9.1"
  },
  "dependencies": {
    "aws-cdk-lib": "2.102.0",
    "constructs": "^10.0.0",
    "@aws-cdk-containers/ecs-service-extensions": "2.0.1-alpha.183"
  }
}

The files above serve the following purpose:

• package.json - This file is used by NPM or Yarn to identify and install all the required dependencies:
• tsconfig.json - Configures the TypeScript settings for the project:
• cdk.json - Tells CDK what command to run, and provides a place to pass other contextual settings to CDK.

Run the following commands to install dependencies and setup your AWS account for the deployment:

npm install
npm run-script cdk bootstrap

Create the CDK App ​

Now create the following file to define a basic CDK application:

import ecs = require('aws-cdk-lib/aws-ecs');
import cdk = require('aws-cdk-lib');
import {
  Container,
  Environment,
  HttpLoadBalancerExtension,
  FireLensExtension,
  Service,
  ServiceDescription
} from '@aws-cdk-containers/ecs-service-extensions';

class ECSStack extends cdk.Stack {
  constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // Create an environment to deploy a service in.
    const environment = new Environment(this, 'production');

    // Build out the service description
    const nameDescription = new ServiceDescription();

    // Define the container for the service.
    nameDescription.add(new Container({
      cpu: 1024,
      memoryMiB: 2048,
      trafficPort: 80,
      image: ecs.ContainerImage.fromRegistry('nathanpeck/name'),
      environment: {
        PORT: '80',
      },
    }));

    // Create a load balancer and attach it to the
    // container's traffic port.
    nameDescription.add(new HttpLoadBalancerExtension());

    // Route logs with FireLens
    nameDescription.add(new FireLensExtension());

    // Use the service description to make a service
    // inside of the environment.
    new Service(this, 'name', {
      environment: environment,
      serviceDescription: nameDescription,
    });
  }
}

const app = new cdk.App();
new ECSStack(app, 'ECSStack');
app.synth();

FireLens is added to the deployed application by importing the FireLensExtension class, and then calling ServiceDescription.add() to add an instance of the FireLensExtension to the ServiceDescription.

You can preview and then deploy this CDK application by running:

npm run-script cdk diff
npm run-script cdk deploy

Verify that it worked ​

Open the ECS console and find the running task that was launched. Either view the task details in the console or use the AWS CLI to view task details with the following command:

aws ecs describe-tasks \
   --cluster your-cluster-name-here \
   --tasks your-task-arn-here \
   --query 'tasks[].containers[].{name: name, lastStatus: lastStatus, image: image}'

You will see output similar to this:

[
    {
        "image": "nathanpeck/name",
        "lastStatus": "RUNNING",
        "name": "app"
    },
    {
        "image": "906394416424.dkr.ecr.us-east-2.amazonaws.com/aws-for-fluent-bit:latest",
        "lastStatus": "RUNNING",
        "name": "firelens"
    }
]

This shows that the FireLens container called aws-for-fluent-bit has been launched as a sidecar to accompany your task's application container. Each instance of your application container will be accompanied by a lightweight Fluent Bit process which captures stdout and stderr logs, attaches metadata to them, and routes the logs to CloudWatch.

You can also view the logs for the Fluent Bit sidecar itself in the ECS console. Look for lines similar to this in the Fluent Bit logs:

time="2023-04-20T15:30:41Z" level=info msg="[cloudwatch 0] Log stream name/app-firelens-fcfc4e8175d848e496380e8e8de88e47 does not exist in log group name-logs"	firelens
4/20/2023, 11:30:41 AM	time="2023-04-20T15:30:41Z" level=info msg="[cloudwatch 0] Created log stream name/app-firelens-fcfc4e8175d848e496380e8e8de88e47 in group name-logs"

FireLens has automatically created a CloudWatch log group for the application, and has begun piping logs into it. Find the referenced log group and open it in CloudWatch to see the logs within. You should see a log output similar to this:

{
    "container_id": "fcfc4e8175d848e496380e8e8de88e47-527074092",
    "container_name": "app",
    "ecs_cluster": "ECSStack-productionenvironmentclusterC6599D2D-oR4P0udMijYx",
    "ecs_task_arn": "arn:aws:ecs:us-east-2:209640446841:task/ECSStack-productionenvironmentclusterC6599D2D-oR4P0udMijYx/fcfc4e8175d848e496380e8e8de88e47",
    "ecs_task_definition": "ECSStacknametaskdefinitionCB1493C3:2",
    "log": "Listening on port 80!",
    "source": "stdout"
}

FireLens automatically configures Fluent Bit to attach ECS metadata to each line of log output. You can see whether ECS container logs were written to stdout or stderr, and you can also see rich information about which ECS task version wrote the log line.

This feature of FireLens can be particularly helpful when you are running many containers, iterating fast on pushing out new feature versions, and want to track an error back to its source.

Clean Up ​

Tear down the CDK application stack with the following command:

npm run-script cdk destroy

See Also ​

• More examples for AWS FireLens, including advanced logging configurations
• Learn how to build your own custom ECS Service Extension