Schedule a containerized cron job using Amazon ECS and AWS Copilot

About

cron is a command line tool and scheduling system built into Unix-like operating systems, to help you schedule commands to run periodically. You can schedule commands to run on on an interval, or at fixed dates and times.

Amazon EventBridge is a serverless event router and scheduler that can make AWS API calls on a cron schedule.

This pattern will demonstrate how to use Amazon EventBridge to schedule an Amazon ECS RunTask API call to launch a container in AWS Fargate. The configuration will be setup using AWS Copilot, the official command line tool for Amazon ECS.

This example is ideal for implementing:

• An ETL (Extract, Transform, Load) job for syncing production data to an database used for running reporting queries
• Generating business reports on a regular basis
• Running cleanup jobs or other similar workloads on a regular interval

Dependencies

In order to use this pattern you will need the following dependencies:

• Docker or other OCI compatible container builder.
• AWS Copilot CLI - Follow the install instructions for your system.

Architecture

The following diagram shows the architecture that will be deployed:

1. An Amazon EventBridge rule is created which has the cron schedule or interval at which you want to run your container.
2. The rule invokes an AWS Step Functions workflow that calls Amazon ECS RunTask API and handles retries and timeouts
3. Amazon ECS RunTask launches the container of your choice on AWS Fargate capacity.

This architecture is designed to be serverless and not have a single point of failure. Unlike a traditional cron job that is tied to a specific instance in a specific availability zone, all components of this architecture are multi AZ. EventBridge and Step Functions utilize multiple AZ’s for high availability, and AWS Fargate is configured to span multiple AZ’s in a VPC so that it can launch your container with high availability.

Instructions

First we need to create a Dockerfile to define the cron job to run. To simulate a batch job you can use the following basic Dockerfile which does nothing but sleep for one hour:

FROM public.ecr.aws/docker/library/busybox:stable
CMD sh -c "echo 'Sleeping one hour' && sleep 3600"

To begin deploying the cron job using Copilot run the following command:

copilot init

Copilot will ask what you’d like to name your application. Type cron-app.

What would you like to name your application? [? for help] cron-app

Next Copilot will ask what type of workload you are deploying. Choose Scheduled Job:

  Which workload type best represents your architecture?  [Use arrows to move, type to filter, ? for more help]
    Request-Driven Web Service  (App Runner)
    Load Balanced Web Service   (Internet to ECS on Fargate)
    Backend Service             (ECS on Fargate)
    Worker Service              (Events to SQS to ECS on Fargate)
  > Scheduled Job               (Scheduled event to State Machine to Fargate)

As the name of the job type cron-job.

What do you want to name this job? [? for help] cron-job

AWS Copilot will ask what Dockerfile to run. Use the example Dockerfile from above, or supply your own custom Dockerfile.

Which Dockerfile would you like to use for cron-job?  [Use arrows to move, type to filter, ? for more help]
> ./Dockerfile
  Enter custom path for your Dockerfile
  Use an existing image instead

To demonstrate job scheduling choose Rate.

  How would you like to schedule this job?  [Use arrows to move, type to filter, ? for more help]
  > Rate
    Fixed Schedule

Copilot will ask what rate you’d like to run the job at. You can type 1h for 1 hour interval between running jobs.

  How long would you like to wait between executions? [? for help] (1h30m) 1h

Last but not least type y when Copilot asks if you are ready to deploy a test environment.

 Would you like to deploy a test environment? [? for help] (y/N)

Manifest

Take a look at the generated manifest file copilot/cron-job/manifest.yml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

# The manifest for the "cron-job" job.
# Read the full specification for the "Scheduled Job" type at:
#  https://aws.github.io/copilot-cli/docs/manifest/scheduled-job/

# Your job name will be used in naming your resources like log groups, ECS Tasks, etc.
name: cron-job
type: Scheduled Job

# Trigger for your task.
on:
  # The scheduled trigger for your job. You can specify a Unix cron schedule or keyword (@weekly) or a rate (@every 1h30m)
  # AWS Schedule Expressions are also accepted: https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html
  schedule: "@every 1h"
retries: 3        # Optional. The number of times to retry the job before failing.
timeout: 1h       # Optional. The timeout after which to stop the job if it's still running. You can use the units (h, m, s).

# Configuration for your container and task.
image:
  # Docker build arguments. For additional overrides: https://aws.github.io/copilot-cli/docs/manifest/scheduled-job/#image-build
  build: Dockerfile

cpu: 256       # Number of CPU units for the task.
memory: 512    # Amount of memory in MiB used by the task.

# Optional fields for more advanced use-cases.
#
#variables:                    # Pass environment variables as key value pairs.
#  LOG_LEVEL: info

#secrets:                      # Pass secrets from AWS Systems Manager (SSM) Parameter Store.
#  GITHUB_TOKEN: GITHUB_TOKEN  # The key is the name of the environment variable, the value is the name of the SSM parameter.

# You can override any of the values defined above by environment.
#environments:
#  prod:
#    cpu: 2048               # Larger CPU value for prod environment.

Some extra configuration that you can customize:

• retries - If the job fails to start up as expected, how many times would you like to retry.
• timeout - A maximum lifespan for the task if it is taking too long. You can use this to prevent multiple copies of the job from running parallel with each other if the duration of one job extends past the interval that would trigger the next job.
• cpu and memory - How much resources your cron job requires.

Extra commands

If you would like to invoke a job instantly you can do so with:

copilot job run

View the logs for your job with:

copilot job logs

Cleanup

Run the following command to tear down the pattern:

copilot job delete