cron is a command line tool and scheduling system built into Unix-like operating systems, to help you schedule commands to run periodically. You can schedule commands to run on on an interval, or at fixed dates and times.
Amazon EventBridge is a serverless event router and scheduler that can make AWS API calls on a cron schedule.
This pattern will demonstrate how to use Amazon EventBridge to schedule an Amazon ECS
RunTask API call to launch a container in AWS Fargate. The configuration will be setup using AWS Copilot, the official command line tool for Amazon ECS.
This example is ideal for implementing:
- An ETL (Extract, Transform, Load) job for syncing production data to an database used for running reporting queries
- Generating business reports on a regular basis
- Running cleanup jobs or other similar workloads on a regular interval
In order to use this pattern you will need the following dependencies:
- Docker or other OCI compatible container builder.
- AWS Copilot CLI - Follow the install instructions for your system.
The following diagram shows the architecture that will be deployed:
- An Amazon EventBridge rule is created which has the cron schedule or interval at which you want to run your container.
- The rule invokes an AWS Step Functions workflow that calls Amazon ECS
RunTaskAPI and handles retries and timeouts
- Amazon ECS
RunTasklaunches the container of your choice on AWS Fargate capacity.
This architecture is designed to be serverless and not have a single point of failure. Unlike a traditional
cron job that is tied to a specific instance in a specific availability zone, all components of this architecture are multi AZ. EventBridge and Step Functions utilize multiple AZ's for high availability, and AWS Fargate is configured to span multiple AZ's in a VPC so that it can launch your container with high availability.
First we need to create a
Dockerfile to define the cron job to run. To simulate a batch job you can use the following basic Dockerfile which does nothing but sleep for one hour:
CMD sh -c "echo 'Sleeping one hour' && sleep 3600"
To begin deploying the cron job using Copilot run the following command:
Copilot will ask what you'd like to name your application. Type
What would you like to name your application? [? for help] cron-app
Next Copilot will ask what type of workload you are deploying. Choose
Which workload type best represents your architecture? [Use arrows to move, type to filter, ? for more help]
Request-Driven Web Service (App Runner)
Load Balanced Web Service (Internet to ECS on Fargate)
Backend Service (ECS on Fargate)
Worker Service (Events to SQS to ECS on Fargate)
> Scheduled Job (Scheduled event to State Machine to Fargate)
As the name of the job type
What do you want to name this job? [? for help] cron-job
AWS Copilot will ask what
Dockerfile to run. Use the example
Dockerfile from above, or supply your own custom
Which Dockerfile would you like to use for cron-job? [Use arrows to move, type to filter, ? for more help]
Enter custom path for your Dockerfile
Use an existing image instead
To demonstrate job scheduling choose
How would you like to schedule this job? [Use arrows to move, type to filter, ? for more help]
Copilot will ask what rate you'd like to run the job at. You can type
1h for 1 hour interval between running jobs.
How long would you like to wait between executions? [? for help] (1h30m) 1h
Last but not least type
y when Copilot asks if you are ready to deploy a test environment.
Would you like to deploy a test environment? [? for help] (y/N)
Take a look at the generated manifest file
# The manifest for the "cron-job" job.
# Read the full specification for the "Scheduled Job" type at:
# Your job name will be used in naming your resources like log groups, ECS Tasks, etc.
type: Scheduled Job
# Trigger for your task.
# The scheduled trigger for your job. You can specify a Unix cron schedule or keyword (@weekly) or a rate (@every 1h30m)
# AWS Schedule Expressions are also accepted: https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html
schedule: "@every 1h"
retries: 3 # Optional. The number of times to retry the job before failing.
timeout: 1h # Optional. The timeout after which to stop the job if it's still running. You can use the units (h, m, s).
# Configuration for your container and task.
# Docker build arguments. For additional overrides: https://aws.github.io/copilot-cli/docs/manifest/scheduled-job/#image-build
cpu: 256 # Number of CPU units for the task.
memory: 512 # Amount of memory in MiB used by the task.
# Optional fields for more advanced use-cases.
#variables: # Pass environment variables as key value pairs.
# LOG_LEVEL: info
#secrets: # Pass secrets from AWS Systems Manager (SSM) Parameter Store.
# GITHUB_TOKEN: GITHUB_TOKEN # The key is the name of the environment variable, the value is the name of the SSM parameter.
# You can override any of the values defined above by environment.
# cpu: 2048 # Larger CPU value for prod environment.
Some extra configuration that you can customize:
retries- If the job fails to start up as expected, how many times would you like to retry.
timeout- A maximum lifespan for the task if it is taking too long. You can use this to prevent multiple copies of the job from running parallel with each other if the duration of one job extends past the interval that would trigger the next job.
memory- How much resources your cron job requires.
If you would like to invoke a job instantly you can do so with:
copilot job run
View the logs for your job with:
copilot job logs
Run the following command to tear down the pattern:
copilot job delete