Run Monte Carlo simulations at scale with AWS Step Functions and AWS Fargate

Introduction

Organizations across financial services and other industries have business processes that require executing the same business logic across billions of records for their machine learning and compliance needs. Many organizations rely on internal custom orchestration systems or big data frameworks to coordinate the parallel processing of their business logic across many parallel compute nodes. The maintenance and operation of orchestration systems can require significant effort from development resources or even require additional internal dedicated teams to manage these tools. Organizations also often manage large clusters of compute resources for executing business logic at scale requiring significant operational and infrastructure investments.

In this post, we demonstrate how to build an application capable of handling datasets containing billions of records, operating on a compute environment with minimal configuration and running tasks resiliently at scale. It combines …