fbpx

AWS Batch has quickly become one of the most sought after ways to run batch computing in the cloud. Powered by AWS ECS, AWS Batch provides scheduling and queuing capabilities that enable developers, scientists, and engineers to easily and efficiently run hundreds of thousands of batch computing jobs.  The service offers managed compute environments, provisioning and scheduling work for you.

When combined with Spotinst Elastigroup, AWS Batch clusters inherit the full power of cost efficient spot instance management and scale.

Components of AWS Batch

Job

A unit of work, such as a shell script, linux executable or docker image that is submitted to the service. It runs as a containerized application with AWS ECS.

Job Definitions

Specifications on how the jobs will run, such as CPU and memory requirements, IAM Role permissions, and container settings.

Job Queues

Each job is submitted to a job queue. The queue ensures all tasks are held until compute capacity is ready to process the tasks. Priority can be assigned to each job queue to ensure sensitive tasks are processed ahead of general workloads.

Compute Environment

The resources that comprise the AWS ECS cluster that will process each Batch job.

The Power of Elastigroup

The following diagram illustrates how Spotinst scales an Elastigroup in accordance with the demands of batch jobs and their requirements.

 

  1. The Spotinst Autoscaler launches the best fit instances to support jobs requirements
  2. Elastigroup continues to monitor jobs and resources
  3. When jobs finish, Elastigroup will react and scale down the environment

 

Getting Started

Go to Spotinst and simply enable the Elastigroup integration with AWS Batch.

Existing Elastigroup

For an existing Elastgroup that already uses ECS, go to Edit Configuration and choose a batch queue:

 

 

View tasks pending in Elastgroup overview:

 

New Elastigroup

For new Elastigroup, open the creation wizard and choose “Fargate/ECS”:

On General tab: 

  1. Fill name
  2. Choose desire region
  3. Choose desire cluster
  4. According to available batch queue, choose desire to run in Elastigroup

Continue with the creation wizard and finalize any Elatigroup settings.

Once created, the magic will start and AWS Batch jobs will be handled by Elastigroup.