Best Practices for Job Scheduler Configuration in AWS Using RDS
Job scheduling in the cloud poses several unique challenges. When managing batch processes on premise (i.e. with a client-server architecture), scale is relatively predictable. Our on premise customers usually have a good idea of the resources they need to automate all their batch processes. Managers of cloud-based infrastructure, however, often face a different set of constraints.
Batch Job and pg_cron AWS Scheduling
The scale of cloud infrastructure tends to vary more significantly and at less predictable time intervals. Cloud managers need to provision and decommission machines at will, according to business demands. As a result, job scheduling and workload automation needs to be more agile, and to adapt to the VMs where critical batch processes are run.
The key to maximizing the efficiency of job scheduling on Amazon Web Services (AWS) is the separation of scheduler application and its database. Amazon Relational Database Service (Amazon RDS) is a cost-effective way to offload the database tasks integral to a centralized scheduler. RDS, being a cloud based solution, has the benefit of being highly available across your chosen region, or regions. Installing the scheduling engine on EC2, while managing the underlying database within RDS creates a streamlined scheduler with the flexibility to handle nearly any combination of jobs and workflows.
In our experience, RDS offers a dead simple method to stand up an instance of SQL Server without any of the tasks associated with building and configuring a cluster environment in AWS or within your on premise infrastructure. Preparing an instance of JAMS Scheduler on Windows Server 2012 R2, and deploying the required database to RDS, takes less than one hour.
On to Scaling
Continuously and quickly scaling the model above for increased utilization can be accomplished in AWS by building a JAMS Agent into the bootstrap, or with other deployment tools such as Puppet, Chef or even Desired State Configuration (DSC). In our test, we built the JAMS Agent directly into the Bootstrap and utilized a custom PowerShell cmdlet from our open-source JAMSAWS Module, Start-JAMSEC2 (forkable on GitHub), to quickly kick start additional environments. The JAMSAWS Module also has features for starting and stopping AWS instances, which helps keep hourly cloud costs to a minimum.