JAMS is an enterprise automation solution intended to manage and maintain all of your batch processing needs. This document details the design choices made to balance flexibility with performance and security.
A typical implementation includes the following scheduler environments:
- Development/Test (DEV)
- Quality Assurance (QA)
- Production* (PROD)*Production environments may include a redundant “failover” instance for disaster recovery.
JAMS provides an infrastructure to execute Jobs and Setups. A Setup adds conditional logic and workflow control to a group of individual jobs and provides users with the ability to schedule that group of Jobs as one. Unless otherwise indicated, the term “Jobs” can mean either Jobs or Setups.
To manage Jobs efficiently, JAMS employs several mechanisms:
- Folders – Logically store and group Jobs, Setups, Triggers and Variables
- Queues and Resources – Enforces load balancing and concurrent processing limits of your Jobs
- Parameters and Variables – Store globally accessible values that can be defined once and utilized across many Jobs. Enables ad-hoc submission without affecting the integrity of a Job’s source or settings.
The primary purpose of Folders is to give each department its own environment while retaining the benefits of a single centralized scheduler. The secondary purpose is to enable IT teams to set default permissions and Properties for all Jobs that exist within those Folders and gain the benefit of an inheritance structure.
Active Directory users or groups are applied to each Folder to define the authorization level for anyone utilizing JAMS. Common Authorization Matrices can be found below.
Queues and Resources
Queues control the execution of Jobs on various machines. A single Queue may include multiple machines to improve performance and provide redundancy. Targeting a Job to a Queue, rather than targeting a Job to a machine, enables JAMS to maximize the efficiency of a Jobs schedule.
Queues also enable you to set a concurrent Job processing limit through the use of Queue Slots. As additional slots become available within a Queue Definition, JAMS will submit Jobs in a round-robin fashion to each node defined within a Queue.
Queues can also be utilized for single threading of Job processing to allow only a single occurrence of a Job to run at once. By creating a single-threaded queue you can set a Job limit to 1, to ensure there is never a risk of more than one of the same Job running at a time.
Lastly a batch queue can be configured to automatically update with any new public IP addresses from Instances of JAMS Agents residing on Amazon AWS EC2.
Resources assign a user-defined value given to a set of Jobs. The total available Resources will generally be set to a total value of 100, users then have the ability to define the amount of total Resources a single Job will be allowed to consume. If a Job requests more Resources than are available, the Job will be held until enough Resources are free.
Parameters and Variables
Through the use of Parameters and Variables, Jobs can be made more flexible and portable. They can even prompt business users for specific values without burdening those users with editing a Job’s source.
Parameters store values for individual Jobs or Folders, whereas Variables store values that can be accessed across the scheduler.
All three environments are designed with at least one JAMS Primary Scheduler and a minimum of two machines as JAMS Agents. The JAMS Primary Scheduler offloads the processing of a majority of Jobs to any of the remote JAMS Agents. JAMS Agents distribute processing across the environment, ensuring physical resources are utilized efficiently.
The Development environment controls jobs for both the development and test networks. In this example, the primary Active Directory groups are CORPXAM, CORPXDEVELOPERS and CORPXADMINISTRATORS. The infrastructure is built with two general categories of machines – JAMS Agents and one JAMS Scheduler.
Initially, two Folders are created. One will be used by the groups CORPXAM and CORPXDEVELOPERS, the other exclusively by the CORPXADMINISTRATORS group.
From a scheduling perspective the Quality Assurance environment (QA) is the most complex of all environments. There are three reasons for this:
- The JAMS Scheduler is used for testing: scripts intended for production use are tested here;
- The JAMS Scheduler is used for “production” scripts which are needed to actually run this environment;
- There are several environments onsite and in the cloud, catered by one JAMS Scheduler.
The QA environment itself will be similar to the Test environment with a single JAMS Scheduler and two Agent machines.
Apart from the Folder shared by CORPXAM and CORPXDEVELOPERS, there will be a separate Folder for each Test environment for production Jobs; these Jobs will only be editable by the CORPXADMINISTRATORS group.
The Production Environment will be redundant with one JAMS Scheduler as the primary and a second as a Failover Scheduler. The primary JAMS Scheduler and Agents are located in a local datacenter with an on premises SQL Server. The other Failover Scheduler and Agents are located in an offsite disaster recovery datacenter. The database will be placed on an existing SQL Server 2014 machine, PRODSQL1.
There will be one Folder shared by CORPXAM and CORPXDEVELOPERS, one for CORPXADMINISTRATORS and one for the Primary Production Jobs for which the Job source will also be maintained through the deployment process. No modifications to Jobs will be permitted within the Production Environment.
JAMS has extensive reporting capabilities that are deployed and usable in all environments. Reports help determine the future (projected) schedule of Jobs, show the performance of previously run Jobs and provide various Audit Trail details for all Jobs. Reporting can be handled ad-hoc or automated through a JAMS Report Job.
Appendix A – Authorization Matrices
This appendix specifies the Job authorization levels for different Folders.
The JAMS Folder is for jobs internal to JAMS only and will therefore only be used by the JAMS Administrators.