The Transition from Job Scheduling to Workload Automation
Job Scheduling Unifies Enterprise Computing
Systems administrators have always needed some ability to schedule the jobs, workflows, and tasks of the enterprise. Without job schedulers, one would need an army of operators to execute repetitive processes. Even the oldest mainframes included some basic functionality to schedule recurring tasks. Most modern operating systems offer rudimentary scheduling functionality through applications such as Windows Task Scheduler and Cron. In addition, business applications such as SQL Server, SAP, Oracle, and SAS, offer some kind of scheduling capabilities. In order to scale with the needs of a business, and align applications with the actual work, systems administrators must routinely exploit all the capabilities of native schedulers.
Native Job Scheduling Tools. Close, but…
While platform- and application-based job scheduling tools provide considerable advantages over manual processing, they offer little to administrators whose responsibilities extend beyond a single application or platform. Automating workflows and jobs at the business level requires a holistic view of every application and platform. Enterprises repeatedly face this challenge when the features of an individual scheduler prevent them from achieving a broad automation goal, such as an end-to-end invoicing process, or a customer provisioning process.
Native job scheduling tools tend to be simplistic, offering the ability to run jobs based upon a simple schedule, with fairly basic monitoring of success and failure. Some native solutions, notably SQL Server Agent, offer multi-step jobs with some success/failure logic, and incorporate reasonably complete logging so that a job’s progress can be tracked and archived. Few organizations, however, can conform all of their jobs to SQL Agent’s scheduling paradigm; their business processes involve many different technologies.
Native tools tend to also function in isolation: SQL Server jobs are managed in SQL Server. Windows jobs are managed in Task Scheduler. Oracle jobs are managed in Oracle. Defining jobs across these differing tools creates a disjointed set of rules for processes that the enterprise would like to be able to manage in a uniform way. Why should a critical Oracle job be given a different set of priorities than a Linux job if their impact on the business is the same? Managing and modifying business processes becomes extremely difficult, as systems administrators struggle to find a comprehensive view of their jobs. Attempts to combine these disparate jobs into larger workflows often create haphazard homegrown systems that compromise security and are only understood by one or two employees.
Requirements for an Enterprise Job Scheduler
In order to satisfy the requirements cited by most systems administrators, an enterprise jobs scheduler (a.k.a. workload automation solution) must have the following features:
- Supports Jobs on Multiple Operating Systems and Applications
The solution should be able to initiate the execution of as many job types as possible to enable workload orchestration. These job types may include scripts written in languages such as Python, PowerShell, and Windows batch. Job types may also include tasks executed on common business platforms, such as PeopleSoft, Banner, SAP, Informatica, and more. - Central or Distributed Job Execution
Enterprise job schedulers must offer the flexibility to execute jobs either from a central server, or from an agent that can reside on a remote machine. - Workflow Logic
When executing a collection of jobs, or job steps, the solution must be able to check that predecessor jobs have completed successfully, and that external dependencies, such as the presence or availability of a file, are met before proceeding to subsequent steps. Administrators must also be able to configure workflow steps to run in series, in parallel, or according to conditional logic. - Programmatic Operation
Enterprise job schedulers must also provide secure access to all its functionality by means of an API. - Consolidation
Job scheduling software must be able to collect and consolidate job data, so that administrators can monitor and manage jobs from a central console. - Auditability
Administrators must be able to log and query all actions managed from or executed by the solution. The solution should provide reporting and archiving capabilities that satisfy standards set forth by both external regulating bodies and internal auditors. - Security
Each object within the solution – jobs, folders, views, servers, etc. – must be securable in the same way as any other technology asset. Add, change, delete, and inquire settings should be easily assigned to these objects, and logically inherited. Ideally, job scheduling software should be able to leverage security settings from an existing authentication system, such as Active Directory. - High Availability
A centralized workload automation solution must include failover/high availability architecture that ensures business-critical tasks continue to execute even if the organization suffers a significant interruption to its network, or a failure of its primary host machine. As some enterprises may not initially have the resources needed to configure failover, the feature should be easy to implement at a later date, and it should inherit the majority of its settings and definitions from a primary job scheduler that is already in use.
JAMS Enterprise Job Scheduling Software
JAMS was originally developed as a job scheduler, but has continuously expanded its capabilities to evolve into a comprehensive workload automation solution. This distinction arises from a deliberate change in architecture that responds to events and conditions as quickly as it responds to the current time. JAMS job scheduling software supports the execution of jobs and workflows through a variety of agents, including ones for Windows, Linux, UNIX, OpenVMS, System i, and zOS. If needed, JAMS supports the execution of jobs locally.
JAMS supports a wide range of technologies to execute jobs. It supports scripting languages such as Windows batch, UNIX shell, SQL queries, VBScript, Perl, Python, and PowerShell. In addition, JAMS job definitions can store tasks that are unique to business applications, such as SAP, PeopleSoft, VMware, JDEdwards, SAS, Informatica, and Netezza. Given their prevalence in many workflows, JAMS natively supports direct execution of file transfers and SSIS packages.
JAMS provides an enterprise with the flexibility to interact with the schedule by whatever client it chooses. The customizable GUI console gives administrators and operators alike a well-organized set of views to define, monitor, and report on jobs. For users with administrative privileges, it provides powerful utilities to configure security, high availability, logging, and auditing, according to the requirements of the business. JAMS can also be managed through a variety of non-GUI methods, including a PowerShell module, a .NET API, and a REST API. All of these management options are included with JAMS, and are updated simultaneously, so that an enterprise has access to all features at the same time, regardless of the method(s) used to manage the scheduler.
The JAMS job scheduler supports high availability or “failover” capability from day one through the use of a backup execution engine.
How JAMS Automates: Basic Principles
The basic unit of work managed by JAMS is the “job”. What is contained within a job may vary within a given enterprise. The number of processes you choose to include within this basic unit is up to you, but the job scheduler treats the job as an atomic unit – one that is irreducible.
Each job managed within JAMS is run by an execution method, such as Windows, Linux, or OpenVMS. The source of each job, which may be a script, a form, or a reference, is stored securely in its definition, along with all of the scheduling properties needed to execute it.
Jobs can be grouped into sequences. Sequences are groups of jobs that execute in a specific order. By grouping jobs together in sequences, enterprises can create multi-step workflows that are aligned with the business process, even if individual jobs execute on different platforms, or relate to different business systems. They can specify settings, such as security permissions, notifications, and dependencies at the parent level; all jobs within that sequence will inherit those properties.
Precision Scheduling. Close does not count.
Once you store the source of a job in JAMS, you gain a nearly infinite number of ways to configure its execution. A job’s execution properties can be broken down into three general categories.
- Continuous Dates and Times – JAMS can schedule jobs according to very basic combinations (e.g. Tuesdays at 9:00 PM) but can also schedule them according to complex combinations that include date patterns (e.g. every other week) and exceptions (e.g. not the last day of the month).
- Dependencies – JAMS jobs can be restricted to run only under certain conditions, such as the completion of other jobs, the presence of a file, or an execution window that you define.
- Triggers – Jobs can also be configured to detect events from other jobs, applications, or file systems. Unlike polling, triggers provide an instantaneous method to kick off jobs based on events such as the arrival of a file, or the completion of another job.
- Non-Continuous Dates/Times – Discrete dates and times, such as company holidays, and maintenance periods can be used to dictate a job’s execution
Flexible Job Definitions
The architecture of the JAMS job scheduling software is designed with modularity in mind. Rather than entering an explicit value for a given job property, users can enter a parameter that is a reference to a centrally stored value, or a value that can be entered by a user at runtime. If the parameter’s value needs to be changed at a later date, users need only change the parameter once, and every job referencing that parameter will inherit the change. Parameters greatly reduce the time needed to update jobs when values such as connection strings and file paths are changed.
JAMS also supports the use of variables, global values that can be referenced across all jobs. Variables can be used “as is” or they can also be used together with parameters.
Notifications: Beyond Success and Failure of Scheduled Jobs
JAMS provides enterprises with a comprehensive system of alerting for any job or sequence. It can produce detailed email notifications for any event related to a job, including all of the nuanced states that may signal the need for intervention. Jobs can be configured to generate alerts on conditions such as “running too long”, “running too quickly”, and “waiting too long on a dependency”. While these states do not constitute failure, they often represent issues that need to be addressed.
Notification and logging can also be configured to use Syslog and SNMP to route alerts to monitoring frameworks such as SCOM, and HP Open View.
Job Scheduling Reporting and Auditing
JAMS collects extensive information about each job’s execution and stores it in one central location, eliminating the need for separate reporting tools. This job history can easily be stored in the SQL Server Express database included with JAMS, but most enterprise customers opt to use SQL Server or highly scalable SQL offering through AWS or Azure, especially when their industry requires a large amount of history data to be available to regulators.
JAMS includes numerous report templates for analyzing job performance metrics such as average run time and resource consumption. Custom reports can draw from any data within the JAMS database, and can display performance data in a variety of tables, charts, and graphs.
Monitors, in the desktop and web client, allow users to see the real-time status of all jobs in one central location. The Projected Schedule view offers a unique perspective of an enterprise’s upcoming jobs. By combining historical performance data with the current execution status of jobs, administrators can identify and avoid potential conflicts associated with automating across a diverse set of platforms and applications.
Migration
Fortra continuously invests in conversion utilities to expedite the transition from native tools and legacy schedulers, to a centralized workload automation solution. Conversion utilities for Windows Task Scheduler and SQL Agent can be invoked from directly within JAMS. External utilities are available for Cron tabs and for jobs defined in proprietary formats, including many created by large vendors.
Programmatic Operation: APIs and PowerShell
The JAMS enterprise job scheduler exposes its functionality by means of several programmatic interfaces: a RESTful API, a .NET API and a PowerShell module. Each of these interfaces enables custom applications to interact directly with JAMS, without ever opening the GUI client. For example, an administrator may want to define a new job for each new customer it creates in its CRM. The customer creation process can include a set of instructions that specifies the creation of the new job in JAMS. Using the API, enterprises can develop lightweight tools that allow business users to submit jobs without any need to deploy a client.
Learn everything you need to know about JAMS when you watch our comprehensive Feature Tour.