Introduction to Workload Automation

Thank you everyone for joining us for today’s webinar, Introduction to Workload Automation. My name is Cody Gondyke, I’m from Fortra, and we make JAMS Job Scheduler, the Enterprise Job Scheduling Solution that centralizes and automates your critical processes. 

Joining me today is JAMS Support Engineer and Workload Automation Expert, Robert Newman. Rob has over a decade of experience supporting SQL Server, he’s helped to implement JAMS at dozens of organizations – and provided training to many more, he’s presented JAMS and automation at events and webinars – including some webinars with MSSQLTips, and he’s a co-author of our JAMS University Curriculum.

Today, I’m going to give you a quick summary of who we are and walk you through some of the Workload Automation fundamentals. Then, Rob is going to take you through all of the great, more technical details on today’s call –really, how you can use a product like JAMS to automate your workloads, and some of the features and functionality you’ll have access to with workload automation.

The chat window, by default, will be over on the right hand side of your screen – go ahead and post something there now to try it out and introduce yourself. We’ll be monitoring that chat for questions as we go, and if we don’t get to your question right away, don’t worry – we will take some time at the end of our webinar to answer as many of those questions as we can. Let’s get started.

Workload Automation is our focus today, but it’s always good to know that here at Fortra, we really can solve any –and every – Security and Automation need you have. I like to say that if you have an IT need, we have a solution.

No matter where you are, you probably have a Fortra office close to home, as you can see here, and we do have dedicated JAMS offices in the United States, the United Kingdom, and Australia – which means you can get 24×7 critical support for our JAMS Workload Automation software.

Our team has deployed JAMS Workload Automation to hundreds of customers worldwide, and you can see a small sample of our customers here. I like to say that’s a general rule that you will interact with a JAMS customer today, whether you’re watching a show on your Vizio TV, building a birdfeeder with Milwaukee powertools, checking the forecast with Accuweather, or maybe having a run in Nike sneakers.

We also have an incredibly strong partner network as you can see here – We’re a Microsoft Technology Partner, we have strategic partners like Oracle, IBM, and Cisco, and we belong to IT organizations like PCI for security. 

And then the last thing I have to say about us here is that everything we do at Fortra is for your success, from incorporating your needs directly into our development cycle, to offering professional services and support services that jump-start your workload automation. That’s enough about us – Let’s talk about WLA.

So, I’ll start off here by saying something really obvious. No business exists in a bubble, and no process exists in a bubble. Chances are, you have half a dozen applications in use, right now, that just don’t talk to each other – and we actually find that’s especially true when an application has an internal scheduler like JD Edwards or SAP, because it’s “easy” to set up some simple jobs inside of the application, even though it hurts the process overall. There’s also a good chance you have multiple platforms. Even if you work on Windows and don’t touch UNIX boxes, processes need to run on each. And of course, you’re generally going to have multiple servers where applications are running. Having everything separated out between applications and people and departments and locations brings a lot of challenges. It’s easy for things to get out of hand, or, worse – for things to get siloed, so that there’s no cross-communication. Interaction between those siloes – whether they’re applications or platforms, becomes manual work. One of the core principles of Workload Automation software is bringing those things together so there’s one brain, one interface, handling automation for all processes.

Another key piece is that you should know what’s running, and how it’s doing, for all the processes in your environment. You should have a single pane of glass where you can see that your Informatica job is 30% of the way through it’s normal runtime, and you should be able to go into workload automation software and see what the schedule is going to look like later today, later next week, or a month from now. I’ve lost track of the number of times I’ve heard of something like Visio being used to manually write down what next week’s schedule should look like. So, the second key to Workload Automation software is giving you good information on everything that’s running, and automatically generating projections for you on what’s going to run down the line.

The third key is automatic detection of the things that could be problematic for you and your business, and the ability to create automatic responses to any of those possible failure points. Let’s say you have a backup process that normally takes… half an hour, and it finishes in 30 seconds and says it was successful, it finished without error. You would know that something is up, because there’s no way it ran that fast and actually did what it needed to. So, with workload automation, you have a property or setting that puts you in control to say- if this backup finishes in less than half it’s normal runtime, say that it finished with an error, because I know it wasn’t really successful. The second part of this concept, the automatic response, is even more exciting. You know that when the backup finishes too fast, it just needs to be re-run and it’ll finish successfully. Re-running jobs, or taking certain actions with “cleanup” jobs in response to a failure not only reduces the amount of time you need to spend checking on things, it makes your jobs (and the schedule) more resilient. You should never have to spend an hour or an afternoon cleaning up because a network blip caused a job to fail. And with workload automation, that job is going to be smart enough to re-run itself so you don’t have to manually respond.

Now, I know there’s at least one person on the webinar thinking “that’s great, but not everything is mathematical, you can’t set up paths for everything, sometimes you need a human response”. – and you’re exactly right, and that brings us to our next key – Alerting and Auditing. How and when you learn about those unique process events is monumentally important, which is why workload automation solutions like JAMS come with built-in options for alerting through email, SMS, chat alerts to applications like Slack and Microsoft Teams, and the ability to hook into third party tools like SCOM. Along the same lines, if something does fail, workload automation software is going to keep a record of that failure – and the job log for you, so you know what happened when. And that auditing also extends to the processes themselves. If your operators are alerted that an ETL process failed, they should be able to see that Joe Smith edited the ETL process earlier today, and roll back to a version before his edits – JAMS gives you all those tools.

The next key concept is really part of the larger principle that a workload automation solution should make your entire environment easy to manage. Variables and Parameters are a piece of that principle, because they let you templatize jobs and pass key information between steps in a process. The example coming to mind here is ETL for schools, where a similar financial aid process is running multiple time, with just the aid year, student class, and priority number changing each time. If you’re managing individual jobs for every iteration, you’ve got dozens of one-off processes. If you’re using workload automation, you’re managing one job definition that’s running smarter with variables to set those values at runtime.

The last key is smarter, event-based scheduling. Timed schedules are great, they work, and we have tons of options for those, but the real key to workload automation is the what, how, and why things run. Chaining jobs together in a process within JAMS gives you immediate benefit because you know Job 2 isn’t going to run until Job 1 finishes successfully – no more pillow/pray scheduling where you say Job 1 should finish in 20 minutes so schedule job 2, 30 minutes out. You eliminate the manual file watching by setting a trigger to do it for you, and kick off the job once it sees a file in that folder. I could go on, but I know you’re itching to hear a proper expert talk about this stuff. I’ll hand things over to Rob now, and he’ll get into all the great detail I glossed over. Rob?

When you think of a process, you probably think of a single step or action within a specific application, like JD Edwards or SAP.  In JAMS we like to think of a process as a collection of steps or actions.  It doesn’t matter if they run on Unix or Widows, SAP or Informatica, SQL or Oracle.  We call this grouping of jobs a Sequence. First you need to run the Daily Data Download and upon successful completion, run your Informatica Data Sync process, followed by your SAP Jobs and so on.  With JAMS you simply use the Submit Job activity to add or inject any of your cross platform jobs to a Sequence, or workload process.

I was recently discussing SQL Agent with a customer showing her how we can convert her SQL Agent jobs directly into JAMS.  and How we can incorporate them into a Sequence and easily add her SAP jobs and your Informatica jobs.  It’s a Sequence like that that really illustrates the idea that we can truly run everything directly from within JAMS. The dependency within the sequence is set automatically thus eliminating pillow processing.  There is no longer any need to schedule extra time between jobs to ensure the Informatica job finished before the SAP jobs starts.  In addition to the sequential dependency you can also, run steps in parallel, add additional dependencies and even add specific failure actions to your Sequences.

Now, by default the Sequence will halt at any failing step and send a notification.  This will allow you fix whatever the cause of the error and resume at, or continue past, the point of error.
Each of the steps within may have it’s own unique set of requirements which, may be specific to only this run.  Maybe a specific connection string or a different set of credentials. With our Connection Store you can preconfigure connections to your Database servers, your SFTP servers, your Email servers as well as your S3 buckets.  You can also chain a set of credentials to these connections.
And of course those Credentials are stored securely in the JAMS Password vault. So, JAMS not only has the power to schedule all of your jobs and processes, it’s also stores your Jobs source code, the credentials needed to run those jobs and the connections needed to communicate and integrate with your remote systems and applications, no matter what application or platform they touch.

Workload Automation ensures that enterprises aren’t just putting jobs on a timer and hoping for the best. So, let’s take a look at how you can see what jobs are running, where they’re running and even how they’re running.

The JAMS Monitor is that single pane of glass, showing you jobs that are currently executing or are scheduled to be executed, jobs that pending pre-requisites like a file or another job or even an email, or jobs that have either completed or are halted and waiting for you to take some action. This monitor view will allow you see all jobs running on your SQL Production box, or all jobs using your domain admin credential, and even all manually submitted jobs, like Joe resetting his password.

You can also see elapsed time and percentage complete stats. These are based on the run times stats JAMS collects for each execution. So, you will know that the WLA Job is abut 79% complete, or more importantly, one of the SAP jobs has been running about 3 times longer than normal. Of course, you can have JAMS automatically notify you of the runaway condition, but I will talk more about that later.

Now that we have a good idea of what’s running and what it means, let’s take a look into the future. The JAMS Projected Schedule will give you an analytical view into your future Job Schedule. This is where you can see what’s going to run today, tomorrow or even next month. You can pinpoint your busy times to ensure you not scheduling a dozen financial reports during your month end processing. You can set your view by folder, job, planned or repeated, and its easy to drill down to that specific SAP Morning job so you can be assured it’s going to run at 9:00 am every workday. This is all great and fantastic, but how about we look into how JAMS can be your eyes, ears and even your fingers.

What good would a Workload Automation and Job Scheduling tool be if it doesn’t automatically detect and automatically respond to problems? When a job fails, JAMS needs to be able to detect that failure and also needs to know what to do based on that failure. JAMS automatically captures your jobs standard out and standard error streams and can take actions based on those exit codes and even on specific messages parsed from the log file.

So, when your Applications shutdown Job fails with a final severity of Error, JAMS will know not to start the backup job. Or, if your Applications Shutdown job logs an “Application not running” message, JAMS will know it is OK to start the backup job. JAMS will also raise specific events when your jobs run long, like 300% longer than the average previous successful runs, or has a short run where you know it couldn’t possible be successful that quickly, and if your file transfer job is waiting too long for that dependent file or even a dependent job, then a stalled event will be raised.

When an Event occurs, you can run a notification job or a recovery job. What would you like JAMS to do if your db backup job fails? Well, why did it fail? If the error is the backup drive is unavailable, then run a recovery job to restore connectivity to that drive. If another backup job is currently running, then wait till that job has finished before executing. Or, what if it’s just the occasional connection error, and you know re-running the job is all you need to do, then have JAMS automatically retry the job, and, if the error indicates there are still active transactions in the DB, then run a response job to remind your users the db is coming down for backup.

Yes, your Workload Automation software should be sending your team Alerts and Notifications. They need to be both timely & informative.  As they are paramount to ensuring your workload processes run to completion.
Those same logs, error codes, error messages and events we previously talked about can also be used to generate Emails or SMS messages. They can send events to SCOM, ServiceNOW, Teams, Slack or just about any ChatOps application API. This is particularly important, when, say a Sequence Halts. Your support staff really needs that notification to include everything needed to successfully determine where the error occurred, what the error was, and maybe even some in-house documentation directing them to a resolution – and you even get the log file. And Having that log file really helped me with the user I was talking about earlier – the one with the great sequence. Her SQL Agent jobs were running SSRS Reports and with every failure their operations staff needed to VPN into the network and then log in to SQL to see went wrong.  Now, the JAMS log tells them everything they need to know and they no longer even need access to SQL.

The Job logs and history records are automatically retained for troubleshooting, tracking and audit purposes.  Additionally, JAMS auditing also tracks all object changes and even identifies source code revisions.  So, when that job that normally runs successful, suddenly stops working you can immediately identify the it was changed yesterday and easily roll back to any previous version – and, the native auditing features in JAMS not only meet your in-house needs, it also satisfies your auditors.
 
Workload Automation should minimize the number of clicks needed to manage your environment. Folder base properties, an inheritance model & even friendly display names, all make for managing your JAMS environment seamless.
There is no need to manually set values on every job or even for every run. You can effortlessly run the same jobs from multiple folders and even multiple environments, all with different values and without actually editing the jobs.

This means that your File Transfer Jobs in the Bank of America folder will transfer the files to Bank of America using BoA credentials. However, that same job in the Chase Manhattan Folder, will now automatically send the files to Chase Manhattan using the Chase Manhattan credentials. Not only does this also provide for seamless promotability from your Dev to QA to Production environments, the Jobs will also inherit the appropriate Security controls, this ensures that your production jobs can only be accessed by your production operations team and, that your developers cannot run the payroll jobs.

Another way to efficiently manage JAMS and run your Jobs is to setup Template Jobs. These can used to empower users the ability to dynamically enter values without needed access to the source code or even the remote application, or allow them to run an ad-hoc file transfer giving them the ability to dynamically pick the desired bank and even the specific daily file.

So, to recap – Workload automation software is where you should be unifying all processes from all platforms and applications. It’s your one-stop shop to know what’s running now and what will run in the future. It should automatically detect possible issues – and have programmatic responses to handle them. It should have alerting and auditing capabilities that give you what you need when human interaction is necessary. It should make managing your environment easy and simple – and quick. It should allow you to control when your jobs run, based on other events in your environment. 

That covers quite enough JAMS topics for now, back to you Cody.