Parameter Powered Batch Automation: JAMS at ESCO Advisors
ESCO Advisors provides retail energy supply companies with the mission-critical data needed to maximize operational efficiency and make intelligent decisions related to the purchase and sale of energy. ESCO Advisors’ clients depend on the company’s Risk Management / Demand Forecasting, Customer Relationship Management, Broker Relationship Management and Third Party Verification systems to reach sales goals and meet their needs. The company’s clients operate across all utilities in deregulated electricity and natural gas markets across the United States.
ESCO Advisors’ batch automation needs center on SQL stored procedures and high volume, multi-step FTP processes.
Once the retail energy supply industry saw the value of ESCO Advisors’ applications, the company’s client base grew quickly, outpacing their existing tools for batch processing, SQL Server Agent and Windows Task Scheduler. “As new clients signed on, we were tying up members of our development team with IT administration tasks, using valuable development time chasing down issues inherent to, or covered up by, SQL Agent and Task Scheduler,” said CIO Christopher Waring. “This wasn’t the most effective use of their time.” Bringing on those new clients was also a challenge, as it could take as many as three days and require multiple members of the company’s IT staff just to set up all the needed processes.
In addition to their core solutions, the company maintains each client’s archive and forecast data in an up-to-date repository that can be accessed by their business intelligence (BI) tools. Providing this valuable service required a reliable overnight ETL process for each data set.
ESCO Advisors’ proprietary applications draw content from multiple data stores and run millions of calculations against the data every day. For each energy retailer, they collect data such as weather forecasts, usage history, market prices, energy supply, and billed revenue per customer. Hundreds of file transfer sequences aggregate data from dozens of different utilities and ISOs across the United States. Once the source data for a client is complete, ESCO Advisors applies its logic, and publishes it for immediate use in each retailer’s portal. With rapid changes in energy prices and fluctuating demand for electricity and gas, even a small delay could have a significant impact on a client’s business.
When issues occurred in one of the company’s workflows, it was not uncommon to recruit multiple members of the IT staff to review log files or check processes on various servers. “We were spending too much time researching logs for the cause of failure,” said Waring.
After evaluating several scheduling solutions, ESCO Advisors selected JAMS Job Scheduler to centrally manage its rapidly growing automation infrastructure. Key to the decision was JAMS’ ability to parameterize job definitions.
Over the course of a single week, the company set up each of its critical batch processes in JAMS. It leveraged the built-in SQL Conversion Utility to quickly convert SQL stored procedures. They then configured a “base job” for every possible job they might run for a given client, based on the products they purchased.
Once Waring’s team established and tested the base jobs, they replaced hard-coded values in job definitions with parameters. Values such as database connection strings, FTP locations, folder paths and lists of executables were converted to parameters. This made it possible to define them quickly when bringing a new client on board, or to update them programmatically according to output received from other processes.
Monitoring jobs centrally through JAMS provided ESCO Advisors with the intelligible information it needed to resolve job failures. Rather than just delivering a notification that a job failed, JAMS identifies the root cause. According to Waring, “Now, we start our research knowing that we had a ‘failed import’ or ‘file unavailable.’ When applications return a string for an error code, we can map that string to the notifications we use to administer our IT environment.”
ESCO Advisors has made significant gains in efficiency in two major areas as a result of implementing JAMS. The company has minimized the time needed to onboard a new client from “days to hours” and its development team is rarely needed to research support issues.
“The onboarding process is now one part of one person’s responsibility,” stated Waring. When a new client signs on, the company easily references its base jobs and adds the client-specific parameters to build their customized processing chain.
If any of the company’s 700+ daily jobs fail, the resulting notification provides the IT team with the time of failure, the exit code, the server name and scores of other data points that make resolution quick and easy.
By leveraging JAMS, ESCO Advisors has laid the groundwork for efficient growth. Whether that growth is the result of new client acquisition, new product offerings, or both, the company is prepared to automate the process.