High-Velocity Data Warehouse: Workload Automation at Prologis
How to Deliver Real-Time Data to Meet the Needs of a Dynamic Environment
Producing data at your fingertips can be an elusive goal—especially in a complex data environment. But when done well, it establishes a framework for organizations to apply lessons learned and practical insights in their own business.
The story of Prologis, a global leader in logistics real estate, provides a modern example of how to design and optimize a data management system to support at-the-ready business intelligence (BI) and analytics. And its data management journey has opened possibilities and revealed the critical infrastructure required to make at-your-fingertips data within reach across the organization.
In this white paper, we will examine how Prologis brought together the right business intelligence tech stack, including high-velocity data warehouse workload automation from JAMS, to drive lasting value through data and to empower real-time decision making among users. We will also provide some leading practices you can use in your own organization, particularly when it comes to scheduling in a complex data center with real-time demands from the business.
The Prologis Data Management Optimization Journey
When Prologis made the decision to transform its infrastructure and move onto a modern platform that shifted to a new data management approach, it sought the right combination of solutions and services. This included support for its new data warehouse and data lake that would be the foundation for self-service business intelligence and reporting.
The Complementary Role of Data Warehouses and Data Lakes
Data warehouses and data lakes typically meet different needs within the organization. While a data warehouse relies on information that is defined and cleaned in advance to then analyze data from line of business applications or transactional systems, a data lake can store native data without having to structure it first. In turn, a data lake can produce business intelligence and analytics from both structured and unstructured data, and provide a high degree of flexibility in making the data usable for better decision making across the business.
According to an Aberdeen Report, of those organizations that had a well-implemented data lake, 65 percent were able to influence time-to-market, 54 percent experienced an ability to mitigate organizational risk, 24 percent experienced an increase in organic growth, and 15 percent reported an increase in operating profit. Similar to the strategy Prologis employed, many organizations recognize the need for leveraging both data warehouses and data lakes, and are evolving their infrastructure to incorporate both use cases.
Re-Imagining Infrastructure: Moving Beyond Its Legacy Tech Stack
For quite some time, Prologis had relied on its historical data warehouse implementation, governed by an enterprise job scheduler for its ETL orchestration. ETL involves the extraction, transformation, and loading of data—these three processes together move data from one or many databases to a specific repository. The primary goal then within Prologis was to open the possibility for the business to have data within reach, empowering individuals to start answering their own questions for better decision making.
The organization began concepting what a modern infrastructure would look like, identifying Snowflake, a leading data cloud company, to become the center of its data management solution, serving as the home for its data lake and data warehouse. It also positioned Informatica Intelligent Cloud Services (IICS) for both point-to-point enterprise integrations of multiple applications exchanging data, and for all the warehouse workloads and sourcing workloads into the data lake.
In addition, Prologis wanted to avoid another business intelligence reporting tool that had a baked-in semantic layer—the layer in which all the business rules are applied to the underlying data coming out of the database. The organization preferred a solution that would leverage a reusable SQL-based engine, so that long term, semantics would not be locked up in a proprietary code. That’s where the Denodo tool entered the picture for data virtualization, the core technology that enables modern data integration for data management solutions.
Discovering and Implementing the Ideal Data Warehouse Workload Orchestration Solution
Once Prologis identified these new components of its data management solution, it required the need for orchestrating high velocity job scheduling. After looking at cloud-based solutions, Prologis realized these tools offered more limitations than benefits, especially in terms of cost-predictability. Specifically, in the interest of cost control, Prologis would have had to build sub-optimal architectures to gain the benefits of controlling overall costs—at the expense of flexibility and scalability.
That’s when the global real estate company came upon JAMS, leading workload automation and job scheduling software that runs, monitors, and manages jobs and workflows to support critical business processes. What really made JAMS shine to Prologis was its versatile functionality that could handle the intensity of job scheduling required within the complex data environment. Simply, JAMS can plug into anything, and integrate complex processes across multiple applications and platforms.
How JAMS Is Used within the Holistic Business Intelligence Tech Stack
Within Prologis, JAMS is now serving IICS to land all of the data first and also serving Denodo for the data virtualization layer, with the ability to tell Denodo to complete its tasks and make data available to users. JAMS also is orchestrating data loads into the existing data warehouse—multi-step orchestration during which data comes and goes across many different operations. Prologis uses JAMS scheduling capabilities to ensure the data set is backed up and completed successfully within a predefined window without affecting report delivery.
JAMS also supports kicking off jobs within Snowflake. Specifically, JAMS launches stored procedures that are tied to the completion of the IICS job landing data into the data lake. Upon successful completion of this job, JAMS calls the stored procedures in Snowflake. Once this is completed, JAMS then kicks off a simple check query to ensure the stored procedure worked.
When looking into the future with Snowflake, JAMS will perform additional advanced orchestration for a green/blue loading mechanism, where two copies of the data warehouse will exist. One copy will be attached to the reporting engine and another to the loading engine, where the four-hour exercise for orchestrating data loading will occur on the offline schema. JAMS will play a prominent role in ensuring that swapping of green/blue objects are performed at the appropriate time in execution lineage as well as help perform a number of checks to make sure data will attach to the correct data warehouse.
Practical Insights from Prologis: Applying Best Practices for Leveraging Workload Automation in Data Management
The Prologis story offers applicable insights and a best case example of how to turn the vision of optimized data management into a reality. Rather than migrating slowly and dipping its toes into data cloud solutions, the global real estate company purposefully designed and implemented an incredible tech stack that delivers business intelligence reports in real-time with predictable cost and reliability. Simply, Prologis put in the hard work of designing, implementing, and now optimizing its new data management infrastructure. Perhaps most important, it recognized the connective role that a flexible workload automation solution would play across these data cloud solutions. So what lessons can be learned and applied to your own organization?
1) Define Your Vision and Stick to It
Prologis recognized the obstacle that its legacy tech stack presented to the company’s vision and sought to redefine what its data management solution could look like. Rather than settling for the status quo, it pursued a vision of finding and landing on best-of-breed solutions available, tailored for data management and business intelligence. The company also recognized that deriving the most value required the right workload automation solution—one that could orchestrate complex processes across each of its new data management tools. Rather than looking for a vendor specific job scheduling solution, Prologis found that JAMS offered impartial orchestration that could integrate into any platform and work across data management solutions.
2) Establish What Success Looks Like
Developing the value proposition of your collective solutions and defining what success looks like is an essential step when embarking on a project of this scope. Prologis identified that real-time data delivery, improved decision making at the business level, better operational management, and self-service reporting were the critical outcomes in its journey. In other words, for Prologis, this project was all about making data available at your fingertips. Without JAMS, Prologis would not have been able to achieve these business benefits, especially around self-service. In fact, the company reports this is the first time it has seen self-service reporting actually work. Understanding what true success looks like—and the steps it takes to get there—are critical in knowing what you are attempting to deliver.
3) Find True Partners Along the Way
Prologis has had an undeniable ability to identify and partner with some incredible vendors to build its data management solution. From IICS and Denodo to Snowflake and Fortra’s JAMS, the real estate company recognizes the value of the partner relationship in developing a truly tailored solution. Even when encountering challenges along the way, partners like Fortra have demonstrated their ability to respond to any issues in a timely manner. And Prologis has taken the opportunity to make these relationships even more strategic and chart a course as a leader in data management adoption. The team at Prologis recognizes JAMS workload automation as delivering on everything advertised and also maturing its relationship to ensure mutually beneficial success—both now and in the future.
