SAP DI (Data Intelligence) and the Future of Business Data Platforms

For the last couple of years, SAP Data Intelligence has been the company’s consolidated solution for data processing, metadata cataloging, machine learning development and operations use cases. Data Intelligence has seen some major improvements and has turned into a reliable all-in-one solution for both data engineering and data science teams. These accomplishments will become available to a broader audience in the near future as SAP has revealed a revised data management product roadmap in August 2022: Data Intelligence will be merged into the Data Warehouse Cloud.

In light of this drastic but ultimately smart move, we invite you to join us in briefly revisiting the main features of Data Intelligence and its development trajectory, and glancing at the potential future of the SAP business data platform.

The Current State of Data Intelligence

Every company's digital processes and intelligent analytics are unique - as are their requirements for the necessary data management platform. There are numerous options for data preparation, automation and analysis. The decision is made depending on the systems involved, the nature of the data and the available expertise. The SAP all-in-one solution Data Intelligence boasts a very useful combination of components, which - especially in the SAP context - offer a fast and flexible way for data management. For projects in the field of Artificial Intelligence (AI) and Machine Learning (ML), there is also significant potential for business users thanks to the intuitive, supporting interface.

Data Intelligence Architecture: The system is a collection of functional modules covering a wide range of data management tasks.
Data Intelligence can be deployed in any cloud provider infrastructure user Kubernetes cluster technolog

SAP Data Intelligence is the company’s easy-to-understand data management, processings, and orchestration solution that can be deployed on-premise in the one’s own data center, in the private cloud or via all major cloud service providers. The system adheres to current IT trends such as self-service analytics and automatic scaling. These features ares implemented in a stable backend which combines SAP's own systems and open source technologies such as Kubernetes and Docker. The application area is the integration, processing and enrichment of data, primarily for analytics processes. The data is linked across multiple workflows and endpoints and is available centrally via the web interface.

Data Intelligence Modeler: The drag-and-drop data pipeline modeling and orchestration tool of SAP Data Intelligence allows users
to extract, transform, and load data from and to various sources. For more complex operations, Python or R code can be executed within pipelines.

SAP Data Intelligence as a
data management solution for AI applications

Data Intelligence does not hide its patchwork nature but promotes the different specialized components in the central user interface:  “Metadata Explorer”, “ML Data Manager”, “ML Scenario Manager”, and “Modeler” - just to name a few. The Modeler component is SAP’s mature graphical data pipeline editor and orchestration service. The drag-and-drop pipeline builder contains many useful, out-of-box applicable operators. Among connectors for all major database and storage source systems and protocols, use of the latest algorithms for advanced users is enabled through Python and R library integration. For tech-savvy business users, the barrier to entry for data projects can be lowered. Similar to Tableau and SAP Analytics Cloud (SAC) in dashboarding, user-centricity and fast usability are used to induce a company-wide rethinking regarding data processes

The Roadmap for Data Intelligence

The SAP Data Intelligence solution has already been offered since 2019 as a combination of its predecessor SAP Data Hub and SAP Leonardo Machine Learning. The numerous components such as the Modeler, Metadata Explorer and ML Scenario Manager are closely interlinked and are intended to optimally accompany the development phases of an AI project and bundle the results centrally. The platform can be complemented by SAP AI Business Services, which provide, for example, automated machine learning (AutoML) via an independent API.

SAP data management product roadmap: Data Intelligence will be merged into Data Warehouse Cloud in 2023 as part of the “Data Suite” project.

During its present life cycle, Data Intelligence has seen a long list of features added to the system. Many of these features have improved the capabilities of the data pipeline Modeler component. Tracing of data flow through pipelines and debugging functionality have been added as well as analysis tools for messages traversing through the different operators within a pipeline. New operators have been included to allow connection to ever more remote systems and protocols, making Data Intelligence version 3.2 a well-rounded, fully-featured data processing and orchestration, and machine learning operations platform.

With the strategic decision to merge Data Intelligence into Data Warehouse Cloud, SAP is tidying up the overall architecture of their data management solutions. Instead of maintaining data integration components in both systems, Data Intelligence will take over that role in the combined “Data Suite” as the project is currently called. Maybe less prominent but a hidden champion of the Data Intelligence components, its Metadata Explorer component will also become available to DWC users. This data cataloging solution offers an intuitive user experience to define data models, semantic metadata, and data governance aspects like access control for specific data objects. Considering the trends of the larger data management ecosystem, easily accessible metadata management will become a major factor in the foreseeable future.

The move to merge Data Intelligence into Data Warehouse Cloud will mark the end of DI as a standalone product. As this is still very fresh news, the timeline is subject to change. According to current plans, the integration of the two systems is scheduled to begin in the first quarter of 2023.

ML Scenario Manager: The self-service machine learning operations platform will become available to all Data Warehouse Cloud users in the near future.

The Future of Business Data Platforms

The strategic decision by SAP to further combine their standalone products into a single cloud-based data management solution aligns well with the latest trends in the business data ecosystem: As data processing software components have reached a maturity level where most commons use cases can be easily covered, the focus of attention shifts to the higher abstraction layer of actual user experience. Data processing platforms should no longer only cater to IT staff - database admins and software engineers by training - but to the business users. Enabling business users to create data models and analyses on their own increases efficiency and accuracy of business intelligence processes. Semantics of business data don’t have to be translated and spelled out for IT staff to create the correct data model and queries. As data literacy of business department staff increases, this trend strengthens ownership not only of the source data but of the entire value chain right up to the decisions made.

This data democratization trend is paralleled by the architectural concept called Data Mesh. Data Mesh architecture promotes a central self-service data platform and data governance framework. Within these infrastructural boundaries, business departments work autonomously and create so-called data products which are shared with other domains. Merging the data processing capabilities of SAP Data Intelligence into Data Warehouse Cloud fits perfectly with this pattern. We have already highlighted DWC Spaces as a means to enable self-service data management for business users in a previous blog post. Now this mechanism will be extended with mature data processing, analysis, and machine learning operations tools developed in the Data Intelligence branch. The results will be a cloud business data platform capable of pushing modern data-driven organizations to the next level of efficiency.

If you want to know more about the Data Mesh architecture and how DWC can help your organization's digital transformation, subscribe to our NextLytics blog as more details will be shared in the coming weeks.

Do you have questions about this or other topics? Send us a message or make an appointment for an exchange. We are happy to assist you in the strategic selection of a data management system - open source architecture or licensed - and support you in the implementation and conceptual design of initial use cases.