Databricks Broadens Data Integration Capabilities with Launch of Lakeflow Connect

Databricks has announced the general availability of Lakeflow Connect, a tool that enables no-code data ingestion from SaaS applications and databases.

As part of the launch, the company has introduced connectors for Salesforce Sales Cloud and Workday, marking them as the first of many planned connectors to provide a simple and scalable way to ingest data into Databricks. The aim is to simplify data integration for organizations.

Last year, Databricks introduced LakeFlow Connect as a comprehensive solution for data ingestion, transformation, and orchestration. With its latest announcement, Lakeflow Connect has transitioned from its preview phase to full general availability. Lakeflow Connect is powered by serverless compute and is available across major cloud providers, including AWS, Azure, and GCP.

Efficient data integration plays a critical role in the modern data-driven world, where businesses depend on quick and efficient access to diverse information sources. It ensures that data is unified and accurate. It also helps break down silos that can hinder productivity and allow organizations to derive actionable insights from their data.

However, organizations often face challenges in data integration, including navigating unique and frequently changing APIs and managing infrastructure for tools like virtual machines or streaming technologies. The complexities of working with diverse data sources add to the challenge.

(Ye-Liew/Shutterstock)

As a result, teams spend excessive time building, optimizing, and patching together pipelines, which slows down projects. It can also lead to fragmented ETL architectures that are difficult to govern.

“Today, it’s difficult to build and maintain ingestion pipelines for SaaS applications yourself because each API is unique and changes often,” shared Databricks. “Meanwhile, database ingestion can be even more complex, often requiring extra infrastructure such as virtual machines for extraction tools, Kafka for streaming, and a strong understanding of change data capture (CDC) technologies for efficient ingestion.”

By introducing no-code connectors, Databricks aims to remove some of the complexities of managing diverse data sources and changing APIs. The data cloud and AI vendor claims that the serverless architecture eliminates the need for additional infrastructure, like virtual machines or Kafka for streaming. The platform also manages schema evolution, third-party API upgrades, and provides built-in observability with alerts that can reduce complexity and enable optimized workflows.

The Salesforce connector is designed to enable organizations to import CRM data for applications like customer behavior insights and sales trend analysis. It accommodates specific Salesforce features, such as custom objects and formula fields.

This functionality builds on the strategic partnership between Databricks and Salesforce, established in 2023, which focuses on creating a seamless, zero-ETL integration experience between their platforms. The connector represents a step forward in aligning Salesforce’s CRM capabilities with Databricks’ advanced analytics and AI tools.

The new Workday connector facilitates data ingestion from custom reports, helping businesses centralize employee workforce analytics. Both connectors integrate with other Databricks Lakeflow engineering suite, which includes Delta Live Tables for data transformation and Workflows for pipeline orchestration.

The Databricks Asset Bundles (DABs) are used to manage continuous integration and delivery, while the open-sourced Unity Catalog provides tools to organize and govern data and pipelines. Additionally, the Lakehouse Monitoring feature tracks data quality throughout the workflows.

Databricks shared a few customer success stories of early adopters, including Porsche Holding Salzburg (Porsche), one of the largest automotive manufacturers in Europe. Porsche leveraged the LakeFlow Connect Salesforce connector to overcome data integration challenges and achieve seamless access to customer data stored in Salesforce Sales Cloud.

“Using the Salesforce connector from Lakeflow Connect helps us close a critical gap for Porsche from the business side on ease of use and price,” emphasized Lucas Salzburger, Project Manager, Porsche Holding. “On the customer side, we’re able to create a completely new customer experience that strengthens the bond between Porsche and the customer with a unified and not fragmented customer journey.”

In the last couple of years, Databricks has been keen to solidify its presence in the data integration and analytics space. In October 2023, the company acquired Arcion Labs for $100 million, signaling its intent to enhance its capabilities in real-time data replication and change data capture (CDC). This acquisition brought Arcion’s expertise in connectors for over 20 enterprise databases and data warehouses into the Databricks ecosystem.

The Arcion acquisition played a key role in the development of LakeFlow Connect. Arcion’s advanced CDC technology gave Lakeflow Connect the ability to handle quick and efficient data replication, supporting both real-time and on-demand data transfers.

We can expect Databricks to announce more connectors in this year’s Databricks’ Data + AI Summit, scheduled for June in San Francisco, CA. Databricks shared that connectors for Google Analytics, Microsoft SQL Server, Oracle NetSuite, PostgreSQL, ServiceNow, and SharePoint are part of its roadmap.

Thriving in the Second Wave of Big Data Modernization

Confluent and Databricks Join Forces to Bridge AI’s Data Gap

The post Databricks Broadens Data Integration Capabilities with Launch of Lakeflow Connect appeared first on BigDATAwire.