Atlan’s Big Plans for Reimagining the Data Control Plane

If you’re in the hunt for an enterprise data catalog, you may want to keep Atlan on your speed dial, as the young company is quickly gathering momentum. In fact, Atlan recently nabbed the number one overall rating from Forrester in the space. But as Atlan’s CEO tells us, there’s a lot more innovation yet to come from data catalogs.

In its recent Forrester Wave for Enterprise Data Catalogs, Forrester gave the Atlan Enterprise Data Catalog a score of 4.20 and gave the company’s strategy a score of 4.50, both of which were higher than all 11 other data catalog vendors in the report. The only category in which Atlan did not nab first place was market presence, where the six-year-old company scored a 2.00, well behind bigger, older, and more established players in this market.

“Even as a recent entrant in this market, Atlan’s Third-Gen Data Catalog is quickly outpacing established players by adeptly anticipating and addressing strategic customer needs through automation,” Forrester lead analyst Jayesh Chaurasia wrote. “Atlan is a visionary player with a clear, ambitious goal: to become the data and AI control plane enabling complex business use cases.”

The Forrester analysts liked several aspects of Atlan’s current offering, including its capability to enable data democratization and self-service through automated metadata tracking, its use of GenAI to assist with discovery, its end-to-end lineage tracking, “and a Netflix-like personalized experience to all business and technical personas.”

Forrester Wave for Enterprise Data Catalogs 3Q-2024 (Image courtesy: Forrester)

Atlan scored low in revenue, which isn’t surprising for a relatively new entrant into the space. However, Forrester gave it a decent score for the number of customers, which shows the company is gaining ground.

Three Types of Data Catalogs

In a recent interview with Datanami, Atlan CEO and co-founder Prukalpa Shankar discussed the state of the data catalog market in 2024, the challenges the company had to overcome to reach its current state, and where she sees the future taking data catalogs.

“I think maybe the most overused word of the last couple of years has been catalog,” Shankar said. “When someone asks me something, I say, what do you mean when you say the word ‘catalog,’ because it means slightly different things.”

There are three types of data catalos, Shankar said, starting with catalogs that store and expose technical metadata to enable applications to share data, such as AWS‘s Amazon Glue (Snowflake’s new Polaris catalog and Databricks’ Unity Catalog would also fit the bill here).

“Metadata is becoming incredibly important for driving downstream use cases,” Shankar said. “And so we’re seeing vendors across the space opening up their metadata APIs, almost like, in the application world, what single sign-on was for SaaS applications. We’re seeing metadata becoming the single sign-on for the data world.”

The second type of data catalog is the data dictionary version, which gets closer to the users and requires a better user experience, Shankar said. Tableau led the way with its Tableau Data Catalog, which allows users to discover what various metrics meant within the context of the BI environment so they can make sense of it. Another product like this is dbt Labs Explorer, she said.

“The third is what we think of ourselves, which is more of the control plane version of a catalog,” Shankar said. “The foundation of the control plane is in your metadata layer, which is being able to bring together metadata from all of these ecosystems, stitch it together, make it intelligent, make sense of it, but then drive the use cases” across these ecosystems.

Data Control Planes

The control plane version of a data catalog that Atlan builds must be able to handle a large diversity of data, users, and tools. Data of all types; users like data analysts, data engineers, and data scientists; and tools ranging from BI products to ETL and data transformation tools to data warehouses and date lakes, all must work with this system.

(Panchenko Vladimir/Shutterstock)

As Forrester points out, Atlan has done a good job of handling the current ecosystem of data, tools, and users. The company has tapped AI and machine learning to automate metadata tracking where it can, thereby lifting the burden of manually stitching and staging data off the shoulders of data stewards.

“Three years ago, I had written this article called Data Catalog 3.0…that said metadata is becoming big data and we need to think about the foundational computation systems of metadata the way we thought about big data,” she said. “The interesting thing, three years later is, I don’t think it’s becoming big data. It is big data. We have customers who, in their starting week, are bringing in millions of assets into [the catalog] The scale of what we are dealing with from a metadata perspective is a whole different scale than what existed five or six years ago.”

The automation of metadata tracking is important now, but it will become even more important in the future, as the volume and variety of use cases that data catalogs must manage expands ever outward and upward.

“In two years from now, our data consumers will be LLMs [large language models] and in this LLM stack, there’s a whole different world of problems that we’re dealing with,” Shankar said. “We’re probably not going to only stick to a single foundational model. We’re going to have significant multiple deployments across architectures. We’ll deal with unstructured data. And the only thing that that’s preventing us from getting to that world is the concept of AI-ready data.”

Solving Data Management

The foundational challenges in data management haven’t changed in more than 25 years, Shankar said. Getting the right data to the right place at the right time remains the ultimate goal. But of course, the type of data, and the places that people want to consume it–not to mention the timeline (i.e. now)–have changed a lot, which is part and parcel of the challenge faced not just by data catalog vendors like Atlan, but the data management field as a whole.

Prukalpa Sankar is the co-founder and CEO of Atlan

Recent industry events, such as the emergence of Apache Iceberg as a standard for table formats and the Iceberg REST API for connecting to metadata catalogs, such as Snowflake’s Polaris and Databricks’ Unity Catalog, are good for customers. Shankar hopes that drives the discussion toward greater openness higher up the data catalog stack, and eventually into the control plane.

“I’m very bullish about the version of the world that is moving to more and more open standards,” Shankar said. “There have been now foundational improvements I think from a from a data lake layer, with open standards from your data itself, so you can bring your own compute. I think the same will happen in the metadata layer.”

Customers naturally want to avoid lock-in, whether it’s a cloud lock-in, database lock-in, table format lock-in, or data catalog lock-in. Even if the Atlan product is not open source, Shankar said that Atlan strives to be open with its platform, and to open up access to its metadata. “The more players start opening up metadata, the more customers start asking for it,” she said.

Atlan uses a graph database to help it make sense of the different types of metadata that it tracks. That includes table metadata, operational metadata from data pipelines, lineage metadata from SQL transformations, and compliance metadata, which is tracked as tags. By collecting and tracking all this metadata as graph and exposing it through the control plane, Atlan is able to deliver better visibility and access to customers.

“I had a customer the other day who said ‘Data storage is cheap. Data confusion is not,’” Shankar said. “And if you see the evolution, the final leg is that our end users are using data, trusting data, [and embarking upon] data-driven decision making.

“The final leg actually is still very similar to what it used to be 15 years ago, despite having the ecosystem going through three layers of technology transformations,” she continued. “And I think we’re now finally at a point where we can solve for the final leg. I think that’s the last step to the problem statement that needs to be solved.”

Active Metadata – The New Unsung Hero of Successful Generative AI Projects

Atlan Plants Itself in the Middle of the Data Governance Map with $105M Round

The post Atlan’s Big Plans for Reimagining the Data Control Plane appeared first on Datanami.