Iceberg Rises, Catalogs Compete: How AI is Reshaping Data Architecture
Artificial intelligence (AI) is transforming how data is stored, accessed, and processed, driving a shift toward data lakehouse architectures. These architectures separate storage and tracking from computation, allowing flexible access to data using open table formats like Apache Iceberg, Delta Lake, and Apache Hudi. This design is increasingly essential for training and fine-tuning AI models that require fast, scalable access to diverse datasets.
Initially, the industry saw a “Table Format War” between these formats—a competition centered on vendor adoption and feature sets. Apache Iceberg has since emerged as a leader, with growing support from companies like AWS, Google, Databricks, Snowflake, Upsolver, and Dremio.
Welcome to the Lakehouse Catalog Era
As consensus forms around Iceberg and other formats, attention is now shifting to a new frontier: the “lakehouse catalog war.” Rather than conflict, this is a race to develop feature-rich catalog platforms that can serve as the governance backbone of lakehouse environments.
Lakehouse catalogs, such as Apache Polaris, Nessie, Gravitino, Unity Catalog, and Lakekeeper, manage assets like tables, views, namespaces, functions, and models. They provide centralized access control, support hybrid and multi-cloud interoperability, and streamline governance through automation and optimization tools.
The Iceberg REST Catalog Specification: A Game Changer
A major step in standardizing catalog interaction is the Iceberg REST Catalog Specification. This protocol defines how compute engines communicate with catalogs to read and write Iceberg tables. Platforms that fully implement it—like Polaris, Nessie, and Gravitino—can offer better compatibility and easier integration across the ecosystem.
Notably, Unity Catalog primarily supports reading through the specification but not writing, as it natively supports Delta Lake. It offers a feature called “Uniform” to enable Iceberg compatibility by mirroring Delta tables, making them accessible to Iceberg-first tools such as Dremio and Snowflake.
Four Key Trends Shaping the Catalog Wars
- REST Catalog Specification Enhancements: The spec is expected to evolve, introducing features like a Scan Planning Endpoint that can shift more optimization work to the catalog itself while maintaining open interoperability.
- Growth of Managed Services: Managed catalog services—like Snowflake’s Open Catalog and Dremio’s Hybrid Catalog built on Polaris—are simplifying deployment and adding features like automated performance tuning. The more managed capabilities a catalog provides, the greater its market traction.
- Supporting Ecosystem Expansion: The breadth of tools that integrate with a catalog will determine its success. Vendor support, ease of integration, and user-centric features will be decisive factors.
- Advanced Feature Sets: Future catalogs may offer lineage tracking, observability, and support for user-defined functions (UDFs), facilitating consistent governance across platforms and teams while supporting AI workflows and cross-departmental collaboration.
Conclusion: Charting the Future of Data Architecture
As data lakehouses become essential to AI and analytics strategies, the tools that govern them—catalogs—are entering a phase of rapid innovation. With Apache Iceberg establishing itself as a dominant format, catalogs like Polaris, Gravitino, Nessie, and Unity Catalog are now competing to define how assets are tracked, secured, and managed in an increasingly distributed world.
The evolving Iceberg REST Catalog Specification and the rise of managed catalog services will play a pivotal role in determining the future of data interoperability. These developments offer organizations the opportunity to harness the full power of lakehouse architectures in a secure, scalable, and AI-ready manner.
As the “catalog wars” gain momentum, the decisions made today will shape how businesses manage data in tomorrow’s AI-driven world.