For years, a “data lake” table meant a folder full of Parquet files and a lot of hope. There was no real schema enforcement, no way to safely have two jobs write to the same table at once, and adding a column often meant an engineer quietly rewriting years of historical data to keep everything consistent. Apache Iceberg exists to fix exactly that, by giving a plain folder of files the guarantees you’d normally only get from a real database table.

If you’ve followed our Trino guides, Iceberg is the other half of the modern data lakehouse story, Trino is the query engine, and Iceberg is increasingly the table format that engine reads from. This tutorial covers what Iceberg actually is, how its metadata layer works, the features that make it more than “just Parquet files,” and a hands-on quickstart using Trino so you can see it working rather than just reading about it.

What Is Apache Iceberg?

(cite index=”35-1″>Apache Iceberg is an open table format originally created at Netflix to fix limitations of Hive tables at scale, most notably that Hive tracks table state by listing directories, which becomes slow and unsafe with concurrent writers. (cite index=”37-1″>It was originally built by Netflix engineers in 2017 to manage petabytes of data on S3, adding a smart metadata layer on top of plain files to provide ACID transactions, schema evolution, and time travel.

(cite index=”34-1″>Originally developed at Netflix and donated to the Apache Software Foundation in 2018, Iceberg graduated to become a top-level Apache project in May 2020. Since then, it’s become the closest thing the data engineering world has to a standard for lakehouse table formats, with native support across every major cloud platform and query engine.

The core idea is deceptively simple. (cite index=”34-1″>Think of it this way: traditional data lakes store files in directories, but they don’t really understand what a “table” is. Iceberg changes that by adding a metadata layer that tracks exactly what files belong to your table, what the schema looks like, and how the data is organized. (cite index=”31-1″>Apache Iceberg doesn’t store data in tables itself. Instead, it organizes data files to present them as a single, coherent table to whatever query engine is reading them.

The Problem Iceberg Actually Solves

It’s worth being specific about what broke with the older approach, since it explains every design decision that follows. (cite index=”32-1″>Apache Iceberg solves a deceptively simple problem: how to turn a collection of files into something that behaves like a database table. Before formats like Iceberg existed, the standard approach (commonly associated with Hive tables) tracked what belonged to a table by listing directories on disk and inferring structure from file paths and naming conventions.

That approach has real limits at scale. (cite index=”45-1″>Under the older Hive connector model, a query first has to call the metastore to get partition locations, then call the underlying file system to list all the data files inside each partition, and only then read metadata from each individual file before a query can even begin executing. Listing directories gets slower as tables grow, provides no protection against two writers colliding, and gives the engine no shortcuts for figuring out which files can be safely skipped for a given query.

How Iceberg’s Architecture Works

(cite index=”29-1″>An Apache Iceberg table has three layers organized hierarchically: the catalog layer sits at the top, followed by the metadata layer, which includes metadata files, the manifest list, and manifest files, with the data layer at the bottom. Understanding this stack, from the bottom up, is the single most useful thing you can do to actually understand Iceberg rather than just its feature list.

The Data Layer

At the bottom sit your actual data files, typically Parquet, though ORC and Avro are also supported. (cite index=”28-1″>There’s nothing special about their format, they live in whatever storage you’re using, S3, GCS, HDFS, or a local filesystem. Iceberg doesn’t reinvent file formats, it adds intelligence about how those files relate to each other.

The Metadata Layer

This is where the actual magic happens, and it’s built from three distinct pieces.

Manifest files track individual data files. (cite index=”28-1″>A manifest tracks individual data files, but it’s not just a list of paths, each entry records the file’s storage location, its format, its partition values, the record count, and column-level statistics, specifically the lower and upper bounds for each column. Those per-column bounds are what let a query engine skip an entire file without opening it, if you’re filtering for order_date > '2026-01-01' and a file’s manifest entry shows a maximum order_date of 2025-12-15, the engine knows immediately that file can’t possibly contain matching rows.

Manifest lists group manifests together into a snapshot. (cite index=”30-1″>The manifest list tracks all manifest files for the current snapshot, and those manifest files track subsets of data files with statistics about each one. (cite index=”28-1″>This two-level pruning, first eliminating manifest groups, then eliminating individual files using column statistics, is why Iceberg query planning is so much faster than the old approach of listing directories and scanning files one by one.

Metadata files hold the table’s global state. (cite index=”28-1″>This is where the current schema lives, alongside an array of every previous schema version, each with an ID, and the same applies to partition specs. Each snapshot in the metadata file references the schema ID and partition spec ID it was written with, so when the engine reads files written under an older layout, it knows exactly how to interpret them. (cite index=”28-1″>The metadata file also maintains the full snapshot history, the current snapshot and all previous ones, each pointing to its own manifest list, which is the mechanism underneath time travel.

The Catalog Layer

(cite index=”33-1″>The catalog is a centralized system responsible for managing and organizing table metadata. Query engines like Spark, Trino, and Hive don’t directly reference the metadata files themselves, they ask the catalog where the current one lives. (cite index=”34-1″>A catalog, like Hive Metastore or Nessie, holds a single pointer to your table’s current master metadata file.

That single pointer is the linchpin of the entire system. (cite index=”36-1″>The catalog’s atomic update mechanism ensures only one metadata file is ever considered “current,” preventing split-brain scenarios during concurrent writes. Every write creates a brand-new metadata file, and only once it’s fully written does the catalog atomically swap its pointer over to it. A reader either sees the old, fully consistent state or the new, fully consistent state, never something in between.

Snapshots: The Foundation of Time Travel and ACID

(cite index=”31-1″>A snapshot is the set of manifest files valid at a specific point in time. Every change you make to the data creates a new snapshot with updated manifest files and metadata. (cite index=”30-1″>Snapshot isolation and ACID transactions follow directly from this: every write creates a new snapshot, so readers always see a consistent view, and concurrent writers can be safely reconciled rather than corrupting each other’s changes.

(cite index=”31-1″>Iceberg follows snapshot-based querying, meaning you can access the entire set of data files as they existed at a specific timestamp, allowing you to access historical data and roll back to a previous version in case of data loss. Nothing about this requires a separate backup system or manual versioning discipline, it’s a structural property of how every write to an Iceberg table works.

Key Features Beyond the Architecture

Schema evolution. (cite index=”35-1″>Columns can be added, dropped, renamed, or reordered without rewriting any existing data files. Because column identity is tracked by a stable internal ID rather than by name or position, renaming a column is a metadata-only operation, no multi-terabyte rewrite job required.

Partition evolution and hidden partitioning. Iceberg tracks partition specs the same way it tracks schemas, versioned and evolvable. You can change how a table is partitioned going forward without touching historical data, and because partitioning is handled transparently by Iceberg rather than requiring users to know the physical layout, queries don’t need to reference partition columns explicitly the way older Hive-style tables often required.

Fast scan planning through statistics. (cite index=”34-1″>Iceberg’s metadata layer enables efficient query planning by storing partition and column-level statistics for every data file, allowing query engines to aggressively prune files during planning, eliminating files that don’t contain relevant data before any actual scanning begins.

Broad engine support. (cite index=”33-1″>Query engines including Spark, Trino, and Hive can all read and write Iceberg tables through a shared, open specification, rather than each engine needing its own proprietary table format.

Table versioning and rollback. (cite index=”33-1″>Tables can be rolled back to a previously stable state, using exactly the same snapshot mechanism that powers time-travel queries.

Iceberg Catalogs: Where Tables Are Registered

Choosing a catalog is one of the first real decisions in any Iceberg deployment, since it determines how tools discover and safely update your tables. Common options include a Hive Metastore for teams already running Hadoop-ecosystem infrastructure, AWS Glue Data Catalog for teams on AWS, a JDBC catalog backed by a plain relational database for a lightweight setup with minimal infrastructure, a REST catalog implementing Iceberg’s open catalog API for maximum portability across tools, and Nessie, which layers Git-like, multi-table versioned branching and commits on top of the standard catalog model.

For local learning and quick experimentation, which is exactly what the quickstart below uses, a REST catalog paired with local object storage is usually the fastest path to a fully working setup without needing an existing Hadoop or cloud environment.

Hands-On Quickstart: Iceberg Tables With Trino

The fastest way to actually understand snapshots, schema evolution, and time travel is to create a real table and watch these mechanics happen. This walkthrough uses Trino together with a lightweight REST catalog and local S3-compatible storage, all runnable on a single machine with Docker.

Step 1: Set Up the Environment

Create a docker-compose.yml with three services: MinIO for S3-compatible object storage, an Iceberg REST catalog server, and Trino itself:

services:
  minio:
    image: quay.io/minio/minio
    command: server /data --console-address ":9001"
    ports:
      - "9000:9000"
      - "9001:9001"
    environment:
      MINIO_ROOT_USER: admin
      MINIO_ROOT_PASSWORD: password

  rest:
    image: tabulario/iceberg-rest
    ports:
      - "8181:8181"
    environment:
      CATALOG_WAREHOUSE: s3a://warehouse/
      CATALOG_IO__IMPL: org.apache.iceberg.aws.s3.S3FileIO
      CATALOG_S3_ENDPOINT: http://minio:9000

  trino:
    image: trinodb/trino
    ports:
      - "8080:8080"
    volumes:
      - ./catalog:/etc/trino/catalog

Then configure Trino’s Iceberg catalog to talk to that REST server, matching the properties format Trino expects:

connector.name=iceberg
iceberg.catalog.type=rest
iceberg.rest-catalog.uri=http://rest:8181
iceberg.rest-catalog.warehouse=s3a://warehouse/
fs.native-s3.enabled=true
s3.endpoint=http://minio:9000
s3.region=us-east-1
s3.path-style-access=true
s3.aws-access-key=admin
s3.aws-secret-key=password

(cite index=”43-1″>This exact combination of a REST catalog, S3-compatible storage, and Trino’s Iceberg connector is a widely used pattern for local Iceberg experimentation, since it doesn’t require an existing Hive Metastore or cloud account to get started.

Save that properties file as catalog/iceberg.properties and start everything:

docker compose up -d

Step 2: Create a Schema and a Table

Connect using the Trino CLI:

docker exec -it  trino

Then create a schema and your first Iceberg table:

CREATE SCHEMA iceberg.retail;

CREATE TABLE iceberg.retail.orders (
  order_id BIGINT,
  customer_id BIGINT,
  order_date DATE,
  amount DECIMAL(10,2)
)
WITH (
  format = 'PARQUET',
  partitioning = ARRAY['month(order_date)']
);

That partitioning = ARRAY['month(order_date)'] clause is hidden partitioning in action. You never reference a separate partition column in your queries, Iceberg derives the partition transparently from order_date itself.

Step 3: Insert Data and Query It

INSERT INTO iceberg.retail.orders VALUES
  (1, 101, DATE '2026-01-15', 49.99),
  (2, 102, DATE '2026-01-20', 129.50),
  (3, 101, DATE '2026-02-03', 19.99);

SELECT * FROM iceberg.retail.orders WHERE order_date >= DATE '2026-02-01';

Notice the filter works directly against order_date, with no awareness needed of how the underlying files are physically laid out on disk.

Step 4: Try Schema Evolution

Add a column without rewriting any existing data:

ALTER TABLE iceberg.retail.orders ADD COLUMN status VARCHAR;

SELECT * FROM iceberg.retail.orders;

The three existing rows now show NULL for status, and every file written before this change remains untouched on disk. Only the table’s metadata changed.

Step 5: Explore Snapshots and Time Travel

Every table Iceberg manages exposes its own history through metadata tables. List every snapshot the table has ever had:

SELECT * FROM iceberg.retail."orders$snapshots";

Grab a snapshot_id from that result and query the table exactly as it looked at that point, before your schema change:

SELECT * FROM iceberg.retail.orders FOR VERSION AS OF 8947822938471625861;

You can also travel by timestamp rather than snapshot ID:

SELECT * FROM iceberg.retail.orders FOR TIMESTAMP AS OF TIMESTAMP '2026-01-15 10:00:00';

That single query, run against live production data, requires no separate backup system, no restore process, and no downtime, it’s simply reading an older, still fully intact snapshot.

Step 6: Roll Back and Clean Up

If a bad write needs undoing, Iceberg’s built-in procedures handle it directly:

CALL iceberg.system.rollback_to_snapshot('retail', 'orders', 8947822938471625861);

(cite index=”46-1″>And once you’ve accumulated enough history that old snapshots are no longer useful, a maintenance procedure removes the snapshots, metadata, and data files older than a given retention window, keeping table metadata from growing unbounded over time:

CALL iceberg.system.expire_snapshots('retail', 'orders', TIMESTAMP '2025-12-01 00:00:00');

Iceberg vs. Plain Hive Tables

The comparison worth internalizing as a beginner: a Hive-style table is really just a convention, a directory structure and a metastore entry describing it, with the engine trusting that convention rather than verifying it directly. An Iceberg table is a fully specified, self-describing structure where every file that belongs to the table, and every historical version of the table, is explicitly tracked. That difference is what eliminates the “list every file in this directory and hope nothing changed mid-scan” problem that made concurrent writes and fast query planning genuinely hard under the older model.

Common Beginner Mistakes to Avoid

• Treating Iceberg tables like ordinary folders of Parquet files. Never modify or delete files directly in storage. Every change needs to go through a query engine so the metadata layer stays in sync with what’s actually on disk.
• Never running snapshot expiration. Every write creates a new snapshot, and old ones accumulate indefinitely unless you actively expire them. On a frequently updated table, skipping this maintenance step leads to metadata bloat and slower query planning over time.
• Over-partitioning a small table. Hidden partitioning makes it easy to add partition transforms without thinking hard about them. A table with far more partitions than it has meaningful data volume per partition creates a large number of small files, which hurts query performance rather than helping it.
• Assuming every catalog behaves identically. A Hive Metastore catalog, a Glue catalog, and a REST catalog all implement the same specification, but operational details like locking behavior and latency can differ meaningfully between them, especially under concurrent write load.

Who Should Use Apache Iceberg?

Iceberg tends to be the right choice for teams that:

• Are building a data lakehouse and want database-like reliability, ACID transactions, safe concurrent writes, without giving up the flexibility and cost profile of object storage.
• Need to evolve table schemas or partitioning strategies over time without expensive, disruptive data rewrites.
• Query the same tables from multiple engines, Spark for ETL, Trino for interactive analytics, and want one shared table format instead of engine-specific copies of the same data.
• Need auditability or the ability to reproduce a query’s results exactly as they looked at a specific point in the past.
• Are already using or evaluating Trino and want a table format that plays natively with it rather than relying on older, directory-listing-based Hive tables.

Frequently Asked Questions

Is Apache Iceberg a database? No. Iceberg is a table format, a metadata specification that sits on top of files in object storage or a filesystem. It doesn’t run as its own server or storage engine, query engines like Trino, Spark, and Flink read and write Iceberg tables directly.

Who created Apache Iceberg? It was originally built by engineers at Netflix starting in 2017, donated to the Apache Software Foundation in 2018, and became a top-level Apache project in May 2020.

How does time travel actually work under the hood? Every write to an Iceberg table creates a new, immutable snapshot rather than modifying data in place. Older snapshots, and the manifest lists and files they point to, remain intact and queryable until explicitly expired, which is what lets you query a table exactly as it existed at a past point in time or snapshot ID.

Does adding a column require rewriting my data? No. Iceberg tracks columns by a stable internal ID rather than by name or position, so adding, dropping, renaming, or reordering columns is a metadata-only operation that doesn’t touch existing data files.

What file formats does Iceberg support? Parquet, ORC, and Avro are all supported for the underlying data files. Parquet is the most commonly used in practice for analytical workloads.

What’s the difference between a manifest file and a manifest list? A manifest file tracks individual data files along with their statistics and partition values. A manifest list groups a set of manifest files together into a single snapshot, acting as that snapshot’s table of contents.

Which catalog should I use for Iceberg? It depends on your existing infrastructure. Hive Metastore suits teams already on Hadoop-ecosystem tooling, AWS Glue suits AWS-native environments, a JDBC catalog is a lightweight option backed by a plain database, a REST catalog offers maximum portability across tools, and Nessie adds Git-like branching and multi-table transactions on top of the standard model.

Can I query Iceberg tables with more than one engine at the same time? Yes, and this is one of Iceberg’s core selling points. Because the table format itself is the shared source of truth rather than something baked into a single engine, Spark, Trino, Flink, and others can all read and write the same Iceberg tables safely, coordinated through the catalog’s atomic update mechanism.

Final Thoughts

Apache Iceberg’s real contribution isn’t any single feature, it’s turning “a folder of files” into something with the reliability guarantees people actually expect from a table: consistent reads, safe concurrent writes, a full history you can query, and schema changes that don’t require rewriting the past. Once the three-layer metadata architecture clicks, catalog, metadata layer, data files, features like time travel and instant rollback stop looking like magic and start looking like the obvious consequence of how the system is built.

The quickstart above is worth actually running rather than just reading, watching a FOR VERSION AS OF query pull back data exactly as it looked before a schema change makes the whole architecture concrete in a way no amount of reading about manifest lists can substitute for. For the authoritative, continuously updated specification and configuration reference, Apache Iceberg’s official documentation is the best place to go deeper, and our Trino architecture guide is worth revisiting once you’re querying Iceberg tables at real scale, since dynamic filtering and fault-tolerant execution both interact directly with the file-pruning statistics Iceberg’s manifests provide.

For more breakdowns of open-source data infrastructure and DevOps tooling like this one, keep exploring the guides on CourseDrill.