Apache Iceberg and Delta Lake both solve the “a folder of files isn’t a table” problem well, but neither one started life solving the specific problem Apache Hudi was built around: how do you continuously stream row-level changes, inserts, updates, and deletes, into a data lake at high volume without either rewriting massive files constantly or falling hopelessly behind the source system. If you’ve read our Iceberg vs Delta Lake comparison, Hudi is the third major open table format worth knowing, and it earns its place with a genuinely different origin story and a sharper focus on upserts and CDC-style ingestion than either of the other two.

This tutorial covers what Hudi actually is, how its timeline and table-type model works, the indexing system that makes upserts fast at scale, and a hands-on quickstart using Spark SQL so you can watch an upsert and an incremental query actually happen rather than just reading about them.

What Is Apache Hudi?

(cite index=”48-1″>Apache Hudi, which stands for Hadoop Upserts Deletes and Incrementals, is an open-source data management framework developed by Uber in 2016 in response to the need for efficient processing and management of large real-time data volumes. Its open-sourcing by Uber and subsequent adoption by the Apache Software Foundation in 2019 solidified its role as a leading framework in the big data landscape.

(cite index=”53-1″>Hudi is a data lakehouse storage framework that enables record-level insert, update, and delete operations on cloud object stores, originally built at Uber for CDC ingestion at scale. It was the first technology to make mutable operations practical on immutable storage, paving the way for modern data lakehouses as we know them. (cite index=”50-1″>At Uber, having one of the world’s largest transactional data lakes has given the team direct incentive to push Hudi further, using advanced primitives like incremental pull to build chained incremental pipelines that reduce the compute footprint of jobs which would otherwise perform large, full scans.

That’s the throughline worth holding onto: where Iceberg and Delta Lake both treat “make writes and reads reliable on a data lake” as the primary goal, Hudi treats “make row-level upserts and CDC ingestion genuinely efficient at scale” as its primary goal, with everything else built around that.

How Hudi’s Architecture Works

(cite index=”47-1″>Apache Hudi maintains the timeline of all activity performed on the dataset to provide instantaneous views of the dataset. Hudi organizes datasets into a directory structure under a base path similar to Hive tables, broken up into partitions, and each partition record is distributed into multiple files.

The Timeline

(cite index=”51-1″>Hudi adopts an MVCC design, where a compaction action merges logs and base files to produce new file slices, and a cleaning action gets rid of unused or older file slices to reclaim space. Every meaningful action on a Hudi table, a commit, a delta commit, a compaction, a rollback, a savepoint, gets recorded as an instant on this timeline, giving the table a full, queryable history of everything that’s ever happened to it, similar in spirit to Iceberg’s snapshot history but organized as its own dedicated structure rather than a metadata tree.

(cite index=”52-1″>Hudi maintains this transaction log split into an active timeline, which has limited history to ensure fast access, and an archived timeline, which is more expensive to access during normal reads and writes. Hudi 1.0 introduced LSM trees for timeline management specifically to improve write efficiency and reduce the overhead of tracking that growing history over time.

Record Keys and the Hoodie Key

This is the mechanism that makes upserts fast, and it’s worth understanding directly rather than treating as a black box. (cite index=”51-1″>Hudi provides efficient upserts by mapping a given hoodie key, the combination of a record key and a partition path, consistently to a file id, via an indexing mechanism. This mapping between record key and file group never changes once the first version of a record has been written to a file.

In plain terms: once Hudi knows record order_id=42 lives in file group abc123, an upsert for that same record always routes to that same file group, forever. This is what lets an upsert operation skip scanning the entire table to find the row it needs to update, the index tells it exactly where to look.

Base Files and Log Files

(cite index=”54-1″>Hudi stores data in base files, typically Parquet, with delta log files for recent changes, all tracked by the timeline of commits. Base files hold the columnar, query-optimized version of your data. Log files, written in a row-oriented format, hold recent changes that haven’t yet been folded into a base file. Which of these a query actually has to touch, and how often log files get merged back into base files, is exactly what the table type decision below controls.

Table Types: Copy-on-Write vs. Merge-on-Read

Unlike Iceberg, where copy-on-write and merge-on-read are effectively per-operation delete strategies, Hudi treats this choice as a property of the table itself, decided up front based on the workload it’s meant to serve.

(cite index=”49-1″>Copy-on-Write (CoW) tables rewrite entire files on updates. All data is stored in base files, providing zero read amplification and excellent query performance. This is the default table type, ideal for read-heavy workloads. (cite index=”50-1″>Via copy-on-write, updates simply version and rewrite the files by performing a synchronous merge during the write itself, so by the time a write completes, every reader sees clean, fully-merged Parquet files with no extra work required at query time.

(cite index=”49-1″>Merge-on-Read (MOR) tables append updates to log files instead of rewriting base files, with compaction asynchronously merging those logs into base files later. MOR provides faster writes but requires merging during reads, making it suitable for write-heavy streaming scenarios. (cite index=”50-1″>The merge-on-read table type stores data using a combination of columnar and row-based file formats, with updates logged to delta files and later compacted into new versions of the columnar files, either synchronously or asynchronously.

(cite index=”51-1″>The intention behind merge-on-read specifically is enabling near real-time processing directly on top of the data lake, rather than copying data out to a specialized system that may not handle the data volume well, with a secondary benefit of reduced write amplification by avoiding a synchronous merge on every batch.

Query Types

Hudi tables can be read in a few distinct ways, and picking the right one matters for both correctness and cost.

(cite index=”51-1″>Snapshot queries see the latest state of the table as of a given commit or compaction action. For a merge-on-read table, this exposes near-real-time data by merging the base and delta files of the latest file slice on the fly, while for a copy-on-write table, it’s effectively a drop-in replacement for querying plain Parquet, since the files are already fully merged.

(cite index=”51-1″>Incremental queries only return new data written to the table since a given commit or compaction, which is the feature that makes Hudi genuinely distinctive as a CDC backbone. Instead of a downstream job re-scanning an entire table to find what changed, it asks Hudi directly for “everything written since commit X” and gets back exactly that, and nothing more.

A read-optimized query type also exists specifically for merge-on-read tables, reading only the compacted, columnar base files and skipping any un-compacted log data, trading a small amount of freshness for the fastest possible read performance.

Indexing: How Hudi Finds Records Fast

(cite index=”58-1″>Hudi’s indexing subsystem is designed to speed up snapshot queries and is maintained automatically by writes, tracking file listings along with column-level and partition-level statistics to help plan queries efficiently.

(cite index=”60-1″>The classic index type is Bloom, where a Bloom filter eliminates the dependency on an external system entirely by storing the filter directly in the footer of a Parquet data file, letting a write operation quickly check whether a given record key might already exist in a specific file without an external lookup service.

(cite index=”46-1″>Newer index types extend well beyond that, with support for bloom indexes, expression indexes, and secondary indexes, dramatically reducing query scanning costs, and (cite index=”58-1″>Hudi 1.x added record-level indexing mechanisms built on row-oriented file formats and bloom filters, giving point lookups on a primary key an especially fast path.

You can watch indexing’s effect directly. (cite index=”62-1″>Without an index on a filtered column, a query might scan every file in the table to find matching rows. After creating a bloom filter expression index on that column, the same query’s file scan count drops sharply, from scanning every file down to only the ones actually containing matching values, a concrete, visible demonstration of what the index is actually buying you.

Concurrency Control

Multi-writer support has historically been one of the trickier areas across all lakehouse table formats, and Hudi’s most recent major version made a real leap here. (cite index=”46-1″>Hudi 1.0 introduced Non-Blocking Concurrency Control (NBCC), allowing multiple writers to concurrently write to the same table without the more restrictive optimistic-locking conflicts that earlier versions of Hudi, and other formats, have historically relied on for concurrent writer coordination.

Hands-On Quickstart: Upserts and Incremental Queries With Spark SQL

Hudi’s most mature, fully-featured integration remains Apache Spark, so that’s what this quickstart uses. The goal is to actually see an upsert route to an existing record and an incremental query pull back only what changed.

Step 1: Start Spark With the Hudi Extension

spark-sql \
  --packages org.apache.hudi:hudi-spark3.5-bundle_2.12:1.0.2 \
  --conf spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension \
  --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog \
  --conf spark.serializer=org.apache.spark.serializer.KryoSerializer

That extension and catalog configuration is what teaches Spark SQL to understand USING HUDI table definitions and Hudi-specific table properties directly.

Step 2: Create a Table With a Record Key and Precombine Field

CREATE TABLE hudi_indexed_table (
  ts BIGINT,
  uuid STRING,
  rider STRING,
  driver STRING,
  fare DOUBLE,
  city STRING
) USING HUDI
OPTIONS (
  primaryKey = 'uuid',
  preCombineField = 'ts',
  type = 'cow'
)
PARTITIONED BY (city);

(cite index=”63-1″>The precombine field is used to de-duplicate multiple records within the same batch being ingested, keeping the record with the highest value for that field when duplicates for the same key show up together. (cite index=”63-1″>The record key field is required and uniquely identifies a record within each partition, the piece that makes the hoodie-key-to-file-group mapping described earlier actually work.

Step 3: Insert Data

INSERT INTO hudi_indexed_table VALUES
  (1000, 'c8abbe79-8d89-47ea-b4ce-4d224bae5bfa', 'rider-A', 'driver-M', 29.5, 'san_francisco'),
  (1001, 'e3cf430c-889d-4015-bc98-59bdce1e530c', 'rider-B', 'driver-N', 18.75, 'sunnyvale');

Step 4: Perform an Upsert

Update the fare for an existing ride, using the same record key:

UPDATE hudi_indexed_table SET fare = 34.0 WHERE uuid = 'c8abbe79-8d89-47ea-b4ce-4d224bae5bfa';

Because this table’s record key maps that uuid to a specific, already-known file group, Hudi doesn’t scan the whole table to apply this change, it goes straight to the relevant file group, exactly the mechanism described in the architecture section above.

Step 5: Query the Current Snapshot

SELECT * FROM hudi_indexed_table WHERE uuid = 'c8abbe79-8d89-47ea-b4ce-4d224bae5bfa';

You should see the updated fare of 34.0, reflecting the merged, current state of that record.

Step 6: Try a Point Lookup With the Record-Level Index

SET hoodie.metadata.record.index.enable=true;

SELECT * FROM hudi_indexed_table
WHERE uuid = 'c8abbe79-8d89-47ea-b4ce-4d224bae5bfa';

(cite index=”59-1″>Turning on the record-level index this way accelerates point queries specifically, letting Hudi jump straight to the relevant record rather than scanning partition-level file listings first.

Step 7: Create a Secondary Index and Watch Pruning in Action

CREATE INDEX record_index ON hudi_indexed_table (uuid);
CREATE INDEX idx_rider ON hudi_indexed_table (rider);

SELECT * FROM hudi_indexed_table WHERE rider = 'rider-B';

(cite index=”59-1″>A secondary index depends on the record index existing first, which is why the record index gets created before it here. Once in place, filtering on rider benefits from the same kind of file-level pruning that made the earlier point lookup fast.

Step 8: Run an Incremental Query

This is the feature that sets Hudi apart most clearly from a typical data lake table. First, capture a starting commit timestamp before making more changes, then insert or update a few more records, and query only what changed since that point:

SELECT * FROM hudi_table_changes('hudi_indexed_table', 'earliest', '20260101000000');

That single query pulls back exactly the rows that changed since the given point, without scanning the entire table, the same mechanism a downstream CDC consumer or an incremental ETL job would lean on in production.

Common Beginner Mistakes to Avoid

• Forgetting the precombine field. Without one, duplicate records arriving in the same batch have no defined way to resolve which version wins, leading to inconsistent results depending on processing order.
• Choosing copy-on-write for a write-heavy streaming workload. COW’s synchronous merge on every write becomes a real bottleneck under high-frequency ingestion. That’s precisely the scenario merge-on-read exists for.
• Neglecting compaction on merge-on-read tables. Log files accumulate on MOR tables by design, and skipping regular compaction means every read has to merge an ever-growing pile of logs, steadily degrading query performance over time.
• Assuming the record key can change. Once a record key maps to a file group, that mapping is permanent for that record’s life in the table. Changing what constitutes a record’s key generally requires a full table rewrite, not a simple update.
• Ignoring HoodieStreamer for CDC ingestion. Building custom Kafka-to-Hudi ingestion logic from scratch, rather than using Hudi’s own purpose-built streaming ingestion tool, is usually reinventing something Hudi already ships.

Hudi vs. Iceberg vs. Delta Lake, Briefly

All three formats now handle the fundamentals, ACID transactions, schema evolution, time travel, reasonably well, and the meaningful differences increasingly come down to what each one was originally optimized for. Iceberg leans toward the broadest multi-engine ecosystem and flexible partition evolution, covered in our Iceberg vs Delta Lake comparison. Delta Lake leans toward the deepest integration within Spark and Databricks specifically. Hudi leans hardest into record-level upserts, CDC ingestion, and incremental processing as first-class, heavily optimized capabilities rather than capabilities bolted on after the fact, a reasonable consequence of being built at Uber specifically to solve that exact problem at scale.

Who Should Use Apache Hudi?

Hudi tends to be the right fit for teams that:

• Run CDC pipelines ingesting continuous row-level changes from operational databases into a data lake.
• Need genuinely fast upserts and deletes on large tables, where re-scanning or rewriting large amounts of data on every change isn’t acceptable.
• Want built-in incremental query support so downstream jobs can process only what changed since their last run, rather than re-scanning full tables repeatedly.
• Are comfortable with Spark as the primary engine, since Hudi’s most complete feature support and tooling remain concentrated there.
• Value a purpose-built streaming ingestion tool, HoodieStreamer, over assembling custom ingestion logic from scratch.

Frequently Asked Questions

What does the name “Hudi” actually stand for? Hadoop Upserts Deletes and Incrementals, reflecting its original purpose: efficient upserts, deletes, and incremental data processing on top of Hadoop-style distributed storage.

Who created Apache Hudi? It was originally built at Uber in 2016 to handle large-scale, real-time data processing needs, and it was donated to and adopted by the Apache Software Foundation in 2019.

What’s the difference between Hudi’s Copy-on-Write and Merge-on-Read table types? Copy-on-Write rewrites entire data files synchronously on every update, giving fast, simple reads at the cost of more expensive writes, and is the default, best suited to read-heavy workloads. Merge-on-Read appends changes to separate log files instead, making writes much faster at the cost of needing to merge those logs during reads, better suited to write-heavy, near-real-time ingestion.

What is an incremental query in Hudi? It’s a query type that returns only the records that changed since a specified commit or compaction, rather than the entire table, letting downstream jobs process just the delta rather than re-scanning everything on every run.

Do I need Spark to use Apache Hudi? Spark has the deepest and most mature integration, but Hudi also integrates with Flink, Hive, and Presto/Trino to varying degrees, so Spark isn’t a strict requirement, just the environment where Hudi’s full feature set is most consistently available.

What is the precombine field used for? It resolves which version of a record wins when duplicate keys appear within the same batch of incoming data, typically by keeping the record with the highest value in that field, such as the most recent timestamp.

Can multiple writers write to the same Hudi table at once? Yes, especially as of Hudi 1.0, which introduced Non-Blocking Concurrency Control, allowing multiple writers to write to the same table concurrently with fewer of the coordination conflicts earlier versions could run into.

How is Hudi different from Iceberg and Delta Lake? All three now cover the core lakehouse fundamentals reasonably well, but Hudi was built from the outset specifically around fast row-level upserts, deletes, and incremental processing for CDC-style workloads, whereas Iceberg emphasizes broad multi-engine ecosystem support and Delta Lake emphasizes deep integration within Spark and Databricks.

Final Thoughts

Apache Hudi’s real distinguishing idea is treating a data lake table less like an append-only archive and more like something that can absorb a continuous, high-volume stream of row-level changes efficiently, the exact problem Uber needed solved at a scale most companies never reach, but that a growing number of teams running serious CDC pipelines run into regardless of their overall size. The timeline, hoodie-key-to-file-group mapping, and table-type choice covered here are what make that possible, and running the quickstart above, watching an upsert route directly to an existing file group and an incremental query pull back only what changed, is the fastest way to see why that architecture matters in practice rather than just in theory.

For the authoritative, continuously updated reference on every configuration option and index type covered here, Apache Hudi’s official documentation is the best place to go deeper.

For more breakdowns of open-source data infrastructure and DevOps tooling like this one, keep exploring the guides on CourseDrill.