feat: Variant Support by c-thiel · Pull Request #2188 · apache/iceberg-rust

c-thiel · 2026-02-28T14:09:03Z

Which issue does this PR close?

Variant Support.
Arrow value support is currently missing as I am unsure how we want to extend Literal

What changes are included in this PR?

Core: Variant Type

crates/iceberg/src/spec/datatypes.rs — new Variant type
crates/iceberg/src/spec/values/literal.rs — Variant literal value
crates/iceberg/src/spec/schema/ — visitor, index, pruning, mod, id reassigner all handle Variant
crates/iceberg/src/spec/table_metadata.rs — metadata support

Avro

crates/iceberg/src/avro/schema.rs — read/write Variant in Avro

Arrow

crates/iceberg/src/arrow/schema.rs — map Variant to Arrow type
crates/iceberg/src/arrow/reader.rs — read Variant from Arrow
crates/iceberg/src/arrow/value.rs — Arrow value conversion
Minor fixes in caching_delete_file_loader.rs and nan_val_cnt_visitor.rs

Parquet

crates/iceberg/src/writer/file_writer/parquet_writer.rs — write Variant columns

Tests & Dev

crates/integration_tests/tests/read_variant.rs — new integration test for reading Variant data
dev/spark/provision.py — Spark provisioning to generate Variant test data

Are these changes tested?

Sure! Even integration tested :)

c-thiel · 2026-03-02T08:59:54Z

        let table_creation = TableCreation::builder()
            .name(name.clone())
            .schema(iceberg_schema)
+            .format_version(format_version)


Before this change existing tests where rightfully failing as we used to create a V2 table with a NS Timestamp column:
https://github.com/apache/iceberg-rust/actions/runs/22522306915/job/65248930667

This new logic determines the min format version required and uses that - but at least V2. Thus we switch now to V3 for ns timestamps.

c-thiel · 2026-03-18T07:06:01Z

@CTTY @liurenjie1024 @Xuanwo this would be ready for review!

CTTY

Thanks for the feature! Just took a look.

Also the test seems to be failing

c-thiel · 2026-04-10T01:38:10Z

@CTTY ready for another round!

CTTY

Mostly LGTM! Left some minor comments

Co-authored-by: Shawn Chang <yxchang@amazon.com>

c-thiel · 2026-05-13T07:26:07Z

@CTTY ready for another round!

CTTY

LGTM, thanks for working on this!

cc @blackmwk to also take a pass

nssalian · 2026-05-29T19:58:22Z

+
+    /// Check that all types in this schema are supported by the given format version.
+    ///
+    /// Mirrors Java's `Schema.checkCompatibility()`. Returns an error listing every


Accuracy fix on the doc comment: this says it "Mirrors Java's Schema.checkCompatibility()", but Java's version also rejects a non-null initial-default when formatVersion < 3, which isn't checked here (iceberg-go #932 guards variant defaults too). Could we narrow it to "Mirrors the type-version portion of Java's Schema.checkCompatibility()" so it doesn't read as full parity? Adding the actual default check is fine as a follow-up.

Good catch! The additional check was easy to add. I added it in a separate Commit so its easy to revert if it turns out to require more discussions:

abaa832

mbutrovich · 2026-05-29T20:04:03Z

        Ok(hive_type)
    }
+
+    fn variant(&mut self, _v: &iceberg::spec::VariantType) -> Result<Self::T> {


I see this was already discussed back in March and the call was to use "variant" (@CTTY's suggestion, with the Glue rendering as unknown: variant), citing apache/iceberg#15220 as open. iceberg-java has since landed on "unknown" for VARIANT in HiveSchemaUtil.java:170 (PR apache/iceberg#15964, April 28). Worth revisiting? Or is the decision still that the rendered shape here is fine? Not pushing either way, just flagging that Java's direction shifted after the original discussion. CC @nssalian

I would lean towards matching with Java. "unknown" is what the metastore-facing type string should be here.

Switch to "unknown" in 3c229b1

mbutrovich · 2026-05-29T20:10:02Z

+
+    /// Returns the minimum format version required for the type.
+    #[inline(always)]
+    pub fn min_format_version(&self) -> FormatVersion {


This currently doesn't recurse: Type::Struct(...).min_format_version() returns V1 even if the struct contains a Variant or TimestampNs. It happens to give the right answer at the schema level only because Schema::min_format_version iterates id_to_field.values(), which already includes every nested field as its own entry.

A SchemaVisitor would fit the codebase's pattern (the PR already extends visitors in seven places) and make the per-Type method correct for any input. Roughly:

struct MinFormatVersionVisitor; impl SchemaVisitor for MinFormatVersionVisitor { type T = FormatVersion; fn schema(&mut self, _: &Schema, v: FormatVersion) -> Result<FormatVersion> { Ok(v) } fn field(&mut self, _: &NestedFieldRef, v: FormatVersion) -> Result<FormatVersion> { Ok(v) } fn r#struct(&mut self, _: &StructType, results: Vec<FormatVersion>) -> Result<FormatVersion> { Ok(results.into_iter().max().unwrap_or(FormatVersion::V1)) } fn list(&mut self, _: &ListType, v: FormatVersion) -> Result<FormatVersion> { Ok(v) } fn map(&mut self, _: &MapType, k: FormatVersion, v: FormatVersion) -> Result<FormatVersion> { Ok(k.max(v)) } fn primitive(&mut self, p: &PrimitiveType) -> Result<FormatVersion> { Ok(match p { PrimitiveType::TimestampNs | PrimitiveType::TimestamptzNs => FormatVersion::V3, _ => FormatVersion::V1, }) } fn variant(&mut self, _: &VariantType) -> Result<FormatVersion> { Ok(FormatVersion::V3) } }

Then Type::min_format_version is visit_type(self, &mut MinFormatVersionVisitor) and Schema::min_format_version reuses the same visitor over the whole tree. Net effect: same public surface, recursion is correct for any Type, and the per-variant table lives in one place rather than split between the flat match here and the id_to_field iteration in Schema::min_format_version.

Good catch — I fixed the recursion. I went with a direct recursive match in Type::min_format_version (struct/list/map arms recurse) rather than a SchemaVisitor: it achieves both of your goals — correct for any Type and the per-type version table living in one place — without the extra visitor struct and Result plumbing (this fold is infallible). Schema::min_format_version now just maps over the top-level fields and relies on that recursion, so the old id_to_field split is gone too. Added a unit test for the nested case.

Let me know what you think!

mbutrovich · 2026-05-29T20:14:24Z

 fn assign_fresh_ids_to_type(field_type: &Type, next_id: &mut i32) -> Type {
    match field_type {
-        Type::Primitive(_) => field_type.clone(),
+        Type::Primitive(_) | Type::Variant(_) => field_type.clone(),


Style mismatch with id_reassigner.rs:105: that file does Type::Variant(v) => Ok(Type::Variant(v)), this one folds Variant into the primitive arm with field_type.clone(). Both work since VariantType is Copy. Picking one would be slightly easier on future readers.

Done — split it into an explicit Type::Variant(v) => Type::Variant(*v) arm so it matches id_reassigner.rs.

mbutrovich · 2026-05-29T20:15:25Z

            Ok(())
        }
+
+        fn variant(&mut self, _v: &crate::spec::VariantType) -> Result<Self::T> {


Applies to crates/iceberg/src/spec/schema/index.rs:57, 153, 305 and crates/iceberg/src/spec/schema/prune_columns.rs:242 and crates/catalog/glue/src/schema.rs:186 and crates/catalog/hms/src/schema.rs:143

Minor: crate::spec::VariantType inline rather than use'd. The codebase has mixed precedent here (some files use inline FQNs too) so feel free to ignore, but use would match the surrounding files' imports more closely.

useing it now :)

mbutrovich · 2026-05-29T20:16:07Z

+    }

-        Ok(())
+    fn variant(&mut self, _v: &VariantType) -> Result<Self::T> {


Not really a comment on this line, but: iceberg-java's TypeToMessageType#variant writes the Parquet group with LogicalTypeAnnotation.variantType(VARIANT_SPEC_VERSION). The Rust write path here doesn't add that annotation, so files written by iceberg-rust would carry a plain Struct(Binary, Binary) without the variant logical type marker. The integration tests are read-only against Spark-written data, so it isn't caught. Worth a tracking issue, or already on the roadmap?

this PR is unshredded variants only, with two follow-ups:

Write annotation (your comment): we emit a plain Struct(Binary,Binary) without variantType(...) since variant_experimental is off. Doesn't break Java read-back — it resolves variant by field-id, not the annotation — but I'll track adding it.

Shredded reads: a typed_value sub-field was being silently dropped (corrupt data). Added a guard that returns FeatureUnsupported + a test; full shredding reconstruction is a follow-up.

Guard added here:
b702f5a

Regarding 1)
iceberg-rust writes via AsyncArrowWriter, which derives the Parquet schema from the Arrow schema. In parquet 58.1.0, that path only emits the VARIANT annotation when the field carries the parquet_variant_compute::VariantType extension type and variant_experimental is enabled (otherwise logical_type_for_struct is a stub returning None). I couldn't find a public per-field hook to inject the annotation onto a plain Struct(Binary,Binary).

So the real cost is: enable variant_experimental + attach the extension type to the field. Two risksthat I se:

Turning on the feature may change how the reader decodes a VARIANT-annotated group (native VariantArray instead of Struct{metadata,value}) — could break the current read path that expects the struct.

New experimental dep surface.

Not sure how we should proceed. I think this maybe should be a separate issue?

I created this for now:
#2546

…g data

# Conflicts: # crates/iceberg/public-api.txt

mbutrovich

Thanks for addressing the feedback @c-thiel! This LGTM! Looking forward to putting some queries through this.

nssalian

lgtm. Thanks for the work @c-thiel!

c-thiel added 2 commits February 28, 2026 15:05

feat: Variant Support

488298d

fix: TableCreation uses correct format version

466b071

c-thiel commented Mar 2, 2026

View reviewed changes

brgr-s mentioned this pull request Mar 4, 2026

Geo type changes #2019

Closed

Merge branch 'main' into feat/variant-support

a70950e

CTTY reviewed Mar 19, 2026

View reviewed changes

Comment thread crates/iceberg/src/avro/schema.rs Outdated

Comment thread crates/iceberg/src/arrow/reader.rs Outdated

Comment thread crates/iceberg/src/avro/schema.rs

Comment thread crates/catalog/glue/src/schema.rs Outdated

c-thiel added 2 commits April 10, 2026 02:04

Merge branch 'main' into feat/variant-support

3ca9ebe

add nesting support, add Glue & HMS

4269b7d

Merge apache/main into feat/variant-support

c18ff16

nssalian mentioned this pull request May 6, 2026

EPIC: v3 Support Tracking #2411

Open

Shekharrajak mentioned this pull request May 10, 2026

feat: add PrimitiveType::Variant to iceberg spec #2423

Open

Merge branch 'main' into feat/variant-support

dffb01c

CTTY reviewed May 13, 2026

View reviewed changes

Comment thread crates/iceberg/src/spec/schema/mod.rs Outdated

Comment thread crates/iceberg/src/spec/datatypes.rs Outdated

Comment thread crates/iceberg/src/spec/table_metadata.rs Outdated

c-thiel and others added 4 commits May 13, 2026 08:33

Improve "invalid schema" error message

7f663e1

Co-authored-by: Shawn Chang <yxchang@amazon.com>

Merge branch 'main' into feat/variant-support

d7d9933

address comments

7d46d0c

Merge branch 'main' into feat/variant-support

1c93191

CTTY approved these changes May 14, 2026

View reviewed changes

Comment thread crates/iceberg/src/avro/schema.rs Outdated

c-thiel added 3 commits May 20, 2026 12:24

Merge branch 'origin/main' into feat/variant-support

0c341b8

assert variant record schema with let-else

71ef18a

Merge branch 'main' into feat/variant-support

1c54224

dannycjones mentioned this pull request May 29, 2026

Tracking issues of Iceberg Rust 0.10.0 Release #2527

Open

17 tasks

nssalian reviewed May 29, 2026

View reviewed changes

mbutrovich reviewed May 29, 2026

View reviewed changes

fix(arrow): reject shredded variant reads instead of silently droppin…

b702f5a

…g data

c-thiel mentioned this pull request May 31, 2026

Varaint Type: Add logical type annotation for writer #2546

Open

c-thiel added 7 commits May 31, 2026 16:17

refactor(spec): address variant review comments

3c229b1

feat(spec): reject non-null initial defaults below v3

abaa832

improve comment

09bff5a

Merge branch 'main' into feat/variant-support

d4cf655

regenerate public-api

18e608a

ci: re-trigger (flaky linker SIGBUS)

069b19f

Merge remote-tracking branch 'apache/main' into feat/variant-support

ff3145b

# Conflicts: # crates/iceberg/public-api.txt

mbutrovich approved these changes Jun 2, 2026

View reviewed changes

nssalian approved these changes Jun 3, 2026

View reviewed changes

Conversation

c-thiel commented Feb 28, 2026

Which issue does this PR close?

What changes are included in this PR?

Core: Variant Type

Avro

Arrow

Parquet

Tests & Dev

Are these changes tested?

Uh oh!

c-thiel Mar 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

c-thiel commented Mar 18, 2026

Uh oh!

CTTY left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

c-thiel commented Apr 10, 2026

Uh oh!

CTTY left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

c-thiel commented May 13, 2026

Uh oh!

CTTY left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

c-thiel May 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mbutrovich May 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mbutrovich May 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

c-thiel May 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

c-thiel May 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mbutrovich left a comment

Choose a reason for hiding this comment

Uh oh!

c-thiel Mar 2, 2026 •

edited

Loading

CTTY left a comment •

edited

Loading

c-thiel May 31, 2026 •

edited

Loading

mbutrovich May 29, 2026 •

edited

Loading

mbutrovich May 29, 2026 •

edited

Loading

c-thiel May 31, 2026 •

edited

Loading

c-thiel May 31, 2026 •

edited

Loading