[FEA][Story] Convert JSON reader output to Apache VARIANT representation

[abigalekim]

[FEA][Story] Convert JSON representations (table/strings) to Parquet VARIANT representation

Background

Apache's VARIANT logical type provides a binary representation for semi-structured data. The work to ingest these columns and expose their materialized type as a struct of two uint8 byte lists (shown below) is already being done at #22312. cuDF Spark is also working on supporting the Variant type in Spark RAPIDS (NVIDIA/cudf-spark#15171).

struct<
  metadata: list<uint8>,
  value: list<uint8>,
  ...
>

The JSON reader (read_json) currently parses JSON inputs into a table of the set of columns along with relevant metadata. For record style inputs, top-level JSON object fields become columns with primitive types (e.g., int, bool, string, float), JSON arrays are represented as list columns, and nested JSON objects are represented as struct columns. This story tracks adding a GPU-accelerated conversion path from the table produced by read_json into the Apache VARIANT representation. This allows users who previously have their data stored as serialized JSON object strings to start to use Variant functionality.

Plan

Status	PR	Scope
⏳ #23400, #23614	Core infrastructure for scalar, non-nested JSON to variant converter	Public cudf::io::parquet::experimental::{encode_variant, encode_strings_to_variant} API that takes in a table produced by JSON reader (cudf::io::read_json) and converts it to a variant column. Supports only scalar, non-nested types (NULLS, integers, floating-point, boolean, strings). Only supports JSON data < 2GB. encode_strings_to_variant should take in a string column of JSON strings and use the same underlying logic.
⏳ #TBD	List support	Extend the converter to handle JSON arrays represented as cuDF list columns. Encode each list row as a Variant array, including scalar lists, null lists, empty lists, and null elements within lists. Support nested list children only where the child type is already supported by the converter.
⏳ #TBD	Nested structs	Extend the converter to handle JSON objects represented as cuDF struct columns. Walk the nested table/column hierarchy using the JSON reader metadata for field names and recursively encode struct children as Variant object fields.
⏳ #TBD	Large data support	Extend the converter to support Variant payloads larger than the 2GB column-size limit by adding more value columns when a single value column cannot contain all encoded bytes.
⏳ #TBD	Variant converter microbenchmarks	NVBench divergence suite with scalar encoding, nested object encoding, array heavy inputs for the kernels in PRs 1-4.
⏳ #TBD	End-to-end example workload	Standalone cpp/examples/variant_workload/ exercising the read_json → encode_variant flow on representative VARIANT data, demonstrating the public API on realistic inputs.
⏳ #TBD	Shredding-aware variant conversion	Add support for shredded columns in the converter, where selected fields are materialized as typed columns alongside the unshredded metadata and value representation. Add public cudf::io::parquet::experimental::encode_variant_shredded that takes in a table produced by JSON reader and a list of field names to be converted into shredded columns.

Out of Scope (future stories)

• Variant writer: Producing Variant-annotated Parquet output. The logic used for conversion into Variant and the encoding should be reusable when we implement a writer.

References

• read_json API
• Apache Parquet VariantEncoding.md and VariantShredding.md