fix: [branch-1.0] backport five bug fixes for 1.0.1

[andygrove]

Which issue does this PR close?

Backports five bug fixes from main to branch-1.0 for the 1.0.1 patch release, plus one clippy chore needed to make branch-1.0 CI green at all. Closes #5334 on branch-1.0.

Rationale for this change

branch-1.0 diverged from main at #5240, and 79 commits have landed on main since. Four fixes were already backported in #5321. Of the remaining fix: commits, these five address correctness bugs or outright job failures that are present in 1.0.0, and none of them depend on post-1.0 features.

Backported	Fixes	Impact
#5185	#5334 (priority:critical)	Boolean-to-decimal cast produces an invalid Decimal128 when 10^scale does not fit the target precision.
#5376	#5270 (priority:critical)	arrays_overlap compares raw IEEE 754 bit patterns on flat float arrays, so Comet silently returns a different answer than Spark when a NaN is present.
#5364	Part of #5149 (priority:critical)	to_time / try_to_time do not match Spark's whitespace trimming.
#5392	#5391	Native shuffle fails with a 2GB task serialization OOM on jobs with very many partitions.
#5443	#5208	Native make_date rejects years that Spark accepts.

#5185 is titled feat: on main because it removes the native cast rather than repairing it, but it is the fix for a critical correctness bug and belongs in a patch release.

What changes are included in this PR?

Six clean cherry-picks, in main order, with original authorship and (cherry picked from commit ...) trailers preserved. No changes were needed to adapt any of them to branch-1.0.

Also included: #5400 (clippy for Rust 1.98)

The five bug fixes alone left CI red, and not because of the backports. branch-1.0 sets RUST_VERSION: stable, stable has since rolled to 1.98.0, and 1.98 added two lints that fire on code already on the branch:

error: using `chunks_exact` with a constant chunk size
  --> spark-expr/src/bloom_filter/spark_bit_array.rs:80:64
error: manual implementation of `isolate_lowest_one`
  --> spark-expr/src/nondetermenistic_funcs/internal/mersenne.rs:152:12

Neither file is touched by any of the five fixes, so this fails on any PR targeting branch-1.0 right now. #5400 fixed both on main and was never backported, so it is included here as a sixth commit.

Deliberately excluded

• fix: release native shuffle reservation after spill failure #5461 (release native shuffle reservation after spill failure) cherry-picks cleanly but does not compile on branch-1.0: its regression test references spill_accounted_input_buffers, repeated_spill_buffer_bytes, and metrics.memory_spilled_bytes, all introduced by fix: report native shuffle write metrics accurately #5370. Backporting it would mean dragging fix: report native shuffle write metrics accurately #5370 along or rewriting the test.
• fix: ignore reader-side parquet.hadoop.vectored.io.enabled in Iceberg native-write detection #5410 is not applicable. It narrows a predicate introduced by feat: detect Iceberg V2 writes and emit fall-back reasons #5298 (Iceberg V2 write fall-back detection), which is not on branch-1.0.
• fix: preserve Spark semantics for dictionary-encoded Parquet inputs and reject dictionary targets #5234, fix: avoid duplicate CheckOverflow evaluation for decimal division #5225, fix: support map casts with NullType elements #5045, fix: guard against silent fail_on_error loss in scalar wiring (#5074) #5359 apply cleanly but are labelled enhancement upstream, not bug fixes.
• fix: report native shuffle write metrics accurately #5370, fix: report native child spill metrics in shuffle tasks #5445, fix: track peak memory before JVM shuffle spill #5463, fix: expose native memory usage to Spark #5408, fix: format NativeMemoryConsumer id in toString #5398, fix: RAII for memory pool registration  #5464 correct Spark UI metrics rather than query results. A patch release is not the place for observability behaviour changes with no correctness gain, and fix: report native child spill metrics in shuffle tasks #5445 conflicts across five files.
• fix: preserve Catalyst nullability and field IDs in native Parquet writes #5369 is a real bug, but spark.comet.parquet.write.enabled defaults to false on branch-1.0 and is documented as highly experimental and not for production, so no 1.0.0 user is exposed.

Still open for 1.0.1

Two milestoned correctness issues have no fix on main yet, so there is nothing to backport for them:

• to_time / try_to_time: native parser rejects 'T12' and '12:30:45.' which Spark accepts #5366 (priority:critical) — fix: match Spark whitespace trimming in to_time and try_to_time #5364 fixes only the whitespace half of to_time. The native parser still rejects 'T12' and '12:30:45.', which Spark accepts.
• CometHashAggregateExec doesn't participate in Spark's AQEPropagateEmptyRelation optimization #4412 (priority:high) — CometHashAggregateExec does not participate in AQEPropagateEmptyRelation.

How are these changes tested?

Each fix carries its own regression tests, which came across with the cherry-picks. Verified locally on this branch:

• cargo clippy --color=never --all-targets --workspace -- -D warnings — clean
• cargo fmt --all -- --check — clean
• cargo check --benches — clean
• cargo test -p datafusion-comet-spark-expr — 610 passed
• cargo test -p datafusion-comet-shuffle — 31 passed
• ./mvnw test-compile — BUILD SUCCESS
• ./mvnw spotless:check scalastyle:check — clean for the parent, comet-common, and comet-spark modules
• CometSqlFileTestSuite — 436 passed, covering all six new/changed .sql fixtures
• CometCastSuite — 168 passed, 8 ignored
• CometArrayExpressionSuite — 46 passed
• CometNativeShuffleSuite — 28 passed
• CometNativeShuffleInputRDDSuite — 1 passed

[comphead]

Thanks @andygrove

[andygrove]

Merged. Thanks @comphead and @sunchao