Describe the enhancement requested
The Arrow IPC Message flatbuffer defines a custom_metadata field ([KeyValue]) that allows attaching arbitrary key-value pairs to individual messages (record batches). The C# implementation currently ignores this field on both read and write.
Motivation
PyArrow has supported writing custom_metadata via write_batch(batch, custom_metadata=...) and reading it via read_next_batch_with_custom_metadata() since v11.0.0. The Rust implementation also added support recently (apache/arrow-rs#9444). The C# library should reach parity with other implementations.
This enables use cases such as:
RPC frameworks embedding method routing info in per-batch metadata
Logging/tracing metadata attached to individual batches
Application-level framing information in streaming IPC
Proposed Changes
Writer: Add WriteRecordBatch(RecordBatch, IReadOnlyDictionary<string, string>) overloads (sync + async) to ArrowStreamWriter that serialize the key-value pairs into the Message's custom_metadata field.
Reader: Expose a LastBatchCustomMetadata property on ArrowStreamReader that returns the custom_metadata from the most recently read batch's Message.
Describe the enhancement requested
The Arrow IPC Message flatbuffer defines a custom_metadata field ([KeyValue]) that allows attaching arbitrary key-value pairs to individual messages (record batches). The C# implementation currently ignores this field on both read and write.
Motivation
PyArrow has supported writing custom_metadata via write_batch(batch, custom_metadata=...) and reading it via read_next_batch_with_custom_metadata() since v11.0.0. The Rust implementation also added support recently (apache/arrow-rs#9444). The C# library should reach parity with other implementations.
This enables use cases such as:
RPC frameworks embedding method routing info in per-batch metadata
Logging/tracing metadata attached to individual batches
Application-level framing information in streaming IPC
Proposed Changes
Writer: Add WriteRecordBatch(RecordBatch, IReadOnlyDictionary<string, string>) overloads (sync + async) to ArrowStreamWriter that serialize the key-value pairs into the Message's custom_metadata field.
Reader: Expose a LastBatchCustomMetadata property on ArrowStreamReader that returns the custom_metadata from the most recently read batch's Message.