feat(query): add New Flight transport

[dqhl76]

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

This PR adds the complete default-off New Flight transport and separates transport mechanics from exchange-domain pipeline code.

It includes:

• reliable inbound and outbound logical streams over Arrow Flight DoExchange
• sequence acknowledgements, replay deduplication, reconnect attempt budgets, reconnect leases, and terminal error propagation
• fragment and statistics stream integration
• per-source backpressure and exchange packet routing
• pipeline adapters for local, legacy remote, and reliable remote exchange paths
• a clear module boundary between transport/legacy, transport/reliable, and exchange-owned adapters

Implementation

The old mixed v1/network module is replaced by two explicit transport implementations:

• v1/transport/legacy: the existing ping-pong transport and outbound buffer
• v1/transport/reliable: protocol framing, reconnect policy, logical inbound/outbound state machines, and the ReliableInboundDelivery boundary

Exchange-specific concerns remain under v1/exchange, including DataBlock serialization, local channels, quota queues, TID/batch routing, pipeline processors, and statistics delivery. The reliable transport operates on FlightData and no longer depends on Settings, DataBlock, pipeline types, or QueryContext.

The setting-to-transport conversion is also owned by the exchange layer, so transport selection remains outside the protocol implementation.

Compatibility and rollout

• enable_experiment_new_flight defaults to 0.

Data stream	Execution handling	Existing Flight	New Flight
Packet-level fragment	Pipeline sink/source processors	do_get	Reliable do_exchange
Block-level fragment	OutboundChannel / InboundChannel	Ping-pong do_exchange	Reliable do_exchange
Statistics	Runtime sender/receiver tasks	do_get	Reliable do_exchange
Local fragment	In-process pipeline channels	No Flight RPC	No Flight RPC

Enable

    SET flight_client_keep_alive_time_secs = 120;
    SET flight_client_keep_alive_interval_secs = 30;
    SET flight_client_keep_alive_retries = 3;

    SET flight_connection_max_retry_times = 5;
    SET flight_connection_retry_interval = 3;
    SET flight_client_timeout = 60;

Tests

• Unit Test
• Logic Test
• Benchmark Test
• No Test - Explain why

Focused tests cover wire compatibility, packet validation, replay deduplication, reconnect lease replacement, reconnect budget exhaustion, sender/receiver failure propagation, legacy ping-pong and buffering, fragment TID routing, statistics completion/failure, and end-to-end exchange setup/cleanup.

Type of change

• Bug Fix (non-breaking change which fixes an issue)
• New Feature (non-breaking change which adds functionality)
• Breaking Change (fix or feature that could cause existing functionality not to work as expected)
• Documentation Update
• Refactoring
• Performance Improvement
• Other (please describe):

AI assistance

• AI usage: An AI coding agent helped implement the reliable inbound/outbound transport, reorganize the legacy and reliable module boundaries, add focused tests, and prepare the PR summary.
• Responsible human: @dqhl76
• The responsible human has read every line of this diff and can explain each change

---

This change is

[dqhl76]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. 🎉

Reviewed commit: d79c7853c7

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[github-actions]

Docker Image for PR

• tag: pr-20397-1120c7b-1787846362

note: this image tag is only available for internal use.

[github-actions]

🤖 CI Job Analysis

Workflow: 33318362674

📊 Summary

• Total Jobs: 92
• Failed Jobs: 1
• Retryable: 0
• Code Issues: 1

❌ NO RETRY NEEDED

All failures appear to be code/test issues requiring manual fixes.

🔍 Job Details

• ❌ linux / test_private_tasks: Not retryable (Code/Test)

---

🤖 About

Automated analysis using job annotations to distinguish infrastructure issues (auto-retried) from code/test issues (manual fixes needed).