[DO NOT MERGE] Import Spark Connect Gateway reference implementation

[viirya]

What changes were proposed in this pull request?

This PR imports the full Spark Connect Gateway reference implementation as a
single squashed commit.

This PR is intentionally too large to review and is NOT meant to be merged.
It exists only as the reference / target end-state so reviewers can see the whole
picture. The implementation will be split into smaller, self-contained,
reviewable PRs that land one at a time (in dependency order, bottom-up). After
each small PR merges, this reference PR is rebased and shrinks until everything
is imported and it can be closed.

The import includes: the Cargo workspace (15 crates under crates/), the Helm
chart (deploy/helm/scg/), e2e examples (deploy/examples/), docs (docs/),
the vendored spark.connect.* protos (proto/), and the full README. Every
imported source file carries the Apache license header; the vendored protos
retain their upstream ASF headers.

Split plan

The import lands as the following small, bottom-up PRs. Each keeps main green
under the full CI matrix (cargo fmt --all --check, cargo clippy --workspace --all-targets -D warnings, cargo build --workspace, cargo test --workspace)
and appends its crate(s) to the workspace members list. This reference PR is
rebased after each one merges.

#	Contents	Depends on	Status
1	Cargo workspace skeleton + vendored protos + genproto	—	✅ merged (#2)
2	routing	1	✅ merged (#3)
3	config	1	open (#4)
4	auth	1	pending
5	observability	1	pending
6	audit + ratelimit	1	pending
7	tenant	4	pending
8	store-memory + pool-static	2	pending
9	store-redis + healthcheck	2	pending
10	pool-k8s	2	pending
11	proxy (src + tests + examples)	1,2,4,5,6,7,8,9	pending
12	gateway binary	1–11	pending
13	Dockerfile, config.example.yaml, full README, docs/, deploy/, test/	12	pending

Dependency layers: genproto, routing, config, auth, observability,
ratelimit, audit are leaves; tenant→auth, and
store-memory/store-redis/pool-static/pool-k8s/healthcheck→routing;
proxy depends on 7 of those (and its tests/examples pull in 4 more, which is
why PRs 8–9 precede 11); gateway depends on all.

Why are the changes needed?

Implements the reference gateway from the approved SPIP (SPARK-58455), moving it
into the apache/spark-connect-gateway subproject repo.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

cargo check --workspace --all-targets and cargo fmt --check pass locally;
helm lint passes on the chart. Per-crate tests come in with the split PRs.

Was this patch authored or co-authored using generative AI tooling?

Yes, co-authored with Claude Code.

[alexandrefimov]

The routing crate landing gives the user-aware placement piece a home, and I have it ready against main: an OwnerLabelStrategy that matches SessionKey::user_id against an owner label on BackendMember, rotates within the subset one user owns, and declines instead of placing the session when user_id is empty or no member carries a matching label. cargo test --workspace is 21/21 with clippy and fmt clean.

It fails closed rather than falling back to round-robin because a wrong placement is not an affinity miss. Spark Connect never uses user_id as the execution identity — the plan runs as the process user of the driver it lands on. On 4.0.3 with a Ranger-enabled Kyuubi engine started as testalice, a client claiming user_id=bob is denied with "user [testalice] does not have [create] privilege": the backend authorizes as its owner, whatever the client says it is.

This is the strategy half only. The other half — labels reaching the pool from discovery — has nothing to attach to yet, since scg-pool-k8s surfaces addresses only.

I can't create a SPARK sub-task myself. Could you file one under SPARK-58455 whenever it fits your import order, or point me at a better venue for the patch?