feat($lookup): sub-pipeline + let (correlated lateral)

[douglasjordan2]

Summary

Implements $lookup with a sub-pipeline + let-bindings, the one remaining "intentionally not implemented" item that was philosophically aligned with the library (compiles to CTEs + a lateral aggregate, no hidden round-trips, stages stay serializable).

collection<Post>("posts")
  .lookup<Comment, Comment, "topComments">({
    from: "comments",
    let: { pid: "$_id" },
    pipeline: (q) =>
      q.match({ $expr: { $eq: ["$post_id", "$$pid"] } })
       .sort({ created_at: -1 })
       .limit(3),
    as: "topComments",
  });

Design

• AST: LookupStage.localField is now optional; new let?: Record<string, Expr>; existing pipeline?: Stage[] is now actually wired up. At least one of localField/pipeline is required (enforced in compileLookup).
• CTE chaining made reentrant: extracted chainCtes(stages, opts, prefix, outerVars) in src/stages.ts. The sub-pipeline gets its own lk0..lkN chain inside the LATERAL subquery. Same one-stage-one-CTE invariant the outer pipeline has, still EXPLAIN-inspectable.
• let threading: StageInput.outerVars is merged into every sub-stage's prevCtx, so $$pid resolves through the existing $$-var machinery in compileExpr. The let expressions are compiled against the outer row, producing correlated references the LATERAL carries into the nested WITH.
• Composability: when both localField/foreignField and pipeline are present, the equality predicate is ANDed onto the sub-pipeline's final select (Mongo 4.4+ semantics).
• Typed builder: .lookup() now has two overloads — the existing <TFrom, TAs> equality form (unchanged) and a new <TFrom, TOut, TAs> callback form pipeline: (q: Pipeline<TFrom>) => Pipeline<TOut>. Result key type is TOut[]. The non-generic implementation signature converts the callback to Stage[] by running it against a fresh sub-builder rooted at from.

Compiled SQL

left join lateral (
  select coalesce(jsonb_agg(doc), '[]'::jsonb) as joined
  from (
    with "lk0" as (select id, data as doc from "comments"),
         "lk1" as (select "lk0".id, "lk0".doc from "lk0" where /* $$pid correlated */),
         "lk2" as (select "lk1".id, "lk1".doc from "lk1" order by /* $sort */),
         "lk3" as (select "lk2".id, "lk2".doc from "lk2" limit 3)
    select doc from "lk3"
  ) __sub
) __lk on true

Tests

• test/pipeline.test.ts — tightened the existing emits lateral join to assert the parenthesized lateral subquery (left join lateral ( … ) __lk on true); two new unit tests assert the nested-WITH shape, lk0..lk3 naming, the __sub wrapper, the "s0".doc #> correlation reference, and the localField-ANDed-with-pipeline form.
• test/integration/lookup-pipeline.test.ts — new file, three end-to-end cases against real Postgres: correlated let+$expr (latest comment per post), localField/foreignField AND-ed with a sub-pipeline (sorted comments per post), and classic equality $lookup unchanged. The correlated case is the load-bearing one — it proves Postgres accepts the nested WITH inside the LATERAL referencing outer-row columns, which was the one design question I'd flagged for real-DB verification.

bun test → 84 pass / 0 fail. bunx tsc --noEmit clean.

Docs

• README — removed "$lookup with a pipeline sub-stage" from What's not (yet) supported; new ### $lookup with a sub-pipeline subsection with the callback example, the compiled-SQL sketch, the let correlation note, and the all-three-type-params caveat.
• CLAUDE.md — updated What's intentionally not implemented and Type-inference quirks.

Notes

The "supply all type params" requirement on the typed builder is unavoidable TS partial-inference behavior — same trap the existing equality .lookup<TFrom, TAs> already has, just with one extra slot. Documented in README and CLAUDE.md.

🤖 Generated with Claude Code