Skip to content

Add true jq streaming mode and optimize row conversion #70

Description

@apstndb

Deferred from local repository review cleanup.

Problem

--jq-input-mode=lazy emits rows incrementally, but it is not constant-memory streaming. The lazy row iterator caches every row it has yielded so .rows can be replayed and .stats can still be accessed later. That makes the mode useful, but the current wording can overpromise for very large result sets.

jqresult.RowToJSON also converts each row by building a one-row spannerpb.ResultSet, marshaling it with protojson, and decoding that JSON back into Go values. This preserves the current ResultSet JSON shape, but it is expensive on large streams.

Proposed work

  • Document or rename the current lazy mode as replayable lazy evaluation with O(n) row cache.
  • Add a true single-pass streaming mode for .rows[]-style filters that does not retain all rows.
  • Define the .stats behavior for true stream mode, such as unsupported, delayed until drain, or a side channel.
  • Add golden tests that prove a direct structpb.Value to jq-value converter is byte-identical to the current protojson path for the supported ResultSet row shapes.
  • Replace the protojson round trip only after the compatibility fixtures are in place.

Acceptance criteria

  • README and help text no longer describe lazy mode as constant-memory streaming.
  • A true streaming path has explicit replay and stats semantics.
  • Existing JSON/YAML output goldens continue to pass.
  • Row conversion benchmarks or focused tests demonstrate the new converter preserves the public output shape.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions