protobuf: load schemas via --registry, including BSR modules - #39
Merged
Conversation
Adds --registry and --include flags to the stripes CLI so the protobuf renderer can resolve message descriptors at run time instead of relying on whatever is compiled into the binary. --schema names the message; --registry repeats and dispatches by value shape: *.binpbset / *.protoset / *.pb FileDescriptorSet bytes *.proto compiled in-process via protocompile buf.build/<owner>/<module>[:ref] shells out to "buf build -o -" --include adds "protoc -I"-style import roots for .proto resolution, so a remote tree of .proto files needs its root supplied only once. All paths accept tigerblock URIs (s3://, gs://, https://, …), and a missing or unresolvable --schema is a hard error before any decoding. Also routes "-f protobuf" on a .json input through protojson, auto- detects .binpb, and preserves the +suffix on application/<type>+json so the protobuf renderer can pick its encoding without a separate Format registration. Loaded files are merged into protoregistry.GlobalFiles for the process so the renderer's existing GlobalFiles-first lookup just works. Tests cover descriptor sets, single .proto files, protoc -I cross- directory resolution, BSR-via-buf (using a stub-buf binary), conflict and missing-file paths, and the new CLI surface end-to-end. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
wrapProtobufString budgeted against the raw terminal width, ignoring
the chain indent that PrefixWriters prepend on output. At deep nesting
(e.g. resource_spans → scope_spans → spans → attributes → value on an
OTLP trace) the wrapped lines overflowed by 2*depth columns and the
terminal soft-wrapped mid-word.
Three fixes together:
- Thread an indentedStyles copy through every PrefixWriter creation
so the recursive renderer sees Width minus the indent it inherits.
- Strip wordwrap's per-line trailing-space padding before joining,
and bump the first-line margin from 4 to 16 chars to cover the
"<field_name>: " prefix.
- Style each wrapped line individually so lipgloss does not treat a
multi-line block as a layout box and pad short lines to the widest
one (which silently re-inserted the same trailing spaces).
Regression test TestProtobufWrapRespectsNestingDepth builds a 4-level-
deep OTLP-shaped message holding a long string and asserts every
rendered line fits within the requested width.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When a source's content type carries a "messageType" MIME parameter
(e.g. application/protobuf; messageType="foo.Bar" — the convention
gRPC and several HTTP-protobuf tools use), the protobuf renderer now
falls back to it when --schema is not explicit. For HTTP(S) sources
that already advertise both the encoding and the message name in a
single header, that means "just point stripes at the URL" works
without --schema or --registry as long as the descriptor is in scope.
Three pieces, intentionally narrow:
- stripes.Func aliases application/x-FOO to application/FOO when the
canonical form is registered. RFC 6648 deprecated the x- prefix
but plenty of tools still emit it (application/x-protobuf is the
historical spelling of application/protobuf). The fallback only
fires when the x- form itself isn't registered, so deliberate
text/x-dockerfile-style registrations still win directly.
- protobuf.rendererFor falls back to params["messagetype"] when
schemaURL is empty (mime.ParseMediaType lowercases parameter
names). Explicit --schema still wins over the MIME parameter.
- cmd/stripes threads tigerblock storage's info.ContentType through
renderOne as a hint, consulted between user --format and the
filename/sniff cascade. --content-type and --format from the user
still win.
Integration test serves a wrapperspb.StringValue payload from the
existing httptest server with content-type application/x-protobuf;
messageType="google.protobuf.StringValue" and asserts the CLI
auto-decodes it. wrapperspb is linked into the stripes binary via the
runtime so the descriptor resolves from GlobalTypes with no --registry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
schema.LoadRegistry's path loop and uriResolver.FindFileByPath both called storage.GetObject one path at a time. For .binpbset reads from s3/https, multi-registry invocations, or .proto trees with several include roots, that turned what could be a single round-trip wall-time into N. Hand both loops to firetiger-oss/concurrent.Run, which fans out across a bounded goroutine pool and yields results in input order so user-precedence on conflicts is preserved and protoregistry.Files writes stay sequential. Restructure LoadRegistry into fetch-then-register: fetchSchemaInput does the per-path I/O (readURI for descriptor sets, bufBuild for buf.build refs, no-op for .proto sources) and runs under concurrent.Run; the outer loop ranges over the in-order results and hands descriptor-set bytes to loadDescriptorSetBytes serially. uriResolver.FindFileByPath similarly fans the readURI calls across its include roots and returns the first successful in-order hit. Test scaffolding gains a STUB_BUF_DELAY_MS env var on stub-buf so the new TestLoadRegistryConcurrentFetch can measure 3 module fetches under a 500ms artificial delay and assert wall-clock stays under 2× delay — concurrent runs land near 1× delay (~650ms), a serial loop would land at ≥3× delay (~1500ms), and the threshold sits cleanly in the gap. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
protoregistry.Files.RangeFiles has the exact shape of an iter.Seq callback (func(T) bool), so Go 1.23+ range-over-func works on it directly. Inline the callback as a for-range loop, drop the outer closed-over err variable (we can return directly from the loop body now), and replace the bool returns with continue / explicit return. Same control flow, less indentation, no closure.
New subpackage protobuf/otlp side-effect-imports the OpenTelemetry protobuf Go bindings (trace/logs/metrics/v1 plus their common/resource imports) so the descriptors land in protoregistry.GlobalTypes at process start. The stripes CLI binary imports the package directly. With this in place, an OTLP payload served via HTTP with "Content-Type: application/x-protobuf; messageType= opentelemetry.proto.trace.v1.TracesData" decodes end-to-end with zero flags — no --registry, no --schema, no buf subprocess. The combination of (a) the previously-added Content-Type messageType auto-detect and (b) pre-registered descriptors collapses to "just point stripes at the URL." Binary-size cost: ~290 KB on the ~77 MB CLI. Acceptable for canonical schemas. Collector wrappers (ExportTraceServiceRequest etc.) are intentionally excluded to avoid pulling grpc-gateway into the runtime closure; users decoding raw OTLP/gRPC export requests can still pass --registry buf.build/opentelemetry/opentelemetry explicitly. Test scaffolding gains a second HTTP endpoint /proto/otlp serving a real TracesData payload and a testscript that hits it with no flags. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pure formatting fix flagged by CI's gofmt check:
- cli_test.go: import group ordering — third-party imports sort
alphabetically by path, so github.com/rogpeppe/... must come
before go.opentelemetry.io/...
- func_test.go: realign trailing // comments after introducing a
shorter row in TestFuncXPrefixAlias.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a way to feed protobuf descriptors into the
stripesCLI at run time, so the protobuf renderer can decode.binpb(and protojson) payloads it wasn't compiled with.--registryrepeats and dispatches by value shape:*.binpbset/*.protoset/*.pb→ FileDescriptorSet bytes*.proto→ compiled in-process viabufbuild/protocompilebuf.build/<owner>/<module>[:ref]→ shells out tobuf build -o -(the BSR path)--includeadds "protoc -I"-style import roots so a remote tree of.protofiles needs its root supplied only once.--schemanames the message; unresolvable schemas are a hard error..binpbextension auto-detects to the protobuf renderer;-f protobufon a.jsoninput goes through protojson.End-to-end OTLP example (real BSR + real
buf):Second commit fixes a longstanding wrap bug: long string values inside deeply-nested messages overflowed the terminal because the wrap budget didn't account for the chain indent added by
PrefixWriters, wordwrap padded each line with trailing spaces, and lipgloss padded multi-line blocks to the widest line.Test plan
go test ./...green (18 packages)protobuf/schemafor descriptor sets,.proto, protoc-I cross-directory resolution, BSR via a stub-buf, conflict/missing/unsupported inputs--registry/--schemahappy path, error paths, and the BSR path via the stub binarybuf.buildwith a non-trivialTracesDatapayload🤖 Generated with Claude Code