Support Spark Connect #745
Open
chenliu0831 wants to merge 8 commits into
Open
Conversation
added 6 commits
June 6, 2026 08:08
Restructures the Spark Connect wire format so the schema is the single source of truth for which fields apply to which constraint or analyzer: the kitchen-sink string-discriminator messages are replaced with `oneof` arms, one per Check / Analyzer builder method, with reused `…Spec` payload submessages. Schema changes: - ConstraintMessage and AnalyzerMessage become Constraint and Analyzer with `oneof body` shapes (one arm per builder method). - Predicate's CompareOp enum gains UNSPECIFIED + COMPARE_OP_ prefix. CheckLevel hoisted to a top-level enum with the same treatment. ConstraintRuleSet introduced (replaces stringly-typed list). - Magic-zero fields (testset_ratio, seed, max_detail_bins, etc.) made proto3 `optional` so server can distinguish "user said 0" from "unset". - Four orphan result enums (VerificationStatus, CheckStatus, ConstraintStatus, MetricEntity) deleted; they were never on the wire. - Compliance analyzer fields renamed instance/predicate (was column/pattern, which mislabeled the actual semantics). - Pair-column constraints use named column_a/column_b (was index into repeated string columns, which silent-dropped malformed inputs). - Every envelope carries wire_format_version for the JAR/wheel pairing guard described in ADR-0004. Scala-side rewrite: - CheckBuilder/AnalyzerBuilder pattern-match on getBodyCase; the compiler enforces exhaustiveness. Old `case unknownType => throw` is unreachable. - DeequRelationPlugin: wire_format_version assertion at envelope unpack; println traces switched to slf4j; rulesFromProto replaces stringly-typed parseConstraintRules; KLLParameters helper centralizes default fallback. Build: - pom.xml ships src/main/protobuf/deequ_connect.proto at META-INF/protobuf/ inside the JAR (per ADR-0003) so python-deequ can extract and run protoc at build time. - dependency-reduced-pom.xml gitignored (it's a shade-plugin artifact). Docs: - CONTEXT.md established as the project glossary. - ADR-0001..0004 record the oneof-arm decision, clean-break policy, stub-delivery mechanism, and cross-repo rollout choreography. See docs/adr/0001..0004 for full design rationale.
A survey of five Spark Connect plugin codebases (Apache Spark itself, Delta Connect, GraphFrames, Apache Iceberg, Apache Sedona) found that ADR-0003 and the wire_format_version guard from ADR-0004 had no community precedent — Spark itself drops .proto files via MergeStrategy.discard, and Spark's only version-adjacent field (client_type) explicitly states it "will not be interpreted by the server." Schema: - Drop wire_format_version field from all four Relation envelopes. Type URL of the unpacked message is the version discriminator. - Renumber field tags now that the version slot is gone. Scala plugin (DeequRelationPlugin): - Drop WIRE_FORMAT_VERSION constant and assertWireFormat() helper. - Match the canonical ExampleRelationPlugin pattern: relation.is / relation.unpack inline, return None for unknown types so Spark's noHandlerFoundForExtension surfaces correctly. Build (pom.xml): - Drop the <resources> block that copied .proto into META-INF/protobuf/. No surveyed plugin ships .proto in JAR; python-deequ will check in the generated _pb2.py instead (graphframes pattern). Schema documentation hygiene (closed-enum markers per AIP-126): - CompareOp, CheckLevel, ConstraintRuleSet annotated as closed. ADRs: - ADR-0003 marked Superseded. - ADR-0004 partially superseded (wire_format_version guard only). - ADR-0005 captures the new evidence-based decisions with citations.
…icial protobuf guide
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
|
Thank you for reporting this. This has been flagged for review by our maintainer team. We'll get back to you as soon as possible. Generated by AI (model: us.anthropic.claude-opus-4-7, prompt: 4d2f5d73) — may not be fully accurate. Reply if this doesn't help. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue #, if available: awslabs/python-deequ#221
Description of changes
Adds a Spark Connect server plugin (
DeequRelationPlugin) so PyDeequ V2 can drive Deequ over a remote Spark session. Spark Connect is the future for language specific bindings like PyDeequ and no JVM access will be required on the client side once this is merged. PyDeequ side change: awslabs/python-deequ#254The Protobuf API designs were refined based on protobuf best practices documented in https://google.aip.dev/100.
Disclaimer: Coauthored with Claude.
Major changes
src/main/protobuf/) — five.protofiles defining theSpark Connect Relation envelopes for verification, analysis, column
profiling, and constraint suggestions, plus a shared
common.proto.src/main/scala/com/amazon/deequ/connect/) —DeequRelationPlugindispatches onAny.is(class)/unpack(class)(the canonical Spark pattern);
CheckBuilderandAnalyzerBuilderdecode the typed
oneofarms into Deequ Check / Analyzer objects.pom.xml) — addsprotobuf-maven-pluginfor codegen, plus ashade rule that relocates
com.google.protobufto match Spark Connect'sshaded namespace.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.