Add pydantic model support for extraction schemas - #77
Closed
jordanalexmeyer wants to merge 3 commits into
Closed
Conversation
Pass a pydantic BaseModel subclass as config["schema"] (or extractor["override_config"]["schema"]) and the SDK converts it to Extend's JSON Schema subset for the request, then validates the extraction output back into model instances (TypedExtractRun). Mirrors the TypeScript SDK's Zod support across the same four integration points: extract(), extract_runs.create_and_poll(), extractors.create()/update(), and extractor_versions.create(). - New wrapper/schema module: ExtendDate/ExtendCurrency/ExtendSignature field types, pydantic_to_extend_schema conversion with SchemaConversionError, typed run wrappers, config detection helpers - Works with pydantic v1 and v2 - Overload signatures give full static inference: result.output.value is typed as the schema model
The wrapper layer re-declares parts of the generated API surface, which can silently go stale when Fern regenerates the SDK with new parameters. True overrides (extract(), extractors.create()/update(), etc.) are already protected because mypy rejects overrides incompatible with the generated superclass, but create_and_poll(), the typed config TypedDicts, and TypedExtractRun had no guard. These tests fail CI whenever a generated create() gains a parameter that create_and_poll() doesn't forward, a config key is missing from the typed TypedDicts, or ExtractRun gains a field TypedExtractRun doesn't mirror. Also fixes drift the new tests caught: parse_runs.create_and_poll() was missing the metadata and data_retention parameters that the generated parse_runs.create() accepts.
Stop silently force-nullabling primitives in the converter — that was defeating the API's 2026-02-09 strict schema validation and could defer user mistakes until after a paid run completed. Non-Optional primitives, enums, and dates now raise SchemaConversionError before any request is sent. Also: - Detect recursive models and raise SchemaConversionError instead of a fatal stack overflow - Recognize typing.Literal on Python 3.8 (distinct from typing_extensions) - Reject field aliases (they caused silent None validation) and Optional array items - Wrap residual output validation failures in ExtractOutputValidationError that preserves the completed ExtractRun (id, dashboard URL, raw output) - Convert pydantic schemas in plain extract_runs.create() so users don't hit a cryptic encoder error - Bump typing_extensions floor to >=4.3.0 for generic TypedDict support
Contributor
Author
|
Superseded by #78, which stacks these pydantic commits on top of a fresh 1.17.0 Fern regeneration from documentation |
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds typed extraction schemas via pydantic, mirroring the TypeScript SDK's Zod support. Pass a
pydantic.BaseModelsubclass asconfig["schema"]and the SDK:nulloption,additionalProperties: false)TypedExtractRun[Model]— a step beyond the TS SDK, which only casts at compile timeIntegration points (full TS parity)
client.extract()(sync + async)client.extract_runs.create_and_poll()(sync + async), includingextractor["override_config"]["schema"]client.extractors.create()/update()(new wrapper clients)client.extractor_versions.create()(new wrapper client)@overloadsignatures give full static inference — mypy resolvesresult.output.valueto the schema model, while untyped (dict) configs keep returning plainExtractRun.Details
wrapper/schema/module:ExtendDate/ExtendCurrency/ExtendSignaturefield types (plaindatetime.dateannotations also map to the date type),pydantic_to_extend_schema()conversion,SchemaConversionErrorcarrying the field path, typed run wrappersstr,float,int,bool,datetime.date,Literal[...]/ string enums, nested models, and lists of these; unsupported constructs (unions, dicts,datetime.datetime) raiseSchemaConversionErrorIS_PYDANTIC_V2compat pattern).fernignore-protectedwrapper/directoryOut of scope (matching TS):
extend:name, per-enum-valueextend:descriptions, typed schemas on plainextract_runs.create().Testing
schema.test.tsconversion suite (~45 cases) plus typed-extraction integration tests for all four endpointspytest tests/wrapper tests/custom: 212 passed (1 pre-existing skip)mypy .: clean (1,134 files); ruff clean on all touched files