SchemaWeaver: JSON schema inference for messy web extractions

Keywords: json schema inference python, pydantic model from dicts, merge json keys, web scraping schema discovery, field synonym clustering, unstructured extraction schema, json schema from samples, product catalog schema python

You scraped product pages and every extractor named price differently. SchemaWeaver maps cost / amount / sale_price to price, drops rare keys, and emits a merged schema plus a Pydantic model.

What is SchemaWeaver?

SchemaWeaver is a small MIT-licensed Python 3.11+ library. You pass a list of dicts (scraper output, LLM extraction dumps, CSS field maps). It clusters synonym keys, counts how often each canonical field appears, and returns a schema you can turn into a Pydantic model.

Piece	Role
normalize_key	Lowercase, snake_case, strip punctuation
canonical_field	Map a raw key through 10 built-in synonym groups
SchemaWeaver.fit	Frequency filter, types, examples, aliases, confidence
DiscoveredSchema.to_pydantic_model	Optional Pydantic fields from those observations

It does not call a network API. The only runtime dependency is Pydantic 2.6+.

Direct answer

How do I build a Pydantic model from scraped JSON whose keys do not agree? Install SchemaWeaver, call SchemaWeaver(min_frequency=0.4).fit(docs), then schema.to_pydantic_model("Product"). Keys that mean the same thing (product_name vs title, cost vs price) collapse to one field. Keys that show up on too few documents are dropped.

Does this emit JSON Schema Draft documents from nested APIs? No. Input is a list of dicts. Nested dict and list values are typed as dict and list. Inner keys are not walked, and there is no Draft-07 / OpenAPI writer.

Do I need an LLM to induce the schema? No. Clustering is a hardcoded English synonym table in schemaweaver/normalize.py. That keeps the library offline and cheap. It also means unknown domain terms stay as their normalized names instead of being guessed.

Why SchemaWeaver?

Hand-writing a Pydantic model is the right call when keys are already clean. Recursive JSON Schema inferencers (for example genson) are the right call when you need nested Draft documents and keys already match. SchemaWeaver sits in the gap: mostly flat extraction dumps with alias drift.

	Hand-written Pydantic	genson / similar inferencers	SchemaWeaver
Field-name drift (cost vs price)	You pick the name	Separate properties	10 synonym groups
Rare junk keys from scrapers	You omit them	Usually kept	Dropped below min_frequency
Nested objects	Full control	Usually recursive	Shallow dict / list types
Output	Your model	JSON Schema	SchemaField list + Pydantic model
Domain bias	None	None	Built-in aliases lean product/catalog
Network / LLM	None	None	None

Tradeoff: the synonym table is English and catalog-shaped. id maps to sku. If that is wrong for your data, rename before fit, or edit SYNONYMS.

How it works

1. Normalize. normalize_key("Sale Price") → sale_price.
2. Cluster. canonical_field looks up the normalized token in SYNONYMS. Hits become one of: price, title, description, url, image, rating, reviews, sku, brand, availability. Misses keep the normalized name.
3. Bucket. For each document, each canonical name stores one value. If the same document has both product_name and title, insertion order wins for the value. Both raw keys still go on aliases.
4. Filter. frequency = count / n_documents. Fields below min_frequency (class default 0.3) are dropped.
5. Score. confidence = min(1.0, frequency + 0.1 * alias_count). Up to three non-null examples are kept.
6. Emit. Fields sort by frequency (desc), then name. to_pydantic_model makes every field optional. Mixed int/float becomes Optional[float]. Mixed scalar/string becomes Optional[str].

dicts  →  normalize  →  synonym cluster  →  frequency filter
                                              │
                                              ▼
                         DiscoveredSchema (aliases, types, examples)
                                              │
                                              ▼
                                    Pydantic model (all optional)

Install

git clone https://github.com/pandeyvishwas51-oss/schemaweaver.git
cd schemaweaver
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

Requirements: Python 3.11+, pydantic>=2.6.0.

Quick start

from schemaweaver import SchemaWeaver

docs = [
    {"product_name": "Widget", "cost": 9.99},
    {"title": "Gadget", "price": 12.5, "brand": "Acme"},
    {"name": "Thing", "amount": 4},
]
schema = SchemaWeaver(min_frequency=0.4).fit(docs)
print(schema.model_dump_json(indent=2))
Product = schema.to_pydantic_model("Product")
print(list(Product.model_fields.keys()))

title and price survive the 0.4 cutoff (they appear in every document). brand appears once in three documents (0.333) and is dropped.

Inspect a single mapping or the full table:

from schemaweaver import SYNONYMS, canonical_field, merge_keys

canonical_field("sale_price")  # "price"
canonical_field("id")          # "sku"
merge_keys(["cost", "amount", "title"])
# {"price": ["cost", "amount"], "title": ["title"]}
print(sorted(SYNONYMS))

CLI

# Built-in four-document demo
schemaweaver
python -m schemaweaver

# Your own JSON array of objects
schemaweaver samples.json --min-frequency 0.4
cat samples.json | schemaweaver --stdin

Demo output (4 documents, default min_frequency=0.3):

{
  "fields": [
    {
      "name": "price",
      "types": ["float", "int"],
      "frequency": 1.0,
      "examples": [9.99, 12.5, 4],
      "aliases": ["amount", "cost", "price"],
      "confidence": 1.0
    },
    {
      "name": "title",
      "types": ["str"],
      "frequency": 1.0,
      "examples": ["Widget", "Gadget", "Thing"],
      "aliases": ["name", "product_name", "title"],
      "confidence": 1.0
    },
    {
      "name": "image",
      "types": ["str"],
      "frequency": 0.75,
      "examples": ["http://x/a.jpg", "http://x/b.jpg", "http://x/c.jpg"],
      "aliases": ["image_url", "img", "thumbnail"],
      "confidence": 1.0
    }
  ],
  "n_documents": 4,
  "coverage_threshold": 0.3
}

brand (1/4) and rating (1/4) fall under the cutoff. Schema JSON is printed on stdout. Generated field names go to stderr so the JSON stays pipeable.

Testing

pip install -e ".[dev]"
pytest -v

FAQ

What field names does SchemaWeaver merge?

Ten groups in SYNONYMS: price (cost, amount, sale_price, mrp, value, ...), title (name, product_name, heading, ...), description, url, image, rating, reviews, sku (id, product_id, asin, item_id, ...), brand, availability. Anything else is only snake_cased.

Why did my id field become sku?

id is in the SKU synonym set. Rename the key before fit, or change SYNONYMS in schemaweaver/normalize.py.

Can I keep rare fields?

Yes. Lower min_frequency. SchemaWeaver(min_frequency=0.0).fit(docs) keeps every canonical field that appeared at least once.

How is confidence computed?

min(1.0, frequency + 0.1 * number_of_distinct_raw_aliases). More aliases bump the score. It is a heuristic, not a statistical test.

Does a document with both title and product_name count twice?

No. Frequency is per document. Both raw names are stored on aliases. The stored example value is the first key in that dict's insertion order.

What Python versions are supported?

Python 3.11+.

Is there a hosted API or PyPI release yet?

This repo is installed from source with pip install -e .. There is a schemaweaver console script after that install.

License

MIT. Free for commercial and personal use.

Contributing

1. Fork and branch.
2. pip install -e ".[dev]"
3. pytest -v
4. Open a PR that says what changed and why.

Useful patches: more synonym groups, a way to pass a custom table into SchemaWeaver without editing source, recursive nested-object discovery.