Skip to content

Extract people and organizations from memory content, not just metadata - #13

Open
ali-ulu wants to merge 1 commit into
mainfrom
claude/remove-demo-data-xdwh4x
Open

Extract people and organizations from memory content, not just metadata#13
ali-ulu wants to merge 1 commit into
mainfrom
claude/remove-demo-data-xdwh4x

Conversation

@ali-ulu

@ali-ulu ali-ulu commented Jul 30, 2026

Copy link
Copy Markdown
Owner

Summary

Entity extraction only ever read connector metadata (attendees, from/to, speakers). That covers calendar/email/transcript imports but left the most common case empty: a note typed by hand, a commit message, a captured decision. Those memories produced zero entities, so the knowledge graph, People/Organizations views, and the trust corroboration signal all stayed blank for anyone not importing a mailbox — the demo corpus was carrying the whole demonstration, and removing it emptied the product.

Adds server/core/text_entities.py with deterministic, high-precision text signals only (no NER model, no LLM, no I/O — consistent with the rest of the entity layer):

Signal Produces
e-mail address in the body person (+ organization via domain)
@handle mention person
Acme Inc / Zephyr Labs organization
met with X / X ile görüştüm person

Wired in at a single shared point (people_in_memory), so the graph, People, and Organizations views all pick it up at once.

Precision over recall. A noisy graph is worse than a sparse one, so:

  • Prose heuristics are skipped entirely for source code — detected by extension, by connector path, or by the shape of the content itself — so a repo import can't mint an entity per capitalised identifier.
  • Generic nouns ("the team", "Support") are rejected as people.
  • Company names matched by the "met with X" phrasing are classified as organizations, not people (you meet with Zephyr Labs, but it isn't a person).
  • A company matched in prose folds into its e-mail domain when both appear in one memory, so Zephyr Labs doesn't fork a node next to zephyrlabs.io.

Known limitation (unchanged by this PR): entity resolution is still per-key, so a company named in prose in one memory and by e-mail domain in another remains two nodes. Cross-memory alias merging needs its own design and isn't attempted here.

The demo-corpus assertion in test_seed_demo.py moves from 21→22 entities: the one addition is the prose-named Zephyr Labs organization, which has no e-mail domain to key off.

Test plan

  • python -m pytest tests/ -q — 620 passed (17 new)
  • New negative tests lock the code-safety guarantee: code yields no prose entities whether detected by extension, by path, or by content shape
  • End-to-end against a live CLI DB — five hand-typed notes (previously producing nothing) now yield 4 people + 2 organizations, with working entity profiles and trust scoring:
[person] Ayse — 1 mentions          [organization] Contoso Technologies — 1 mentions
[person] Elif — 1 mentions          [organization] Zephyrlabs — 1 mentions
[person] deniz — 1 mentions
[person] mert-dev — 1 mentions

Generated by Claude Code

Entity extraction only ever read connector metadata (attendees, from/to,
speakers). That covers calendar/email/transcript imports and leaves the
most common case empty: a note typed by hand, a commit message, a
captured decision. Those memories produced zero entities, so the graph,
People/Organizations views and the trust corroboration signal all stayed
blank for anyone not importing a mailbox — the demo corpus was carrying
the whole demonstration.

Adds server/core/text_entities.py: deterministic, high-precision text
signals only (no NER model, no LLM, no I/O), consistent with the rest of
the entity layer:
  - e-mail addresses in the body -> person (+ organization via domain)
  - @handle mentions             -> person
  - "Acme Inc" / "Zephyr Labs"   -> organization
  - "met with X" / "X ile görüştüm" -> person

Precision is favoured over recall: a noisy graph is worse than a sparse
one. The prose heuristics are skipped entirely for source code, detected
by extension, by connector path, or by the shape of the content itself,
so a repo import can't mint an entity per capitalised identifier. A
company matched by prose folds into its e-mail domain when both appear
in one memory, so "Zephyr Labs" doesn't fork a node next to
zephyrlabs.io.

Wired in at the single shared point (people_in_memory), so the graph,
People and Organizations views all pick it up at once.

Known limitation, unchanged by this commit: entity resolution is still
per-key, so a company named in prose in one memory and by e-mail domain
in another remains two nodes. Cross-memory alias merging needs its own
design.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants