Skip to content

SP2: physical project store - #53

Merged
david-w-t merged 28 commits into
davidwt-com:mainfrom
david-w-t:develop
Aug 9, 2026
Merged

SP2: physical project store#53
david-w-t merged 28 commits into
davidwt-com:mainfrom
david-w-t:develop

Conversation

@david-w-t

Copy link
Copy Markdown
Contributor

SP2 — physical project store

Second of the four-sub-project program that separates the shared environment
ontology from per-project instance space (SP1 landed in #52).

Each registered project now gets its own three Mnesia tables
nodes_<Anchor>, relationships_<Anchor>, counters_<Anchor> — plus its own
nref and rel-id allocators starting at 1. This closes a real nref-collision
defect: before SP2 a project instance and an environment node could occupy the
same integer in the same table.

Design: docs/designs/sp2-physical-project-store-design.md
Plan: docs/superpowers/plans/2026-08-05-sp2-physical-project-store.md

What changed

Module Change
graphdb_ns namespace_of/2, target_namespace/2 take a leading Home; new node_table/1 / rel_table/1
graphdb_project register_project/1 creates the anchor and the three tables; open/1 / require_project/1 replace SP1's open_session/1 / require_session/1; new next_nref/1, next_rel_id_pair/1
graphdb_instance Fully Project-routed end-to-end — every write primitive and all six instance reads
graphdb_mgr Project-taking twins: get_node/2, retire_node/2, unretire_node/2, update_node_avps/3, delete_node/2, mutate/2, verify_caches/1, rebuild_caches/1
graphdb_query Sessions bind a Project; resolve_home/2 resolves bare nrefs

Routing rule: characterization / reciprocal (arc labels), class nodes and
attribute nodes always resolve to the environment; source_nref /
target_nref route through Home :: environment | Project. Instance↔class
membership arcs live in the project's table — including the class→instance row
whose source_nref is an environment class nref.

mutate/1 stays environment-only by design; mutate/2 is the project-aware twin.

Tests

689 passing, 0 failures (544 CT + 145 EUnit), up from 653 pre-SP2.
rebar3 xref added as a CI gate.

Two pre-existing Criticals fixed along the way

Both were invisible to rebar3 compile and to the test suite, because the test
profile defines TEST:

  • graphdb_attr:find_attribute_by_name/2 was exported only under
    -ifdef(TEST) but called from production init/1 paths in
    graphdb_language and graphdb_rules. application:start(graphdb) failed
    with undefrebar3 release produced an unstartable system.
  • Same pattern in graphdb_mgr (validate_avp_updates/1,
    apply_avp_updates/2), making every update_relationship* path undef in
    production.

Verified fixed by booting the default (non-TEST) profile directly. The new
xref CI gate is what prevents this class recurring — nothing else catches it.

Review

Every task was reviewed as it landed, then the whole branch was reviewed end to
end, followed by three fix waves and a scoped re-review. Fixes were verified by
reverting each one and confirming the pre-fix failure reproduced, rather than by
inspection.

Notable defects the process caught (each invisible to the compiler, all fixed):
graphdb_mgr calling a function SP1 had deleted; #q_instances_of{} unable to
see project instances at all; project-bound sessions silently dropping
environment labels; malformed handles crashing shared singleton workers;
register_project/1 not actually being idempotent, with a test that asserted a
property the code lacked.

Known deferrals

All recorded in TASKS.md. One is worth flagging here and is covered in a
follow-up comment on this PR: query traversal silently truncates
environment-only paths under a project-bound session.

Draft pending a decision on that item.

david-w-t and others added 28 commits August 5, 2026 21:12
Write the SP2 (physical project store) design spec: per-project
nodes/relationships/counters tables, in-store nref/rel_id allocation
via dirty_update_counter starting from 1, the graphdb_project handle
(open/1, register_project/1), and home-relative routing in graphdb_ns
(target_namespace/2 replacing the arity-1 forms).

Amend the SP1 namespace-model design (§3) with the home-relative
routing correction surfaced during SP2 brainstorming: kind=instance
does not imply project residency, and source_nref is not always the
row's home (class->instance membership row). Both endpoint fields now
route via target_namespace(Home, TargetKind), verified against every
row shape via the always-populated reciprocal field.

Add a new "Client session" TASKS.md category capturing the deferred
Session concept (per-user/client state bag: primary project, primary
language, other disambiguating state) at generic-intent level only,
per user direction — no design work has been done or should be
inferred. Update the SP2 TASKS.md entry to point at the new spec.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EWukKCbrN8GybaScJGU2kF
15-task plan for the physical project store (per-project nodes/
relationships/counters tables, Project-taking twins for retire_node/
unretire_node/update_node_avps/delete_node, mutate/2, graphdb_query
Project-aware reads). Written via superpowers:writing-plans against
docs/designs/sp2-physical-project-store-design.md.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EWukKCbrN8GybaScJGU2kF
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EWukKCbrN8GybaScJGU2kF
Review finding: a partial ensure_tables/1 failure (e.g. nodes_<A> created,
relationships_<A> creation throws) left open/1 reporting {ok, Project}
instead of {error, no_store}, because tables_exist/1 only checked the
nodes table. Fixed to require all three (nodes/rels/counters) present.
Added open_rejects_project_with_partial_tables/1 covering the partial-
table state; verified it fails against the pre-fix tables_exist/1 and
passes against the fix.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EWukKCbrN8GybaScJGU2kF
…s after T4-T9)

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EWukKCbrN8GybaScJGU2kF
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EWukKCbrN8GybaScJGU2kF
get_instance/2, children/2, compositional_ancestors/2, class_of/2,
class_memberships/2, resolve_value/3 gain a leading Project. All
instance-node and instance-relationship reads route through
graphdb_ns:node_table(Project)/rel_table(Project); the Priority-2
class-bound lookup (graphdb_class:search_class_taxonomy/2) stays
environment-bound, unchanged.

do_add_class_membership/4 (Task 9's untouched surface) is pinned to
the literal environment Home for its do_get_instance/2 call --
graphdb_ns:node_table(environment) resolves to the literal nodes
table, so this preserves exact current behaviour; Task 9 replaces
the pin when it threads a real Project through add_class_membership.

Umbrella compiles with zero errors and zero warnings.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EWukKCbrN8GybaScJGU2kF
…ode gain Project-taking twins

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EWukKCbrN8GybaScJGU2kF
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EWukKCbrN8GybaScJGU2kF
…, project cache audit

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EWukKCbrN8GybaScJGU2kF
@david-w-t

Copy link
Copy Markdown
Contributor Author

Deferred defect: query traversal silently truncates environment-only paths

Filed in TASKS.md under Multi-project sessions. Recording the analysis here
because it is the one open item that could reasonably block this PR, and the fix
is a design decision rather than a patch.

Symptom

A #q_find_path{} whose only route runs through a bootstrap nref returns
{ok, no_path} under a session bound to a project with enough instances,
while returning the correct path under an environment session:

A1=1000003  A2=1000004   (runtime-tier environment attributes; neither is shadowed)
Query: #q_find_path{from=A1, to=A2, max_depth=4, arc_kinds=[taxonomy]}

environment session      => {ok, [A1 -via 23-> 6, 6 -via 24-> A2]}
project, 1 instance      => {ok, [A1 -via 23-> 6, 6 -via 24-> A2]}
project, 21 instances    => {ok, no_path}          <-- silently wrong
                            (nref 6 shadowed by the project? true)

No error is raised. The only signal is a collision warning in the log.

Why it happens

resolve_home/2 was designed for entry-point nrefs, where the query
language genuinely has no characterization context to route on. It guesses:
try the bound project's table, fall back to the environment.

But session_read_arcs/4 (graphdb_query.erl:410) and is_scaffold_node/2
(:931) apply that same guess to arc-discovered nrefs — re-deciding Home
for every node the BFS reaches. When an intermediate node's integer happens to
exist in the project, the walk crosses stores mid-traversal and the frontier
truncates.

Both endpoints in the repro are unshadowed, so this is not the documented
"an endpoint resolves to the project by caller intent" behaviour that
resolve_home_prefers_project_and_logs_on_collision covers. That behaviour is
intentional; this is not.

Why it matters more than it looks

Project allocators start at 1. So any project with ≥6 instances shadows nref
6
, and ≥35 shadows the entire bootstrap scaffold. This is not an edge case
that needs a contrived setup — it is the steady state of any real project.

What the fix requires

The key asymmetry: arc-discovered nrefs are not ambiguous. Unlike an
entry-point nref, an nref reached by traversing an arc arrives with that arc's
characterization in hand, and the arc label carries a target_kind AVP. So
the guess can be replaced with a deterministic lookup:

  1. Route arc-discovered nrefs via graphdb_ns:target_namespace/2 on the arc's
    target_kind, instead of resolve_home/2.
  2. Handle the 29/30 class↔instance membership pair as an explicit
    exception
    — its source and target deliberately live in different Homes
    (the class in the environment, the instance in the project), so the naive
    "reuse the arc's Home" shortcut breaks precisely there. This is the reason
    the fix is not a one-liner and why I did not attempt it inside SP2.
  3. Decide what is_scaffold_node/2 should mean under a project-bound session
    — scaffold-ness is an environment property, so it likely should not consult
    the project at all.
  4. Keep resolve_home/2's contract unchanged for genuine entry points; other
    callers depend on it, and its collision behaviour there is tested.

Worth adding a regression test that asserts an environment-only path is
returned identically under an environment session and under a project-bound
session with a shadowing instance count — that is the invariant that broke.

Scope note

This is a defect in SP2's own work (introduced by the Task 12 resolve_home/2
design), not inherited. I deferred it rather than fixing it because getting
item 2 right is a modelling decision about how membership arcs are traversed,
and I would rather that be a deliberate call than a patch made under the
momentum of a large branch.

@david-w-t
david-w-t marked this pull request as ready for review August 9, 2026 01:03
@david-w-t
david-w-t merged commit bb6c8c6 into davidwt-com:main Aug 9, 2026
1 check passed
david-w-t added a commit to david-w-t/SeerStoneGraphDb that referenced this pull request Aug 9, 2026
Specs the fix for the SP2 defect filed in PR davidwt-com#53: graphdb_query's BFS
re-guesses which store each frontier node lives in, so a project with >=6
instances shadows bootstrap nref 6 and an environment-only path returns
{ok, no_path}.

Two halves. (A) arc-discovered nrefs route through a new pure
graphdb_ns:arc_target_namespace(Home, Kind, Char) keyed on
#relationship.kind, with the 29/30 membership pair distinguished by
characterization. (B) the BFS frontier, visited set, and target comparison
become Home-qualified -- without B, project-6 and environment-6 collapse
into one visited entry and the bare-nref target test reports a false found.

Corrects the fix direction recorded in the PR davidwt-com#53 comment and in TASKS.md,
which said to route via graphdb_ns:target_namespace/2 on the arc's
target_kind. Bootstrap arc labels 21-30 carry no target_kind AVP, and arcs
23/24 are the decisive hops in the repro, so that lookup fails exactly
where routing matters most.

Design only -- no code changes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EWukKCbrN8GybaScJGU2kF
david-w-t added a commit that referenced this pull request Aug 9, 2026
* Design: query-traversal Home routing for arc-discovered nrefs

Specs the fix for the SP2 defect filed in PR #53: graphdb_query's BFS
re-guesses which store each frontier node lives in, so a project with >=6
instances shadows bootstrap nref 6 and an environment-only path returns
{ok, no_path}.

Two halves. (A) arc-discovered nrefs route through a new pure
graphdb_ns:arc_target_namespace(Home, Kind, Char) keyed on
#relationship.kind, with the 29/30 membership pair distinguished by
characterization. (B) the BFS frontier, visited set, and target comparison
become Home-qualified -- without B, project-6 and environment-6 collapse
into one visited entry and the bare-nref target test reports a false found.

Corrects the fix direction recorded in the PR #53 comment and in TASKS.md,
which said to route via graphdb_ns:target_namespace/2 on the arc's
target_kind. Bootstrap arc labels 21-30 carry no target_kind AVP, and arcs
23/24 are the decisive hops in the repro, so that lookup fails exactly
where routing matters most.

Design only -- no code changes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EWukKCbrN8GybaScJGU2kF

* Plan: query-traversal Home routing implementation

Five tasks, TDD throughout. T1 graphdb_ns:arc_target_namespace/3 + EUnit;
T2 Home-qualified BFS (the headline invariant + the false-found case);
T3 home key on cross-store edges; T4 resume/2 continuation-Home gate;
T5 docs + verification gate.

Records one deliberate deviation from the design: its T3 (visiting
project-6 must not suppress environment-6) is not constructible under
this scope, since reaching both keys in one walk needs a store crossing
and the only one available lands on a class, from which the environment
attribute subtree is unreachable by taxonomy. Replaced by T3', which
asserts the same Half-B property observably via #cont_path{} state.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EWukKCbrN8GybaScJGU2kF

* Half A: graphdb_ns:arc_target_namespace/3 routes arc-discovered nrefs

* Half B: Home-qualified BFS frontier, visited set and target

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EWukKCbrN8GybaScJGU2kF

* Disclose home_id on path edges that cross stores

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EWukKCbrN8GybaScJGU2kF

* resume/2 rejects continuations carrying a foreign project Home

Adds validate_cont_homes/2, a caller-side gate on resume/2 that runs
after validate_session_home/1 and before the gen_server:call. Every
home_id() a #cont_path{} carries (target + frontier) must resolve
against the resuming session's own bound project; a mismatch returns
{error, session_project_mismatch} instead of reaching home_of_id/2
inside the graphdb_query singleton.

Closes two review findings tracked from Task 2:
- a continuation resumed under a project-less session no longer risks
  maps:get(project, Session) raising {badkey, project} inside
  handle_call/3 and killing the singleton
- a continuation resumed under a *different* project is rejected
  instead of home_of_id/2 silently discarding the anchor and reading
  the wrong project's tables

Adds three regression tests: t3_continuation_state_is_home_qualified
and t6_resume_round_trip_under_project_session (pass on arrival --
Task 2 already delivered this shape) and
resume_rejects_foreign_project_continuation (drives this fix).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EWukKCbrN8GybaScJGU2kF

* Docs: record query-traversal Home routing as implemented

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EWukKCbrN8GybaScJGU2kF

* Docs: fix stale resolve_home/2 duplicates and test count after review

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EWukKCbrN8GybaScJGU2kF

* Final review fix wave: docs corrections, dead-code removal, hardened frontier gate

Six findings from the whole-branch final review:

1. docs/Architecture.md: correct the false claim that bootstrap arc labels
   21-30 carry target_kind (they don't -- only runtime-created relationship-
   attribute pairs do); note arc-discovered traversal routes via
   graphdb_ns:arc_target_namespace/3 instead, and that target_namespace/2
   has no production caller.
2. graphdb_query.erl: rewrite resume/2's stale header comment -- it no
   longer dispatches through resolve_home/2; validate_session_home/1 is
   still load-bearing because validate_cont_homes/2 depends on it running
   first.
3. apps/graphdb/CLAUDE.md: document resume/2's {error, invalid_project}
   and {error, session_project_mismatch} returns alongside the existing
   {error, snapshot_expired}.
4. graphdb_query.erl: validate_cont_homes/2 folds a frontier-element shape
   check into its lists:all/2 pass, closing a gap where a wrong-arity
   frontier tuple was silently dropped by the old list-comprehension
   generator instead of failing the gate. Adds
   resume_rejects_malformed_frontier_continuation/1 to
   graphdb_query_SUITE's sp2_traversal_home_routing group, asserting both
   the clean error and that the graphdb_query singleton pid survives.
5. graphdb_ns.erl: Rev PA3 revision-history entry for arc_target_namespace/3.
6. graphdb_query.erl: remove the dead `[] -> []` arm in bfs/8's
   depth-exhausted clause -- unreachable because the first bfs/8 clause
   already matches an empty frontier for any depth.

Verification: rebar3 compile / as test compile zero warnings, rebar3 xref
clean, make test-ct-parallel 551/551, rebar3 eunit 151/151.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EWukKCbrN8GybaScJGU2kF

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant