feat: Add reproducible embeddings pipeline and interactive semantic + graph visualization

.gitignore

@@ -3,3 +3,12 @@ node_modules/
 *.log
 .env
 .env.*
+
+# Generated embeddings, vector DB (recomputable)
+generated/
+embeddings/chroma/
+*.chroma
+
+# Do not ignore the source viz template
+!embeddings/viz.html
+

EMBEDDINGS.md

@@ -0,0 +1,152 @@
+# Embeddings & Spatial Visualization
+
+**Reproducible semantic embeddings + hybrid graph visualization for the Marble Skill Taxonomy.**
+
+This contribution adds a fully Dockerized pipeline that turns the hand-authored prerequisite graph into an explorable 3D semantic space while keeping the graph structure visible and interactive.
+
+## Why this exists
+
+The original release deliberately excluded semantic embeddings because they are *derived* and recomputable. This package gives you a clean, versioned, deterministic way to generate them.
+
+The visualization lets you see two complementary structures at once:
+
+- **Semantic diversity** (UMAP projection of embeddings) — concepts that *feel* similar sit close together in space.
+- **The prerequisite web** (directed edges) — the actual pedagogical dependencies a learner must traverse.
+
+By allowing you to dial graph forces on top of the semantic layout, you can literally watch how the declared learning structure sits inside (or stretches across) conceptual neighborhoods.
+
+This often reveals **surprising long-range connections** — hard prerequisites between ideas that are distant in meaning space.
+
+## Quick start (recommended)
+
+```bash
+# From the repository root
+./scripts/generate-viz.sh
+```
+
+This builds the Docker image (if needed), runs the full pipeline, and prints instructions.
+
+After success:
+
+```bash
+cd generated
+python -m http.server 8080
+# Then open http://localhost:8080/viz.html
+```
+
+On macOS:
+
+```bash
+open generated/viz.html
+```
+
+## What you get
+
+- `generated/topics_with_layout.json` — every topic with stable 3D + 2D UMAP coordinates
+- `generated/links.json` — the prerequisite graph (strength + reason)
+- `generated/chroma/` — a queryable Chroma vector database with rich metadata
+- `generated/generation_manifest.json` — full provenance (model, params, timestamp, text strategy)
+- `generated/viz.html` — the interactive explorer (copied from the source template in embeddings/)
+
+Here's what it looks like:
+
+![Screenshot of the 3D semantic + graph explorer](embeddings/screenshot.png)
+
+## The visualization features (highlights)
+
+- **3D force-graph** (WebGL) seeded with UMAP positions
+- Live **Graph pull** slider — morphs between pure semantic layout and graph-respecting layout
+- Click any node → traces full prerequisite ancestor chains (warm/orange) + what it unlocks (green) with animated particles traveling the edges
+- **"Highlight surprising prerequisites"** — surfaces hard edges between semantically distant concepts
+- Search, subject filters, age quick-filters, multiple color modes
+- Rich inspector panel with direct navigation
+- Camera controls, auto-rotate, fit view, reset
+- Fast (1,590 nodes is small) and uses fixed random seeds for reproducibility
+
+## Architecture & reproducibility
+
+**Embedding model** (default): `sentence-transformers/all-MiniLM-L6-v2` (384 dimensions)
+
+**Text strategy** (chosen after experimentation for coherent clusters):
+```
+{subject} — {domain} — {name}. {description}. Evidence of mastery: {evidence joined}.
+```
+
+Age ranges and standards are deliberately left out of the embedding text (they are visualized through other channels).
+
+**Layout**:
+- Primary: UMAP (cosine) → 3D coordinates
+- Secondary: 2D for alternative views
+- Random state fixed at 42
+
+**Vector DB**: Chroma (persistent local directory) with full topic metadata.
+
+Everything is deterministic given the same model + parameters + source data snapshot.
+
+## Running without Docker (advanced)
+
+```bash
+cd embeddings
+python -m venv .venv
+source .venv/bin/activate
+pip install -r requirements.txt
+
+# From repo root
+python embeddings/generate.py \
+  --data-dir data \
+  --output-dir generated \
+  --model sentence-transformers/all-MiniLM-L6-v2
+```
+
+Then serve `generated/viz.html`.
+
+## Trying stronger models
+
+Edit `embeddings/docker-compose.yml` or pass the env:
+
+```bash
+EMBEDDING_MODEL=BAAI/bge-small-en-v1.5 ./scripts/generate-viz.sh
+```
+
+Other good options (via sentence-transformers):
+- `nomic-ai/nomic-embed-text-v1.5`
+- `BAAI/bge-base-en-v1.5`
+
+Update `umap` hyperparameters via CLI flags (`--n-neighbors`, `--min-dist`) and record them in the manifest.
+
+## Interpreting the view
+
+- **Clusters** = semantic neighborhoods (ideas that share vocabulary, structure, or conceptual framing).
+- **Long threads** that cross large distances = pedagogically necessary bridges between conceptually distant domains.
+- Pull the graph force slider high → concepts that *must* be learned together are pulled close even if their surface descriptions differ.
+- The "surprises" button is often the most interesting starting point for curriculum designers.
+
+Example patterns you will likely discover:
+- Foundational counting concepts sit near other early number sense ideas.
+- Some "meta" learning-to-learn skills act as bridges across many subjects.
+- Certain science process skills are surprisingly close to early mathematics despite different language.
+
+## License & attribution
+
+The generated embeddings and visualization are **produced works** based on the open Marble Skill Taxonomy.
+
+When using or sharing views:
+- Credit the source taxonomy (see root README and CITATION.cff).
+- Note the embedding model and generation parameters you used.
+
+The underlying data remains under ODbL 1.0 (structure) + CC BY-SA 4.0 (authored text).
+
+## Extending
+
+- Add more metadata to the Chroma collection (standards links, clusters).
+- Export the layout to other tools (Gephi, Observable, Blender).
+- Train a small classifier on top of the embeddings to predict subject or difficulty.
+- Create 2D printable posters of interesting slices.
+
+Contributions that improve the hybrid layout algorithm, add new insight layers, or improve mobile/web performance are very welcome.
+
+---
+
+This work is a contribution to the open-source Marble Skill Taxonomy project.
+
+Run the pipeline, serve the artifacts, and explore the combined semantic and graph structure.

README.md

@@ -123,3 +123,22 @@ Plus the upstream notices in [PROVENANCE.md](PROVENANCE.md) for any curriculum s
 ## What's *not* here
 
 Deliberately excluded from this release: semantic embeddings (derived, recomputable) and any per-child / user data (never published). See [CHANGELOG.md](CHANGELOG.md).
+
+## Semantic spatial explorer (community contribution)
+
+A Docker-powered pipeline exists to generate embeddings and an interactive visualization that places every micro-topic in 3D semantic space (via UMAP) while overlaying and respecting the prerequisite graph structure.
+
+See [EMBEDDINGS.md](EMBEDDINGS.md) for full documentation.
+
+```bash
+./scripts/generate-viz.sh
+# then open generated/viz.html (or serve the directory)
+```
+
+The explorer lets you:
+- See conceptual clusters (semantic diversity)
+- Trace prerequisite chains that sometimes span large distances in meaning-space
+- Dial graph forces live to watch the "web of learning" pull concepts together
+- Discover surprising long-range dependencies
+
+This is the perfect complement to the existing 3D age-based prerequisite graph. Both are now possible from the same open dataset.

embeddings/Dockerfile

@@ -0,0 +1,31 @@
+# embeddings/Dockerfile
+# Lean, reproducible image for generating semantic embeddings + UMAP layout + Chroma vector DB
+# for the Marble Skill Taxonomy.
+
+FROM python:3.12-slim AS base
+
+# System deps (minimal)
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    build-essential \
+    git \
+    && rm -rf /var/lib/apt/lists/*
+
+WORKDIR /app
+
+# Copy requirements first for better layer caching
+COPY requirements.txt .
+
+# Install Python deps (pinned). sentence-transformers pulls torch CPU by default in slim.
+# For smaller images in future one could use ONNX, but this is reliable and explicit.
+RUN pip install --no-cache-dir -r requirements.txt
+
+# Copy generation code and viz template (the template is source-controlled in embeddings/)
+COPY generate.py .
+COPY viz.html .
+
+# Default command runs the full pipeline.
+# Data is expected at /data (volume), output written to /output (volume)
+ENV PYTHONUNBUFFERED=1
+ENV EMBEDDING_MODEL=sentence-transformers/all-MiniLM-L6-v2
+
+CMD ["python", "generate.py", "--data-dir", "/data", "--output-dir", "/output"]

embeddings/docker-compose.yml

@@ -0,0 +1,20 @@
+# embeddings/docker-compose.yml
+# One-shot generation of embeddings, Chroma vector DB, UMAP layouts, and viz-ready artifacts.
+
+services:
+  generate:
+    build:
+      context: .
+      dockerfile: Dockerfile
+    image: marble-taxonomy-embeddings:latest
+    container_name: marble-embeddings
+    volumes:
+      # Read-only access to the canonical data
+      - ../data:/data:ro
+      # Write artifacts + vector DB here
+      - ../generated:/output
+    environment:
+      - EMBEDDING_MODEL=sentence-transformers/all-MiniLM-L6-v2
+    # Override for custom UMAP params if desired:
+    # command: python generate.py --data-dir /data --output-dir /output --n-neighbors 25 --min-dist 0.1
+    restart: "no"