diff --git a/.gitignore b/.gitignore index ec53a15b..dcf6f2c9 100644 --- a/.gitignore +++ b/.gitignore @@ -3,3 +3,12 @@ node_modules/ *.log .env .env.* + +# Generated embeddings, vector DB (recomputable) +generated/ +embeddings/chroma/ +*.chroma + +# Do not ignore the source viz template +!embeddings/viz.html + diff --git a/EMBEDDINGS.md b/EMBEDDINGS.md new file mode 100644 index 00000000..03ecdedc --- /dev/null +++ b/EMBEDDINGS.md @@ -0,0 +1,152 @@ +# Embeddings & Spatial Visualization + +**Reproducible semantic embeddings + hybrid graph visualization for the Marble Skill Taxonomy.** + +This contribution adds a fully Dockerized pipeline that turns the hand-authored prerequisite graph into an explorable 3D semantic space while keeping the graph structure visible and interactive. + +## Why this exists + +The original release deliberately excluded semantic embeddings because they are *derived* and recomputable. This package gives you a clean, versioned, deterministic way to generate them. + +The visualization lets you see two complementary structures at once: + +- **Semantic diversity** (UMAP projection of embeddings) — concepts that *feel* similar sit close together in space. +- **The prerequisite web** (directed edges) — the actual pedagogical dependencies a learner must traverse. + +By allowing you to dial graph forces on top of the semantic layout, you can literally watch how the declared learning structure sits inside (or stretches across) conceptual neighborhoods. + +This often reveals **surprising long-range connections** — hard prerequisites between ideas that are distant in meaning space. + +## Quick start (recommended) + +```bash +# From the repository root +./scripts/generate-viz.sh +``` + +This builds the Docker image (if needed), runs the full pipeline, and prints instructions. + +After success: + +```bash +cd generated +python -m http.server 8080 +# Then open http://localhost:8080/viz.html +``` + +On macOS: + +```bash +open generated/viz.html +``` + +## What you get + +- `generated/topics_with_layout.json` — every topic with stable 3D + 2D UMAP coordinates +- `generated/links.json` — the prerequisite graph (strength + reason) +- `generated/chroma/` — a queryable Chroma vector database with rich metadata +- `generated/generation_manifest.json` — full provenance (model, params, timestamp, text strategy) +- `generated/viz.html` — the interactive explorer (copied from the source template in embeddings/) + +Here's what it looks like: + +![Screenshot of the 3D semantic + graph explorer](embeddings/screenshot.png) + +## The visualization features (highlights) + +- **3D force-graph** (WebGL) seeded with UMAP positions +- Live **Graph pull** slider — morphs between pure semantic layout and graph-respecting layout +- Click any node → traces full prerequisite ancestor chains (warm/orange) + what it unlocks (green) with animated particles traveling the edges +- **"Highlight surprising prerequisites"** — surfaces hard edges between semantically distant concepts +- Search, subject filters, age quick-filters, multiple color modes +- Rich inspector panel with direct navigation +- Camera controls, auto-rotate, fit view, reset +- Fast (1,590 nodes is small) and uses fixed random seeds for reproducibility + +## Architecture & reproducibility + +**Embedding model** (default): `sentence-transformers/all-MiniLM-L6-v2` (384 dimensions) + +**Text strategy** (chosen after experimentation for coherent clusters): +``` +{subject} — {domain} — {name}. {description}. Evidence of mastery: {evidence joined}. +``` + +Age ranges and standards are deliberately left out of the embedding text (they are visualized through other channels). + +**Layout**: +- Primary: UMAP (cosine) → 3D coordinates +- Secondary: 2D for alternative views +- Random state fixed at 42 + +**Vector DB**: Chroma (persistent local directory) with full topic metadata. + +Everything is deterministic given the same model + parameters + source data snapshot. + +## Running without Docker (advanced) + +```bash +cd embeddings +python -m venv .venv +source .venv/bin/activate +pip install -r requirements.txt + +# From repo root +python embeddings/generate.py \ + --data-dir data \ + --output-dir generated \ + --model sentence-transformers/all-MiniLM-L6-v2 +``` + +Then serve `generated/viz.html`. + +## Trying stronger models + +Edit `embeddings/docker-compose.yml` or pass the env: + +```bash +EMBEDDING_MODEL=BAAI/bge-small-en-v1.5 ./scripts/generate-viz.sh +``` + +Other good options (via sentence-transformers): +- `nomic-ai/nomic-embed-text-v1.5` +- `BAAI/bge-base-en-v1.5` + +Update `umap` hyperparameters via CLI flags (`--n-neighbors`, `--min-dist`) and record them in the manifest. + +## Interpreting the view + +- **Clusters** = semantic neighborhoods (ideas that share vocabulary, structure, or conceptual framing). +- **Long threads** that cross large distances = pedagogically necessary bridges between conceptually distant domains. +- Pull the graph force slider high → concepts that *must* be learned together are pulled close even if their surface descriptions differ. +- The "surprises" button is often the most interesting starting point for curriculum designers. + +Example patterns you will likely discover: +- Foundational counting concepts sit near other early number sense ideas. +- Some "meta" learning-to-learn skills act as bridges across many subjects. +- Certain science process skills are surprisingly close to early mathematics despite different language. + +## License & attribution + +The generated embeddings and visualization are **produced works** based on the open Marble Skill Taxonomy. + +When using or sharing views: +- Credit the source taxonomy (see root README and CITATION.cff). +- Note the embedding model and generation parameters you used. + +The underlying data remains under ODbL 1.0 (structure) + CC BY-SA 4.0 (authored text). + +## Extending + +- Add more metadata to the Chroma collection (standards links, clusters). +- Export the layout to other tools (Gephi, Observable, Blender). +- Train a small classifier on top of the embeddings to predict subject or difficulty. +- Create 2D printable posters of interesting slices. + +Contributions that improve the hybrid layout algorithm, add new insight layers, or improve mobile/web performance are very welcome. + +--- + +This work is a contribution to the open-source Marble Skill Taxonomy project. + +Run the pipeline, serve the artifacts, and explore the combined semantic and graph structure. diff --git a/README.md b/README.md index 93a93575..8a2ceefe 100644 --- a/README.md +++ b/README.md @@ -123,3 +123,22 @@ Plus the upstream notices in [PROVENANCE.md](PROVENANCE.md) for any curriculum s ## What's *not* here Deliberately excluded from this release: semantic embeddings (derived, recomputable) and any per-child / user data (never published). See [CHANGELOG.md](CHANGELOG.md). + +## Semantic spatial explorer (community contribution) + +A Docker-powered pipeline exists to generate embeddings and an interactive visualization that places every micro-topic in 3D semantic space (via UMAP) while overlaying and respecting the prerequisite graph structure. + +See [EMBEDDINGS.md](EMBEDDINGS.md) for full documentation. + +```bash +./scripts/generate-viz.sh +# then open generated/viz.html (or serve the directory) +``` + +The explorer lets you: +- See conceptual clusters (semantic diversity) +- Trace prerequisite chains that sometimes span large distances in meaning-space +- Dial graph forces live to watch the "web of learning" pull concepts together +- Discover surprising long-range dependencies + +This is the perfect complement to the existing 3D age-based prerequisite graph. Both are now possible from the same open dataset. diff --git a/embeddings/Dockerfile b/embeddings/Dockerfile new file mode 100644 index 00000000..486f281e --- /dev/null +++ b/embeddings/Dockerfile @@ -0,0 +1,31 @@ +# embeddings/Dockerfile +# Lean, reproducible image for generating semantic embeddings + UMAP layout + Chroma vector DB +# for the Marble Skill Taxonomy. + +FROM python:3.12-slim AS base + +# System deps (minimal) +RUN apt-get update && apt-get install -y --no-install-recommends \ + build-essential \ + git \ + && rm -rf /var/lib/apt/lists/* + +WORKDIR /app + +# Copy requirements first for better layer caching +COPY requirements.txt . + +# Install Python deps (pinned). sentence-transformers pulls torch CPU by default in slim. +# For smaller images in future one could use ONNX, but this is reliable and explicit. +RUN pip install --no-cache-dir -r requirements.txt + +# Copy generation code and viz template (the template is source-controlled in embeddings/) +COPY generate.py . +COPY viz.html . + +# Default command runs the full pipeline. +# Data is expected at /data (volume), output written to /output (volume) +ENV PYTHONUNBUFFERED=1 +ENV EMBEDDING_MODEL=sentence-transformers/all-MiniLM-L6-v2 + +CMD ["python", "generate.py", "--data-dir", "/data", "--output-dir", "/output"] diff --git a/embeddings/docker-compose.yml b/embeddings/docker-compose.yml new file mode 100644 index 00000000..237862b2 --- /dev/null +++ b/embeddings/docker-compose.yml @@ -0,0 +1,20 @@ +# embeddings/docker-compose.yml +# One-shot generation of embeddings, Chroma vector DB, UMAP layouts, and viz-ready artifacts. + +services: + generate: + build: + context: . + dockerfile: Dockerfile + image: marble-taxonomy-embeddings:latest + container_name: marble-embeddings + volumes: + # Read-only access to the canonical data + - ../data:/data:ro + # Write artifacts + vector DB here + - ../generated:/output + environment: + - EMBEDDING_MODEL=sentence-transformers/all-MiniLM-L6-v2 + # Override for custom UMAP params if desired: + # command: python generate.py --data-dir /data --output-dir /output --n-neighbors 25 --min-dist 0.1 + restart: "no" diff --git a/embeddings/generate.py b/embeddings/generate.py new file mode 100644 index 00000000..94b64f45 --- /dev/null +++ b/embeddings/generate.py @@ -0,0 +1,382 @@ +#!/usr/bin/env python3 +""" +embeddings/generate.py + +Reproducible pipeline: +- Load Marble Skill Taxonomy topics + dependencies +- Construct rich embedding text (name + description + mastery evidence) +- Compute sentence embeddings (default: all-MiniLM-L6-v2) +- Persist to Chroma vector DB with rich metadata +- Compute deterministic UMAP 2D + 3D layouts (semantic diversity) +- Write clean artifacts for visualization + downstream use +- Full provenance manifest + +Designed to run inside Docker with volume mounts or directly with Python. +""" + +import argparse +import json +import os +import time +from datetime import datetime, timezone +from pathlib import Path +from typing import Any, Dict, List + +import numpy as np +from rich.console import Console +from rich.progress import Progress, SpinnerColumn, TextColumn, BarColumn, TimeElapsedColumn +from sentence_transformers import SentenceTransformer +from sklearn.metrics.pairwise import cosine_similarity +import umap +import chromadb +from chromadb.config import Settings + +console = Console() + +DEFAULT_MODEL = "sentence-transformers/all-MiniLM-L6-v2" +RANDOM_STATE = 42 + + +def load_data(data_dir: Path) -> tuple[List[Dict], List[Dict]]: + """Load and validate core taxonomy files.""" + topics_path = data_dir / "topics.json" + deps_path = data_dir / "dependencies.json" + + if not topics_path.exists(): + raise FileNotFoundError(f"topics.json not found at {topics_path}") + if not deps_path.exists(): + raise FileNotFoundError(f"dependencies.json not found at {deps_path}") + + with open(topics_path) as f: + topics_doc = json.load(f) + with open(deps_path) as f: + deps_doc = json.load(f) + + topics = topics_doc["topics"] + dependencies = deps_doc["dependencies"] + + console.log(f"[green]Loaded[/green] {len(topics)} topics and {len(dependencies)} dependencies from {data_dir}") + return topics, dependencies + + +def build_embedding_text(topic: Dict[str, Any]) -> str: + """ + First-principles text construction for a learning micro-topic. + + Goal: Capture both the conceptual essence and the "what mastery looks like" + signal so that embeddings reflect pedagogical ideas, not just surface words. + """ + subject = topic.get("subject", "") + domain = topic.get("domain") or "" + name = topic.get("name") or "" + description = topic.get("description", "").strip() + evidence = topic.get("evidence", []) + + parts = [] + if subject: + parts.append(subject) + if domain: + parts.append(domain) + if name: + parts.append(name) + + header = " — ".join(parts) if parts else "Concept" + + text = f"{header}. {description}" + if evidence: + evidence_str = " | ".join(e.strip() for e in evidence if e and e.strip()) + text += f" Evidence of mastery: {evidence_str}." + + return text.strip() + + +def compute_embeddings( + topics: List[Dict[str, Any]], + model_name: str, + batch_size: int = 64, +) -> np.ndarray: + """Compute dense embeddings with progress and determinism.""" + console.log(f"[bold]Loading embedding model:[/bold] {model_name}") + # device="cpu" explicit for reproducibility in container + model = SentenceTransformer(model_name, device="cpu") + + texts = [build_embedding_text(t) for t in topics] + + embeddings_list: List[np.ndarray] = [] + with Progress( + SpinnerColumn(), + TextColumn("[progress.description]{task.description}"), + BarColumn(), + TextColumn("{task.completed}/{task.total}"), + TimeElapsedColumn(), + console=console, + ) as progress: + task = progress.add_task("Embedding topics", total=len(texts)) + for i in range(0, len(texts), batch_size): + batch = texts[i : i + batch_size] + emb = model.encode( + batch, + show_progress_bar=False, + convert_to_numpy=True, + normalize_embeddings=True, + ) + embeddings_list.append(emb) + progress.update(task, advance=len(batch)) + + embeddings = np.vstack(embeddings_list).astype(np.float32) + dim = model.get_sentence_embedding_dimension() + console.log(f"[green]Computed[/green] embeddings shape: {embeddings.shape} (dim={dim})") + return embeddings, dim + + +def persist_to_chroma( + topics: List[Dict[str, Any]], + embeddings: np.ndarray, + chroma_dir: Path, + collection_name: str = "marble-taxonomy-topics", +) -> None: + """Persist vectors + rich metadata into a local Chroma DB.""" + chroma_dir.mkdir(parents=True, exist_ok=True) + + client = chromadb.PersistentClient( + path=str(chroma_dir), + settings=Settings(anonymized_telemetry=False), + ) + + # Fresh collection for reproducibility in this run + try: + client.delete_collection(collection_name) + except Exception: + pass + + collection = client.create_collection( + name=collection_name, + metadata={"hnsw:space": "cosine", "description": "Marble Skill Taxonomy micro-topics"}, + ) + + ids = [t["id"] for t in topics] + metadatas = [] + documents = [] + + for t in topics: + meta = { + "id": t["id"], + "name": t.get("name"), + "subject": t.get("subject"), + "domain": t.get("domain"), + "type": t.get("type"), + "ageRangeStart": t.get("ageRangeStart"), + "ageRangeEnd": t.get("ageRangeEnd"), + "centrality": t.get("centrality"), + } + metadatas.append(meta) + documents.append(build_embedding_text(t)) + + collection.add( + ids=ids, + embeddings=embeddings.tolist(), + metadatas=metadatas, + documents=documents, + ) + + console.log(f"[green]Persisted[/green] {len(ids)} vectors to Chroma at {chroma_dir} (collection: {collection_name})") + + +def compute_umap_layouts( + embeddings: np.ndarray, + n_neighbors: int = 20, + min_dist: float = 0.08, +) -> Dict[str, np.ndarray]: + """Deterministic UMAP for 3D (primary) and 2D.""" + console.log("[bold]Computing UMAP layouts[/bold] (cosine, fixed random_state=42)...") + + # 3D primary for spatial "distant concepts" experience + reducer3d = umap.UMAP( + n_components=3, + n_neighbors=n_neighbors, + min_dist=min_dist, + metric="cosine", + random_state=RANDOM_STATE, + verbose=False, + ) + coords3d = reducer3d.fit_transform(embeddings) + + reducer2d = umap.UMAP( + n_components=2, + n_neighbors=n_neighbors, + min_dist=min_dist, + metric="cosine", + random_state=RANDOM_STATE, + verbose=False, + ) + coords2d = reducer2d.fit_transform(embeddings) + + console.log(f" 3D layout range: x=[{coords3d[:,0].min():.2f},{coords3d[:,0].max():.2f}] ...") + return {"3d": coords3d, "2d": coords2d} + + +def build_artifacts( + topics: List[Dict[str, Any]], + dependencies: List[Dict[str, Any]], + coords: Dict[str, np.ndarray], + output_dir: Path, + model_name: str, + embedding_dim: int, + umap_params: Dict[str, Any], +) -> None: + """Write the clean, versioned artifacts consumed by the visualization.""" + output_dir.mkdir(parents=True, exist_ok=True) + + # Enriched nodes with layout positions (primary = 3D) + nodes = [] + coords3d = coords["3d"] + coords2d = coords["2d"] + + for i, t in enumerate(topics): + node = { + "id": t["id"], + "name": t.get("name"), + "subject": t.get("subject"), + "domain": t.get("domain"), + "type": t.get("type"), + "ageRangeStart": t.get("ageRangeStart"), + "ageRangeEnd": t.get("ageRangeEnd"), + "centrality": t.get("centrality"), + "x": float(coords3d[i, 0]), + "y": float(coords3d[i, 1]), + "z": float(coords3d[i, 2]), + "x2": float(coords2d[i, 0]), + "y2": float(coords2d[i, 1]), + } + nodes.append(node) + + # Slim links (preserve the pedagogical graph) + links = [] + for d in dependencies: + links.append({ + "source": d["topicId"], + "target": d["prerequisiteId"], + "strength": d["strength"], + "reason": d.get("reason"), + }) + + # Write artifacts + (output_dir / "topics_with_layout.json").write_text( + json.dumps({"nodes": nodes, "count": len(nodes)}, indent=2) + ) + (output_dir / "links.json").write_text( + json.dumps({"links": links, "count": len(links)}, indent=2) + ) + + # Provenance manifest (critical for reproducibility) + manifest = { + "dataset": "Marble Skill Taxonomy", + "taxonomyVersion": "v1", + "generatedAt": datetime.now(timezone.utc).isoformat(), + "model": model_name, + "embeddingDim": embedding_dim, + "numTopics": len(nodes), + "numLinks": len(links), + "umap": { + "n_neighbors": umap_params["n_neighbors"], + "min_dist": umap_params["min_dist"], + "metric": "cosine", + "random_state": RANDOM_STATE, + }, + "layout": "UMAP 3D (primary) + 2D", + "textStrategy": "subject — domain — name. description. Evidence of mastery: ...", + "notes": "Embeddings and layouts are derived and fully reproducible from the open taxonomy data.", + } + (output_dir / "generation_manifest.json").write_text(json.dumps(manifest, indent=2)) + + # Also copy a small README for the generated dir + (output_dir / "README.txt").write_text( + "Generated artifacts for Marble Taxonomy semantic visualization.\n" + "Open viz.html (after generation) or serve this directory.\n" + "See ../EMBEDDINGS.md for full documentation.\n" + ) + + # Copy the source viz.html template into generated/ (the template lives in embeddings/ as source) + viz_dst = output_dir / "viz.html" + import shutil + src = Path(__file__).parent / "viz.html" # embeddings/viz.html (source controlled) + if src.exists(): + shutil.copy(src, viz_dst) + console.log(f"[green]Copied[/green] viz.html template from embeddings/ to generated/") + else: + console.log("[yellow]Warning: embeddings/viz.html not found; visualization will be missing[/yellow]") + + console.log(f"[green]Wrote artifacts[/green] to {output_dir}") + + +def sanity_check(embeddings: np.ndarray, topics: List[Dict[str, Any]]) -> None: + """Lightweight sanity: nearest neighbors for a couple of known concepts.""" + console.log("[bold]Sanity check — nearest neighbors (cosine on embeddings)[/bold]") + # Simple brute force for tiny N + sims = cosine_similarity(embeddings) + + # Pick a few interesting indices + interesting = ["mt_AzTrT5ySCx", "mt_glPPG-kTQY"] # AI in Daily Life, Adding within 100 + id_to_idx = {t["id"]: i for i, t in enumerate(topics)} + + for tid in interesting: + if tid not in id_to_idx: + continue + idx = id_to_idx[tid] + sim_row = sims[idx] + top = np.argsort(sim_row)[::-1][1:4] # skip self + name = topics[idx].get("name") + console.log(f" '{name}' ({tid}) nearest:") + for j in top: + console.log(f" • {topics[j].get('name')} (sim={sim_row[j]:.3f})") + + +def main(): + parser = argparse.ArgumentParser(description="Generate embeddings + layout for Marble taxonomy") + parser.add_argument("--data-dir", type=Path, default=Path("/data"), help="Directory containing topics.json + dependencies.json") + parser.add_argument("--output-dir", type=Path, default=Path("/output"), help="Where to write generated/ artifacts and chroma/") + parser.add_argument("--model", type=str, default=os.environ.get("EMBEDDING_MODEL", DEFAULT_MODEL)) + parser.add_argument("--n-neighbors", type=int, default=20) + parser.add_argument("--min-dist", type=float, default=0.08) + args = parser.parse_args() + + start = time.time() + console.rule("[bold cyan]Marble Taxonomy — Embeddings + Spatial Layout Generator[/bold cyan]") + + data_dir = args.data_dir + output_dir = args.output_dir + chroma_dir = output_dir / "chroma" + + topics, dependencies = load_data(data_dir) + + embeddings, embedding_dim = compute_embeddings(topics, args.model) + + persist_to_chroma(topics, embeddings, chroma_dir) + + coords = compute_umap_layouts( + embeddings, + n_neighbors=args.n_neighbors, + min_dist=args.min_dist, + ) + + build_artifacts( + topics, + dependencies, + coords, + output_dir, + args.model, + embedding_dim, + {"n_neighbors": args.n_neighbors, "min_dist": args.min_dist}, + ) + + sanity_check(embeddings, topics) + + elapsed = time.time() - start + console.rule(f"[bold green]Complete[/bold green] in {elapsed:.1f}s") + console.log(f"Artifacts: {output_dir}") + console.log(f"Vector DB: {chroma_dir}") + console.log("Next: open generated/viz.html or serve the directory.") + + +if __name__ == "__main__": + main() diff --git a/embeddings/requirements.txt b/embeddings/requirements.txt new file mode 100644 index 00000000..7e7d7420 --- /dev/null +++ b/embeddings/requirements.txt @@ -0,0 +1,12 @@ +# embeddings/requirements.txt +# Pinned for reproducibility. CPU-friendly defaults. +sentence-transformers==3.1.1 +umap-learn==0.5.6 +chromadb==0.5.5 +numpy==1.26.4 +tqdm==4.66.5 +# networkx optional for any pre-hybrid layout +networkx==3.3 +scikit-learn==1.5.1 +# For nicer CLI / provenance +rich==13.7.1 diff --git a/embeddings/screenshot.png b/embeddings/screenshot.png new file mode 100644 index 00000000..c622d6fa Binary files /dev/null and b/embeddings/screenshot.png differ diff --git a/embeddings/viz.html b/embeddings/viz.html new file mode 100644 index 00000000..e4e572f9 --- /dev/null +++ b/embeddings/viz.html @@ -0,0 +1,1055 @@ + + + + + + Marble Taxonomy • Semantic + Graph Explorer + + + + + + +
+ +
+ +
+
+
+ M +
+
+
Marble
+
SKILL TAXONOMY • v1
+
+
+
+ 1,590 micro-concepts in semantic space
+ connected by the prerequisite graph +
+
+ + +
+
+ +
⌘K
+
+
+ + +
+
Subjects
+
+ +
+
Color by
+
+ + + +
+
+ + +
+
Layout & Graph Forces
+ +
+
+
+
Graph pull
+
0.35
+
+ +
+ +
+ + +
+ + +
+
+ + +
+
Quick filters
+
+ + + +
+
+
+ + +
+ +
Data: Marble Skill Taxonomy v1 • withmarble.com
+
+
+ + +
+ +
+
+
+ Semantic × Prerequisite Space + 3D UMAP + forces +
+ +
+ +
+ + + +
+
+ + +
+
+ + + +
+ + +
+
Ready
+
+
1,590 nodes • 3,221 edges • UMAP 3D seeded
+
+
+ + +
+
+
Selected concept
+
—
+
+
+ +
+ +
+
DESCRIPTION
+
+
+ + +
+
EVIDENCE OF MASTERY
+
    +
    + + +
    +
    +
    PREREQUISITES
    +
    +
    +
    + +
    +
    +
    UNLOCKS
    +
    +
    +
    + + +
    + +
    + +
    +
    +
    + + + + + + + diff --git a/package.json b/package.json index 54db870c..4995ce55 100644 --- a/package.json +++ b/package.json @@ -7,6 +7,8 @@ "license": "(ODbL-1.0 AND CC-BY-SA-4.0)", "homepage": "https://withmarble.com", "scripts": { - "validate": "node scripts/validate.mjs" + "validate": "node scripts/validate.mjs", + "generate-viz": "bash scripts/generate-viz.sh", + "serve-viz": "cd generated && python3 -m http.server 8080" } } diff --git a/scripts/generate-viz.sh b/scripts/generate-viz.sh new file mode 100755 index 00000000..73426137 --- /dev/null +++ b/scripts/generate-viz.sh @@ -0,0 +1,67 @@ +#!/bin/bash +# +# scripts/generate-viz.sh +# One-command (or near one-command) entrypoint for the full embeddings + viz pipeline. +# +# Usage: +# ./scripts/generate-viz.sh +# # or with Docker Compose from embeddings/ dir +# +# After success: +# open generated/viz.html (macOS) +# or cd generated && python3 -m http.server 8080 + +set -euo pipefail + +ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +EMBEDDINGS_DIR="$ROOT_DIR/embeddings" +GENERATED_DIR="$ROOT_DIR/generated" + +echo "============================================================" +echo " Marble Skill Taxonomy — Semantic + Graph Visualization" +echo " Embeddings + UMAP + 3D Force-Directed Explorer" +echo "============================================================" + +cd "$EMBEDDINGS_DIR" + +echo "" +echo "→ Building Docker image (this may take a few minutes the first time)..." +docker compose build --pull + +echo "" +echo "→ Running generation pipeline inside container..." +docker compose up --build --abort-on-container-exit --remove-orphans + +echo "" +echo "============================================================" +echo "✓ Generation complete." +echo "" +echo "Artifacts written to:" +echo " $GENERATED_DIR/" +echo " $GENERATED_DIR/chroma/ (vector database)" +echo " $GENERATED_DIR/viz.html (if copied) or use the HTML below" +echo "" + +if [[ -f "$GENERATED_DIR/topics_with_layout.json" ]]; then + COUNT=$(python3 -c ' +import json, sys +try: + print(json.load(open(sys.argv[1]))["count"]) +except Exception: + print("unknown") +' "$GENERATED_DIR/topics_with_layout.json" 2>/dev/null) + echo "✓ $COUNT topics with 3D UMAP coordinates ready." +fi + +echo "" +echo "To view the visualization:" +echo " 1. cd $GENERATED_DIR" +echo " 2. python3 -m http.server 8080" +echo " 3. Open http://localhost:8080/viz.html in your browser" +echo "" +echo "On macOS you can also try:" +echo " open $GENERATED_DIR/viz.html" +echo "" +echo "See EMBEDDINGS.md for full documentation, how to change models, and" +echo "how to interpret the 'semantic diversity + prerequisite web' view." +echo "============================================================"