feat(gradebook): export-canvas template overlay for clean Canvas re-import

README.md

@@ -65,7 +65,7 @@ This is a deliberate design principle, not an accident. Proprietary LMS gradeboo
 | `reg-term-create` | `term_create` | Scaffold a term from a YAML term-spec: semester note + per-section class notes + archive manifest skeletons + course-catalog wiring. Idempotent; `--init` writes a stub spec. |
 | `reg-term-finalize` | `term_finalize` | Reconcile grade distributions, flip section statuses to finalized, roll up enrollment-weighted aggregates. Supports `--dry-run`. |
 | `reg-term-archive` | `term_archive` | Build per-section archive bundles (roster → grades → GitHub → gradebook → exams → lectures → syllabus → `manifest.yaml`). `--check` validates an existing bundle for drift. |
-| `reg-gradebook` | `gradebook` + `gradebook_ledger` | Vault-native **grades ledger** (the vault is the grade source of truth; grades flow vault → Canvas). `build` rolls per-component score files (a `components.yaml` registry) into `gradebook.csv` + a `GRADEBOOK.md` ledger (grouped overview + per-assignment subsidiary ledgers + a live per-student view) with source-document reconciliation; `export-canvas` emits a Canvas bulk-upload CSV; the legacy `import` (Canvas → vault) and `dfw` / `dist` / `check` remain. |
+| `reg-gradebook` | `gradebook` + `gradebook_ledger` | Vault-native **grades ledger** (the vault is the grade source of truth; grades flow vault → Canvas). `build` rolls per-component score files (a `components.yaml` registry) into `gradebook.csv` + a `GRADEBOOK.md` ledger (grouped overview + per-assignment subsidiary ledgers + a live per-student view) with source-document reconciliation; `export-canvas` emits a Canvas bulk-upload CSV — bare (`SIS User ID` + graded-component columns) or, with `--template <canvas-export.csv> [--only lab2]`, **overlaid onto a Canvas gradebook export** so it re-imports cleanly into the existing assignment (id-suffixed header match, identity columns + posting row preserved, minimal diff, EC folded in — see [docs/canvas-grades-workflow.md](docs/canvas-grades-workflow.md)); the legacy `import` (Canvas → vault) and `dfw` / `dist` / `check` remain. |
 | `reg-gradescope-stats` | `gradescope_stats` | Per-outcome **item analysis** from Gradescope *Export Evaluations* — per-distractor stats joined to the grading-note `form·Qn·slot` keys; flags dead distractors, over-key distractors, and the miskey alarm. Emits `ITEM_ANALYSIS.md`, a self-contained newspaper/agate **broadsheet**, and a per-student×question `item_scores` matrix. |
 | `reg-exam-build` | `exam_build` | Assemble exam PDFs — single-source `.tex` mode or pack-mode `.yaml` manifest (multi-form A/B/C, per-student individualized, Gradescope products). See below. |
 | `reg-exam-verify` | `exam_verify` | Verify a student exam serial against the register. Confirms which form and which student a paper belongs to. |

docs/README.md

@@ -15,6 +15,7 @@
 | Document | Description |
 |---|---|
 | [gradescope-workflow.md](gradescope-workflow.md) | Step-by-step for taking a lectern-built exam into Gradescope: region/bubble setup, A/B Version Sets, roster import, per-student serial identity verification |
+| [canvas-grades-workflow.md](canvas-grades-workflow.md) | Pushing vault grades to Canvas for one assignment via a Canvas-export **template overlay**: id-suffixed assignment match, `--only` targeting, minimal diff, extra-credit handling |
 
 ## Quick pointers
 

docs/canvas-grades-workflow.md

@@ -0,0 +1,91 @@
+# Pushing Vault Grades to Canvas
+
+How to move grades from the vault (the source of truth) into Canvas for a single
+assignment — a lab or an exam — without disturbing anything else in the Canvas
+gradebook.
+
+The vault is the grade source of truth; grades flow **vault → Canvas**. The clean
+way to do that is a *template overlay*: you export the Canvas gradebook, lay the
+vault's computed scores onto that exact file, and re-import it. Because the import
+file is the export with only your changed cells touched, Canvas updates the
+existing assignment and leaves everything else alone.
+
+Companion: [design/lab-report.md](design/lab-report.md) (feedback delivery, the
+GitHub side of the same grading round). Canvas's own import rules:
+<https://community.instructure.com/en/kb/articles/660862-how-do-i-import-grades-in-the-gradebook>.
+
+---
+
+## TL;DR
+
+```sh
+# 1. In Canvas: Grades → Export → "Export Current Gradebook" → save the CSV.
+# 2. Overlay the vault's scores for one component onto that export:
+reg-gradebook export-canvas \
+    --gradebook gradebook.csv \
+    --course CECS_378 \                     # resolves the schema (or pass --schema)
+    --template <the-canvas-export>.csv \
+    --only lab2 \                           # the component short_name to push
+    --out canvas-<lab2>-import.csv
+# 3. In Canvas: Grades → Import → upload canvas-<lab2>-import.csv → review → save.
+```
+
+Step 2 changes only the cells whose value actually moved. Everything else — the
+identity columns, the other assignment columns, the points/posting row — is passed
+through byte-for-byte.
+
+---
+
+## Why a template overlay (and not a bare CSV)
+
+Canvas matches an imported column to an existing assignment by its **id-suffixed
+header**, e.g. `Lab 2 - Malicious Software (1767131)`. A bare `SIS User ID, Lab 2`
+file (no id) risks creating a *new* manual assignment instead of updating the real
+one. Only the Canvas export carries those ids, the student identity columns
+(`Student, ID, SIS User ID, SIS Login ID, Section`), and the posting row — so the
+export is the only faithful template. `export-canvas --template` overlays onto it:
+
+- **Assignment match** — each export column header is matched to a schema
+  component by stripping the ` (id)` suffix and looking up its `canvas_title`.
+  `--only <short_name…>` restricts the overlay to specific components (the usual
+  case: push one lab or one exam); omit it to overlay every graded component.
+- **Student match** — by `SIS User ID` (zero-padded), against `gradebook.csv`.
+  A student in the export but not in the gradebook (a Canvas "Test Student", a
+  late add) is passed through untouched.
+- **Minimal diff** — a cell is rewritten only when its value actually changed;
+  numerically-equal cells keep the export's original text. The import therefore
+  touches just the grades that moved, which is easy to eyeball before you upload.
+- **Read-only columns** — the export's computed columns (`Current Score`,
+  `Final Grade`, `Override …`) ride along unchanged; Canvas ignores them on import.
+
+## Extra credit
+
+Extra credit is already folded into the component score in the vault, because the
+per-component score file feeds it in (e.g. a lab's `score` column = base + ACE).
+So the overlay carries EC automatically — a Lab 2 base of 96 with +14 ACE lands in
+Canvas as `110`. Canvas accepts scores above the assignment's points-possible, so
+this is the natural single-column home for built-in extra credit. (If you keep a
+*separate* Canvas EC assignment instead, push base-only by building the gradebook
+from base scores, or add the EC as its own component.)
+
+## Bare-CSV mode (no template)
+
+Without `--template`, `export-canvas` emits the minimal `SIS User ID` + one column
+per graded component (header = bare `canvas_title`, no id). This is fine for a
+brand-new assignment you want Canvas to create, or a quick spot-check, but for
+updating existing assignments prefer the template overlay above.
+
+## Worked example — CECS 378 Lab 2 (Su26 §01)
+
+Lab 2 base grades were already in Canvas; only the Task-2 ACE extra credit needed
+to land. Exporting Canvas, then:
+
+```sh
+reg-gradebook export-canvas --gradebook gradebook.csv --course CECS_378 \
+    --template 2026-06-28_Grades-CECS_378_Sec01.csv --only lab2 \
+    --out canvas-lab2-import.csv
+```
+
+produced an import identical to the export except six cells — the six students who
+earned ACE (Badberg 82→91, Bavouset 91→106, Cuevas 92→97, C. Schulte 96→110,
+D. Schulte 92→104, Silaiev 71→75). Re-importing applied exactly those six changes.

lectern/gradebook.py

@@ -631,8 +631,10 @@ def _cmd_export_canvas(args: argparse.Namespace) -> int:
     if schema_path is None:
         sys.exit("export-canvas needs --schema or --course to resolve the schema")
     schema = load_schema(schema_path)
-    export_canvas(args.gradebook, schema, args.out)
-    print(f"→ {args.out}")
+    export_canvas(args.gradebook, schema, args.out,
+                  template=getattr(args, "template", None),
+                  only=getattr(args, "only", None))
+    print(f"→ {args.out}" + (f" (overlaid onto {args.template})" if getattr(args, "template", None) else ""))
     return 0
 
 
@@ -744,6 +746,12 @@ def main(argv: list[str] | None = None) -> int:
     pe.add_argument("--schema", type=Path, help="schema yaml (else resolved by --course)")
     pe.add_argument("--course", help="course code (used only to resolve --schema)")
     pe.add_argument("--out", type=Path, required=True)
+    pe.add_argument("--template", type=Path,
+                    help="a Canvas gradebook EXPORT to overlay scores onto, "
+                         "preserving its exact format for clean re-import")
+    pe.add_argument("--only", nargs="*",
+                    help="with --template, restrict the overlay to these component "
+                         "short_names (e.g. lab2); default = all graded components")
     pe.set_defaults(func=_cmd_export_canvas)
 
     pd = sub.add_parser("dfw", help="roll up DFW rate across sections of a term")

lectern/gradebook_build.py

@@ -10,6 +10,7 @@
 
 import csv
 import json
+import re
 import sys
 from dataclasses import dataclass
 from pathlib import Path
@@ -247,12 +248,104 @@ def build_gradebook(
     return rows
 
 
-def export_canvas(gradebook_csv: Path, schema: GradebookSchema, out: Path) -> None:
-    """Emit a Canvas bulk-upload CSV: 'SIS User ID' + one column per GRADED
-    component (its schema canvas_title), values = raw earned points. A component
-    that no student has graded gets no column (never upload-zeroes ungraded work).
-    A student with no score on an otherwise-graded component gets a blank cell.
+_CANVAS_ID_SUFFIX = re.compile(r"^(.*?)\s*\(\d+\)\s*$")
+
+
+def _fmt_canvas(v) -> str:
+    """Format a numeric score the way Canvas accepts: integers without a trailing
+    `.0`, fractions kept (e.g. 60.0 -> '60', 97.5 -> '97.5')."""
+    try:
+        return f"{float(v):g}"
+    except (TypeError, ValueError):
+        return "" if v is None else str(v)
+
+
+def _overlay_canvas_template(gradebook_csv: Path, schema: GradebookSchema,
+                             template: Path, out: Path, only) -> None:
+    """Overlay vault scores onto a Canvas gradebook EXPORT, preserving its exact
+    shape (identity columns, `Name (assignment_id)` headers, posting/points row).
+    Only assignment columns that map to a schema component — and, if `only` is
+    given, only those short_names — have their cells updated; every other column,
+    the posting row, and any student not in the gradebook are passed through
+    untouched. Match is by SIS User ID. This is the re-import-safe form: Canvas
+    updates the existing assignment (matched by its id-suffixed header) and leaves
+    everything you didn't touch alone.
+    """
+    title_to_short = {c["canvas_title"]: c["short_name"] for c in schema.columns}
+    only = set(only) if only else None
+
+    scores: dict[str, dict] = {}
+    with Path(gradebook_csv).open(encoding="utf-8") as fh:
+        for r in csv.DictReader(fh):
+            scores[pad_student_id(r["student_id"])] = json.loads(r.get("raw_scores") or "{}")
+
+    rows = list(csv.reader(Path(template).open(newline="", encoding="utf-8")))
+    if not rows:
+        sys.exit(f"empty Canvas template: {template}")
+    header = rows[0]
+    if "SIS User ID" not in header:
+        sys.exit(f"Canvas template missing 'SIS User ID' column: {template}")
+    sis_i = header.index("SIS User ID")
+
+    col_short: dict[int, str] = {}
+    for i, h in enumerate(header):
+        m = _CANVAS_ID_SUFFIX.match(h.strip())
+        title = (m.group(1).strip() if m else h.strip())
+        short = title_to_short.get(title)
+        if short and (only is None or short in only):
+            col_short[i] = short
+    if not col_short:
+        sys.exit("no assignment columns in the template match the schema"
+                 + (f" for --only {sorted(only)}" if only else ""))
+
+    out_rows = [header]
+    body = rows[1:]
+    # A leading posting/points row carries no SIS value — preserve it verbatim.
+    if body and (sis_i >= len(body[0]) or not str(body[0][sis_i]).strip()):
+        out_rows.append(body[0]); body = body[1:]
+    for r in body:
+        row = list(r)
+        sid = pad_student_id(r[sis_i]) if sis_i < len(r) else ""
+        raw = scores.get(sid)
+        if raw:
+            for i, short in col_short.items():
+                if i >= len(row) or raw.get(short) is None:
+                    continue
+                new = _fmt_canvas(raw[short])
+                old = str(row[i]).strip()
+                # Minimal diff: keep the template's existing cell (and its exact
+                # formatting) when the value is numerically unchanged; only rewrite
+                # a genuine change, so the import touches just the moved grades.
+                try:
+                    if old != "" and float(old) == float(new):
+                        continue
+                except ValueError:
+                    pass
+                row[i] = new
+        out_rows.append(row)
+
+    Path(out).parent.mkdir(parents=True, exist_ok=True)
+    with Path(out).open("w", newline="", encoding="utf-8") as fh:
+        csv.writer(fh).writerows(out_rows)
+
+
+def export_canvas(gradebook_csv: Path, schema: GradebookSchema, out: Path,
+                  *, template: Path | None = None, only=None) -> None:
+    """Emit a Canvas bulk-upload CSV.
+
+    Default (no template): 'SIS User ID' + one column per GRADED component (its
+    schema canvas_title), values = raw earned points. A component no student has
+    graded gets no column (never upload-zeroes ungraded work); a student with no
+    score on an otherwise-graded component gets a blank cell.
+
+    With `template` (a Canvas gradebook export): overlay the vault scores onto
+    that export's exact structure so the result re-imports cleanly into the same
+    assignment (id-suffixed header), updating only the targeted cells. `only`
+    restricts the overlay to specific component short_names.
     """
+    if template is not None:
+        _overlay_canvas_template(gradebook_csv, schema, Path(template), out, only)
+        return
     short_to_title = {c["short_name"]: c["canvas_title"] for c in schema.columns}
     schema_order = [c["short_name"] for c in schema.columns]
 

tests/test_gradebook_build.py

@@ -1,3 +1,4 @@
+import csv
 import json
 from pathlib import Path
 import pytest
@@ -313,3 +314,106 @@ def test_build_writes_ledger_surfaces(tmp_path, schema_378):
     assert (out / "assignments" / "exam1.md").exists()
     gb = (out / "GRADEBOOK.md").read_text()
     assert "Per-student statements" in gb and "exam1" in gb
+
+
+# ── canvas export: template overlay (re-import preserving Canvas's exact format) ─
+
+CANVAS_TEMPLATE = (
+    "Student,ID,SIS User ID,SIS Login ID,Section,"
+    "Lab 1 - Symmetric Cryptography (1767130),Exam 1 (1767127),Current Score\n"
+    ",,,,,Manual Posting,Manual Posting,\n"
+    '"Kane, Kate",101,040100020,kkane,CECS 378 Sec01,55.00,40.00,80.00\n'
+    '"Pennyworth, Alfreda",102,040100010,apenny,CECS 378 Sec01,30.00,0.00,30.00\n'
+)
+
+
+def _gb_overlay(tmp_path):
+    gb = tmp_path / "gradebook.csv"
+    gb.write_text(
+        "student_id,display_name,enrollment_status,raw_scores,standing_score,"
+        "weighted_score,letter_grade,in_progress,graded_cols,total_cols,flags\n"
+        '040100020,Kate Kane,enrolled,"{""lab1"": 60.0, ""exam1"": 45.0}",80.0,80.0,B,true,2,3,\n'
+        '040100010,Alfreda Pennyworth,enrolled,"{""lab1"": 30.0, ""exam1"": 10.0}",0.0,0.0,F,true,2,3,\n',
+        encoding="utf-8")
+    return gb
+
+
+def test_overlay_updates_only_targeted_assignment(tmp_path, schema_378):
+    schema = load_schema(schema_378)
+    gb = _gb_overlay(tmp_path)
+    tmpl = tmp_path / "canvas_export.csv"; tmpl.write_text(CANVAS_TEMPLATE, encoding="utf-8")
+    out = tmp_path / "import.csv"
+    export_canvas(gb, schema, out, template=tmpl, only=["lab1"])
+    rows = list(csv.reader(out.open(encoding="utf-8")))
+    assert rows[0][5] == "Lab 1 - Symmetric Cryptography (1767130)"  # id-suffixed header kept
+    assert rows[1][5] == "Manual Posting"                            # posting row preserved
+    by = {r[2]: r for r in rows[2:]}
+    assert by["040100020"][5] == "60"        # lab1 overlaid (was 55.00)
+    assert by["040100020"][6] == "40.00"     # exam1 NOT in --only → template value kept
+    assert by["040100020"][0] == "Kane, Kate"  # identity preserved
+    assert by["040100010"][5] == "30.00"     # 30.0 == template 30.00 → unchanged, format kept
+
+
+def test_overlay_no_only_updates_all_known(tmp_path, schema_378):
+    schema = load_schema(schema_378)
+    gb = _gb_overlay(tmp_path)
+    tmpl = tmp_path / "t.csv"; tmpl.write_text(CANVAS_TEMPLATE, encoding="utf-8")
+    out = tmp_path / "o.csv"
+    export_canvas(gb, schema, out, template=tmpl)
+    by = {r[2]: r for r in csv.reader(out.open(encoding="utf-8"))}
+    assert by["040100020"][5] == "60" and by["040100020"][6] == "45"   # both overlaid
+
+
+def test_overlay_preserves_unknown_student_and_posting_row(tmp_path, schema_378):
+    schema = load_schema(schema_378)
+    gb = _gb_overlay(tmp_path)
+    tmpl = tmp_path / "t.csv"
+    tmpl.write_text(CANVAS_TEMPLATE +
+        '"Test, Student",999,,,CECS 378 Sec01,12.00,12.00,12.00\n', encoding="utf-8")
+    out = tmp_path / "o.csv"
+    export_canvas(gb, schema, out, template=tmpl, only=["lab1"])
+    rows = list(csv.reader(out.open(encoding="utf-8")))
+    test_row = [r for r in rows if r[0] == "Test, Student"][0]
+    assert test_row[5] == "12.00"   # blank SIS / not in gradebook → untouched
+
+
+def test_overlay_errors_when_no_matching_columns(tmp_path, schema_378):
+    schema = load_schema(schema_378)
+    gb = _gb_overlay(tmp_path)
+    tmpl = tmp_path / "t.csv"; tmpl.write_text(CANVAS_TEMPLATE, encoding="utf-8")
+    out = tmp_path / "o.csv"
+    import pytest
+    with pytest.raises(SystemExit):
+        export_canvas(gb, schema, out, template=tmpl, only=["lab3"])  # not in template
+
+
+def test_cli_export_canvas_template_overlay(tmp_path, schema_378):
+    gb = _gb_overlay(tmp_path)
+    tmpl = tmp_path / "canvas_export.csv"; tmpl.write_text(CANVAS_TEMPLATE, encoding="utf-8")
+    out = tmp_path / "import.csv"
+    rc = gradebook_main([
+        "export-canvas", "--gradebook", str(gb), "--schema", str(schema_378),
+        "--out", str(out), "--template", str(tmpl), "--only", "lab1",
+    ])
+    assert rc == 0
+    by = {r[2]: r for r in csv.reader(out.open(encoding="utf-8"))}
+    assert by["040100020"][5] == "60" and by["040100020"][6] == "40.00"
+
+
+def test_overlay_minimal_diff_keeps_unchanged_cell_formatting(tmp_path, schema_378):
+    # A student whose gradebook lab1 == the template value keeps the template's
+    # exact string (e.g. "60.00"); only genuinely-changed cells are rewritten.
+    schema = load_schema(schema_378)
+    gb = tmp_path / "gradebook.csv"
+    gb.write_text(
+        "student_id,display_name,enrollment_status,raw_scores,standing_score,"
+        "weighted_score,letter_grade,in_progress,graded_cols,total_cols,flags\n"
+        '040100020,Kate Kane,enrolled,"{""lab1"": 55.0}",80.0,80.0,B,true,1,3,\n'    # == template 55.00
+        '040100010,Alfreda Pennyworth,enrolled,"{""lab1"": 42.0}",0.0,0.0,F,true,1,3,\n',  # changed
+        encoding="utf-8")
+    tmpl = tmp_path / "t.csv"; tmpl.write_text(CANVAS_TEMPLATE, encoding="utf-8")
+    out = tmp_path / "o.csv"
+    export_canvas(gb, schema, out, template=tmpl, only=["lab1"])
+    by = {r[2]: r for r in csv.reader(out.open(encoding="utf-8"))}
+    assert by["040100020"][5] == "55.00"   # unchanged → template formatting preserved
+    assert by["040100010"][5] == "42"      # changed (was 30.00) → rewritten