From f0c11d608f40d9e866caddc0e29034ed92dfc069 Mon Sep 17 00:00:00 2001 From: Weston Voglesonger Date: Mon, 13 Jul 2026 19:04:19 -0400 Subject: [PATCH 1/5] Add the #54 schedule data loop (base): renderer, validator, importer, example Implements the reusable flat-data schedule pipeline from #54 in the shared theme repo, where clinic builds read includes + data from (via the submodule): - _includes/schedule.html: shared, fixed renderer (Week -> Day -> Session); resolves instructor keys against _data/team, drops shadow rows in the participant view. Never edited during a clinic. - schemas/schedule-cohort.schema.json: Tier-0 JSON Schema. - tools/validate_schedule.py: Tier 0 (schema) + Tier 1 (people resolution, non-overlap within track, end>=start). tools/import_schedule.py: wall -> YAML. - .github/workflows/schedule-validate.yml: PR/push gate on _data/schedule/**. - _data/schedule/roles.yml: shared role tokens. - _data/schedule/example/sample.yml: a SYNTHETIC example cohort (not a real clinic) so the validator + renderer are exercised in CI and the format is documented. Per-clinic schedule DATA stays OUT of the shared theme: each clinic keeps its own schedule YAML in its own repo. Where a clinic's build reads that data from is the #56 topology decision. Verified locally: validator 0 errors / 0 warnings; a real Jekyll build renders the example (1 week / 2 days / 8 visible sessions of 9), resolves a person key to a linked name and role tokens to their labels, and drops the shadow row (0 leaks). Part of #54 / epic #58. --- .github/workflows/schedule-validate.yml | 55 ++++ _data/schedule/example/sample.yml | 97 +++++++ _data/schedule/roles.yml | 19 ++ _includes/schedule.html | 95 +++++++ schemas/schedule-cohort.schema.json | 181 +++++++++++++ tools/import_schedule.py | 338 ++++++++++++++++++++++++ tools/validate_schedule.py | 242 +++++++++++++++++ 7 files changed, 1027 insertions(+) create mode 100644 .github/workflows/schedule-validate.yml create mode 100644 _data/schedule/example/sample.yml create mode 100644 _data/schedule/roles.yml create mode 100644 _includes/schedule.html create mode 100644 schemas/schedule-cohort.schema.json create mode 100644 tools/import_schedule.py create mode 100644 tools/validate_schedule.py diff --git a/.github/workflows/schedule-validate.yml b/.github/workflows/schedule-validate.yml new file mode 100644 index 0000000..4dc8a13 --- /dev/null +++ b/.github/workflows/schedule-validate.yml @@ -0,0 +1,55 @@ +# Gate schedule DATA edits (#54). +# +# A Jekyll build stays green even when a schedule is wrong, because Liquid renders +# an undefined value or a broken reference as the empty string. This workflow runs +# tools/validate_schedule.py (Tier 0 schema + Tier 1 referential: people keys must +# resolve against _data/team, sessions non-overlapping within a track, end >= start) +# so a bad edit fails a check with a clear, annotated error instead of silently +# shipping a broken page. Make it a REQUIRED status check in branch protection to +# actually gate merges. +name: Validate schedule data + +on: + pull_request: + paths: + - '_data/schedule/**' + - '_data/team/**' # people data feeds Tier-1 resolution + - 'schemas/**' + - 'tools/validate_schedule.py' + - '.github/workflows/schedule-validate.yml' + push: + branches: ['master'] + paths: + - '_data/schedule/**' + - '_data/team/**' + - 'schemas/**' + - 'tools/validate_schedule.py' + workflow_dispatch: + +permissions: + contents: read + +jobs: + validate: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: '3.11' + + - name: Install validator dependencies + run: python -m pip install --quiet pyyaml jsonschema + + - name: Validate every cohort schedule + run: | + shopt -s nullglob + files=(_data/schedule/*/*.yml) + if [ ${#files[@]} -eq 0 ]; then + echo "No cohort schedule files under _data/schedule// — nothing to validate." + exit 0 + fi + echo "Validating: ${files[*]}" + python3 tools/validate_schedule.py "${files[@]}" --github diff --git a/_data/schedule/example/sample.yml b/_data/schedule/example/sample.yml new file mode 100644 index 0000000..2ee77b0 --- /dev/null +++ b/_data/schedule/example/sample.yml @@ -0,0 +1,97 @@ +# Synthetic example cohort for the #54 schedule data loop. +# +# This is NOT a real clinic schedule. It documents the flat-data format and gives +# the validator (tools/validate_schedule.py) and renderer (_includes/schedule.html) +# something to exercise in CI. Real per-clinic schedules live in their own clinic +# repos; where a clinic's build reads that data from is the #56 topology decision. +# +# Instructor entries are either a role token (see _data/schedule/roles.yml) or a +# person key that resolves to _data/team/.yml. This example uses both. +clinic: example +year: 2099 +status: draft +title: Example Clinic 2099 +timezone: Africa/Johannesburg +display_timezones: + - Africa/Johannesburg + - America/New_York +tracks: + - main +locations: + main-hall: Main Hall + comp-lab: Comp. Lab +faculty: + - faculty + - pulliam +weeks: + - title: "Week 1: Foundations" + open: true + days: + - date: "2099-01-05" + label: "Monday, 5 Jan" + sessions: + - start: "09:00" + end: "09:15" + kind: organizing + track: main + title: "Welcome and orientation" + instructors: [organizers] + location: main-hall + - start: "09:15" + end: "10:30" + kind: lecture + track: main + title: "Introduction to dynamical models" + instructors: [pulliam] + location: main-hall + links: + - {text: "slides", url: "https://example.org/slides/intro"} + - start: "10:30" + end: "11:00" + kind: tea + - start: "11:00" + end: "12:30" + kind: computer-session + track: main + title: "Tutorial 1: R and the SIR model" + instructors: [tutors] + location: comp-lab + - start: "12:30" + end: "13:30" + kind: meal + meal: lunch + - start: "13:30" + end: "15:00" + kind: discussion + track: main + title: "Model assumptions and their consequences" + instructors: [faculty] + location: main-hall + # a faculty-only (shadow) row: present in the source, dropped from the + # rendered participant view by the renderer. + - start: "15:00" + end: "15:30" + kind: organizing + track: main + title: "Faculty debrief" + shadow: true + instructors: [faculty] + notes: + - text: "Bring a laptop with R installed." + - date: "2099-01-06" + label: "Tuesday, 6 Jan" + sessions: + - start: "09:00" + end: "10:30" + kind: lecture + track: main + title: "Stochastic models" + instructors: [pulliam] + location: main-hall + - start: "10:30" + end: "12:00" + kind: group-work + track: main + title: "Project scoping" + instructors: [mentors] + location: main-hall diff --git a/_data/schedule/roles.yml b/_data/schedule/roles.yml new file mode 100644 index 0000000..96b034d --- /dev/null +++ b/_data/schedule/roles.yml @@ -0,0 +1,19 @@ +# Shared role tokens for ICI3D clinic schedules (#54 data loop). +# +# An `instructors:` / `faculty:` entry in a schedule YAML is one of: +# - a person KEY that must resolve to _data/team/.yml (hard-checked, Tier 1a) +# - a ROLE token listed here (allowed, no person record) +# - an external object {name, url} (skipped by Tier 1) +# +# Role tokens are matched case-insensitively by tools/validate_schedule.py, so the +# schedule may write "Tutors" / "Mentors" / "everyone" and they resolve here. +# `tbd` is allowed but emits a non-gating WARNING (a placeholder still to be filled). +roles: + - everyone # the whole cohort (faculty + participants) + - all # synonym for everyone + - faculty # the assembled faculty, unspecified individuals + - mentors # project mentors as a group + - tutors # R/computer-session tutors as a group + - aims-tutors # AIMS tutors as a group + - organizers # the organizing team + - tbd # placeholder — allowed but warns diff --git a/_includes/schedule.html b/_includes/schedule.html new file mode 100644 index 0000000..ffadc21 --- /dev/null +++ b/_includes/schedule.html @@ -0,0 +1,95 @@ +{%- comment -%} + Shared, fixed schedule renderer for the #54 data loop. + + Turns one flat cohort data file (_data/schedule//.yml, validated by + tools/validate_schedule.py) into the rendered schedule. This include is NEVER edited + during a clinic; only the data file is. A bad data edit fails validation, it cannot + break the build, because no Liquid lives in the edited file. + + Usage from a clinic schedule page (layout: clinic): assign the cohort to a variable, + then include schedule.html with data= (the include tag cannot take a + bracket-subscripted value directly, so assign it first). + shadow:true sessions and notes are dropped here (participant view). A separate + faculty build would include them; see DESIGN.md / #56 for where that lives. +{%- endcomment -%} +{%- assign sched = include.data -%} +{%- for week in sched.weeks -%} +
+ {{ week.title }} + {%- for day in week.days -%} +
+

{{ day.label | default: day.date }}

+
    + {%- for s in day.sessions -%} + {%- unless s.shadow -%} + {%- case s.kind -%} + {%- when 'lecture' -%}{%- assign label = 'Lecture' -%} + {%- when 'discussion' -%}{%- assign label = 'Discussion' -%} + {%- when 'computer-session' -%}{%- assign label = 'Comp. Session' -%} + {%- when 'live-coding' -%}{%- assign label = 'Live coding' -%} + {%- when 'group-work' -%}{%- assign label = 'Group Work' -%} + {%- when 'organizing' -%}{%- assign label = 'Organizational' -%} + {%- when 'poster' -%}{%- assign label = 'Posters' -%} + {%- when 'reading' -%}{%- assign label = 'Reading' -%} + {%- when 'social' -%}{%- assign label = 'Social event' -%} + {%- when 'meal' -%}{%- assign label = s.meal | capitalize -%} + {%- when 'coffee' -%}{%- assign label = 'Coffee' -%} + {%- when 'tea' -%}{%- assign label = 'Tea' -%} + {%- when 'break' -%}{%- assign label = 'Free' -%} + {%- when 'todo' -%}{%- assign label = 'TODO' -%} + {%- else -%}{%- assign label = '' -%} + {%- endcase -%} +
  • + {%- if s.start -%}{{ s.start }}{% if s.end %}–{{ s.end }}{% endif %}{%- endif -%} + {%- if label != '' %} {{ label }}{% endif -%} + {%- comment -%} the title's own link (display text == title) is rendered AS the linked title, not repeated below {%- endcomment -%} + {%- assign primary_url = '' -%} + {%- for l in s.links -%}{%- if l.text == s.title and primary_url == '' -%}{%- assign primary_url = l.url -%}{%- endif -%}{%- endfor -%} + {%- if s.title -%} + {%- assign title_html = s.title | markdownify | remove: '

    ' | remove: '

    ' | strip -%} + {% if primary_url != '' %}{{ title_html }}{% else %}{{ title_html }}{% endif %} + {%- endif -%} + {%- comment -%} who: single-track instructor list {%- endcomment -%} + {%- if s.instructors and s.instructors.size > 0 -%} + ( + {%- for key in s.instructors -%} + {%- assign person = site.data.team[key] -%} + {%- if person -%}{{ person.name }}{%- else -%}{{ key }}{%- endif -%} + {%- unless forloop.last %}, {% endunless -%} + {%- endfor -%} + {%- if s.location %}, {{ sched.locations[s.location] | default: s.location }}{% endif -%}) + {%- elsif s.instructors_by_track -%} + ( + {%- for pair in s.instructors_by_track -%} + {{ pair[0] }}: + {%- for key in pair[1] -%} {% assign person = site.data.team[key] %}{% if person %}{{ person.name }}{% else %}{{ key }}{% endif %}{%- unless forloop.last %},{% endunless -%}{%- endfor -%} + {%- unless forloop.last %}; {% endunless -%} + {%- endfor -%}) + {%- elsif s.location -%} + ({{ sched.locations[s.location] | default: s.location }}) + {%- endif -%} + {%- comment -%} extra resource links (everything except the primary title link rendered above) {%- endcomment -%} + {%- assign n_extra = 0 -%} + {%- for l in s.links -%}{%- unless l.url == primary_url and l.text == s.title -%}{%- assign n_extra = n_extra | plus: 1 -%}{%- endunless -%}{%- endfor -%} + {%- if n_extra > 0 %} [{% for l in s.links %}{% unless l.url == primary_url and l.text == s.title %}{{ l.text | default: 'link' }}{% unless forloop.last %} {% endunless %}{% endunless %}{% endfor %}]{% endif -%} + {%- assign has_notes = false -%} + {%- for n in s.notes -%}{%- unless n.shadow %}{%- assign has_notes = true -%}{%- endunless -%}{%- endfor -%} + {%- if has_notes -%} +
      {%- for n in s.notes -%}{%- unless n.shadow -%}
    • {{ n.text | default: n | markdownify | remove: '

      ' | remove: '

      ' | strip }}
      • {%- endunless -%}{%- endfor -%}
    + {%- endif -%} +
    • + {%- endunless -%} + {%- endfor -%} +
+ {%- assign has_day_notes = false -%} + {%- for n in day.notes -%}{%- unless n.shadow %}{%- assign has_day_notes = true -%}{%- endunless -%}{%- endfor -%} + {%- if day.links.size > 0 or has_day_notes -%} +
    + {%- for l in day.links -%}
  • {{ l.text | default: 'link' }}{% if l.deadline %} (by {{ l.deadline }}){% endif %}
    • {%- endfor -%} + {%- for n in day.notes -%}{%- unless n.shadow -%}
  • {{ n.text | default: n | markdownify | remove: '

    ' | remove: '

    ' | strip }}
    • {%- endunless -%}{%- endfor -%} +
+ {%- endif -%} +
+ {%- endfor -%} +
+{%- endfor -%} diff --git a/schemas/schedule-cohort.schema.json b/schemas/schedule-cohort.schema.json new file mode 100644 index 0000000..a194ad9 --- /dev/null +++ b/schemas/schedule-cohort.schema.json @@ -0,0 +1,181 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://ici3d.org/schemas/schedule-cohort.schema.json", + "title": "ICI3D clinic schedule (one cohort-year)", + "type": "object", + "required": ["timezone", "tracks", "weeks"], + "additionalProperties": false, + "properties": { + "clinic": { "type": "string", "pattern": "^[a-z][a-z0-9-]*$" }, + "year": { "type": "integer", "minimum": 2000, "maximum": 2100 }, + "status": { "enum": ["draft", "published", "archive"], "default": "published" }, + "title": { "type": "string" }, + "location_name": { "type": "string" }, + "timezone": { "$ref": "#/$defs/ianaTz" }, + "display_timezones": { "type": "array", "minItems": 1, "uniqueItems": true, + "items": { "$ref": "#/$defs/ianaTz" } }, + "tracks": { "type": "array", "minItems": 1, "uniqueItems": true, + "items": { "type": "string", "minLength": 1 } }, + "faculty": { "type": "array", "uniqueItems": true, + "items": { "$ref": "#/$defs/personKey" } }, + "locations": { "type": "object", + "additionalProperties": { "type": "string", "minLength": 1 } }, + "meal_defaults": { "type": "object", + "additionalProperties": { + "type": "object", "additionalProperties": false, + "required": ["start", "end"], + "properties": { "start": { "$ref": "#/$defs/hhmm" }, + "end": { "$ref": "#/$defs/hhmm" } } } }, + "weeks": { "type": "array", "minItems": 1, + "items": { "$ref": "#/$defs/week" } } + }, + "$defs": { + "hhmm": { "type": "string", "pattern": "^([01][0-9]|2[0-3]):[0-5][0-9]$" }, + "isoDate": { "type": "string", "format": "date", + "pattern": "^[0-9]{4}-[0-9]{2}-[0-9]{2}$" }, + "ianaTz": { "type": "string", + "pattern": "^[A-Za-z][A-Za-z0-9_+\\-]*(/[A-Za-z0-9_+\\-]+)+$" }, + "personKey": { "type": "string", "pattern": "^[a-z][a-z0-9_-]*$" }, + "instructorKey": { "type": "string", "pattern": "^[A-Za-z][A-Za-z0-9_-]*$" }, + + "external": { + "type": "object", + "required": ["name"], + "additionalProperties": false, + "properties": { + "name": { "type": "string", "minLength": 1 }, + "url": { "type": "string", "minLength": 1 } + } + }, + + "instructor": { + "oneOf": [ + { "$ref": "#/$defs/instructorKey" }, + { "$ref": "#/$defs/external" } + ] + }, + + "link": { + "type": "object", + "required": ["url"], + "additionalProperties": false, + "properties": { + "text": { "type": "string", "minLength": 1 }, + "url": { "type": "string", "minLength": 1 }, + "deadline": { "$ref": "#/$defs/hhmm" } + } + }, + + "note": { + "oneOf": [ + { "type": "string", "minLength": 1 }, + { "type": "object", + "required": ["text"], + "additionalProperties": false, + "properties": { + "text": { "type": "string", "minLength": 1 }, + "shadow": { "type": "boolean", "default": false } + } } + ] + }, + + "week": { + "type": "object", + "required": ["days"], + "additionalProperties": false, + "properties": { + "title": { "type": "string" }, + "collapsible": { "type": "boolean", "default": false }, + "open": { "type": "boolean", "default": false }, + "days": { "type": "array", "minItems": 1, "items": { "$ref": "#/$defs/day" } } + } + }, + + "day": { + "type": "object", + "required": ["date"], + "additionalProperties": false, + "properties": { + "date": { "$ref": "#/$defs/isoDate" }, + "label": { "type": "string" }, + "sessions": { "type": "array", "items": { "$ref": "#/$defs/session" } }, + "links": { "type": "array", "items": { "$ref": "#/$defs/link" } }, + "notes": { "type": "array", "items": { "$ref": "#/$defs/note" } } + } + }, + + "session": { + "type": "object", + "required": ["kind"], + "additionalProperties": false, + "properties": { + "kind": { + "enum": ["lecture", "discussion", "computer-session", "live-coding", + "group-work", "organizing", "activity", "poster", "reading", + "social", "meal", "coffee", "tea", "break", + "note", "raw", "todo"] + }, + "start": { "$ref": "#/$defs/hhmm" }, + "end": { "$ref": "#/$defs/hhmm" }, + "track": { "type": "string", "minLength": 1 }, + "tracks": { "type": "array", "minItems": 1, "uniqueItems": true, + "items": { "type": "string", "minLength": 1 } }, + "instructors_by_track": { + "type": "object", + "additionalProperties": { + "type": "array", + "items": { "$ref": "#/$defs/instructor" } } }, + "choice": { "type": "string", "minLength": 1 }, + "title": { "type": "string", "minLength": 1 }, + "shadow": { "type": "boolean", "default": false }, + "guest": { "type": "boolean", "default": false }, + "speaker": { "type": "string", "minLength": 1 }, + "instructors": { "type": "array", "items": { "$ref": "#/$defs/instructor" } }, + "location": { "type": "string", "minLength": 1 }, + "links": { "type": "array", "items": { "$ref": "#/$defs/link" } }, + "notes": { "type": "array", "items": { "$ref": "#/$defs/note" } }, + "time_note": { "type": "string" }, + "meal": { "enum": ["breakfast", "lunch", "dinner"] }, + "text": { "type": "string" }, + "html": { "type": "string" }, + "source": { "type": "string" } + }, + "not": { "required": ["track", "tracks"] }, + "allOf": [ + { + "if": { "properties": { "kind": { "enum": + ["lecture", "discussion", "computer-session", "live-coding", + "group-work", "organizing", "activity", "poster", "reading"] } }, + "required": ["kind"] }, + "then": { "required": ["start", "end", "title"], + "anyOf": [ { "required": ["track"] }, { "required": ["tracks"] } ] } + }, + { + "if": { "properties": { "kind": { "enum": ["coffee", "tea", "break"] } }, + "required": ["kind"] }, + "then": { "required": ["start", "end"] } + }, + { + "if": { "properties": { "kind": { "const": "meal" } }, "required": ["kind"] }, + "then": { "required": ["start", "end", "meal"] } + }, + { + "if": { "properties": { "kind": { "const": "social" } }, "required": ["kind"] }, + "then": { "required": ["title"] } + }, + { + "if": { "properties": { "kind": { "const": "note" } }, "required": ["kind"] }, + "then": { "required": ["text"], "properties": { "shadow": { "const": true } } } + }, + { + "if": { "properties": { "kind": { "const": "raw" } }, "required": ["kind"] }, + "then": { "required": ["html"] } + }, + { + "if": { "properties": { "kind": { "const": "todo" } }, "required": ["kind"] }, + "then": { "required": ["source"] } + } + ] + } + } +} diff --git a/tools/import_schedule.py b/tools/import_schedule.py new file mode 100644 index 0000000..dd6a36d --- /dev/null +++ b/tools/import_schedule.py @@ -0,0 +1,338 @@ +#!/usr/bin/env python3 +"""One-shot migration: an MMED Liquid-wall schedule (schedule//index.md) -> cohort YAML (#54). + +This is a MIGRATION tool, run once per cohort, NOT the live renderer. Its contract is the one +the design rests on: it NEVER silently drops a line. Every source bullet becomes either a typed +session / day-level link / day-level note, or a `kind: todo` row carrying the original text verbatim. +A `todo` parses under the schema but HARD-FAILS Tier 1, so CI turns every line the importer could not +confidently map into a concrete, addressable error for a human to triage (vs. today's silent failures). + + python3 tools/import_schedule.py schedule/2025/index.md --year 2025 \ + --clinic mmed --status published -o _data/schedule/mmed/2025.yml + +Then validate the result: + python3 tools/validate_schedule.py _data/schedule/mmed/2025.yml +""" +from __future__ import annotations +import argparse +import re +import sys +from pathlib import Path + +import yaml + +# ---- {{ token }} vocabulary (from the assign preamble) -> closed kind enum ---- +KIND = { + "lect": ("lecture", {}), "glect": ("lecture", {"guest": True}), + "disc": ("discussion", {}), + "prac": ("computer-session", {}), "labex": ("computer-session", {}), + "labs": ("computer-session", {}), "rtut": ("computer-session", {}), + "ex": ("activity", {}), + "gw": ("group-work", {}), + "lc": ("live-coding", {}), "pc": ("live-coding", {}), + "sc": ("social", {}), + "org": ("organizing", {}), + "post": ("poster", {}), + "catch": ("activity", {}), "proj": ("activity", {}), +} +MEAL = {"bfast": "breakfast", "ssbfast": "breakfast", "lunch": "lunch", + "dinner": "dinner", "ssdinner": "dinner"} +LOGISTIC = {"coffee": "coffee", "tea": "tea", "break": "break"} +LOC = {"main": "main-hall", "lab": "comp-lab", "breakout": "group-breakouts", + "lobby": "lobby", "sections": "sections"} +LOCATIONS_LABEL = {"main-hall": "Main Hall", "comp-lab": "Comp. Lab", + "group-breakouts": "Group Breakouts", "lobby": "Lobby", + "sections": "Section 1 / Section 2"} +MONTHS = {m: i for i, m in enumerate( + ["january", "february", "march", "april", "may", "june", "july", "august", + "september", "october", "november", "december"], 1)} + +RE_SUMMARY = re.compile(r"]*>(.*?)", re.I) +RE_DETAILS_OPEN = re.compile(r"]*\bopen\b", re.I) +RE_DAY = re.compile(r"^#{2,3}\s*Day\s*([0-9A-Za-z]+)\s*(?:\(([^)]*)\))?\s*$", re.I) +RE_TIME = re.compile(r"^(\d{1,2})h(\d{2})\s*[-–]\s*(\d{1,2})h(\d{2})\s*(.*)$") +RE_TOKEN = re.compile(r"\{\{\s*(\w+)\s*\}\}") +RE_SHADOW = re.compile(r"\{:\s*\.shadow\s*\}") # kramdown faculty-only marker; tolerates inner spaces +RE_PEOPLE = re.compile(r'people\s*=\s*"([^"]+)"') +RE_LINK = re.compile(r"\[([^\]]+)\]\(([^)]+)\)") +RE_ANYLINK = re.compile(r"\[([^\]]+)\]\(([^)]*)\)") # also matches empty-url [text]() +RE_DATEPROSE = re.compile(r"(\d{1,2})\s+([A-Za-z]+)") + + +def hhmm(h, m): + return f"{int(h):02d}:{int(m):02d}" + + +def split_people(meta): + """Pull instructor strings out of a trailing meta group like + ({% include instructors people="a, b" %}, {{ main }}) or (everyone, {{ main }}).""" + people = [] + for grp in RE_PEOPLE.findall(meta): + # the source separates co-instructors with both "," and "|" + people += [x.strip() for x in re.split(r"[,|]", grp) if x.strip()] + if not people: + # bare names before the location token: strip tokens/includes, take leading words + bare = RE_TOKEN.sub("", meta) + bare = re.sub(r"\{%.*?%\}", "", bare) + bare = bare.strip().strip("()").strip() + for chunk in re.split(r"[,|]", bare): + c = chunk.strip() + if c and "include" not in c and "instructors" not in c: + people.append(c) + return people + + +def parse_meta_group(rest): + """Given the text after the time range, separate (title+links) from the trailing + ( ...people... {{ loc }} ) meta group. Returns (body_without_meta, location, people).""" + location = None + people = [] + # find the LAST parenthesised group that carries a location token, people=, or an instructors include + best = None + for mo in re.finditer(r"\(([^()]*(?:\([^()]*\)[^()]*)*)\)", rest): + inner = mo.group(1) + if RE_PEOPLE.search(inner) or "instructors" in inner or RE_TOKEN.search(inner) \ + or re.search(r"\b(everyone|all)\b", inner, re.I): + best = mo + if best: + inner = best.group(1) + toks = RE_TOKEN.findall(inner) + location = next((LOC[t] for t in toks if t in LOC), None) + people = split_people(inner) + rest = (rest[:best.start()] + rest[best.end():]).strip() + return rest, location, people + + +def parse_session(rest, source): + """Build a session dict from the text after a time range (kind/title/links/people/location). + + Returns (sess, title, links, location, people, extra, is_todo). An undefined macro + (a {{token}} that is not a known kind/meal/logistic/location, e.g. {{ mlect }}) is NOT + silently coerced to a generic activity; it becomes a loud `todo` for a human to map. + """ + toks = RE_TOKEN.findall(rest) + kind = subtype = None + extra = {} + for t in toks: + if t in KIND: + kind, extra = KIND[t][0], dict(KIND[t][1]) + break + if t in MEAL: + kind, subtype = "meal", MEAL[t] + break + if t in LOGISTIC: + kind = LOGISTIC[t] + break + unknown = [t for t in toks if t not in KIND and t not in MEAL and t not in LOGISTIC and t not in LOC] + if kind is None and unknown: + return {"kind": "todo", "source": source}, None, [], None, [], {}, True + + body, location, people = parse_meta_group(rest) + body = re.sub(r"\{%.*?%\}", "", body) + body = RE_TOKEN.sub("", body).strip() # drop kind/loc tokens left in the title region + + # links: keep only non-empty urls; tidy {url} when display text duplicates the url + links = [] + for (t, u) in RE_ANYLINK.findall(body): + if u: + links.append({"url": u} if t == u else {"text": t, "url": u}) + # title: inline each link's display text, drop urls and any leftover empty () + title = RE_ANYLINK.sub(lambda m: m.group(1), body) + title = re.sub(r"\(\s*\)", "", title) + title = re.sub(r"\s+", " ", title).strip(" -–·,") + title = title or None + + if kind is None: + kind = "activity" # timed but no recognised token -> generic (the 33 untyped 2025 rows) + + sess = {"kind": kind} + if subtype: + sess["meal"] = subtype + return sess, title, links, location, people, extra, False + + +def infer_dates(days, year): + """Fill day['date'] from the prose in the header parens; carry forward +1 when absent.""" + from datetime import date, timedelta + prev = None + for d in days: + iso = None + prose = d.pop("_dateprose", "") or "" + m = RE_DATEPROSE.search(prose) + if m and m.group(2).lower() in MONTHS: + iso = date(year, MONTHS[m.group(2).lower()], int(m.group(1))) + elif prev is not None: + iso = prev + timedelta(days=1) + if iso is not None: + d["date"] = iso.isoformat() + prev = iso + else: + d["date"] = f"{year}-01-01" # placeholder; Tier 0 still requires a real ISO date + d.setdefault("_needs_date", True) + + +def parse(path, year): + weeks = [] + cur_week = None + cur_day = None + last_session = None # most recent timed session, so indented sub-bullets attach to it + todos = 0 + + def new_week(title, is_open): + nonlocal cur_week, cur_day + cur_week = {"title": title, "collapsible": True, "open": bool(is_open), "days": []} + weeks.append(cur_week) + cur_day = None + + def ensure_week(): + if cur_week is None: + new_week("Schedule", False) + return cur_week + + for raw in Path(path).read_text().splitlines(): + line = raw.rstrip() + s = line.strip() + if not s: + continue + + msum = RE_SUMMARY.search(s) + if msum: + new_week(msum.group(1).strip(), bool(RE_DETAILS_OPEN.search(s))) + last_session = None + continue + mday = RE_DAY.match(s) + if mday: + ensure_week() + cur_day = {"label": ("Day " + mday.group(1) + (f" ({mday.group(2).strip()})" if mday.group(2) else "")), + "_dateprose": (mday.group(2) or ""), "sessions": []} + cur_week["days"].append(cur_day) + last_session = None + continue + + if not s.startswith("- "): + continue + indented = raw[:1] in (" ", "\t") + body = s[2:].strip() + shadow = bool(RE_SHADOW.search(body)) + body = RE_SHADOW.sub("", body).strip() + + # an indented sub-bullet is a note on the PRECEDING session, not a day item + if indented and last_session is not None and not RE_TIME.match(body): + if body: + last_session.setdefault("notes", []).append( + {"text": body, "shadow": True} if shadow else body) + continue + + # top-level untimed bullet under a day -> a day-scoped link or note (never dropped) + if cur_day is not None and not RE_TIME.match(body): + only_link = RE_LINK.fullmatch(body) + if only_link: + cur_day.setdefault("links", []).append( + {"text": only_link.group(1).strip(), "url": only_link.group(2).strip()}) + continue + cur_day.setdefault("notes", []).append( + {"text": body, "shadow": True} if shadow else body) + continue + + mt = RE_TIME.match(body) + if not mt: + # no day context and not a time row -> a TODO the human must place + ensure_week() + if cur_day is None: + cur_day = {"label": "Day ?", "_dateprose": "", "sessions": []} + cur_week["days"].append(cur_day) + cur_day["sessions"].append({"kind": "todo", "source": s}) + todos += 1 + continue + + start, end = hhmm(mt.group(1), mt.group(2)), hhmm(mt.group(3), mt.group(4)) + rest = mt.group(5).strip() + sess, title, links, location, people, extra, is_todo = parse_session(rest, s) + sess["start"], sess["end"] = start, end + if is_todo: + todos += 1 + ordered = {k: sess[k] for k in ["start", "end", "kind", "source"] if k in sess} + else: + if title: + sess["title"] = title + if "track" not in sess: + sess["track"] = "main" + if people: + sess["instructors"] = people + if location: + sess["location"] = location + if links: + sess["links"] = links + if shadow: + sess["shadow"] = True + sess.update(extra) + ordered = {k: sess[k] for k in ["start", "end", "kind", "meal", "track", "title", + "instructors", "location", "links", "notes", "shadow", "guest"] + if k in sess} + ensure_week() + if cur_day is None: + cur_day = {"label": "Day ?", "_dateprose": "", "sessions": []} + cur_week["days"].append(cur_day) + cur_day["sessions"].append(ordered) + last_session = ordered + + for w in weeks: + infer_dates(w["days"], year) + for d in w["days"]: + d.pop("_needs_date", None) + # order day keys: date, label, sessions, links, notes + for k in ("date", "label", "sessions", "links", "notes"): + if k in d: + d[k] = d.pop(k) + return weeks, todos + + +class _Dumper(yaml.SafeDumper): + pass + + +def _str_representer(dumper, data): + # force-quote scalars that YAML would otherwise coerce (dates, HH:MM times) + if re.fullmatch(r"\d{4}-\d{2}-\d{2}", data) or re.fullmatch(r"\d{2}:\d{2}", data): + return dumper.represent_scalar("tag:yaml.org,2002:str", data, style="'") + return dumper.represent_scalar("tag:yaml.org,2002:str", data) + + +_Dumper.add_representer(str, _str_representer) + + +def main(argv=None): + ap = argparse.ArgumentParser() + ap.add_argument("source", help="schedule//index.md (the Liquid wall)") + ap.add_argument("--year", type=int, required=True) + ap.add_argument("--clinic", default="mmed") + ap.add_argument("--status", default="published", choices=["draft", "published", "archive"]) + ap.add_argument("-o", "--out", required=True) + args = ap.parse_args(argv) + + weeks, todos = parse(args.source, args.year) + n_sess = sum(len(d.get("sessions", [])) for w in weeks for d in w["days"]) + doc = { + "clinic": args.clinic, + "year": args.year, + "status": args.status, + "title": f"MMED {args.year}", + "timezone": "Africa/Johannesburg", + "display_timezones": ["Africa/Johannesburg"], + "tracks": ["main", "Section 1", "Section 2"], + "locations": LOCATIONS_LABEL, + "meal_defaults": {"breakfast": {"start": "07:45", "end": "08:15"}, + "dinner": {"start": "18:00", "end": "18:30"}}, + "weeks": weeks, + } + Path(args.out).parent.mkdir(parents=True, exist_ok=True) + with open(args.out, "w") as fh: + yaml.dump(doc, fh, Dumper=_Dumper, sort_keys=False, default_flow_style=False, + allow_unicode=True, width=100) + print(f"wrote {args.out}: {len(weeks)} weeks, " + f"{sum(len(w['days']) for w in weeks)} days, {n_sess} sessions, {todos} TODO rows") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/validate_schedule.py b/tools/validate_schedule.py new file mode 100644 index 0000000..bb707d0 --- /dev/null +++ b/tools/validate_schedule.py @@ -0,0 +1,242 @@ +#!/usr/bin/env python3 +"""Validate ICI3D clinic schedule data files (#54 "flat data -> validate -> render" loop). + +Two gating tiers run here; Tier 2 (link-liveness) is a separate advisory job and is NOT run here. + + Tier 0 - schema: JSON Schema (schemas/schedule-cohort.schema.json). Required-fields-per-kind, + the closed `kind` enum, quoted "HH:MM" / "YYYY-MM-DD" strings (an unquoted + date becomes a YAML date object and fails the `type: string` assert here), + declared-track membership of the shape, etc. + Tier 1 - semantic: (a) every instructor/faculty key resolves to a person record or a role token; + (b) sessions do not overlap within a track (scoped: shadow + logistics/social + + same-`choice` alternatives + untimed rows are exempt); + (c) end >= start; (d) every session.track is a declared track; + (e) timezone + display_timezones are real IANA zones; + (f) no `kind: todo` survives in a non-archive cohort (the importer's loud TODO). + +Usage: + python3 tools/validate_schedule.py _data/schedule/mmed/2025.yml [more.yml ...] \ + [--schema schemas/schedule-cohort.schema.json] \ + [--people-dir _data/team] \ + [--roles _data/schedule/roles.yml] [--github] + +Exit status is non-zero if any GATING finding (error) is present. Warnings never gate. +""" +from __future__ import annotations +import argparse +import json +import sys +from collections import defaultdict +from datetime import date as _date +from difflib import get_close_matches +from pathlib import Path + +import yaml +from jsonschema import Draft202012Validator + +try: + from zoneinfo import available_timezones + _IANA = available_timezones() +except Exception: # pragma: no cover - zoneinfo always present on 3.9+ + _IANA = None + +# Kinds exempt from the within-track non-overlap check. The real schedules deliberately +# nest a shadow faculty meeting inside lunch, run a coffee break inside a long lab, etc. +NONOVERLAP_EXEMPT_KINDS = {"meal", "coffee", "tea", "break", "note", "social", "raw", "todo"} +_ALL_LANE = "\x00ALL\x00" # sentinel lane for untracked sessions + + +class Finding: + __slots__ = ("level", "where", "msg", "line") + + def __init__(self, level: str, where: str, msg: str, line: int | None = None): + self.level = level # "error" (gates) or "warning" (advisory) + self.where = where + self.msg = msg + self.line = line + + +def _hhmm_to_min(s): + try: + h, m = str(s).split(":") + return int(h) * 60 + int(m) + except Exception: + return None + + +def _iter_sessions(doc): + """Yield (session, locator) for every session in the document.""" + for wi, week in enumerate(doc.get("weeks") or []): + wt = (week or {}).get("title") or f"Week #{wi}" + for di, day in enumerate(week.get("days") or []): + dd = (day or {}).get("date") or (day or {}).get("label") or f"Day #{di}" + for si, sess in enumerate(day.get("sessions") or []): + if not isinstance(sess, dict): + continue + title = sess.get("title") or sess.get("meal") or sess.get("kind") or "?" + start = sess.get("start") or "--:--" + loc = f"{wt} / {dd} / {start} {sess.get('kind','?')} \"{title}\"" + yield sess, loc, day + + +def _collect_instructor_strings(sess): + """All string (key/role) instructor entries on a session; externals/dicts skipped.""" + out = [] + for item in (sess.get("instructors") or []): + if isinstance(item, str): + out.append(item) + for vals in (sess.get("instructors_by_track") or {}).values(): + for item in (vals or []): + if isinstance(item, str): + out.append(item) + return out + + +def validate_doc(path: Path, schema, people_keys, role_tokens) -> list[Finding]: + findings: list[Finding] = [] + raw = path.read_text() + try: + doc = yaml.safe_load(raw) + except yaml.YAMLError as e: + return [Finding("error", str(path), f"YAML did not parse: {e}")] + if not isinstance(doc, dict): + return [Finding("error", str(path), "top-level YAML is not a mapping")] + + status = doc.get("status", "published") + + # ---- Tier 0: schema ---- + for err in sorted(Draft202012Validator(schema).iter_errors(doc), key=lambda e: list(e.path)): + loc = "/".join(str(p) for p in err.path) or "" + findings.append(Finding("error", f"T0 {loc}", err.message)) + + # If the shape is badly broken, semantic checks would just add noise. + if any(f.level == "error" for f in findings): + # Still run the cheap top-level checks that don't depend on shape integrity. + pass + + declared_tracks = set(doc.get("tracks") or []) + + # ---- Tier 1e: timezone validity ---- + if _IANA is not None: + for tz in [doc.get("timezone")] + list(doc.get("display_timezones") or []): + if tz and tz not in _IANA: + findings.append(Finding("error", "T1e timezone", f"'{tz}' is not a known IANA timezone")) + + # ---- Tier 1a: top-level faculty roster resolution ---- + for key in (doc.get("faculty") or []): + if isinstance(key, str): + _resolve_person(key, "T1a faculty", people_keys, role_tokens, findings) + + # ---- per-session checks (+ gather per-day lanes for the non-overlap check) ---- + # Overlap only matters WITHIN a calendar day, so lanes are keyed per day, then per track. + day_lanes: dict = defaultdict(lambda: defaultdict(list)) # id(day) -> lane -> [(smin,emin,loc,choice)] + for sess, loc, day in _iter_sessions(doc): + kind = sess.get("kind") + smin, emin = _hhmm_to_min(sess.get("start")), _hhmm_to_min(sess.get("end")) + + # 1c: end >= start + if smin is not None and emin is not None and emin < smin: + findings.append(Finding("error", "T1c " + loc, f"end {sess.get('end')} is before start {sess.get('start')}")) + + # 1a: instructor resolution + for key in _collect_instructor_strings(sess): + _resolve_person(key, "T1a " + loc, people_keys, role_tokens, findings) + + # 1d: declared-track closure + sess_tracks = [] + if sess.get("track"): + sess_tracks.append(sess["track"]) + sess_tracks += list(sess.get("tracks") or []) + for t in sess_tracks: + if declared_tracks and t not in declared_tracks: + findings.append(Finding("error", "T1d " + loc, f"track '{t}' is not in the declared tracks {sorted(declared_tracks)}")) + + # 1f: TODO closure + if kind == "todo": + lvl = "error" if status != "archive" else "warning" + findings.append(Finding(lvl, "T1f " + loc, "unresolved importer TODO row: " + (sess.get("source") or "")[:160])) + + # 1b: gather lanes for non-overlap (skip exempt / shadow / untimed) + if kind in NONOVERLAP_EXEMPT_KINDS or sess.get("shadow") or smin is None or emin is None: + continue + for lk in (sess_tracks or [_ALL_LANE]): + day_lanes[id(day)][lk].append((smin, emin, loc, sess.get("choice"))) + + # ---- Tier 1b: non-overlap within each (day, track) lane ---- + # An _ALL_LANE (untracked) session conflicts with everything that day; fold it into every real lane. + for lanes in day_lanes.values(): + real_lanes = [lk for lk in lanes if lk != _ALL_LANE] + for lk in (real_lanes or [_ALL_LANE]): + items = list(lanes.get(lk, [])) + if lk != _ALL_LANE: + items += lanes.get(_ALL_LANE, []) + items.sort(key=lambda x: x[0]) + for i in range(len(items)): + for j in range(i + 1, len(items)): + a, b = items[i], items[j] + if b[0] >= a[1]: # sorted: no later item can overlap a + break + if a[3] is not None and a[3] == b[3]: # same choice => alternatives, not overlap + continue + lane_name = "all tracks" if lk == _ALL_LANE else f"track '{lk}'" + findings.append(Finding("error", "T1b " + a[2], f"overlaps ({lane_name}) with -> {b[2]}")) + + # dedupe identical findings (an _ALL_LANE pair can surface once per real lane) + seen, deduped = set(), [] + for f in findings: + sig = (f.level, f.where, f.msg) + if sig not in seen: + seen.add(sig) + deduped.append(f) + return deduped + + +def _resolve_person(key, where, people_keys, role_tokens, findings): + if key in people_keys: + return + low = key.lower() + if low in role_tokens: + if low == "tbd": + findings.append(Finding("warning", where, "instructor 'tbd' is a placeholder still to be filled")) + return + sugg = get_close_matches(low, people_keys, n=1) + hint = f" (did you mean '{sugg[0]}'?)" if sugg else "" + findings.append(Finding("error", where, f"instructor '{key}' resolves to neither a person (_data/team/{key}.yml) nor a role{hint}")) + + +def main(argv=None): + ap = argparse.ArgumentParser(description="Validate ICI3D clinic schedule data files (Tier 0 + Tier 1).") + ap.add_argument("files", nargs="+", help="schedule YAML file(s)") + here = Path(__file__).resolve().parent.parent + ap.add_argument("--schema", default=str(here / "schemas/schedule-cohort.schema.json")) + ap.add_argument("--people-dir", default=str(here / "_data/team")) + ap.add_argument("--roles", default=str(here / "_data/schedule/roles.yml")) + ap.add_argument("--github", action="store_true", help="also emit ::error/::warning GitHub annotations") + args = ap.parse_args(argv) + + schema = json.loads(Path(args.schema).read_text()) + people_keys = {p.stem for p in Path(args.people_dir).glob("*.yml") if p.stem != "template"} + roles_doc = yaml.safe_load(Path(args.roles).read_text()) or {} + role_tokens = {r.lower() for r in (roles_doc.get("roles") or [])} + + n_err = n_warn = 0 + for f in args.files: + p = Path(f) + findings = validate_doc(p, schema, people_keys, role_tokens) + errs = [x for x in findings if x.level == "error"] + warns = [x for x in findings if x.level == "warning"] + n_err += len(errs) + n_warn += len(warns) + status = "OK" if not errs else f"{len(errs)} ERROR(S)" + print(f"\n=== {f}: {status}{' + ' + str(len(warns)) + ' warning(s)' if warns else ''} ===") + for x in findings: + print(f" [{x.level.upper()}] {x.where}\n {x.msg}") + if args.github: + tag = "error" if x.level == "error" else "warning" + print(f"::{tag} file={f}::{x.where}: {x.msg}") + print(f"\nTOTAL: {n_err} error(s), {n_warn} warning(s) across {len(args.files)} file(s)") + return 1 if n_err else 0 + + +if __name__ == "__main__": + sys.exit(main()) From edcec949fcd80587c5fad33b561c960068f75bdb Mon Sep 17 00:00:00 2001 From: Weston Voglesonger Date: Mon, 13 Jul 2026 20:03:10 -0400 Subject: [PATCH 2/5] Package the validator as a composite action for clinics to pull in Wrap tools/validate_schedule.py in a composite action (.github/actions/validate-schedule) so each clinic repo can run the shared validator by checking out this repo as a submodule and calling the action -- the validation logic (and schema/roles/people-dir defaults, resolved relative to the validator's own location) stays maintained once here. This repo's own schedule-validate.yml now uses the action too, so the theme and every clinic run the identical check. Verified: the action's invocation matches _data/schedule/example/sample.yml and validates 0 errors / 0 warnings. --- .github/actions/validate-schedule/action.yml | 45 ++++++++++++++++++++ .github/workflows/schedule-validate.yml | 34 +++------------ 2 files changed, 52 insertions(+), 27 deletions(-) create mode 100644 .github/actions/validate-schedule/action.yml diff --git a/.github/actions/validate-schedule/action.yml b/.github/actions/validate-schedule/action.yml new file mode 100644 index 0000000..07cfc26 --- /dev/null +++ b/.github/actions/validate-schedule/action.yml @@ -0,0 +1,45 @@ +name: 'Validate ICI3D schedule data' +description: > + Runs the shared schedule validator (Tier 0 JSON Schema + Tier 1 referential: + instructor keys resolve, sessions non-overlapping within a track, end >= start) + on a clinic's schedule YAML. Designed to be called from a clinic repo that + vendors ICI3D.github.io as a submodule, so the validator, schema, role tokens, + and people data all come from one shared source. The clinic workflow just + checks out (with submodules) and calls this action. +inputs: + paths: + description: > + Glob of schedule data files to validate, relative to the calling repo's + root. Defaults to every cohort file under _data/schedule//. + required: false + default: '_data/schedule/*/*.yml' +runs: + using: 'composite' + steps: + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: '3.11' + + - name: Install validator dependencies + shell: bash + run: python -m pip install --quiet pyyaml jsonschema + + - name: Validate schedule data + shell: bash + env: + PATHS: ${{ inputs.paths }} + # validate_schedule.py lives two levels up from this action dir, in the + # shared repo's tools/. Its schema / roles / people-dir defaults are + # resolved relative to its own location, so they point at the shared + # repo's schemas/, _data/schedule/roles.yml, and _data/team. + VALIDATOR: ${{ github.action_path }}/../../../tools/validate_schedule.py + run: | + shopt -s nullglob + files=($PATHS) + if [ ${#files[@]} -eq 0 ]; then + echo "No schedule files matching '$PATHS' — nothing to validate." + exit 0 + fi + echo "Validating: ${files[*]}" + python3 "$VALIDATOR" "${files[@]}" --github diff --git a/.github/workflows/schedule-validate.yml b/.github/workflows/schedule-validate.yml index 4dc8a13..b9b4043 100644 --- a/.github/workflows/schedule-validate.yml +++ b/.github/workflows/schedule-validate.yml @@ -1,12 +1,8 @@ -# Gate schedule DATA edits (#54). +# Validate this repo's own schedule data (#54) using the shared composite action. # -# A Jekyll build stays green even when a schedule is wrong, because Liquid renders -# an undefined value or a broken reference as the empty string. This workflow runs -# tools/validate_schedule.py (Tier 0 schema + Tier 1 referential: people keys must -# resolve against _data/team, sessions non-overlapping within a track, end >= start) -# so a bad edit fails a check with a clear, annotated error instead of silently -# shipping a broken page. Make it a REQUIRED status check in branch protection to -# actually gate merges. +# The same action (.github/actions/validate-schedule) is what clinic repos call +# after checking out ICI3D.github.io as a submodule, so the validation logic is +# maintained once here and pulled in everywhere else. name: Validate schedule data on: @@ -16,6 +12,7 @@ on: - '_data/team/**' # people data feeds Tier-1 resolution - 'schemas/**' - 'tools/validate_schedule.py' + - '.github/actions/validate-schedule/**' - '.github/workflows/schedule-validate.yml' push: branches: ['master'] @@ -24,6 +21,7 @@ on: - '_data/team/**' - 'schemas/**' - 'tools/validate_schedule.py' + - '.github/actions/validate-schedule/**' workflow_dispatch: permissions: @@ -34,22 +32,4 @@ jobs: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - - - name: Set up Python - uses: actions/setup-python@v5 - with: - python-version: '3.11' - - - name: Install validator dependencies - run: python -m pip install --quiet pyyaml jsonschema - - - name: Validate every cohort schedule - run: | - shopt -s nullglob - files=(_data/schedule/*/*.yml) - if [ ${#files[@]} -eq 0 ]; then - echo "No cohort schedule files under _data/schedule// — nothing to validate." - exit 0 - fi - echo "Validating: ${files[*]}" - python3 tools/validate_schedule.py "${files[@]}" --github + - uses: ./.github/actions/validate-schedule From a413930c2ffaf3312b4ab054757611d4d9c5b0ef Mon Sep 17 00:00:00 2001 From: Weston Voglesonger Date: Fri, 17 Jul 2026 15:59:18 -0400 Subject: [PATCH 3/5] Add pre-commit hook for schedule validation; run same hook in CI Contributors can now run the schedule validator locally on every commit (pip install pre-commit && pre-commit install), catching schema and referential errors before pushing. CI runs the same hook via pre-commit/action, so local and CI validation share one source of truth (tools/validate_schedule.py). roles.yml is validator config, not a schedule document, so it is excluded from the matched files. The composite action is left in place for now: clinic repos still call it until they move to the distributed pre-commit hook (follow-up). --- .github/workflows/schedule-validate.yml | 20 +++++++++++++------- .pre-commit-config.yaml | 23 +++++++++++++++++++++++ 2 files changed, 36 insertions(+), 7 deletions(-) create mode 100644 .pre-commit-config.yaml diff --git a/.github/workflows/schedule-validate.yml b/.github/workflows/schedule-validate.yml index b9b4043..b7ca48b 100644 --- a/.github/workflows/schedule-validate.yml +++ b/.github/workflows/schedule-validate.yml @@ -1,8 +1,9 @@ -# Validate this repo's own schedule data (#54) using the shared composite action. +# Validate this repo's own schedule data (#54) via the shared pre-commit hook. # -# The same action (.github/actions/validate-schedule) is what clinic repos call -# after checking out ICI3D.github.io as a submodule, so the validation logic is -# maintained once here and pulled in everywhere else. +# The hook (.pre-commit-config.yaml -> tools/validate_schedule.py) is the SAME +# check contributors run locally with `pre-commit install`, so local and CI +# validation are one source of truth. Clinic repos run the equivalent check on +# their own schedule data (see DESIGN.md for the consumer setup). name: Validate schedule data on: @@ -12,7 +13,7 @@ on: - '_data/team/**' # people data feeds Tier-1 resolution - 'schemas/**' - 'tools/validate_schedule.py' - - '.github/actions/validate-schedule/**' + - '.pre-commit-config.yaml' - '.github/workflows/schedule-validate.yml' push: branches: ['master'] @@ -21,7 +22,7 @@ on: - '_data/team/**' - 'schemas/**' - 'tools/validate_schedule.py' - - '.github/actions/validate-schedule/**' + - '.pre-commit-config.yaml' workflow_dispatch: permissions: @@ -32,4 +33,9 @@ jobs: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - - uses: ./.github/actions/validate-schedule + - uses: actions/setup-python@v5 + with: + python-version: '3.11' + - uses: pre-commit/action@v3.0.1 + with: + extra_args: validate-schedule --all-files diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml new file mode 100644 index 0000000..a3b42f6 --- /dev/null +++ b/.pre-commit-config.yaml @@ -0,0 +1,23 @@ +# Local validation for ICI3D clinic schedule data (#54 "flat data -> validate -> render"). +# +# Contributors editing _data/schedule//.yml get the SAME check CI +# runs, on every commit: +# +# pip install pre-commit && pre-commit install +# +# CI (.github/workflows/schedule-validate.yml) runs this same hook, so local and +# CI validation share one source of truth: tools/validate_schedule.py. roles.yml +# is validator config (allowed role tokens), not a schedule document, so it is +# excluded from the matched files. +repos: + - repo: local + hooks: + - id: validate-schedule + name: Validate ICI3D schedule data (Tier 0 schema + Tier 1 references) + entry: python tools/validate_schedule.py + language: python + additional_dependencies: + - pyyaml + - jsonschema + files: ^_data/schedule/.*\.ya?ml$ + exclude: ^_data/schedule/roles\.ya?ml$ From 37c72621f0ddb06174ecd152b4a1fe2306086f3b Mon Sep 17 00:00:00 2001 From: Weston Voglesonger Date: Fri, 17 Jul 2026 16:14:17 -0400 Subject: [PATCH 4/5] Package the schedule validator so clinic repos consume it via pre-commit Moves the validator, JSON Schema, and role vocabulary into an installable ici3d_schedule package with a validate-schedule console script and a .pre-commit-hooks.yaml. Clinic repos can now pin this repo as a pre-commit hook and validate their schedule data against the shared, versioned schema without vendoring it (no submodule needed): repos: - repo: https://github.com/ICI3D/ICI3D.github.io rev: hooks: - id: validate-schedule Schema and roles ship as package data resolved relative to the module, so the same code works pip-installed (distributed hook) and from a repo checkout (this repo's own hook, CI, transitional composite action). --people-dir now defaults to _data/team in the working repo, matching the renderer's site.data.team lookup: instructor records are clinic-owned. tools/ validate_schedule.py is kept as a thin shim for backward compatibility. --- .github/workflows/schedule-validate.yml | 6 +- .pre-commit-hooks.yaml | 21 ++ ici3d_schedule/__init__.py | 6 + .../data}/roles.yml | 0 .../data}/schedule-cohort.schema.json | 0 ici3d_schedule/validate.py | 253 ++++++++++++++++++ pyproject.toml | 19 ++ tools/validate_schedule.py | 242 +---------------- 8 files changed, 311 insertions(+), 236 deletions(-) create mode 100644 .pre-commit-hooks.yaml create mode 100644 ici3d_schedule/__init__.py rename {_data/schedule => ici3d_schedule/data}/roles.yml (100%) rename {schemas => ici3d_schedule/data}/schedule-cohort.schema.json (100%) create mode 100644 ici3d_schedule/validate.py create mode 100644 pyproject.toml diff --git a/.github/workflows/schedule-validate.yml b/.github/workflows/schedule-validate.yml index b7ca48b..2a96524 100644 --- a/.github/workflows/schedule-validate.yml +++ b/.github/workflows/schedule-validate.yml @@ -11,8 +11,9 @@ on: paths: - '_data/schedule/**' - '_data/team/**' # people data feeds Tier-1 resolution - - 'schemas/**' + - 'ici3d_schedule/**' - 'tools/validate_schedule.py' + - 'pyproject.toml' - '.pre-commit-config.yaml' - '.github/workflows/schedule-validate.yml' push: @@ -20,8 +21,9 @@ on: paths: - '_data/schedule/**' - '_data/team/**' - - 'schemas/**' + - 'ici3d_schedule/**' - 'tools/validate_schedule.py' + - 'pyproject.toml' - '.pre-commit-config.yaml' workflow_dispatch: diff --git a/.pre-commit-hooks.yaml b/.pre-commit-hooks.yaml new file mode 100644 index 0000000..e9a5f78 --- /dev/null +++ b/.pre-commit-hooks.yaml @@ -0,0 +1,21 @@ +# Distributed pre-commit hook for ICI3D clinic repos (#54). A clinic validates its +# schedule data against the shared, versioned schema WITHOUT vendoring it, by adding +# to its own .pre-commit-config.yaml: +# +# repos: +# - repo: https://github.com/ICI3D/ICI3D.github.io +# rev: +# hooks: +# - id: validate-schedule +# +# pre-commit installs this package (schema + role tokens bundled) into an isolated +# environment and runs it on the clinic's _data/schedule/*.yml files. Instructor keys +# resolve against the clinic's own _data/team (the same records the renderer reads via +# site.data.team); override with `args: [--people-dir, some/other/dir]` if needed. +- id: validate-schedule + name: Validate ICI3D schedule data (Tier 0 schema + Tier 1 references) + description: Schema + referential validation for ICI3D clinic schedule YAML. + entry: validate-schedule + language: python + files: ^_data/schedule/.*\.ya?ml$ + exclude: ^_data/schedule/roles\.ya?ml$ diff --git a/ici3d_schedule/__init__.py b/ici3d_schedule/__init__.py new file mode 100644 index 0000000..4d19844 --- /dev/null +++ b/ici3d_schedule/__init__.py @@ -0,0 +1,6 @@ +"""ICI3D clinic schedule tooling (#54 "flat data -> validate -> render" loop). + +Ships the schedule JSON Schema and the allowed role-token vocabulary as package +data (ici3d_schedule/data/) and exposes the validator behind the `validate-schedule` +console script, which the distributed pre-commit hook (.pre-commit-hooks.yaml) runs. +""" diff --git a/_data/schedule/roles.yml b/ici3d_schedule/data/roles.yml similarity index 100% rename from _data/schedule/roles.yml rename to ici3d_schedule/data/roles.yml diff --git a/schemas/schedule-cohort.schema.json b/ici3d_schedule/data/schedule-cohort.schema.json similarity index 100% rename from schemas/schedule-cohort.schema.json rename to ici3d_schedule/data/schedule-cohort.schema.json diff --git a/ici3d_schedule/validate.py b/ici3d_schedule/validate.py new file mode 100644 index 0000000..915cdfb --- /dev/null +++ b/ici3d_schedule/validate.py @@ -0,0 +1,253 @@ +#!/usr/bin/env python3 +"""Validate ICI3D clinic schedule data files (#54 "flat data -> validate -> render" loop). + +Two gating tiers run here; Tier 2 (link-liveness) is a separate advisory job and is NOT run here. + + Tier 0 - schema: JSON Schema (ici3d_schedule/data/schedule-cohort.schema.json). Required-fields-per-kind, + the closed `kind` enum, quoted "HH:MM" / "YYYY-MM-DD" strings (an unquoted + date becomes a YAML date object and fails the `type: string` assert here), + declared-track membership of the shape, etc. + Tier 1 - semantic: (a) every instructor/faculty key resolves to a person record or a role token; + (b) sessions do not overlap within a track (scoped: shadow + logistics/social + + same-`choice` alternatives + untimed rows are exempt); + (c) end >= start; (d) every session.track is a declared track; + (e) timezone + display_timezones are real IANA zones; + (f) no `kind: todo` survives in a non-archive cohort (the importer's loud TODO). + +Usage (installed console script, or via the repo shim tools/validate_schedule.py): + validate-schedule _data/schedule/mmed/2025.yml [more.yml ...] \ + [--schema ...] [--roles ...] (both default to this package's bundled copies) \ + [--people-dir _data/team] (defaults to _data/team in the current repo) \ + [--github] + +The JSON Schema and the role-token list ship *inside this package* (ici3d_schedule/ +data/), so a clinic repo that pins the pre-commit hook validates against the shared, +versioned schema without vendoring it. People records (--people-dir) are clinic-owned +and resolved from the working repo, matching the renderer's site.data.team lookup. + +Exit status is non-zero if any GATING finding (error) is present. Warnings never gate. +""" +from __future__ import annotations +import argparse +import json +import sys +from collections import defaultdict +from datetime import date as _date +from difflib import get_close_matches +from pathlib import Path + +import yaml +from jsonschema import Draft202012Validator + +# Schema + role tokens ship as package data (ici3d_schedule/data/), one dir up-tree +# from this module. This path resolves identically whether the package is pip-installed +# (site-packages/ici3d_schedule/data) or run from a repo checkout, so the distributed +# pre-commit hook and the in-repo script share one source of truth. +_PKG_DATA = Path(__file__).resolve().parent / "data" + +try: + from zoneinfo import available_timezones + _IANA = available_timezones() +except Exception: # pragma: no cover - zoneinfo always present on 3.9+ + _IANA = None + +# Kinds exempt from the within-track non-overlap check. The real schedules deliberately +# nest a shadow faculty meeting inside lunch, run a coffee break inside a long lab, etc. +NONOVERLAP_EXEMPT_KINDS = {"meal", "coffee", "tea", "break", "note", "social", "raw", "todo"} +_ALL_LANE = "\x00ALL\x00" # sentinel lane for untracked sessions + + +class Finding: + __slots__ = ("level", "where", "msg", "line") + + def __init__(self, level: str, where: str, msg: str, line: int | None = None): + self.level = level # "error" (gates) or "warning" (advisory) + self.where = where + self.msg = msg + self.line = line + + +def _hhmm_to_min(s): + try: + h, m = str(s).split(":") + return int(h) * 60 + int(m) + except Exception: + return None + + +def _iter_sessions(doc): + """Yield (session, locator) for every session in the document.""" + for wi, week in enumerate(doc.get("weeks") or []): + wt = (week or {}).get("title") or f"Week #{wi}" + for di, day in enumerate(week.get("days") or []): + dd = (day or {}).get("date") or (day or {}).get("label") or f"Day #{di}" + for si, sess in enumerate(day.get("sessions") or []): + if not isinstance(sess, dict): + continue + title = sess.get("title") or sess.get("meal") or sess.get("kind") or "?" + start = sess.get("start") or "--:--" + loc = f"{wt} / {dd} / {start} {sess.get('kind','?')} \"{title}\"" + yield sess, loc, day + + +def _collect_instructor_strings(sess): + """All string (key/role) instructor entries on a session; externals/dicts skipped.""" + out = [] + for item in (sess.get("instructors") or []): + if isinstance(item, str): + out.append(item) + for vals in (sess.get("instructors_by_track") or {}).values(): + for item in (vals or []): + if isinstance(item, str): + out.append(item) + return out + + +def validate_doc(path: Path, schema, people_keys, role_tokens) -> list[Finding]: + findings: list[Finding] = [] + raw = path.read_text() + try: + doc = yaml.safe_load(raw) + except yaml.YAMLError as e: + return [Finding("error", str(path), f"YAML did not parse: {e}")] + if not isinstance(doc, dict): + return [Finding("error", str(path), "top-level YAML is not a mapping")] + + status = doc.get("status", "published") + + # ---- Tier 0: schema ---- + for err in sorted(Draft202012Validator(schema).iter_errors(doc), key=lambda e: list(e.path)): + loc = "/".join(str(p) for p in err.path) or "" + findings.append(Finding("error", f"T0 {loc}", err.message)) + + # If the shape is badly broken, semantic checks would just add noise. + if any(f.level == "error" for f in findings): + # Still run the cheap top-level checks that don't depend on shape integrity. + pass + + declared_tracks = set(doc.get("tracks") or []) + + # ---- Tier 1e: timezone validity ---- + if _IANA is not None: + for tz in [doc.get("timezone")] + list(doc.get("display_timezones") or []): + if tz and tz not in _IANA: + findings.append(Finding("error", "T1e timezone", f"'{tz}' is not a known IANA timezone")) + + # ---- Tier 1a: top-level faculty roster resolution ---- + for key in (doc.get("faculty") or []): + if isinstance(key, str): + _resolve_person(key, "T1a faculty", people_keys, role_tokens, findings) + + # ---- per-session checks (+ gather per-day lanes for the non-overlap check) ---- + # Overlap only matters WITHIN a calendar day, so lanes are keyed per day, then per track. + day_lanes: dict = defaultdict(lambda: defaultdict(list)) # id(day) -> lane -> [(smin,emin,loc,choice)] + for sess, loc, day in _iter_sessions(doc): + kind = sess.get("kind") + smin, emin = _hhmm_to_min(sess.get("start")), _hhmm_to_min(sess.get("end")) + + # 1c: end >= start + if smin is not None and emin is not None and emin < smin: + findings.append(Finding("error", "T1c " + loc, f"end {sess.get('end')} is before start {sess.get('start')}")) + + # 1a: instructor resolution + for key in _collect_instructor_strings(sess): + _resolve_person(key, "T1a " + loc, people_keys, role_tokens, findings) + + # 1d: declared-track closure + sess_tracks = [] + if sess.get("track"): + sess_tracks.append(sess["track"]) + sess_tracks += list(sess.get("tracks") or []) + for t in sess_tracks: + if declared_tracks and t not in declared_tracks: + findings.append(Finding("error", "T1d " + loc, f"track '{t}' is not in the declared tracks {sorted(declared_tracks)}")) + + # 1f: TODO closure + if kind == "todo": + lvl = "error" if status != "archive" else "warning" + findings.append(Finding(lvl, "T1f " + loc, "unresolved importer TODO row: " + (sess.get("source") or "")[:160])) + + # 1b: gather lanes for non-overlap (skip exempt / shadow / untimed) + if kind in NONOVERLAP_EXEMPT_KINDS or sess.get("shadow") or smin is None or emin is None: + continue + for lk in (sess_tracks or [_ALL_LANE]): + day_lanes[id(day)][lk].append((smin, emin, loc, sess.get("choice"))) + + # ---- Tier 1b: non-overlap within each (day, track) lane ---- + # An _ALL_LANE (untracked) session conflicts with everything that day; fold it into every real lane. + for lanes in day_lanes.values(): + real_lanes = [lk for lk in lanes if lk != _ALL_LANE] + for lk in (real_lanes or [_ALL_LANE]): + items = list(lanes.get(lk, [])) + if lk != _ALL_LANE: + items += lanes.get(_ALL_LANE, []) + items.sort(key=lambda x: x[0]) + for i in range(len(items)): + for j in range(i + 1, len(items)): + a, b = items[i], items[j] + if b[0] >= a[1]: # sorted: no later item can overlap a + break + if a[3] is not None and a[3] == b[3]: # same choice => alternatives, not overlap + continue + lane_name = "all tracks" if lk == _ALL_LANE else f"track '{lk}'" + findings.append(Finding("error", "T1b " + a[2], f"overlaps ({lane_name}) with -> {b[2]}")) + + # dedupe identical findings (an _ALL_LANE pair can surface once per real lane) + seen, deduped = set(), [] + for f in findings: + sig = (f.level, f.where, f.msg) + if sig not in seen: + seen.add(sig) + deduped.append(f) + return deduped + + +def _resolve_person(key, where, people_keys, role_tokens, findings): + if key in people_keys: + return + low = key.lower() + if low in role_tokens: + if low == "tbd": + findings.append(Finding("warning", where, "instructor 'tbd' is a placeholder still to be filled")) + return + sugg = get_close_matches(low, people_keys, n=1) + hint = f" (did you mean '{sugg[0]}'?)" if sugg else "" + findings.append(Finding("error", where, f"instructor '{key}' resolves to neither a person (_data/team/{key}.yml) nor a role{hint}")) + + +def main(argv=None): + ap = argparse.ArgumentParser(description="Validate ICI3D clinic schedule data files (Tier 0 + Tier 1).") + ap.add_argument("files", nargs="+", help="schedule YAML file(s)") + ap.add_argument("--schema", default=str(_PKG_DATA / "schedule-cohort.schema.json")) + ap.add_argument("--people-dir", default="_data/team", + help="clinic-owned people records (default: _data/team in the current repo)") + ap.add_argument("--roles", default=str(_PKG_DATA / "roles.yml")) + ap.add_argument("--github", action="store_true", help="also emit ::error/::warning GitHub annotations") + args = ap.parse_args(argv) + + schema = json.loads(Path(args.schema).read_text()) + people_keys = {p.stem for p in Path(args.people_dir).glob("*.yml") if p.stem != "template"} + roles_doc = yaml.safe_load(Path(args.roles).read_text()) or {} + role_tokens = {r.lower() for r in (roles_doc.get("roles") or [])} + + n_err = n_warn = 0 + for f in args.files: + p = Path(f) + findings = validate_doc(p, schema, people_keys, role_tokens) + errs = [x for x in findings if x.level == "error"] + warns = [x for x in findings if x.level == "warning"] + n_err += len(errs) + n_warn += len(warns) + status = "OK" if not errs else f"{len(errs)} ERROR(S)" + print(f"\n=== {f}: {status}{' + ' + str(len(warns)) + ' warning(s)' if warns else ''} ===") + for x in findings: + print(f" [{x.level.upper()}] {x.where}\n {x.msg}") + if args.github: + tag = "error" if x.level == "error" else "warning" + print(f"::{tag} file={f}::{x.where}: {x.msg}") + print(f"\nTOTAL: {n_err} error(s), {n_warn} warning(s) across {len(args.files)} file(s)") + return 1 if n_err else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/pyproject.toml b/pyproject.toml new file mode 100644 index 0000000..4031e3d --- /dev/null +++ b/pyproject.toml @@ -0,0 +1,19 @@ +[build-system] +requires = ["setuptools>=61"] +build-backend = "setuptools.build_meta" + +[project] +name = "ici3d-schedule" +version = "0.1.0" +description = "Validator, JSON Schema, and role vocabulary for ICI3D clinic schedule data (#54 flat-data loop)." +requires-python = ">=3.9" +dependencies = ["pyyaml", "jsonschema>=4.18"] + +[project.scripts] +validate-schedule = "ici3d_schedule.validate:main" + +[tool.setuptools] +packages = ["ici3d_schedule"] + +[tool.setuptools.package-data] +ici3d_schedule = ["data/*.json", "data/*.yml"] diff --git a/tools/validate_schedule.py b/tools/validate_schedule.py index bb707d0..fcc71e6 100644 --- a/tools/validate_schedule.py +++ b/tools/validate_schedule.py @@ -1,242 +1,16 @@ #!/usr/bin/env python3 -"""Validate ICI3D clinic schedule data files (#54 "flat data -> validate -> render" loop). +"""Backward-compatible shim: the validator now lives in the ici3d_schedule package. -Two gating tiers run here; Tier 2 (link-liveness) is a separate advisory job and is NOT run here. - - Tier 0 - schema: JSON Schema (schemas/schedule-cohort.schema.json). Required-fields-per-kind, - the closed `kind` enum, quoted "HH:MM" / "YYYY-MM-DD" strings (an unquoted - date becomes a YAML date object and fails the `type: string` assert here), - declared-track membership of the shape, etc. - Tier 1 - semantic: (a) every instructor/faculty key resolves to a person record or a role token; - (b) sessions do not overlap within a track (scoped: shadow + logistics/social - + same-`choice` alternatives + untimed rows are exempt); - (c) end >= start; (d) every session.track is a declared track; - (e) timezone + display_timezones are real IANA zones; - (f) no `kind: todo` survives in a non-archive cohort (the importer's loud TODO). - -Usage: - python3 tools/validate_schedule.py _data/schedule/mmed/2025.yml [more.yml ...] \ - [--schema schemas/schedule-cohort.schema.json] \ - [--people-dir _data/team] \ - [--roles _data/schedule/roles.yml] [--github] - -Exit status is non-zero if any GATING finding (error) is present. Warnings never gate. +Kept so `python3 tools/validate_schedule.py ...` keeps working for this repo's own +pre-commit hook, CI, and the (transitional) composite action, while clinic repos run +the same logic via the pip-installed `validate-schedule` console script. Schema and +role tokens are bundled in the package, so both invocation paths share one source. """ -from __future__ import annotations -import argparse -import json import sys -from collections import defaultdict -from datetime import date as _date -from difflib import get_close_matches from pathlib import Path -import yaml -from jsonschema import Draft202012Validator - -try: - from zoneinfo import available_timezones - _IANA = available_timezones() -except Exception: # pragma: no cover - zoneinfo always present on 3.9+ - _IANA = None - -# Kinds exempt from the within-track non-overlap check. The real schedules deliberately -# nest a shadow faculty meeting inside lunch, run a coffee break inside a long lab, etc. -NONOVERLAP_EXEMPT_KINDS = {"meal", "coffee", "tea", "break", "note", "social", "raw", "todo"} -_ALL_LANE = "\x00ALL\x00" # sentinel lane for untracked sessions - - -class Finding: - __slots__ = ("level", "where", "msg", "line") - - def __init__(self, level: str, where: str, msg: str, line: int | None = None): - self.level = level # "error" (gates) or "warning" (advisory) - self.where = where - self.msg = msg - self.line = line - - -def _hhmm_to_min(s): - try: - h, m = str(s).split(":") - return int(h) * 60 + int(m) - except Exception: - return None - - -def _iter_sessions(doc): - """Yield (session, locator) for every session in the document.""" - for wi, week in enumerate(doc.get("weeks") or []): - wt = (week or {}).get("title") or f"Week #{wi}" - for di, day in enumerate(week.get("days") or []): - dd = (day or {}).get("date") or (day or {}).get("label") or f"Day #{di}" - for si, sess in enumerate(day.get("sessions") or []): - if not isinstance(sess, dict): - continue - title = sess.get("title") or sess.get("meal") or sess.get("kind") or "?" - start = sess.get("start") or "--:--" - loc = f"{wt} / {dd} / {start} {sess.get('kind','?')} \"{title}\"" - yield sess, loc, day - - -def _collect_instructor_strings(sess): - """All string (key/role) instructor entries on a session; externals/dicts skipped.""" - out = [] - for item in (sess.get("instructors") or []): - if isinstance(item, str): - out.append(item) - for vals in (sess.get("instructors_by_track") or {}).values(): - for item in (vals or []): - if isinstance(item, str): - out.append(item) - return out - - -def validate_doc(path: Path, schema, people_keys, role_tokens) -> list[Finding]: - findings: list[Finding] = [] - raw = path.read_text() - try: - doc = yaml.safe_load(raw) - except yaml.YAMLError as e: - return [Finding("error", str(path), f"YAML did not parse: {e}")] - if not isinstance(doc, dict): - return [Finding("error", str(path), "top-level YAML is not a mapping")] - - status = doc.get("status", "published") - - # ---- Tier 0: schema ---- - for err in sorted(Draft202012Validator(schema).iter_errors(doc), key=lambda e: list(e.path)): - loc = "/".join(str(p) for p in err.path) or "" - findings.append(Finding("error", f"T0 {loc}", err.message)) - - # If the shape is badly broken, semantic checks would just add noise. - if any(f.level == "error" for f in findings): - # Still run the cheap top-level checks that don't depend on shape integrity. - pass - - declared_tracks = set(doc.get("tracks") or []) - - # ---- Tier 1e: timezone validity ---- - if _IANA is not None: - for tz in [doc.get("timezone")] + list(doc.get("display_timezones") or []): - if tz and tz not in _IANA: - findings.append(Finding("error", "T1e timezone", f"'{tz}' is not a known IANA timezone")) - - # ---- Tier 1a: top-level faculty roster resolution ---- - for key in (doc.get("faculty") or []): - if isinstance(key, str): - _resolve_person(key, "T1a faculty", people_keys, role_tokens, findings) - - # ---- per-session checks (+ gather per-day lanes for the non-overlap check) ---- - # Overlap only matters WITHIN a calendar day, so lanes are keyed per day, then per track. - day_lanes: dict = defaultdict(lambda: defaultdict(list)) # id(day) -> lane -> [(smin,emin,loc,choice)] - for sess, loc, day in _iter_sessions(doc): - kind = sess.get("kind") - smin, emin = _hhmm_to_min(sess.get("start")), _hhmm_to_min(sess.get("end")) - - # 1c: end >= start - if smin is not None and emin is not None and emin < smin: - findings.append(Finding("error", "T1c " + loc, f"end {sess.get('end')} is before start {sess.get('start')}")) - - # 1a: instructor resolution - for key in _collect_instructor_strings(sess): - _resolve_person(key, "T1a " + loc, people_keys, role_tokens, findings) - - # 1d: declared-track closure - sess_tracks = [] - if sess.get("track"): - sess_tracks.append(sess["track"]) - sess_tracks += list(sess.get("tracks") or []) - for t in sess_tracks: - if declared_tracks and t not in declared_tracks: - findings.append(Finding("error", "T1d " + loc, f"track '{t}' is not in the declared tracks {sorted(declared_tracks)}")) - - # 1f: TODO closure - if kind == "todo": - lvl = "error" if status != "archive" else "warning" - findings.append(Finding(lvl, "T1f " + loc, "unresolved importer TODO row: " + (sess.get("source") or "")[:160])) - - # 1b: gather lanes for non-overlap (skip exempt / shadow / untimed) - if kind in NONOVERLAP_EXEMPT_KINDS or sess.get("shadow") or smin is None or emin is None: - continue - for lk in (sess_tracks or [_ALL_LANE]): - day_lanes[id(day)][lk].append((smin, emin, loc, sess.get("choice"))) - - # ---- Tier 1b: non-overlap within each (day, track) lane ---- - # An _ALL_LANE (untracked) session conflicts with everything that day; fold it into every real lane. - for lanes in day_lanes.values(): - real_lanes = [lk for lk in lanes if lk != _ALL_LANE] - for lk in (real_lanes or [_ALL_LANE]): - items = list(lanes.get(lk, [])) - if lk != _ALL_LANE: - items += lanes.get(_ALL_LANE, []) - items.sort(key=lambda x: x[0]) - for i in range(len(items)): - for j in range(i + 1, len(items)): - a, b = items[i], items[j] - if b[0] >= a[1]: # sorted: no later item can overlap a - break - if a[3] is not None and a[3] == b[3]: # same choice => alternatives, not overlap - continue - lane_name = "all tracks" if lk == _ALL_LANE else f"track '{lk}'" - findings.append(Finding("error", "T1b " + a[2], f"overlaps ({lane_name}) with -> {b[2]}")) - - # dedupe identical findings (an _ALL_LANE pair can surface once per real lane) - seen, deduped = set(), [] - for f in findings: - sig = (f.level, f.where, f.msg) - if sig not in seen: - seen.add(sig) - deduped.append(f) - return deduped - - -def _resolve_person(key, where, people_keys, role_tokens, findings): - if key in people_keys: - return - low = key.lower() - if low in role_tokens: - if low == "tbd": - findings.append(Finding("warning", where, "instructor 'tbd' is a placeholder still to be filled")) - return - sugg = get_close_matches(low, people_keys, n=1) - hint = f" (did you mean '{sugg[0]}'?)" if sugg else "" - findings.append(Finding("error", where, f"instructor '{key}' resolves to neither a person (_data/team/{key}.yml) nor a role{hint}")) - - -def main(argv=None): - ap = argparse.ArgumentParser(description="Validate ICI3D clinic schedule data files (Tier 0 + Tier 1).") - ap.add_argument("files", nargs="+", help="schedule YAML file(s)") - here = Path(__file__).resolve().parent.parent - ap.add_argument("--schema", default=str(here / "schemas/schedule-cohort.schema.json")) - ap.add_argument("--people-dir", default=str(here / "_data/team")) - ap.add_argument("--roles", default=str(here / "_data/schedule/roles.yml")) - ap.add_argument("--github", action="store_true", help="also emit ::error/::warning GitHub annotations") - args = ap.parse_args(argv) - - schema = json.loads(Path(args.schema).read_text()) - people_keys = {p.stem for p in Path(args.people_dir).glob("*.yml") if p.stem != "template"} - roles_doc = yaml.safe_load(Path(args.roles).read_text()) or {} - role_tokens = {r.lower() for r in (roles_doc.get("roles") or [])} - - n_err = n_warn = 0 - for f in args.files: - p = Path(f) - findings = validate_doc(p, schema, people_keys, role_tokens) - errs = [x for x in findings if x.level == "error"] - warns = [x for x in findings if x.level == "warning"] - n_err += len(errs) - n_warn += len(warns) - status = "OK" if not errs else f"{len(errs)} ERROR(S)" - print(f"\n=== {f}: {status}{' + ' + str(len(warns)) + ' warning(s)' if warns else ''} ===") - for x in findings: - print(f" [{x.level.upper()}] {x.where}\n {x.msg}") - if args.github: - tag = "error" if x.level == "error" else "warning" - print(f"::{tag} file={f}::{x.where}: {x.msg}") - print(f"\nTOTAL: {n_err} error(s), {n_warn} warning(s) across {len(args.files)} file(s)") - return 1 if n_err else 0 - +sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) +from ici3d_schedule.validate import main # noqa: E402 if __name__ == "__main__": - sys.exit(main()) + raise SystemExit(main()) From 271a817244057f0b90fb64a2724e645b20827099 Mon Sep 17 00:00:00 2001 From: Weston Voglesonger Date: Fri, 17 Jul 2026 16:15:37 -0400 Subject: [PATCH 5/5] Document the Path A consumer setup (remote_theme + pre-commit) Adds ici3d_schedule/README.md describing the two-channel split (Jekyll remote_theme for rendering, this pre-commit hook for validation) and the clinic-owned _data/team model, and fixes the workflow comment to point at it instead of a DESIGN.md that does not exist. --- .github/workflows/schedule-validate.yml | 2 +- ici3d_schedule/README.md | 48 +++++++++++++++++++++++++ 2 files changed, 49 insertions(+), 1 deletion(-) create mode 100644 ici3d_schedule/README.md diff --git a/.github/workflows/schedule-validate.yml b/.github/workflows/schedule-validate.yml index 2a96524..b4ef460 100644 --- a/.github/workflows/schedule-validate.yml +++ b/.github/workflows/schedule-validate.yml @@ -3,7 +3,7 @@ # The hook (.pre-commit-config.yaml -> tools/validate_schedule.py) is the SAME # check contributors run locally with `pre-commit install`, so local and CI # validation are one source of truth. Clinic repos run the equivalent check on -# their own schedule data (see DESIGN.md for the consumer setup). +# their own schedule data via the distributed pre-commit hook (see ici3d_schedule/README.md). name: Validate schedule data on: diff --git a/ici3d_schedule/README.md b/ici3d_schedule/README.md new file mode 100644 index 0000000..201bc59 --- /dev/null +++ b/ici3d_schedule/README.md @@ -0,0 +1,48 @@ +# ici3d_schedule — schedule validation for ICI3D clinics (#54) + +The **flat-data schedule loop**: a clinic edits one YAML file per cohort +(`_data/schedule//.yml`), a validator gates it (schema + referential +checks), and a fixed Liquid include renders it. Editing data can never break the +build, because no Liquid lives in the edited file — a bad edit fails validation instead. + +This package ships the **validator**, the **JSON Schema**, and the allowed **role +vocabulary**, so any clinic repo validates against the shared, versioned schema +without vendoring it. + +## Two independent channels (Path A) + +Rendering and validation are distributed separately, both from this repo: + +| Concern | Mechanism | What the clinic adds | +|---|---|---| +| Render `schedule.html` | Jekyll `remote_theme` | `remote_theme: ICI3D/ICI3D.github.io` + `jekyll-remote-theme` plugin | +| Validate schedule data | this pre-commit hook | a `.pre-commit-config.yaml` entry (below) | + +A Jekyll theme distributes `_includes`/`_layouts`/`_sass`/`assets` only — not `_data` +and not this Python validator. So schedule **data** and **people records** +(`_data/team`, which the renderer reads as `site.data.team`) are **clinic-owned**, and +the validator resolves instructors against the clinic's own `_data/team`. + +## Local validation (contributors) + + pip install pre-commit + pre-commit install + +Every commit that touches `_data/schedule/**` now runs the same check CI runs. + +## Consuming from a clinic repo + +`.pre-commit-config.yaml`: + + repos: + - repo: https://github.com/ICI3D/ICI3D.github.io + rev: + hooks: + - id: validate-schedule + +pre-commit installs this package (schema + roles bundled) in an isolated environment +and runs it on the clinic's `_data/schedule/*.yml`. Instructor keys resolve against the +clinic's `_data/team`; point elsewhere with `args: [--people-dir, some/dir]`. + +Run the identical check in CI with `pre-commit/action`, so local and CI validation are +one source of truth.