diff --git a/.github/CI-MAINTAINER.md b/.github/CI-MAINTAINER.md new file mode 100644 index 000000000..69324a203 --- /dev/null +++ b/.github/CI-MAINTAINER.md @@ -0,0 +1,145 @@ +# CI Maintainer Notes + +Operational steps that a `gramps-project/addons-source` maintainer +needs to handle once PR 820 (and the branch-neutral follow-up) lands. +The pipeline is otherwise self-driving — this document is only about +the rough edges. + +For day-to-day addon-release maintenance see [MAINTAINERS.md](../MAINTAINERS.md); +for the contributor-facing summary of what a green CI check means on +an unreleased branch see [CONTRIBUTING.md](../CONTRIBUTING.md#work-towards-a-merge). + +## Contents + +1. [One-time setup when the PR first merges](#one-time-setup-when-the-pr-first-merges) +2. [Creating a new maintenance branch](#creating-a-new-maintenance-branch) +3. [When a Gramps minor release lands on PyPI](#when-a-gramps-minor-release-lands-on-pypi) +4. [Diagnostic log markers](#diagnostic-log-markers) +5. [Optional future-proofing knobs](#optional-future-proofing-knobs) + +## One-time setup when the PR first merges + +### 1. Make the `gramps-ci` GHCR package public + +`docker-build.yml` pushes images to +`ghcr.io/gramps-project/addons-source/gramps-ci:` using the +workflow's `GITHUB_TOKEN`. GHCR creates the package as **private** the +first time. Same-repo CI keeps working (token covers own packages), +but **fork PRs cannot pull the image** because their `GITHUB_TOKEN` +has no read access to private packages in `gramps-project`. Fork-PR +container jobs would fail at "Initialize containers" with an +authentication error. + +Fix once, immediately after the first `Build Docker Images` run +finishes: + +1. Go to +2. Under "Danger Zone" → "Change visibility" → set to **Public** + +Every existing and future `gramps-ci:` tag inherits public +visibility from this single setting. + +### 2. Expect the first-push race on `maintenance/gramps60` + +The first push event after merge fires both workflows in parallel: + +- `Build Docker Images` builds and pushes `gramps-ci:gramps60` + (~5 min cold). +- `CI` runs `setup` (~2 s), then its container jobs try to pull the + image. + +Because both start on the same push, the CI container jobs race the +image push and may fail at "Initialize containers" the first time. +This race only happens once per branch: + +1. Wait for `Build Docker Images` to complete. +2. Open the failed CI run → click "Re-run failed jobs". +3. Subsequent pushes find the image already in GHCR — no race. + +This is also explained in the header comment of `ci.yml`. + +## Creating a new maintenance branch + +When a new Gramps minor series goes into development and addons need +a corresponding branch (e.g. `maintenance/gramps62` once 6.2 opens): + +``` +git branch maintenance/gramps62 maintenance/gramps61 +git push origin maintenance/gramps62 +``` + +No workflow edits required. The workflows derive everything from +`github.ref_name`. The first push fires the same race described +above — re-run the failed CI jobs once `Build Docker Images` +finishes. + +The setup job's regex (`gramps[0-9][0-9]`) requires a two-digit +suffix. When Gramps 10.0 opens this regex needs updating in two +places (`ci.yml` setup job and `docker-build.yml` params step). + +## When a Gramps minor release lands on PyPI + +The hybrid Dockerfile auto-detects PyPI availability: + +- `pip install "gramps==X.Y.*"` succeeds → image installs the tagged + PyPI release (`::notice::` log line). +- pip reports "No matching distribution found" → image falls back to a + SHA-pinned `git clone` of `gramps-project/gramps@maintenance/grampsNN` + at the SHA captured by `docker-build.yml`'s params step + (`::warning::` log line). + +When `gramps==6.1.0` is finally published to PyPI, the next image +rebuild on `maintenance/gramps61` silently switches from "git tip" to +"PyPI release" — no maintainer action needed. + +To **immediately** rebuild against the new PyPI release without +waiting for the next push: + +1. Open the `Build Docker Images` workflow in the Actions tab. +2. "Run workflow" → select `maintenance/grampsNN` → Run. + +The `::notice::` line in the build log confirms the switch. + +## Diagnostic log markers + +When investigating a CI failure, the install-step output in +`Build Docker Images` carries these annotations: + +| Annotation | Meaning | +| --- | --- | +| `::notice::installed gramps==X.Y.* from PyPI` | Released-path install. CI is testing against a tagged release. | +| `::warning::no gramps==X.Y.* on PyPI; installing from gramps-project/gramps@maintenance/grampsNN at ` | Unreleased-branch fallback. CI is testing against the upstream branch tip at ``. Visible to contributors via the CONTRIBUTING.md note. | +| `::error::no gramps==X.Y.* on PyPI and GRAMPS_FALLBACK_SHA is unset` | `git ls-remote` in `docker-build.yml`'s params step returned no SHA for `maintenance/grampsNN` on `gramps-project/gramps`. The matching upstream branch is missing, or the addons-source branch is misnamed. | +| `::error::pip install gramps failed (non-version reason)` | pip failed for a network/registry reason, not because the version is missing. Captured stderr is dumped after the line. The build does **not** fall back to git in this case — by design, so a transient PyPI hiccup cannot silently flip a released branch into "git tip" mode. | + +Other useful entry points: + +- `ci.yml` setup job log shows the derived `branch_suffix` and + `ci_image`. A failure here means the branch name doesn't match + `maintenance/gramps[0-9][0-9]`. +- `docker-build.yml` params step log shows the captured + `fallback_sha`. An empty value means upstream `gramps-project/gramps` + has no matching maintenance branch (warning issued; image build will + fail iff the fallback is needed). + +## Optional future-proofing knobs + +Not needed today; record here in case they ever come up. + +### Upstream gramps repo URL + +The Dockerfile hardcodes `https://github.com/gramps-project/gramps.git` +as the fallback source. If the project ever reorgs or renames, this +URL must change in one place (`.github/docker/gramps-ci/Dockerfile`). +It could be parameterised as a build arg (`ARG GRAMPS_UPSTREAM_REPO`) +with the current URL as default, but a hardcoded value keeps the +Dockerfile simpler and a rename is a sufficiently large event that +editing one string is not the bottleneck. + +### GHCR tag retention + +`docker-build.yml` pushes both a moving `gramps-ci:` tag (e.g. +`gramps60`) and a per-commit `gramps-ci:-` tag. +The moving tag is always overwritten; the SHA tags accumulate over +time. Set a retention policy on the GHCR package settings page if +the count becomes inconvenient — the moving tags are what CI consumes. diff --git a/.github/docker/gramps-ci/Dockerfile b/.github/docker/gramps-ci/Dockerfile index 3ef3b8945..38a4e9a28 100644 --- a/.github/docker/gramps-ci/Dockerfile +++ b/.github/docker/gramps-ci/Dockerfile @@ -1,6 +1,9 @@ # .github/docker/gramps-ci/Dockerfile # -# Unified Gramps 6.0 CI image. Includes everything jobs need: +# Gramps CI image. The Gramps minor series (6.0, 6.1, …) is picked at +# build time via the GRAMPS_SERIES build arg, so the same Dockerfile +# produces gramps-ci:gramps60, gramps-ci:gramps61, … without per-branch +# edits. Includes everything jobs need: # - Python + pip-installed Gramps, PyGObject, pycairo # - GTK typelibs (so addon modules that `from gi.repository import Gtk` # at module load time are importable — widgets still need xvfb to render) @@ -13,11 +16,44 @@ # `--init` (or use a container runtime that injects tini) because xvfb-run # hangs if it inherits PID 1. # +# Gramps install path. PyPI is tried first; when no gramps==${SERIES}.* +# release exists (e.g. while 6.1 is still in development) the build +# falls back to a SHA-pinned git clone of +# gramps-project/gramps@maintenance/gramps${SERIES_NODOT}. Other pip +# failures (network, etc.) are NOT silently retried so a transient +# PyPI outage cannot flip a normally-released branch to "test against +# moving tip" mode. docker-build.yml captures the upstream SHA via +# git ls-remote and passes it in as GRAMPS_FALLBACK_SHA; the SHA +# becomes part of the buildx cache key so a moved upstream tip +# actually re-runs the install layer. +# +# Local build (released series): +# docker build --build-arg GRAMPS_SERIES=6.0 .github/docker/gramps-ci +# +# Local build (unreleased series, e.g. 6.1): +# sha=$(git ls-remote https://github.com/gramps-project/gramps.git \ +# refs/heads/maintenance/gramps61 | awk '{print $1}') +# docker build --build-arg GRAMPS_SERIES=6.1 \ +# --build-arg GRAMPS_FALLBACK_SHA=$sha \ +# .github/docker/gramps-ci +# ARG PYTHON_VERSION=3.12 FROM python:${PYTHON_VERSION}-slim +# Gramps minor series to install (e.g. 6.0, 6.1). No default — must be +# passed explicitly so a wrong default cannot silently produce an image +# for the wrong Gramps series. docker-build.yml derives this from the +# branch ref. +ARG GRAMPS_SERIES +# Commit SHA on gramps-project/gramps@maintenance/grampsNN. Only used +# when no gramps==${GRAMPS_SERIES}.* release exists on PyPI. Ignored +# otherwise. Empty default is intentional — on released branches the +# fallback never fires. +ARG GRAMPS_FALLBACK_SHA="" +RUN [ -n "$GRAMPS_SERIES" ] || { echo "GRAMPS_SERIES is required (e.g. 6.0)"; exit 1; } + LABEL org.opencontainers.image.source="https://github.com/gramps-project/addons-source" -LABEL org.opencontainers.image.description="Unified Gramps 6.0 CI image (Python, Gramps, GTK typelibs, xvfb)" +LABEL org.opencontainers.image.description="Gramps ${GRAMPS_SERIES} CI image (Python, Gramps, GTK typelibs, xvfb)" RUN apt-get update && apt-get install -y --no-install-recommends \ libgirepository-2.0-dev \ @@ -26,6 +62,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \ gir1.2-pango-1.0 \ gir1.2-gdkpixbuf-2.0 \ gir1.2-atk-1.0 \ + gir1.2-gexiv2-0.10 \ gcc \ pkg-config \ python3-dev \ @@ -37,13 +74,39 @@ RUN apt-get update && apt-get install -y --no-install-recommends \ xauth \ && rm -rf /var/lib/apt/lists/* -RUN pip install --no-cache-dir \ - PyGObject \ - pycairo \ - "gramps>=6.0,<6.1" \ - orjson \ - ruff \ - dbf +# Addon runtime deps (dbf, networkx, lxml, svgwrite, boto3, etc.) are +# NOT baked in here — ci.yml's "Install addon runtime deps (derived from +# requires_mod)" step pip-installs them at CI runtime from every +# .gpr.py's requires_mod list, matching what Gramps' Addon Manager does +# for an end user. Keeps .gpr.py the single source of truth. +RUN pip install --no-cache-dir PyGObject pycairo orjson ruff + +# Install gramps: PyPI first, SHA-pinned git clone as fallback. +RUN /bin/bash -eo pipefail <<'BASH' +suffix_nodot="${GRAMPS_SERIES//./}" +if pip install --no-cache-dir "gramps==${GRAMPS_SERIES}.*" 2>/tmp/pip.err; then + echo "::notice::installed gramps==${GRAMPS_SERIES}.* from PyPI" +elif grep -q "No matching distribution found for gramps==" /tmp/pip.err; then + if [ -z "${GRAMPS_FALLBACK_SHA}" ]; then + echo "::error::no gramps==${GRAMPS_SERIES}.* on PyPI and GRAMPS_FALLBACK_SHA is unset" + exit 1 + fi + echo "::warning::no gramps==${GRAMPS_SERIES}.* on PyPI; installing from gramps-project/gramps@maintenance/gramps${suffix_nodot} at ${GRAMPS_FALLBACK_SHA}" + mkdir -p /tmp/gramps + cd /tmp/gramps + git init -q + git remote add origin https://github.com/gramps-project/gramps.git + git fetch --depth 1 origin "${GRAMPS_FALLBACK_SHA}" + git checkout FETCH_HEAD + pip install --no-cache-dir . + cd / + rm -rf /tmp/gramps +else + echo "::error::pip install gramps failed (non-version reason); aborting rather than silently switching to git tip" + cat /tmp/pip.err + exit 1 +fi +BASH RUN apt-get purge -y gcc python3-dev pkg-config && apt-get autoremove -y diff --git a/.github/environment.yml b/.github/environment.yml index 9ca19f57f..f1c7c9ae7 100644 --- a/.github/environment.yml +++ b/.github/environment.yml @@ -6,7 +6,17 @@ dependencies: - pygobject - gtk3 - pip + # Addon runtime deps (dbf, networkx, lxml, svgwrite, boto3, etc.) are + # installed at CI runtime by ci.yml's auto-derive step from .gpr.py + # requires_mod — single source of truth. Keep only the stable base + # here (Gramps + orjson for plugin registration). + # + # conda-forge has no gramps 6.1 yet, so on a maintenance/gramps61 (or later) + # branch this resolves to 6.0.x — i.e. the Windows lane validates addons + # against conda-forge's newest in-range gramps, not the branch's exact series + # (the Linux CI image git-builds the exact series; conda-Windows cannot — see + # ci.yml's "Report gramps-vs-branch series" step). When 6.1 reaches + # conda-forge the pin picks it up automatically. - pip: - "gramps>=6.0,<6.1" - orjson - - dbf diff --git a/.github/scripts/addon_system_deps.py b/.github/scripts/addon_system_deps.py new file mode 100644 index 000000000..6e3cef514 --- /dev/null +++ b/.github/scripts/addon_system_deps.py @@ -0,0 +1,209 @@ +#!/usr/bin/env python3 +"""Single source of truth for addon *system* dependencies in CI. + +Addons declare three dependency kinds in their ``.gpr.py``: + +* ``requires_mod`` — importable Python modules. pip-installable; ci.yml already + auto-derives these from the ``.gpr.py`` files. Nothing to do here. +* ``requires_gi`` — GObject-introspection typelibs (e.g. ``GooCanvas``). +* ``requires_exe`` — system executables (e.g. ``dot`` from graphviz). + +The latter two are *system* packages: not pip-installable, named differently per +platform, and Gramps' own ``Requirements`` only *checks* them (never installs). +This module maps each declared ``requires_gi`` namespace / ``requires_exe`` name +to its package on each CI platform and scans the addons for what they declare, +so ci.yml derives the install list from one place instead of a hand-kept list. + +Platform availability is asymmetric and encoded here: the GTK 3 addon libs +(goocanvas, osm-gps-map, gexiv2) exist on Debian/apt but **not on conda-forge**, +so the conda (Windows) lane cannot install them — addons needing them skip there +by necessity. A ``conda`` value of ``None`` records that. + +Pure stdlib so it runs anywhere in CI without bootstrapping. + +CLI:: + + addon_system_deps.py --platform apt # space-separated install list + addon_system_deps.py --platform conda # (only packages available there) + addon_system_deps.py --unmapped . # declared deps with no map entry; exit 1 if any +""" + +# ------------------------ +# Python modules +# ------------------------ +from __future__ import annotations + +import argparse +import ast +import glob +import os +import re +import sys + +# --------------------------------------------------------------------------- +# The map. Keys are what addons declare; values give the package per platform. +# A None value means "no package provides this on that platform" (so it is not +# installed there and an addon needing it is expected to skip). +# --------------------------------------------------------------------------- + +# requires_gi namespace -> package providing the typelib, per platform. +GI_PACKAGES: dict[str, dict[str, str | None]] = { + "GExiv2": {"apt": "gir1.2-gexiv2-0.10", "conda": None}, + "GooCanvas": {"apt": "gir1.2-goocanvas-2.0", "conda": None}, + "OsmGpsMap": {"apt": "gir1.2-osmgpsmap-1.0", "conda": None}, + # PlaceCoordinateGramplet declares GeocodeGlib 1.0, but modern distros ship + # only the 2.0 typelib and conda-forge ships none; the addon has no tests. + # Recorded so the drift-guard recognises the namespace; not installed. + "GeocodeGlib": {"apt": None, "conda": None}, +} + +# requires_exe executable -> package providing it, per platform. +EXE_PACKAGES: dict[str, dict[str, str | None]] = { + "dot": {"apt": "graphviz", "conda": "graphviz"}, +} + +PLATFORMS = ("apt", "conda") + + +# ------------------------------------------------------------ +# +# scanning +# +# ------------------------------------------------------------ +_GI_RE = re.compile(r"requires_gi\s*=\s*(\[[^\]]*\])") +_EXE_RE = re.compile(r"requires_exe\s*=\s*(\[[^\]]*\])") + + +def _gpr_files(root: str) -> list[str]: + return sorted(glob.glob(os.path.join(root, "*", "*.gpr.py"))) + + +def _literal(src: str): + try: + return ast.literal_eval(src) + except (ValueError, SyntaxError): + return [] + + +def _scan(root: str, pattern: re.Pattern, first_of_tuple: bool) -> set[str]: + found: set[str] = set() + for path in _gpr_files(root): + try: + text = open(path, encoding="utf-8").read() + except OSError: + continue + for match in pattern.finditer(text): + for entry in _literal(match.group(1)): + if first_of_tuple and isinstance(entry, (tuple, list)): + entry = entry[0] if entry else None + if entry: + found.add(entry) + return found + + +def scan_gi_namespaces(root: str) -> set[str]: + return _scan(root, _GI_RE, first_of_tuple=True) + + +def scan_executables(root: str) -> set[str]: + return _scan(root, _EXE_RE, first_of_tuple=False) + + +def addon_requirements(addon_dir: str) -> tuple[set[str], set[str]]: + """Return (gi_namespaces, executables) declared by a single addon dir.""" + gi: set[str] = set() + exe: set[str] = set() + for path in sorted(glob.glob(os.path.join(addon_dir, "*.gpr.py"))): + try: + text = open(path, encoding="utf-8").read() + except OSError: + continue + for match in _GI_RE.finditer(text): + for entry in _literal(match.group(1)): + ns = entry[0] if isinstance(entry, (tuple, list)) else entry + if ns: + gi.add(ns) + for match in _EXE_RE.finditer(text): + for entry in _literal(match.group(1)): + if entry: + exe.add(entry) + return gi, exe + + +# ------------------------------------------------------------ +# +# derivation +# +# ------------------------------------------------------------ +def packages(platform: str) -> list[str]: + """All install-by-name packages available for a platform (full mapped set).""" + pkgs: list[str] = [] + for table in (GI_PACKAGES, EXE_PACKAGES): + for entry in table.values(): + pkg = entry.get(platform) + if pkg: + pkgs.append(pkg) + return sorted(set(pkgs)) + + +def unmapped(root: str) -> tuple[set[str], set[str]]: + """Declared deps with no entry in the maps at all (drift).""" + return ( + scan_gi_namespaces(root) - set(GI_PACKAGES), + scan_executables(root) - set(EXE_PACKAGES), + ) + + +def addon_satisfiable_on(addon_dir: str, platform: str) -> bool: + """ + True if every system dep the addon declares has a package on this platform. + + Used by the test runner to tell an *expected* platform skip (a declared dep + that simply is not packaged here, e.g. goocanvas on conda) from a suspicious + all-skip that should fail. + """ + gi, exe = addon_requirements(addon_dir) + for ns in gi: + entry = GI_PACKAGES.get(ns) + if entry is None or entry.get(platform) is None: + return False + for name in exe: + entry = EXE_PACKAGES.get(name) + if entry is None or entry.get(platform) is None: + return False + return True + + +# ------------------------------------------------------------ +# +# CLI +# +# ------------------------------------------------------------ +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--platform", choices=PLATFORMS) + parser.add_argument( + "--unmapped", + metavar="ROOT", + help="print declared GI/exe deps with no map entry; exit 1 if any", + ) + args = parser.parse_args(argv) + + if args.unmapped is not None: + gi, exe = unmapped(args.unmapped) + for ns in sorted(gi): + print(f"gi:{ns}") + for name in sorted(exe): + print(f"exe:{name}") + return 1 if (gi or exe) else 0 + + if args.platform: + print(" ".join(packages(args.platform))) + return 0 + + parser.error("nothing to do: pass --platform or --unmapped") + return 2 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/.github/scripts/gi_bootstrap/sitecustomize.py b/.github/scripts/gi_bootstrap/sitecustomize.py new file mode 100644 index 000000000..9f5cb090d --- /dev/null +++ b/.github/scripts/gi_bootstrap/sitecustomize.py @@ -0,0 +1,29 @@ +"""Pin the GObject-introspection versions, like the Gramps GUI launcher. + +Put this directory on ``PYTHONPATH`` for a test step and the interpreter imports +this ``sitecustomize`` at startup — before any test (or subprocess it spawns) +imports a ``gramps.gui`` module. + +Why: gramps pins its GI versions in the GUI launcher (``gramps/gui/grampsgui.py`` +calls ``gi.require_version`` for Pango/PangoCairo/Gtk at import). A test that +imports a ``gramps.gui.*`` module directly never runs that launcher, so Gtk/Pango +get imported with no version pinned first — emitting a ``PyGIWarning`` and, on a +host where GTK 4 is the default, risking the wrong stack. This shim performs the +same bootstrap, so tests run under the supported GTK 3 stack. + +Used for the discover-based / subprocess-loading steps (e.g. plugin +registration), where the bootstrap must be inherited via ``PYTHONPATH`` by every +spawned interpreter. The in-process unit/integration runner +(``run_addon_tests.py``) does the same ``require_version`` itself. +""" + +try: + import gi + + for _ns, _ver in (("Pango", "1.0"), ("PangoCairo", "1.0"), ("Gtk", "3.0")): + try: + gi.require_version(_ns, _ver) + except (ValueError, AttributeError): + pass +except ImportError: + pass diff --git a/.github/scripts/run_addon_tests.py b/.github/scripts/run_addon_tests.py new file mode 100644 index 000000000..6491aa0e5 --- /dev/null +++ b/.github/scripts/run_addon_tests.py @@ -0,0 +1,210 @@ +#!/usr/bin/env python3 +"""Run per-addon unit tests with a GI bootstrap, a timeout, and honest skips. + +Replaces a bare ``python -m unittest `` in CI. It does three things +plain unittest does not: + +1. **GI version bootstrap.** Before any test imports a ``gramps.gui`` module, it + calls ``gi.require_version`` for Pango/PangoCairo/Gtk — the set the Gramps + GUI launcher (``gramps/gui/grampsgui.py``) pins at startup. A direct test + import never runs that launcher, so without this the first + ``from gi.repository import Gtk`` (in gramps core) warns and risks the wrong + GTK on a host where GTK 4 is the default. + +2. **A per-module timeout.** Each module runs in its own subprocess with a wall + clock. A test that hangs (e.g. a DB import that blocks on a platform) would + otherwise hang the whole CI job indefinitely — neither plain unittest nor + xmlrunner has a timeout. A module that exceeds the limit is killed and + reported as a FAILURE, so the job stays bounded and names the culprit. + +3. **Honest skip accounting.** unittest exits 0 when every test SKIPS, so a + wholly-skipped module reads as a pass. This runner FAILS such a module — + UNLESS the addon's declared system deps are unavailable on this platform + (e.g. goocanvas/osm-gps-map are not on conda-forge), in which case the skip + is expected and tolerated (the map lives in ``addon_system_deps.py``). + +Usage:: + + run_addon_tests.py --platform apt Addon.tests.test_x Other.tests.test_y + run_addon_tests.py --platform conda Addon.tests.test_x + +Exit code is non-zero if any module is a hard failure (test failure/error, +timeout, or an unexpected all-skip on a platform where the addon's deps are +available). +""" + +# ------------------------ +# Python modules +# ------------------------ +from __future__ import annotations + +import argparse +import os +import subprocess +import sys +import unittest + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +import addon_system_deps as deps # noqa: E402 + +# Per-module wall clock. Generous enough for a legitimate DB-backed suite, small +# enough that a hung test is caught promptly instead of running to the job cap. +# Overridable via env for tuning/testing. +MODULE_TIMEOUT_S = int(os.environ.get("RUN_ADDON_TESTS_TIMEOUT", "300")) + +# Markers the worker prints so the parent can read the outcome without needing +# xmlrunner (820's env has only stdlib unittest). +_OK = "__RESULT__ ok" +_LOADERROR = "__RESULT__ loaderror" + + +def _bootstrap_gi() -> None: + """Pin the GI versions the Gramps GUI launcher pins, before tests import.""" + try: + import gi + except ImportError: + return + for namespace, version in (("Pango", "1.0"), ("PangoCairo", "1.0"), ("Gtk", "3.0")): + try: + gi.require_version(namespace, version) + except (ValueError, AttributeError): + pass + + +# ------------------------------------------------------------ +# +# worker: runs ONE module in this (sub)process +# +# ------------------------------------------------------------ +def _run_worker(modname: str) -> int: + """Run a single module; print a machine-readable result line; exit 0. + + The parent classifies pass/fail from the printed counts and its own platform + knowledge, so the worker always exits 0 (a non-zero exit would be + indistinguishable from an interpreter crash). + """ + _bootstrap_gi() + try: + suite = unittest.defaultTestLoader.loadTestsFromName(modname) + except Exception as exc: # import-time failure + print(f"{_LOADERROR} {exc!r}", flush=True) + return 0 + result = unittest.TextTestRunner(verbosity=2).run(suite) + broke = len(result.failures) + len(result.errors) + print( + f"{_OK} tests={result.testsRun} skipped={len(result.skipped)} broke={broke}", + flush=True, + ) + return 0 + + +# ------------------------------------------------------------ +# +# parent: spawns a timed worker per module and classifies the outcome +# +# ------------------------------------------------------------ +def _classify(modname: str, platform: str, root: str) -> tuple[bool, str]: + """Run one module in a timed subprocess. Return (is_hard_failure, summary).""" + addon = modname.split(".", 1)[0] + satisfiable = deps.addon_satisfiable_on(os.path.join(root, addon), platform) + + proc = subprocess.Popen( + [sys.executable, os.path.abspath(__file__), "--worker", modname], + stdout=subprocess.PIPE, + stderr=None, # stream the test output straight to the CI log + text=True, + ) + try: + # communicate() enforces the wall clock and reaps the process. + stdout, _ = proc.communicate(timeout=MODULE_TIMEOUT_S) + except subprocess.TimeoutExpired: + proc.kill() + proc.communicate() + return True, f" FAIL {modname} — timed out after {MODULE_TIMEOUT_S}s (hung)" + + out_lines = stdout.splitlines() + for line in out_lines: # echo the worker's result marker into the log + print(line) + + result_line = next( + (ln for ln in reversed(out_lines) if ln.startswith("__RESULT__")), "" + ) + + if result_line.startswith(_LOADERROR): + if satisfiable: + return True, f" FAIL {modname} — load error" + return False, ( + f" skip {modname} — not loadable on {platform} " + f"(addon system deps unavailable here)" + ) + + if not result_line.startswith(_OK): + return True, f" FAIL {modname} — no result (worker crashed)" + + fields = dict(tok.split("=", 1) for tok in result_line.split()[2:] if "=" in tok) + ran = int(fields.get("tests", 0)) + skipped = int(fields.get("skipped", 0)) + broke = int(fields.get("broke", 0)) + + if broke: + return True, f" FAIL {modname} — {broke} failed/errored" + if ran > 0 and skipped == ran: + if satisfiable: + return True, ( + f" FAIL {modname} — all {ran} tests skipped " + f"(degraded coverage; deps ARE available on {platform})" + ) + return False, ( + f" skip {modname} — all {ran} skipped, expected " + f"(addon system deps unavailable on {platform})" + ) + if skipped: + return False, f" ok {modname} — {ran} tests, {skipped} skipped" + return False, f" ok {modname} — {ran} tests" + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--platform", choices=deps.PLATFORMS) + parser.add_argument( + "--root", + default=".", + help="addons-source root holding the / dirs (default: cwd)", + ) + parser.add_argument( + "--worker", + metavar="MODULE", + help="internal: run this single module and print its result line", + ) + parser.add_argument("modules", nargs="*", help="dotted test modules to run") + args = parser.parse_args(argv) + + if args.worker: + return _run_worker(args.worker) + + if not args.platform: + parser.error("--platform is required in parent mode") + if not args.modules: + print("No per-addon unit test modules found") + return 0 + + hard_failures: list[str] = [] + summary: list[str] = [] + for modname in args.modules: + failed, line = _classify(modname, args.platform, args.root) + summary.append(line) + if failed: + hard_failures.append(modname) + + print("\n=== addon test summary ===") + for line in summary: + print(line) + + if hard_failures: + print(f"\n{len(hard_failures)} module(s) failed: {hard_failures}") + return 1 + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 1a79cda8b..d606be7b9 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1,35 +1,101 @@ name: CI +# Branch-neutral workflow: the image tag, make.py argument, and Gramps +# series pin are all derived from the branch ref at runtime, so the same +# file runs unchanged on maintenance/gramps60, maintenance/gramps61, and +# any future maintenance/grampsNN branch. +# +# NOTE for maintainers — first push to a new maintenance branch: the +# corresponding gramps-ci: image does not exist in GHCR yet, so +# the container jobs in this workflow will fail at "Initialize containers" +# on the very first run. The companion docker-build.yml workflow fires +# on the same push and builds/pushes the image (~5 min cold). After it +# finishes, re-run the failed CI jobs (Actions tab → this run → "Re-run +# failed jobs") and they will pull the now-existing image. This race +# happens only on initial branch creation — every subsequent push to +# that branch finds the image already in GHCR. +# +# Full operational runbook (GHCR visibility, PyPI-release transitions, +# diagnostic log markers, etc.): .github/CI-MAINTAINER.md + on: push: - branches: [maintenance/gramps60] + branches: [maintenance/gramps**] pull_request: - branches: [maintenance/gramps60] - -env: - CI_IMAGE: ghcr.io/${{ github.repository }}/gramps-ci:gramps60 + branches: [maintenance/gramps**] jobs: + # ----------------------------------------------------------------- + # Setup — derive the branch suffix (gramps60 / gramps61 / …) from + # the ref. On push events github.ref_name is the branch being + # pushed; on pull_request events github.base_ref is the target + # branch. Either way the suffix is what follows "maintenance/". + # ----------------------------------------------------------------- + setup: + name: Setup + runs-on: ubuntu-latest + outputs: + branch_suffix: ${{ steps.compute.outputs.branch_suffix }} + ci_image: ${{ steps.compute.outputs.ci_image }} + steps: + - id: compute + shell: bash + run: | + ref="${{ github.base_ref || github.ref_name }}" + suffix="${ref#maintenance/}" + case "$suffix" in + gramps[0-9][0-9]) ;; + *) echo "::error::unexpected ref '$ref' (suffix '$suffix')"; exit 1 ;; + esac + echo "branch_suffix=$suffix" >> "$GITHUB_OUTPUT" + echo "ci_image=ghcr.io/${{ github.repository }}/gramps-ci:$suffix" >> "$GITHUB_OUTPUT" + # ----------------------------------------------------------------- # Lint (ci container) # ----------------------------------------------------------------- lint: name: Lint + needs: setup runs-on: ubuntu-latest - # Non-blocking until the existing ruff E9/F63/F7/F82 errors across the - # addon set are cleaned up in a follow-up PR. Flip this off in that PR. - continue-on-error: true container: - image: ghcr.io/${{ github.repository }}/gramps-ci:gramps60 + image: ${{ needs.setup.outputs.ci_image }} steps: - uses: actions/checkout@v4 - name: Run ruff (syntax and import errors only) - run: ruff check --select=E9,F63,F7,F82 --no-fix --exclude='*.gpr.py' . + # Skip addon directories whose every register() in .gpr.py sets + # include_in_listing=False — those addons are not built or released + # by make.py, so CI does not gate on their lint state (per Gary + # Griffin's request on PR #820). To re-enable lint gating for an + # addon, set include_in_listing=True on at least one register() + # call in its descriptor (or remove the field — Gramps' default + # is True). Repeated inline rather than centralised so each job + # step stays self-contained. + shell: bash + run: | + is_active() { + local addon="$1" g + for g in "$addon"/*.gpr.py; do + [ -f "$g" ] || continue + grep -qE 'include_in_listing[[:space:]]*=[[:space:]]*True' "$g" && return 0 + grep -qE 'include_in_listing[[:space:]]*=' "$g" || return 0 + done + return 1 + } + excludes="" + for d in */; do + d="${d%/}" + ls "$d"/*.gpr.py >/dev/null 2>&1 || continue + is_active "$d" || excludes="$excludes --exclude=$d" + done + ruff check --select=E9,F63,F7,F82 --no-fix --exclude='*.gpr.py' $excludes . - name: Check trailing whitespace in Python files run: | - if git --no-pager grep --color -n --full-name '[ \t]$' -- '*.py'; then + # Use PCRE (-P): in BRE/ERE the bracket expression [ \t] is the + # set { space, backslash, 't' } — git grep matches anything ending + # in 't', not just whitespace. -P makes \t a tab. + if git --no-pager grep --color -n --full-name -P '[ \t]+$' -- '*.py'; then echo "::error::Trailing whitespace found in Python files" exit 1 fi @@ -46,11 +112,23 @@ jobs: steps: - uses: actions/checkout@v4 - - name: Check all addons have po/template.pot + - name: Check all listed addons have po/template.pot + # Skip include_in_listing=False addons (see lint job for rationale). + shell: bash run: | + is_active() { + local addon="$1" g + for g in "$addon"/*.gpr.py; do + [ -f "$g" ] || continue + grep -qE 'include_in_listing[[:space:]]*=[[:space:]]*True' "$g" && return 0 + grep -qE 'include_in_listing[[:space:]]*=' "$g" || return 0 + done + return 1 + } failed=0 for gpr in */*.gpr.py; do addon_dir="$(dirname "$gpr")" + is_active "$addon_dir" || continue if [ ! -d "$addon_dir/po" ]; then echo "::error::$addon_dir is missing po/ directory" failed=1 @@ -60,7 +138,7 @@ jobs: fi done if [ "$failed" -eq 0 ]; then - echo "All addons have po/template.pot" + echo "All listed addons have po/template.pot" fi exit $failed @@ -69,17 +147,39 @@ jobs: # ----------------------------------------------------------------- compile-check: name: Compile Check + needs: setup runs-on: ubuntu-latest container: - image: ghcr.io/${{ github.repository }}/gramps-ci:gramps60 + image: ${{ needs.setup.outputs.ci_image }} steps: - uses: actions/checkout@v4 - - name: Compile all Python files (excluding .gpr.py) + - name: Compile all Python files in listed addons (excluding .gpr.py) + # Skip include_in_listing=False addons (see lint job for rationale). shell: bash run: | + is_active() { + local addon="$1" g + for g in "$addon"/*.gpr.py; do + [ -f "$g" ] || continue + grep -qE 'include_in_listing[[:space:]]*=[[:space:]]*True' "$g" && return 0 + grep -qE 'include_in_listing[[:space:]]*=' "$g" || return 0 + done + return 1 + } + skipped="" + for d in */; do + d="${d%/}" + ls "$d"/*.gpr.py >/dev/null 2>&1 || continue + is_active "$d" || skipped="$skipped $d" + done failed=0 while IFS= read -r f; do + skip=0 + for s in $skipped; do + case "$f" in ./$s/*) skip=1; break;; esac + done + [ "$skip" = 1 ] && continue if ! python3 -m py_compile "$f" 2>&1; then failed=1 fi @@ -91,15 +191,42 @@ jobs: # ----------------------------------------------------------------- unit-test-linux: name: Unit Tests (Linux) + needs: setup runs-on: ubuntu-latest - # Non-blocking until the currently-broken addon unit modules (import - # failures, stale API usage) are sorted out in follow-up PRs. - continue-on-error: true container: - image: ghcr.io/${{ github.repository }}/gramps-ci:gramps60 + image: ${{ needs.setup.outputs.ci_image }} steps: - uses: actions/checkout@v4 + - name: Install addon system deps (derived from requires_gi / requires_exe) + # System deps (GI typelibs, executables) are not pip-installable and + # gramps only *checks* them, so the image cannot bake them generically + # (its build context excludes addons-source). Derive the apt set from + # every .gpr.py via the single-source map and install it — the container + # runs as root. Mirrors the requires_mod derivation below. + shell: bash + run: | + pkgs=$(python3 .github/scripts/addon_system_deps.py --platform apt) + if [ -n "$pkgs" ]; then + echo "→ addon system deps (apt): $pkgs" + apt-get update + apt-get install -y --no-install-recommends $pkgs + else + echo "no requires_gi / requires_exe declarations found" + fi + + - name: Validate addon system deps are mapped + # Every requires_gi / requires_exe an addon declares must have an entry + # in addon_system_deps.py, so the install list never silently drifts + # from what addons declare (the system-dep analogue of the requires_mod + # find_spec gate below). + shell: bash + run: | + python3 .github/scripts/addon_system_deps.py --unmapped . || { + echo "::error::Addon(s) declare requires_gi/requires_exe with no entry in .github/scripts/addon_system_deps.py — add a mapping row." + exit 1 + } + - name: Install addon runtime deps (derived from requires_mod) # Auto-derive the union of requires_mod across every .gpr.py in # the repo. Mirrors Gramps' Addon Manager install path @@ -138,6 +265,58 @@ jobs: echo "no requires_mod declarations found" fi + - name: Validate requires_mod names against Gramps' dep gate + # Cross-check: every requires_mod entry that pip successfully + # installed in the previous step must also resolve via + # find_spec(), since that is what Gramps' Addon Manager calls + # (gramps/gen/utils/requirements.py:check_mod). A name that + # pip-installs but does not import is a declaration bug — e.g. + # requires_mod=["Pillow"] when the importable name is "PIL". + # Pip-install failures upstream are skipped: those are + # system-dep / image gaps, not PR-caused. + shell: bash + run: | + python3 - <<'PY' + import ast, glob, re, subprocess, sys + from importlib.util import find_spec + + pat = re.compile(r"requires_mod\s*=\s*(\[[^\]]*\])") + names = set() + for f in glob.glob("*/*.gpr.py"): + try: + text = open(f, encoding="utf-8").read() + except OSError: + continue + for m in pat.finditer(text): + try: + names.update(ast.literal_eval(m.group(1))) + except (ValueError, SyntaxError): + pass + + bad = [] + for name in sorted(names): + installed = subprocess.run( + [sys.executable, "-m", "pip", "show", name], + capture_output=True, + ).returncode == 0 + if not installed: + print(f"~ {name} (pip-install failed earlier, skipping)") + continue + if find_spec(name) is None: + bad.append(name) + print(f"x {name} (pip-installed but find_spec returned None)") + else: + print(f"ok {name}") + + if bad: + print() + print(f"::error::Wrong requires_mod names: {bad}") + print("These pip-install but are not importable. requires_mod is") + print("consumed by gramps' check_mod() via find_spec(), so the") + print("importable module name is required (e.g. 'PIL', not 'Pillow').") + sys.exit(1) + PY + - name: Run per-addon unit tests # Filename convention (all OSes): # test_*.py — general (any OS) @@ -150,14 +329,25 @@ jobs: # shell: bash — the container's default shell is /bin/sh # (dash on python:3.12-slim), which does not support the # ${var//pattern/repl} and ${var%.py} parameter expansions - # used below. Falls silently under continue-on-error. + # used below. shell: bash env: PYTHONPATH: . run: | + is_active() { + local addon="$1" g + for g in "$addon"/*.gpr.py; do + [ -f "$g" ] || continue + grep -qE 'include_in_listing[[:space:]]*=[[:space:]]*True' "$g" && return 0 + grep -qE 'include_in_listing[[:space:]]*=' "$g" || return 0 + done + return 1 + } modules="" for f in */tests/test_*.py; do [ -f "$f" ] || continue + addon="${f%%/*}" + is_active "$addon" || continue case "$(basename "$f")" in test_integration*) continue ;; test_windows_*) continue ;; @@ -171,7 +361,14 @@ jobs: done if [ -n "$modules" ]; then echo "Running unit tests:$modules" - python3 -m unittest -v $modules + # xvfb-run: some addons create a Gtk style context at import and + # need a display (else a hard Gtk-ERROR abort, not a clean skip). + # run_addon_tests.py: pins the GI versions like gramps' launcher + # (so gramps.gui imports load GTK 3, no PyGIWarning) and fails a + # wholly-skipped module unless the addon's deps are unavailable on + # this platform. + xvfb-run -a --server-args="-screen 0 1920x1080x24" \ + python3 .github/scripts/run_addon_tests.py --platform apt --root . $modules else echo "No per-addon unit test modules found" fi @@ -181,9 +378,8 @@ jobs: # ----------------------------------------------------------------- unit-test-windows: name: Unit Tests (Windows) + needs: setup runs-on: windows-latest - # Non-blocking for the same reason as unit-test-linux. - continue-on-error: true defaults: run: shell: bash -el {0} @@ -204,6 +400,47 @@ jobs: mamba list | head -30 python -c "import gramps, gi; print('deps OK')" + - name: Report gramps-vs-branch series (Windows lane caveat) + # The Linux lane git-builds the branch's exact gramps in its CI image + # (.github/docker/gramps-ci/Dockerfile, PyPI-first/git-fallback). The + # conda-forge Windows lane CANNOT match that: conda-forge has no gramps + # 6.1 yet, and gramps' own Windows build targets MSYS2 UCRT64, not conda + # — building 6.1 from git here fails in gramps' build hook (build_intl's + # `msgfmt --xml` cannot locate the shared-mime-info/appstream ITS rules, + # which are absent in the conda env). So on a maintenance/gramps61 (or + # later) branch this lane validates addons against conda-forge's newest + # in-range gramps (6.0.x today) rather than the branch's series. This + # step surfaces that honestly; it does NOT fail. Addon tests that depend + # on series-exact gramps behaviour skip themselves on Windows (e.g. + # TMGimporter's real-DB import tests) and run on the Linux lane instead. + # When gramps 6.1 reaches conda-forge, environment.yml's pin picks it up + # and this caveat disappears on its own. + run: | + suffix="${{ needs.setup.outputs.branch_suffix }}" # e.g. gramps61 + digits="${suffix#gramps}" # e.g. 61 + want="${digits:0:1}.${digits:1}" # e.g. 6.1 + have="$(python -c 'from gramps.version import major_version; print(major_version)')" + if [ "$want" = "$have" ]; then + echo "conda-forge gramps $have matches branch series $want — addons tested against the branch's gramps" + else + echo "::warning::Windows lane: branch targets gramps $want but conda-forge ships $have; addons here are validated against $have. Full $want coverage is on the Linux lane (its CI image git-builds $want). See step comment for why conda-Windows cannot build $want." + fi + + - name: Install addon system deps (derived, conda-available subset) + # Only the conda-forge-available subset (e.g. graphviz). The GTK 3 addon + # GI libs (goocanvas/osm-gps-map/gexiv2) are NOT on conda-forge, so the + # map returns None for them on conda and they are not installed; addons + # needing them skip on Windows by necessity, which run_addon_tests + # tolerates (--platform conda). + run: | + pkgs=$(python .github/scripts/addon_system_deps.py --platform conda) + if [ -n "$pkgs" ]; then + echo "→ addon system deps (conda): $pkgs" + mamba install -y -c conda-forge $pkgs + else + echo "no conda-available addon system deps to install" + fi + - name: Install addon runtime deps (derived from requires_mod) # See unit-test-linux for rationale. Uses `python` (conda-forge # env) to match the surrounding Windows job style. @@ -234,15 +471,71 @@ jobs: echo "no requires_mod declarations found" fi + - name: Validate requires_mod names against Gramps' dep gate + # See unit-test-linux for rationale. Uses `python` (conda-forge + # env) to match the surrounding Windows job style. + run: | + python - <<'PY' + import ast, glob, re, subprocess, sys + from importlib.util import find_spec + + pat = re.compile(r"requires_mod\s*=\s*(\[[^\]]*\])") + names = set() + for f in glob.glob("*/*.gpr.py"): + try: + text = open(f, encoding="utf-8").read() + except OSError: + continue + for m in pat.finditer(text): + try: + names.update(ast.literal_eval(m.group(1))) + except (ValueError, SyntaxError): + pass + + bad = [] + for name in sorted(names): + installed = subprocess.run( + [sys.executable, "-m", "pip", "show", name], + capture_output=True, + ).returncode == 0 + if not installed: + print(f"~ {name} (pip-install failed earlier, skipping)") + continue + if find_spec(name) is None: + bad.append(name) + print(f"x {name} (pip-installed but find_spec returned None)") + else: + print(f"ok {name}") + + if bad: + print() + print(f"::error::Wrong requires_mod names: {bad}") + print("These pip-install but are not importable. requires_mod is") + print("consumed by gramps' check_mod() via find_spec(), so the") + print("importable module name is required (e.g. 'PIL', not 'Pillow').") + sys.exit(1) + PY + - name: Run per-addon unit tests # See filename-convention note in unit-test-linux. The Windows # job runs test_*.py except test_linux_* and test_integration_*. env: PYTHONPATH: . run: | + is_active() { + local addon="$1" g + for g in "$addon"/*.gpr.py; do + [ -f "$g" ] || continue + grep -qE 'include_in_listing[[:space:]]*=[[:space:]]*True' "$g" && return 0 + grep -qE 'include_in_listing[[:space:]]*=' "$g" || return 0 + done + return 1 + } modules="" for f in */tests/test_*.py; do [ -f "$f" ] || continue + addon="${f%%/*}" + is_active "$addon" || continue case "$(basename "$f")" in test_integration*) continue ;; test_linux_*) continue ;; @@ -256,7 +549,10 @@ jobs: done if [ -n "$modules" ]; then echo "Running unit tests:$modules" - python -m unittest -v $modules + # No xvfb on Windows (GTK renders natively). run_addon_tests pins + # the GI versions and tolerates addons whose GI deps are not on + # conda-forge (they skip here by platform necessity). + python .github/scripts/run_addon_tests.py --platform conda --root . $modules else echo "No per-addon unit test modules found" fi @@ -267,13 +563,28 @@ jobs: integration-test: name: Integration Tests (Gramps) runs-on: ubuntu-latest - needs: [unit-test-linux] + needs: [setup, unit-test-linux] container: - image: ghcr.io/${{ github.repository }}/gramps-ci:gramps60 + image: ${{ needs.setup.outputs.ci_image }} options: --init steps: - uses: actions/checkout@v4 + - name: Install addon system deps (derived from requires_gi / requires_exe) + # Same as unit-test-linux: derive the apt set from the single-source + # map and install it (container runs as root). The plugin registration + # test subprocess-loads each addon module, so the GI typelibs those + # modules import must be present here too. (Mapping is drift-guarded in + # unit-test-linux, which this job needs:, so no duplicate gate here.) + shell: bash + run: | + pkgs=$(python3 .github/scripts/addon_system_deps.py --platform apt) + if [ -n "$pkgs" ]; then + echo "→ addon system deps (apt): $pkgs" + apt-get update + apt-get install -y --no-install-recommends $pkgs + fi + - name: Install addon runtime deps (derived from requires_mod) # See unit-test-linux for rationale. The plugin registration test # subprocess-loads each addon's module, which imports its @@ -306,14 +617,70 @@ jobs: echo "no requires_mod declarations found" fi + - name: Validate requires_mod names against Gramps' dep gate + # See unit-test-linux for rationale. + shell: bash + run: | + python3 - <<'PY' + import ast, glob, re, subprocess, sys + from importlib.util import find_spec + + pat = re.compile(r"requires_mod\s*=\s*(\[[^\]]*\])") + names = set() + for f in glob.glob("*/*.gpr.py"): + try: + text = open(f, encoding="utf-8").read() + except OSError: + continue + for m in pat.finditer(text): + try: + names.update(ast.literal_eval(m.group(1))) + except (ValueError, SyntaxError): + pass + + bad = [] + for name in sorted(names): + installed = subprocess.run( + [sys.executable, "-m", "pip", "show", name], + capture_output=True, + ).returncode == 0 + if not installed: + print(f"~ {name} (pip-install failed earlier, skipping)") + continue + if find_spec(name) is None: + bad.append(name) + print(f"x {name} (pip-installed but find_spec returned None)") + else: + print(f"ok {name}") + + if bad: + print() + print(f"::error::Wrong requires_mod names: {bad}") + print("These pip-install but are not importable. requires_mod is") + print("consumed by gramps' check_mod() via find_spec(), so the") + print("importable module name is required (e.g. 'PIL', not 'Pillow').") + sys.exit(1) + PY + - name: Run plugin registration tests # shell: bash for consistency with the surrounding steps; the # current command uses no bashisms, but keeps this block safe # against future edits. Container default is /bin/sh → dash. + # + # gi_bootstrap on PYTHONPATH pins the GI versions (like the gramps GUI + # launcher) for this process AND the addon-module subprocesses this test + # spawns, so gramps.gui imports load GTK 3 without a PyGIWarning. + # + # NOT run under xvfb: this test only *loads* (imports) addon modules in + # subprocesses and tolerates load failures; it does not render. Giving it + # a display made an addon load hang on the (absent) AT-SPI accessibility + # bus until the per-load timeout. Imports that build a Gtk style context + # are exercised under xvfb in the unit/integration test runs instead. shell: bash env: - PYTHONPATH: . - run: python3 -m unittest discover -s tests -p "test_*.py" -t . -v + PYTHONPATH: .github/scripts/gi_bootstrap:. + run: | + python3 -m unittest discover -s tests -p "test_*.py" -t . -v - name: Run per-addon integration tests # shell: bash — see unit-test-linux for rationale; the @@ -323,16 +690,28 @@ jobs: env: PYTHONPATH: . run: | + is_active() { + local addon="$1" g + for g in "$addon"/*.gpr.py; do + [ -f "$g" ] || continue + grep -qE 'include_in_listing[[:space:]]*=[[:space:]]*True' "$g" && return 0 + grep -qE 'include_in_listing[[:space:]]*=' "$g" || return 0 + done + return 1 + } modules="" for f in */tests/test_integration*.py; do [ -f "$f" ] || continue + addon="${f%%/*}" + is_active "$addon" || continue mod="${f%.py}" mod="${mod//\//.}" modules="$modules $mod" done if [ -n "$modules" ]; then echo "Running per-addon integration tests:$modules" - python3 -m unittest -v $modules + xvfb-run -a --server-args="-screen 0 1920x1080x24" \ + python3 .github/scripts/run_addon_tests.py --platform apt --root . $modules else echo "No per-addon integration test modules found" fi @@ -342,9 +721,10 @@ jobs: # ----------------------------------------------------------------- build: name: Build + needs: setup runs-on: ubuntu-latest container: - image: ghcr.io/${{ github.repository }}/gramps-ci:gramps60 + image: ${{ needs.setup.outputs.ci_image }} steps: - uses: actions/checkout@v4 @@ -359,4 +739,4 @@ jobs: GRAMPSPATH: ${{ steps.gramps-path.outputs.path }} run: | mkdir -p ../download - python3 make.py gramps60 build all + python3 make.py "${{ needs.setup.outputs.branch_suffix }}" build all diff --git a/.github/workflows/docker-build.yml b/.github/workflows/docker-build.yml index 9289949b2..6ebd19ed5 100644 --- a/.github/workflows/docker-build.yml +++ b/.github/workflows/docker-build.yml @@ -2,9 +2,11 @@ name: Build Docker Images on: push: - branches: [maintenance/gramps60] - paths: - - '.github/docker/**' + # No paths filter on purpose. Buildx layer cache makes the rebuild + # a ~20-30 s no-op when nothing under .github/docker/ has changed, + # in exchange for guaranteeing gramps-ci: exists on the + # first push to any newly-created maintenance branch. + branches: [maintenance/gramps**] workflow_dispatch: env: @@ -32,14 +34,44 @@ jobs: - name: Set up Docker Buildx uses: docker/setup-buildx-action@v3 + - name: Compute branch parameters + # Derive the image-tag suffix, Gramps minor series, and upstream + # fallback SHA from the branch ref. Same validation as ci.yml's + # setup job: anything outside maintenance/grampsNN fails fast. + # The fallback SHA is the current tip of gramps-project/gramps + # at the matching maintenance branch; the Dockerfile only uses + # it when no gramps==${series}.* release exists on PyPI. The + # SHA is part of the buildx cache key so a moved upstream tip + # actually re-runs the install layer (otherwise gramps61 CI + # would stay on the same stale gramps revision build after + # build). + id: params + shell: bash + run: | + ref="${{ github.ref_name }}" + suffix="${ref#maintenance/}" + case "$suffix" in + gramps[0-9][0-9]) ;; + *) echo "::error::unexpected ref '$ref' (suffix '$suffix')"; exit 1 ;; + esac + # gramps60 → 6.0, gramps61 → 6.1, gramps62 → 6.2, … + series="${suffix:6:1}.${suffix:7}" + fallback_sha=$(git ls-remote https://github.com/gramps-project/gramps.git "refs/heads/maintenance/${suffix}" | awk '{print $1}') + if [ -z "$fallback_sha" ]; then + echo "::warning::upstream gramps-project/gramps has no maintenance/${suffix} branch; fallback path will fail if PyPI lacks gramps==${series}.*" + fi + echo "suffix=$suffix" >> "$GITHUB_OUTPUT" + echo "series=$series" >> "$GITHUB_OUTPUT" + echo "fallback_sha=$fallback_sha" >> "$GITHUB_OUTPUT" + - name: Docker metadata id: meta uses: docker/metadata-action@v5 with: images: ${{ env.REGISTRY }}/${{ env.REPO }}/gramps-ci tags: | - type=raw,value=gramps60 - type=sha,prefix=gramps60- + type=raw,value=${{ steps.params.outputs.suffix }} + type=sha,prefix=${{ steps.params.outputs.suffix }}- - name: Build and push gramps-ci uses: docker/build-push-action@v6 @@ -48,5 +80,8 @@ jobs: push: true tags: ${{ steps.meta.outputs.tags }} labels: ${{ steps.meta.outputs.labels }} + build-args: | + GRAMPS_SERIES=${{ steps.params.outputs.series }} + GRAMPS_FALLBACK_SHA=${{ steps.params.outputs.fallback_sha }} cache-from: type=gha cache-to: type=gha,mode=max diff --git a/ImportGramplet/tests/__init__.py b/ImportGramplet/tests/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/ImportGramplet/tests/test_xml_death_fallback.py b/ImportGramplet/tests/test_xml_death_fallback.py new file mode 100644 index 000000000..0c09e9028 --- /dev/null +++ b/ImportGramplet/tests/test_xml_death_fallback.py @@ -0,0 +1,192 @@ +# +# Gramps - a GTK+/GNOME based genealogy program +# +# Copyright (C) 2026 Gramps Development Team +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# + +""" +Repro-or-close test for bug 13420: Text Import gramplet death event +is not shown as the person's death fallback until re-validated in the +editor. + +The reporter's input was XML produced by a no-code Android tool with +UUID handles and longer-than-normal change-times; no developer +reproduced the symptom on conformant Gramps XML. Per the triage +verdict in `triage/batches/batch-03-confirmed-velocity/issue_13420.md` +the decisive test is: + + - Build CONFORMANT minimal Gramps XML (one person, Birth + Death + events both role Primary, standard Gramps handles) + - Import via the addon's `AtomicGrampsParser` (the wrapper the + ImportGramplet uses for its XML branch) + - Assert person.get_death_ref() returns the Death event ref + WITHOUT manual editor re-save + +If clean XML imports correctly → reporter's XML was malformed; close +13420 as cannot-reproduce. If clean XML still fails → real addon bug, +fix needed. + +The fallback-setting logic lives in gramps CORE at +`gramps/plugins/importer/importxml.py:1434-1473` (`start_eventref`) +and is shared between the standard XML import and AtomicGrampsParser +(AtomicGrampsParser only overrides `parse()` for an atomic DbTxn +wrapper, not the per-element handlers). The note above +`start_eventref` itself spells out the precondition: "We count here +on events being already parsed prior to parsing people or families. +This code will fail if this is not true." So the test fixture lists +events BEFORE people. +""" + +import os +import shutil +import sys +import tempfile +import unittest + +# AtomicGrampsParser pulls `from gi.repository import Gtk` at module +# load (via ImportGramplet.py). Pin Gtk to 3.0 before importing; skip +# cleanly if PyGObject / GTK 3 aren't available. +try: + import gi + + gi.require_version("Gtk", "3.0") + gi.require_version("Gdk", "3.0") +except (ImportError, ValueError) as err: + raise unittest.SkipTest("GTK 3.0 / PyGObject not available: %s" % err) + +# Make sure addon modules are importable from the parent directory. +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) + + +# Minimal CONFORMANT Gramps XML. Events listed before people, both +# events role Primary, standard Gramps handles (no UUIDs). Mirrors +# the structure in `example/gramps/example.gramps`. +CLEAN_XML = """ + + +
+ + + Bug 13420 Test + +
+ + + Birth + + Birth of Test, Person + + + Death + + Death of Test, Person + + + + + M + + Person + Test + + + + + + +""" + + +class TestImportGrampletDeathFallback(unittest.TestCase): + """Repro-or-close for bug 13420.""" + + def setUp(self): + # Per Sqlite/tests/test_sqlite.py: create a sqlite in-memory- + # equivalent db in a tempdir, import the XML, exercise. + from gramps.gen.db.utils import make_database # pylint: disable=import-outside-toplevel + + self.db_dir = tempfile.mkdtemp(prefix="bug13420_") + self.database = make_database("sqlite") + self.database.load(self.db_dir) + + def tearDown(self): + try: + self.database.close() + except Exception: # pylint: disable=broad-except + pass + shutil.rmtree(self.db_dir, ignore_errors=True) + + def test_clean_xml_imports_set_death_ref_via_atomic_parser(self): + """Clean XML through AtomicGrampsParser must set the + person's death_ref to the Death event. + + This is the repro-or-close decision point for bug 13420. + If it passes, the reporter's malformed XML (UUID handles) + was the cause; if it fails, the addon's atomic parser is + skipping the fallback wiring. + """ + from io import BytesIO # pylint: disable=import-outside-toplevel + import time # pylint: disable=import-outside-toplevel + from gramps.cli.user import User # pylint: disable=import-outside-toplevel + + # Importing the addon transitively imports Gtk -- this is why + # the test is gated above on gi.require_version("Gtk", "3.0"). + from ImportGramplet.ImportGramplet import AtomicGrampsParser # pylint: disable=import-outside-toplevel + + user = User() + change = int(time.time()) + parser = AtomicGrampsParser(self.database, user, change) + ifile = BytesIO(CLEAN_XML.encode("utf-8")) + parser.parse(ifile) + + # Look the person up by handle, not by gramps_id: core's importer + # normalises gramps_ids to the database's configured width (the XML + # "I00001" becomes "I0001" under the default "I%04d"), so a literal + # gramps_id lookup is brittle and fails on any default-config tree. + # Exactly one person was imported, so fetch it directly. + handles = list(self.database.get_person_handles()) + self.assertEqual( + len(handles), 1, "exactly one person must be imported" + ) + person = self.database.get_person_from_handle(handles[0]) + + death_ref = person.get_death_ref() + self.assertIsNotNone( + death_ref, + "Person.get_death_ref() must return the Death event ref " + "after clean-XML import via AtomicGrampsParser. If this " + "assertion fails, bug 13420 reproduces on conformant XML " + "and is a real addon defect.", + ) + + death_event = self.database.get_event_from_handle(death_ref.ref) + self.assertIsNotNone(death_event) + self.assertEqual(str(death_event.get_type()), "Death") + + # Birth ref must also be wired up -- same logic in core + # start_eventref. If birth is fine but death is not, the bug + # is type-specific; if both fail, it's the whole fallback + # path. + birth_ref = person.get_birth_ref() + self.assertIsNotNone(birth_ref) + birth_event = self.database.get_event_from_handle(birth_ref.ref) + self.assertEqual(str(birth_event.get_type()), "Birth") + + +if __name__ == "__main__": + unittest.main()