From c6aa10e0c17575e73094364f7926b7f778587b3f Mon Sep 17 00:00:00 2001 From: Eduard Ralph Date: Mon, 20 Apr 2026 19:52:11 +0200 Subject: [PATCH 1/3] CI: auto-derive addon pip deps from requires_mod in .gpr.py MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Dockerfile bakes in only `dbf`, but addons declare a wider set of Python deps in their .gpr.py `requires_mod` lists (networkx, psycopg2, pygraphviz, lxml, svgwrite, boto3, litellm, life_line_chart, psycopg). Without these installed, per-addon unit tests and the plugin- registration subprocess load fail with ImportError/NameError. Add a pre-test step to unit-test-linux, unit-test-windows, and integration-test that globs every *.gpr.py, extracts the requires_mod union via ast.literal_eval, and pip-installs each package one at a time. Per-package install (not batched) keeps a single build failure (pygraphviz without graphviz-dev, psycopg2 without libpq-dev) from aborting the rest — the affected addon's tests will skip or fail in isolation without blocking others. Mirrors Gramps' Addon Manager install path (gramps/gui/plug/_windows.py __on_install_clicked → req.install → gen/utils/requirements.py), keeping .gpr.py files as the single source of truth for addon deps. New addon deps do not need a parallel update to the Dockerfile or this workflow. --- .github/workflows/ci.yml | 100 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 100 insertions(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 92bd05ff6..cdc67a128 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -100,6 +100,44 @@ jobs: steps: - uses: actions/checkout@v4 + - name: Install addon runtime deps (derived from requires_mod) + # Auto-derive the union of requires_mod across every .gpr.py in + # the repo. Mirrors Gramps' Addon Manager install path + # (gramps/gui/plug/_windows.py __on_install_clicked → req.install → + # gen/utils/requirements.py). Keeps .gpr.py files as the single + # source of truth for addon deps — no parallel list to maintain + # in the image or workflow. Best-effort: a package needing exotic + # system deps (pygraphviz → graphviz-dev, psycopg2 → libpq-dev) + # may fail here; the affected addon's tests will skip or fail in + # isolation without blocking the rest. + shell: bash + run: | + addon_mods=$(python3 - <<'PY' + import ast, glob, re + pat = re.compile(r"requires_mod\s*=\s*(\[[^\]]*\])") + mods = set() + for f in glob.glob("*/*.gpr.py"): + try: + text = open(f, encoding="utf-8").read() + except OSError: + continue + for m in pat.finditer(text): + try: + mods.update(ast.literal_eval(m.group(1))) + except (ValueError, SyntaxError): + pass + print(" ".join(sorted(mods))) + PY + ) + if [ -n "$addon_mods" ]; then + echo "→ addon deps: $addon_mods" + for mod in $addon_mods; do + pip install "$mod" || echo "× $mod failed to install (continuing)" + done + else + echo "no requires_mod declarations found" + fi + - name: Run per-addon unit tests env: PYTHONPATH: . @@ -152,6 +190,36 @@ jobs: mamba list | head -30 python -c "import gramps, gi; print('deps OK')" + - name: Install addon runtime deps (derived from requires_mod) + # See unit-test-linux for rationale. Uses `python` (conda-forge + # env) to match the surrounding Windows job style. + run: | + addon_mods=$(python - <<'PY' + import ast, glob, re + pat = re.compile(r"requires_mod\s*=\s*(\[[^\]]*\])") + mods = set() + for f in glob.glob("*/*.gpr.py"): + try: + text = open(f, encoding="utf-8").read() + except OSError: + continue + for m in pat.finditer(text): + try: + mods.update(ast.literal_eval(m.group(1))) + except (ValueError, SyntaxError): + pass + print(" ".join(sorted(mods))) + PY + ) + if [ -n "$addon_mods" ]; then + echo "→ addon deps: $addon_mods" + for mod in $addon_mods; do + pip install "$mod" || echo "× $mod failed to install (continuing)" + done + else + echo "no requires_mod declarations found" + fi + - name: Run per-addon unit tests env: PYTHONPATH: . @@ -189,6 +257,38 @@ jobs: steps: - uses: actions/checkout@v4 + - name: Install addon runtime deps (derived from requires_mod) + # See unit-test-linux for rationale. The plugin registration test + # subprocess-loads each addon's module, which imports its + # requires_mod packages. + shell: bash + run: | + addon_mods=$(python3 - <<'PY' + import ast, glob, re + pat = re.compile(r"requires_mod\s*=\s*(\[[^\]]*\])") + mods = set() + for f in glob.glob("*/*.gpr.py"): + try: + text = open(f, encoding="utf-8").read() + except OSError: + continue + for m in pat.finditer(text): + try: + mods.update(ast.literal_eval(m.group(1))) + except (ValueError, SyntaxError): + pass + print(" ".join(sorted(mods))) + PY + ) + if [ -n "$addon_mods" ]; then + echo "→ addon deps: $addon_mods" + for mod in $addon_mods; do + pip install "$mod" || echo "× $mod failed to install (continuing)" + done + else + echo "no requires_mod declarations found" + fi + - name: Run plugin registration tests env: PYTHONPATH: . From 8d2654a04b0d4c0f19497ba8f49221f2b1005aa6 Mon Sep 17 00:00:00 2001 From: Eduard Ralph Date: Mon, 20 Apr 2026 20:19:00 +0200 Subject: [PATCH 2/3] =?UTF-8?q?CI:=20remove=20dbf=20from=20image/env=20?= =?UTF-8?q?=E2=80=94=20installed=20via=20auto-derive=20now?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit With ci.yml's auto-derive step in place (previous commit), dbf is installed at CI runtime from TMGimporter's .gpr.py requires_mod list. Keeping it baked into the Dockerfile and environment.yml in parallel would defeat the "single source of truth = .gpr.py" goal and drift the moment a new addon declares an additional dep. Remove dbf from both; leave the stable base (PyGObject, pycairo, Gramps, orjson, ruff) since those are not addon deps. Add a comment pointing readers at the auto-derive step so future edits do not re-bake runtime deps back in. --- .github/docker/gramps-ci/Dockerfile | 8 ++++++-- .github/environment.yml | 5 ++++- 2 files changed, 10 insertions(+), 3 deletions(-) diff --git a/.github/docker/gramps-ci/Dockerfile b/.github/docker/gramps-ci/Dockerfile index 3ef3b8945..d5102ac99 100644 --- a/.github/docker/gramps-ci/Dockerfile +++ b/.github/docker/gramps-ci/Dockerfile @@ -37,13 +37,17 @@ RUN apt-get update && apt-get install -y --no-install-recommends \ xauth \ && rm -rf /var/lib/apt/lists/* +# Addon runtime deps (dbf, networkx, lxml, svgwrite, boto3, etc.) are +# NOT baked in here — ci.yml's "Install addon runtime deps (derived from +# requires_mod)" step pip-installs them at CI runtime from every +# .gpr.py's requires_mod list, matching what Gramps' Addon Manager does +# for an end user. Keeps .gpr.py the single source of truth. RUN pip install --no-cache-dir \ PyGObject \ pycairo \ "gramps>=6.0,<6.1" \ orjson \ - ruff \ - dbf + ruff RUN apt-get purge -y gcc python3-dev pkg-config && apt-get autoremove -y diff --git a/.github/environment.yml b/.github/environment.yml index 9ca19f57f..dd9e85603 100644 --- a/.github/environment.yml +++ b/.github/environment.yml @@ -6,7 +6,10 @@ dependencies: - pygobject - gtk3 - pip + # Addon runtime deps (dbf, networkx, lxml, svgwrite, boto3, etc.) are + # installed at CI runtime by ci.yml's auto-derive step from .gpr.py + # requires_mod — single source of truth. Keep only the stable base + # here (Gramps + orjson for plugin registration). - pip: - "gramps>=6.0,<6.1" - orjson - - dbf From 28febdcd2becbd31ff2f75df98c355303380531a Mon Sep 17 00:00:00 2001 From: Eduard Ralph Date: Mon, 20 Apr 2026 20:40:46 +0200 Subject: [PATCH 3/3] CI: add shell: bash to unit-test-linux + integration-test steps MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Root cause of "Unit Tests (Linux)" and "Integration Tests (Gramps)" failures was not broken test modules — the steps never invoked unittest. The container's default shell is /bin/sh (dash on python:3.12-slim), and the inline scripts use bash-only parameter expansions (${f%.py}, ${mod//\//.}) to build the dotted module list. Dash fails with "Bad substitution" on the first such line; the rest of the script never runs. continue-on-error: true masked this as a generic job failure for two CI rounds. Add "shell: bash" explicitly to: - unit-test-linux / Run per-addon unit tests (bashisms) - integration-test / Run per-addon integration tests (bashisms) - integration-test / Run plugin registration tests (no bashisms today, but consistent and future-proof) Compile Check already sets shell: bash. Windows jobs inherit bash via defaults.run at the job level. No other steps affected. --- .github/workflows/ci.yml | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index cdc67a128..0787cf355 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -139,6 +139,11 @@ jobs: fi - name: Run per-addon unit tests + # shell: bash — the container's default shell is /bin/sh + # (dash on python:3.12-slim), which does not support the + # ${var//pattern/repl} and ${var%.py} parameter expansions + # used below. Falls silently under continue-on-error. + shell: bash env: PYTHONPATH: . run: | @@ -290,11 +295,19 @@ jobs: fi - name: Run plugin registration tests + # shell: bash for consistency with the surrounding steps; the + # current command uses no bashisms, but keeps this block safe + # against future edits. Container default is /bin/sh → dash. + shell: bash env: PYTHONPATH: . run: python3 -m unittest discover -s tests -p "test_*.py" -t . -v - name: Run per-addon integration tests + # shell: bash — see unit-test-linux for rationale; the + # ${var//pattern/repl} and ${var%.py} expansions below are + # bash-only. + shell: bash env: PYTHONPATH: . run: |