Add CI/CD pipeline with container-based testing (Feature Request ID 9393 in Mantis)#820
Conversation
|
Added Note to https://gramps-project.org/bugs/view.php?id=9393 requesting testing and feedback of this PR. |
|
Corrected to conform to the agents.md guidelines, including using unittest instead of pytest |
|
Can you provide the commands to test the CI workflow. And a usage of the test harness. |
|
How did the code work without the imports you have added |
|
As per comment on another PR, the CLAUDE.md file should not be here as it just duplicates Agents.md in the main Gramps repository meaning any changes etc. will have to be made in two place and it will be hard to keep the files in step. |
|
Thanks for fixing so many bugs and problems, but there are too many completely different changes in a single commit. Should they not be made in several commits(or even separate PRs). |
|
Yeah, I wanted more at one go, I can see that it might not be the right approach. @kulath , there are many Lint issues and I chose to fix them them instead of dropping the Lint. I thought this might get some pushback. We can do it the other way around - I'll remove the Linting and we can make issues out of them as preperation for activating them. |
517129d to
b677454
Compare
Introduce GitHub Actions CI inside a shared Docker image on ghcr.io, plus a native Windows runner for cross-platform unit-test coverage, and a shared unittest harness with Gramps-backed fixtures that verifies every addon registers, loads, and exposes valid plugin metadata. CI infrastructure ----------------- - .github/docker/gramps-ci/Dockerfile — Python 3.12 + Gramps 6.0 (pip) + PyGObject + GTK typelibs + xvfb/xauth + ruff, dbf, intltool, gettext, git. GTK lives in the base so addon modules that do `from gi.repository import Gtk` at load time are importable; xvfb and xauth are bundled for tests that actually render. - .github/workflows/docker-build.yml — rebuilds the image on .github/docker/** changes or via workflow_dispatch. - .github/workflows/ci.yml — seven jobs: lint (ruff E9/F63/F7/F82 + trailing whitespace), addon-structure (every addon has po/template.pot), compile-check (py_compile on every .py), unit-test-linux (container), unit-test-windows (native, conda+pip), integration-test (container with --init so xvfb-run does not hang), build (make.py gramps60 build all). - .github/environment.yml — hybrid conda+pip env for Windows. Gramps is not on conda-forge, so pygobject/gtk3 come from conda and gramps/orjson/dbf come from pip. Shared test harness ------------------- - tests/__init__.py — GPL header. - tests/gramps_test_env.py — sys.path / GRAMPS_RESOURCES bootstrap and two unittest base classes: GrampsTestCase (session-cached plugin manager + registry via setUpClass) and GrampsDbTestCase (same plus a fresh in-memory SQLite DB per test). - tests/test_plugin_registration.py — four unittest.TestCase classes covering plugin registration, subprocess-isolated module loading (crash-safe), required metadata (gramps_target_version=6.0, valid id/name/version), and import/export entry-function smoke tests. Gate policy ----------- All seven jobs run on every push and PR. Four are marked continue-on-error: true so they surface issues without blocking merges while the existing tree is cleaned up: - lint (~79 pre-existing ruff E9/F63/F7/F82 errors) - addon-structure (4 addons missing po/template.pot) - unit-test-linux (some addon test modules fail to import today) - unit-test-windows (same) compile-check, integration-test, and build are blocking from day one. Each non-blocking gate will be flipped to blocking in the same follow-up PR that fixes its underlying issues, so the tightening is incremental and visible in history. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
b677454 to
774a9ac
Compare
|
I've significantly reduced the scope of this PR based on @kulath's feedback about "too many completely different changes." It's now strictly CI infrastructure — 7 files, one commit. What's in the PR
Everything else from the earlier revision (lint fixes across ~29 addons, po/template.pot stubs for the four addons missing them, TMGimporter tests, the SurnameMappingGramplet.grp.py → .gpr.py rename, CLAUDE.md) has been removed and will be submitted as separate PRs. Gate Policy All seven jobs run on every push and PR. To avoid showering innocent contributors with red CI for pre-existing tree state, four jobs are marked continue-on-error: true so they surface issues without blocking merges: lint — non-blocking (~79 pre-existing ruff E9/F63/F7/F82 errors) In order to activate all gates, there will be some rework of existing addons needed |
|
@GaryGriffin — here are the commands for each CI job, lifted straight from ci.yml. Run from a checkout of addons-source with Gramps importable (either PYTHONPATH / GRAMPS_RESOURCES pointing at a Gramps checkout, or inside the CI container image). Lint — syntax/import errors plus trailing whitespace ruff check --select=E9,F63,F7,F82 --no-fix --exclude='.gpr.py' . Addon structure — every addon must have po/template.pot for gpr in /.gpr.py; do Compile check — py_compile on every .py find . -name '.py' ! -name '.gpr.py' ! -path './.git/' ! -path '/pycache/*' Per-addon unit tests — matches both the Linux and Windows jobs export PYTHONPATH=. Integration tests — plugin registration plus per-addon integration export PYTHONPATH=. per-addon integration uses the same loop as the unit block above |
Sorry, what I meant was, is there a way to invoke the CI process (not the commands that are invoked by the CI process) manually to test the github actions. I did try the invoked commands: Lint: I dont have ruff on my Mac. How would I invoke the CI lint process within github as an action. Addon Structure: if I manually invoke the commands on my Mac that is in the comment, it fails (due to escaping special chars needed for comments). I need to use:
Results: 4 issues, as you stated. I do not know if it is a requirement that the template.pot exists. If there are no translated strings, then there shouldn't need to be one, I think. Maybe I am wrong. Compile Check: same issue with escaping special chars. I think the gpr.py should actually be included. I need to use:
Results: 4 issues - Themes, Query, HouseTimelineGramplet, and lxml Integration Tests: failed with error |
|
Can you tell me what you are trying to do or understand? I'm not sure if I'm giving you the right answers. The tests run in containers stored on ghcr.io containing all the necessary tools. It's optimized for steady-state but is kind of a pain for this specific PR, because you can't really test it in this environment. The upside is that the image is stable across hundreds of normal PRs. It only churns when someone explicitly wants the CI environment to change.
You will have to make the images publically available for the PRs to work, but I don't think that's an issue. There will be some maintainance work needed to be done once you flip over to branch61 as default, I'd see what can be done to make it a bit easier then, assuming you like the current process. In order for you to verify how it works, take a poke at my forked repo where I've merged the PR. This is how you can do it
|
I should have been more explicit. I am trying to test the implementation before merging/publishing this PR. Understanding the resources needed and the frequency of activity is part of that. And making sure that it functions completely and as expected. I was hoping for something like a github command to activate the action so I can see it work. Given your complete description (much appreciated), I think I am going to have to yield to @Nick-Hall to review/merge/publish this one since it impacts the repo actions in ways far above my knowledge of github. For instance, I dont know if these are the right blocking/non-blocking decisions. Or if the scope should just be the Addon impacted by the PR rather than all Addons. |
|
@GaryGriffin - I thought of adding a manual button for everything, but that only works if it's merged with the default branch, lol |
The Dockerfile bakes in only `dbf`, but addons declare a wider set of Python deps in their .gpr.py `requires_mod` lists (networkx, psycopg2, pygraphviz, lxml, svgwrite, boto3, litellm, life_line_chart, psycopg). Without these installed, per-addon unit tests and the plugin- registration subprocess load fail with ImportError/NameError. Add a pre-test step to unit-test-linux, unit-test-windows, and integration-test that globs every *.gpr.py, extracts the requires_mod union via ast.literal_eval, and pip-installs each package one at a time. Per-package install (not batched) keeps a single build failure (pygraphviz without graphviz-dev, psycopg2 without libpq-dev) from aborting the rest — the affected addon's tests will skip or fail in isolation without blocking others. Mirrors Gramps' Addon Manager install path (gramps/gui/plug/_windows.py __on_install_clicked → req.install → gen/utils/requirements.py), keeping .gpr.py files as the single source of truth for addon deps. New addon deps do not need a parallel update to the Dockerfile or this workflow.
With ci.yml's auto-derive step in place (previous commit), dbf is installed at CI runtime from TMGimporter's .gpr.py requires_mod list. Keeping it baked into the Dockerfile and environment.yml in parallel would defeat the "single source of truth = .gpr.py" goal and drift the moment a new addon declares an additional dep. Remove dbf from both; leave the stable base (PyGObject, pycairo, Gramps, orjson, ruff) since those are not addon deps. Add a comment pointing readers at the auto-derive step so future edits do not re-bake runtime deps back in.
Root cause of "Unit Tests (Linux)" and "Integration Tests (Gramps)"
failures was not broken test modules — the steps never invoked
unittest. The container's default shell is /bin/sh (dash on
python:3.12-slim), and the inline scripts use bash-only parameter
expansions (${f%.py}, ${mod//\//.}) to build the dotted module list.
Dash fails with "Bad substitution" on the first such line; the rest
of the script never runs. continue-on-error: true masked this as a
generic job failure for two CI rounds.
Add "shell: bash" explicitly to:
- unit-test-linux / Run per-addon unit tests (bashisms)
- integration-test / Run per-addon integration tests (bashisms)
- integration-test / Run plugin registration tests (no bashisms today,
but consistent and future-proof)
Compile Check already sets shell: bash. Windows jobs inherit bash
via defaults.run at the job level. No other steps affected.
The Windows unit-test job hung on TMGimporter's DB-backed tests because
make_database("sqlite").load(":memory:", None) deadlocks under the
conda-forge GTK + pip Gramps combination. Rather than patch the hang,
introduce a filename convention so per-addon authors can declare OS
scope up front:
test_*.py general (every OS)
test_linux_*.py Linux-only
test_windows_*.py Windows-only
test_integration_*.py Linux-only, full-pipeline/DB-backed (pre-existing)
unit-test-linux skips test_windows_* and test_integration_*;
unit-test-windows skips test_linux_* and test_integration_*.
Applied to TMGimporter: the 13 DB-backed classes in tests/test_libtmg.py
move to tests/test_linux_libtmg.py (along with the _Rec/_table/_make_db/
_add_person/_MockUser helpers they use). The 7 pure-logic classes
(TestStripTmgCodes, TestTmgDateToGrampsDate, TestNumTo{Month,Date},
TestParseDate, TestRepoTypeFromName, TestUrlFromName) stay in
test_libtmg.py and will run on every OS.
Locally all 175 tests still pass via run-addon-unit.sh TMGimporter.
|
I just reworked the CI approach a bit to be able to differentiate between Base, Linux & Windows test specifically. Also added automated dependency loading for future unit tests. The Unit Tests currently fail because of an issue in Websearch, but that needs to be adressed seperately. |
Operational document for the gramps-project/addons-source maintainer
covering the new steps introduced by PR 820 + the branch-neutral
follow-up:
- One-time setup: make the gramps-ci GHCR package public (so fork
PR contributors can pull the image), and expect the first-push
race on maintenance/gramps60 immediately after merge.
- Creating a new maintenance branch: a plain `git branch && git
push` plus a one-time re-run of the failed CI jobs after the
image is built.
- When a Gramps minor release lands on PyPI: nothing to do (the
hybrid Dockerfile auto-detects); optional workflow_dispatch to
rebuild immediately.
- Diagnostic log markers: ::notice:: / ::warning:: / ::error::
annotations emitted from docker-build.yml and the Dockerfile,
with their respective root causes.
- Future-proofing knobs: upstream repo URL and GHCR tag retention,
only mentioned in case they ever become relevant.
The ci.yml header NOTE about the first-push race now points to this
runbook for the full picture (GHCR visibility, log markers, etc.).
Verified:
- YAML parses
- All TOC anchors in the new file resolve to existing headers
- ../MAINTAINERS.md and ../CONTRIBUTING.md (relative links in the
new file) both exist
- CONTRIBUTING.md#work-towards-a-merge anchor exists
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds tests/test_addon_dependencies.py, an independent detector for the
class of bug behind Mantis 13707 (WebConnect packs importing libwebconnect
without declaring depends_on=["libwebconnect"], so install fails when
libwebconnect is absent). Picked up automatically by the integration-test
job's `python3 -m unittest discover -s tests -p "test_*.py"` — no ci.yml
change.
Design
- Independent of Gramps' loader. Does NOT import gramps.gen.plug —
no PluginRegister, no BasePluginManager, no PluginData. Using
Gramps' loader would test addons through the very dependency
resolver whose leniency let 13707 ship, and would tie this test to
Gramps' internal, unstable API.
- Reads every *.gpr.py via an exec-shim of its own: a permissive
globals dict (LOAD_GLOBAL → __missing__ → sentinel, so plugin-type
constants like GRAMPLET / REPORT / TOOL never NameError) plus a
fake register() that records each call's kwargs. Builds the
addon-provided-module set and an id → {modules, depends_on,
requires_mod, dir} map.
- Isolated-load each plugin's registered module in a fresh
subprocess with sys.path = [target_dir] + dep_dirs (declared
depends_on ids resolved → their directories via the index). Uses
`python3 -I`, cwd=/tmp, PYTHONPATH stripped, and strips "" from
sys.path defensively — without those, PEP 420 implicit namespace
packages let sibling addons import as empty packages and silently
defeat the isolation. Gramps remains reachable from system
site-packages, as intended: the isolation is addon-from-addon, not
addon-from-Gramps.
- Pins GI namespace versions Gramps itself pins (Gtk 3.0, PangoCairo
1.0, etc.) before importing the addon. Not Gramps' loader — just
matching the runtime conditions a plugin is loaded under, so
addons whose top-level `from gi.repository import X` is
version-sensitive don't generate false (c) failures from ambiguous
GI defaults.
- Classifies failures: (a) missing name is an addon-provided module
not in this addon's depends_on — FINDING, fails the test;
(b) missing name is in this addon's requires_mod — environment
concern (PR 820's auto-derive owns it), ignored;
(c) anything else — logged separately, not a finding. A given
stderr may name multiple missing modules; (a) wins over (b) wins
over (c) so the highest-signal finding is reported.
Limitation, stated in the module docstring: catches undeclared
dependencies that manifest at LOAD time (top-level imports). MISSES
lazily-imported deps — a sibling addon imported inside a function not
called at module load. No false positives, not exhaustive.
Verified
- Synthetic positive/negative: with depends_on=["libwebconnect"]
stripped from a USWebConnectPack copy, detector flags bucket (a)
`libwebconnect`; with the declaration left in, rc=0 clean load.
(Both checks live under /tmp/, not committed; they validate the
classifier and isolation end-to-end.)
- Full tree run in gramps-ci-local:gramps60-test (PR 820's CI image,
Gramps 6.0.8): 187 plugins indexed, 159 pass, 0 (a), 7 (b)
ignored, 21 (c) logged. Every WebConnect pack on this branch
already declares depends_on=["libwebconnect"] — the 13707
declaration is in effect on gramps60, so the test is green here.
On gramps61 the situation may differ and the remediation round
will be informed by re-running this detector there.
This commit is the detector + the regression check. It does NOT apply
any depends_on fixes — the WebConnect remediation is a separate later
round on maintenance/gramps61.
Issue #13707.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
@GaryGriffin - I added a quality gate that checks for internal dependencies to avoid errors like 13707; the good news is that isn't a often recurring issue. |
`test_target_version_is_6_0` hardcoded "6.0" in the assertion and its name, so the same harness copied to maintenance/gramps61 (via the bootstrap commit 458ebd0) flagged every 6.1-targeted addon as mistargeted. Switch the prefix to ``f"{VERSION_TUPLE[0]}.{VERSION_TUPLE[1]}"``, read from the loaded ``gramps.version``, so the same test works on every maintenance branch — the CI image's Gramps install is always the series that branch's addons should target, which is exactly what we want to assert. Rename the method to ``test_target_version_matches_gramps_install`` to match the new semantics, and update the failure message to include the expected prefix instead of a fixed "6.0". Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three related fixes for the same addon:
1) Plugin descriptor rename:
`SurnameMappingGramplet.grp.py` -> `SurnameMappingGramplet.gpr.py`.
The file is named `.grp.py` (typo). Gramps loads only `*.gpr.py`
plugin descriptors, so the addon has never been registered with
Gramps — it does not appear in the Add Gramplet menu today.
2) Python 2 / Gramps 3 era pre-namespace imports:
import gtk # PyGTK, lowercase
from gen.plug import Gramplet
Neither resolves on Python 3 / Gramps 5+. Once the descriptor is
renamed and Gramps actually tries to register the addon, the
module would fail import with `ModuleNotFoundError: No module
named 'gen'`. Migrate to:
from gi.repository import Gtk
from gramps.gen.plug import Gramplet
Rename every `gtk.*` reference in the file (25 sites) to
`Gtk.*` to match.
3) Python 2 `unicode()` usages:
self.dbstate.db.set_name_group_mapping(unicode(surname), ...)
`unicode` is a Py2 builtin removed in Py3 (the default string
type is already Unicode; it's just `str`). Replace all eight
`unicode(...)` calls with `str(...)`.
PR #820's body called out items (1) and (3) under "Renamed
SurnameMappingGramplet.grp.py -> .gpr.py" and "Py2 leftovers:
... unicode() -> str()". Item (2) was the gap an earlier revision
of this PR missed — the .grp.py -> .gpr.py rename without the
import migration would have made the addon visible to Gramps but
still unloadable.
Behavioural impact:
- The rename makes Gramps register a previously-invisible plugin.
Users with this addon installed will see "Surname Mapping"
appear in their Gramplet bar — the original authorial intent.
- The import migration unblocks module loading on Py3.
- The unicode -> str swap is a no-op on Py3 (both call-site
results are identical) and fixes the latent NameError on the
on-edit/remove paths.
Out of scope for this PR (separate follow-up): the addon's
`init()` / `build_gui()` methods still use several PyGTK-era
GTK 2 APIs (`Gtk.Toolbar.insert_stock`, `Gtk.STOCK_*`,
`Gtk.DIALOG_MODAL`, the deprecated `Gtk.Table.attach(... xoptions=...)`
form, etc.) that don't work as-is on modern PyGObject. Those only
fire when a user actually opens the gramplet — they don't block
plugin registration. Fixing them needs a wider GTK 2 -> GTK 3 API
audit best done as a separate PR.
Add a regression test in `SurnameMappingGramplet/tests/` that
imports the module via the explicit submodule path (addon dir and
impl module share the name — namespace-package shadowing, same
trap as libaccess; see gramps bug 0012691 family). Asserts the
`SurnameMappingGramplet` class is a Gramplet subclass.
Verified via the testbed's `run-addon-unit.sh SurnameMappingGramplet`:
Before fix: ModuleNotFoundError: No module named 'gen' (FAIL)
After fix: 1 test, OK
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
This push adds four CI fixes so the per-addon test pipeline actually exercises addons that need a display or system packages, instead of silently skipping them and staying green. Each change is small and independent — here is what each solves and why it is needed. 1. Install addon system dependencies (
|
… skips The CI ran per-addon tests but quietly no-op'd for any GUI/graphviz addon: the image shipped no goocanvas/osm-gps-map/graphviz, tests were never run under xvfb, nothing pinned the GTK version before gramps.gui imports, and an all-skipped module still exited 0. Close all four gaps. System deps (requires_gi / requires_exe): - .github/scripts/addon_system_deps.py is the single source mapping each declared GI typelib / executable to its per-platform package (apt and conda), and scans the .gpr.py files. The Linux jobs derive the apt set at runtime and install it (the container runs as root; the image build context excludes addons-source so it cannot bake them). A drift-guard step fails if an addon declares a dep with no map entry — the system-dep analogue of the existing requires_mod find_spec gate. - The GTK 3 addon libs (goocanvas, osm-gps-map, gexiv2) are not on conda-forge, so the map records conda=None for them; the Windows lane installs only the available subset (graphviz) and those addons skip there by platform necessity. Display + GI bootstrap: - Per-addon test runs are wrapped in xvfb-run (addons that build a Gtk style context at import need a display, else a hard Gtk-ERROR). - .github/scripts/run_addon_tests.py and a gi_bootstrap sitecustomize pin Pango/PangoCairo/Gtk the way gramps/gui/grampsgui.py does, so a test importing a gramps.gui.* module loads GTK 3 instead of warning / risking GTK 4. The sitecustomize covers the subprocess-loading plugin registration test via PYTHONPATH. Honest skip accounting: - run_addon_tests.py replaces bare `unittest` for the per-addon runs and FAILS a wholly-skipped module, UNLESS the addon's declared system deps are unavailable on the platform (e.g. goocanvas on conda), in which case the skip is expected and tolerated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
aac468e to
7f8a839
Compare
|
Follow-up: per-module timeout (run_addon_tests). Validating the above against
Re-validated on the fork CI: |
conda-forge has no gramps 6.1 yet, so environment.yml's "gramps<6.1" pin resolves to 6.0.x on every branch. On maintenance/gramps61 the Windows lane therefore validates addons against gramps 6.0.x, not the branch's series. Git-building the matching gramps in the conda env (as the Linux CI image does) is not viable: gramps' own Windows build targets MSYS2 UCRT64, not conda, and the wheel build fails in build_intl when `msgfmt --xml` cannot locate the shared-mime-info/appstream ITS rules absent from the conda env. Add a non-failing "Report gramps-vs-branch series" step to the Windows job that surfaces the caveat in the log on every gramps61+ run. Addon tests needing series-exact gramps behaviour skip themselves on Windows (e.g. TMGimporter's real-DB import tests) and run on the Linux lane, which git-builds the branch's exact gramps. When 6.1 reaches conda-forge the pin picks it up and the caveat disappears. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per the maintainer's earlier review ("too many completely different changes
in a single commit"), this PR should carry only the CI infrastructure
(workflows, image, the tests/ harness, run_addon_tests.py /
addon_system_deps.py / gi_bootstrap). Two standalone pieces that had
re-grown here move to their own follow-up PRs:
- tests/test_addon_dependencies.py — the undeclared-depends_on dependency
detector (Mantis-13707 class), a standalone feature; ships in its own PR
with its per-module isolated-load loop parallelised.
- the TMGimporter test rename (test_libtmg.py -> test_linux_libtmg.py) — a
per-addon test edit; ships in its own PR as the test_<os>_* convention
demonstrator.
This commit removes both from gramps-project#820, leaving only the CI infrastructure.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Split out of the CI-pipeline PR (gramps-project#820) per the one-change-per-PR review: this is a standalone feature, not CI infrastructure. Detects addons that import another addon's module without declaring it in depends_on (the Mantis-13707 class), by loading each registered module in an isolated subprocess and classifying the failure. The isolated loads run in a bounded thread pool (configurable; default scales with CPUs) instead of one 30s-timeout subprocess per module serially — so the detector's wall-clock stays bounded on the full ~144-addon set (review finding R-F). Logic split into tests/addon_dependencies.py (the engine) + its test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Split out of the CI-pipeline PR (gramps-project#820) per the one-change-per-PR review: this is a standalone feature, not CI infrastructure. Detects addons that import another addon's module without declaring it in depends_on (the Mantis-13707 class), by loading each registered module in an isolated subprocess and classifying the failure. The isolated loads run in a bounded thread pool (configurable; default scales with CPUs) instead of one 30s-timeout subprocess per module serially — so the detector's wall-clock stays bounded on the full ~144-addon set (review finding R-F). Logic split into tests/addon_dependencies.py (the engine) + its test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Split out of the CI-pipeline PR (#820) per the one-change-per-PR review: this is a per-addon test edit, not CI infrastructure. Renames TMGimporter/tests/test_libtmg.py -> test_linux_libtmg.py to follow the test_<os>_* naming convention the CI harness uses to scope per-OS test runs (the convention #820 introduces; this is its demonstrator). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This PR added tests/__init__.py as a plain package marker; maintenance/gramps60 independently gained a tests/__init__.py that pins the GTK/GDK 3.0 introspection versions for the repo-root test suite. That add/add is the PR's only merge conflict. Adopt the maintenance/gramps60 version verbatim — the GI-version pin is the functional, canonical content and supersedes the bare marker — so the file is byte-identical on both sides and the conflict resolves with no merge commit.
The addon CI suite has a load check, test_load_all_addon_modules, that collects every addon that fails to load into a "hard failures" list but then only logs a warning about them. Its only assertion is that at least one plugin was found, so a real addon that fails to load lets CI pass green while the failure scrolls by unread. The check named a failure class it could never fail on. Make that check gate, the way its two siblings in the same file already do: TestImportPluginSmoke and TestExportPluginSmoke both self.fail on their findings. A non-dependency hard load failure now fails the test directly; dependency skips and subprocess crashes (typically a missing display server in CI) stay advisory and are still only logged. A regression test drives the real production method with a synthetic always-failing addon injected at its load seams and asserts the run now fails rather than passing silently.
The CI image purged the build toolchain (gcc, python3-dev, pkg-config) right after the Gramps install, and the source-built addon requires_mod that have no wheel on a CI platform (pygraphviz, psycopg2, psycopg) had no system package provisioned on either lane. So pip installing those at CI runtime failed for a missing compiler or header/library, the failure was swallowed by the install step's "|| echo ... (continuing)", and an addon that hard-imports them (e.g. NetworkChart, the PostgreSQL backends) ran a silently degraded suite while the job still reported green. Keep the build toolchain in the image and derive each source-built module's system package from the addons' .gpr.py through the same single-source map as requires_gi/requires_exe, installing it before the runtime pip step on both lanes: the apt lane gets the -dev headers/libpq and compiles the extension, and the conda lane installs the prebuilt conda-forge package so the Windows suites run instead of skipping. A declared requires_mod must now be classified as wheel-only or source-built, and the mapping drift guard fails CI on any module that is neither, so a newly added source-built dependency cannot quietly reopen the coverage gap. The affected addon suites either run with their deps present or fail honestly when a dep cannot be provisioned -- never a silent green.
The addon CI workflow inlined the same requires_mod derivation as an identical Python heredoc in three jobs (unit-test-linux, unit-test-windows, integration-test) and the is_active() bash filter verbatim across six job steps, with the find_spec name-gate triplicated alongside. A one-line change to any of them meant a three- to six-site edit, and the copies could silently diverge. Move the requires_mod derivation and the find_spec validator into a single .github/scripts/addon_python_deps.py that every job calls (--install-list / --check-resolves), and the is_active() filter into .github/scripts/active_addons.sh that each filtering step sources. The .gpr.py files stay the single source of truth, so the derived module list and the active-addon set are unchanged. The module also centralises the import-to-distribution install map (PIL -> Pillow) on the install side only; the find_spec gate keeps validating the raw declared import name, exactly as Gramps does at runtime via check_mod(). No requires_mod=["PIL"] exists in the tree today, so the derived install list is byte-identical to the old heredoc output and the change is behaviour-preserving. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Note that if/when the mini-installer from gramps-project/gramps#2308 is merged, that it uses slightly different names (pip names rather than import names). |
|
I plan to review #2308 next and it will probably be merged. |
…#2308
The CI install step maps requires_mod import names to PyPI distribution names
(e.g. PIL→Pillow). gramps PR #2308 ("PyPI wheel installer for addon module
dependencies") adds the authoritative table _IMPORT_TO_PYPI in
gramps/gen/utils/pypi.py with ~12 entries; this copy had one. Once #2308 merges,
an addon declaring e.g. cv2 / yaml / sklearn in requires_mod installs correctly
in Gramps but this CI's install union would pip install the bare import name and
fail the addon-unit step.
Mirror the 11 missing entries now (all current, correct PyPI names — valid
independent of #2308) so CI installs the same distribution Gramps does. Closes
the drift flagged on PR 820 (upstream PR 2308). No live addon declares an
at-risk name today, so this is pre-emptive. When #2308 merges, single-source the
table from gramps' module instead of hand-mirroring it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
@dsblank - I've added some stuff on that account to PR 820 now |
This is part of the answer to issue 9393, using the "X framebuffer" approach (Option 1 of the feature request).
Scope
This PR is strictly CI infrastructure. After review feedback from @kulath (2026-04-18) — "too many completely different changes in a single commit" — all per-addon code/lint/structure work that was originally bundled here has been split into one-PR-per-addon submissions; see the Companion PRs section at the bottom. This PR adds only the workflow files, the container image, the shared CI scripts, and a small shared test harness.
Thanks also to @GaryGriffin for the early testing feedback on bug 9393 and for asking the questions that led to the standalone per-job reproduction commands below.
What's in the PR
CI infrastructure
.github/docker/gramps-ci/Dockerfile— Python 3.12 + Gramps 6.0 + PyGObject + GTK typelibs (incl.gir1.2-gexiv2-0.10for EditExifMetadata / PhotoTaggingGramplet) + xvfb/xauth + ruff, intltool, gettext, git. GTK lives in the base image so addon modules that dofrom gi.repository import Gtkat module load are importable; xvfb/xauth are bundled for tests that actually render. Gramps is installed from PyPI for released series, or from a SHA-pinnedgramps-project/gramps@maintenance/grampsNNsnapshot for an unreleased branch (the series is parameterised via theGRAMPS_SERIESbuild arg)..github/workflows/docker-build.yml— rebuilds the image on.github/docker/**changes or viaworkflow_dispatch, deriving the image tag and Gramps series from the branch ref. Default-branch-only; gates on GHCR write access..github/workflows/ci.yml— eight jobs:make.pyseries suffix from the branch ref (maintenance/grampsNN), so the rest of the matrix is branch-neutral.ruff --select=E9,F63,F7,F82+ trailing-whitespace check (usesgrep -Pso\tmatches a real tab, not a literalt— see commit205b21cfor the regex bug fix).po/template.pot.python3 -m py_compileon every.py.<addon>/tests/test_*.py(skipstest_windows_*/test_integration_*) viarun_addon_tests.py(GI bootstrap + per-module timeout + honest skip accounting).tests/test_plugin_registration.py+<addon>/tests/test_integration_*.py, Linux-only, container--initsoxvfb-rundoesn't hang.make.py gramps60 build all..github/environment.yml— hybrid conda+pip environment for Windows. Gramps isn't on conda-forge, sopygobject/gtk3come from conda; only the stable base (gramps+orjson) comes from pip. Addon runtime deps (dbf,networkx,lxml, …) are no longer pinned here —ci.ymlauto-derives and installs them at runtime from each.gpr.py'srequires_mod(single source of truth;dbfdropped in commit8d2654a).Shared CI scripts
.github/scripts/addon_system_deps.py— single source of truth for addon system dependencies. Maps eachrequires_gitypelib /requires_exeexecutable declared in a.gpr.pyto its package on each CI platform (apt vs conda), and scans the tree soci.ymlderives the install list from one place instead of a hand-kept list. Records platform asymmetry (the GTK 3 libs — goocanvas, osm-gps-map, gexiv2 — exist on apt but not conda-forge). Pure stdlib..github/scripts/run_addon_tests.py— per-addon unit/integration runner that replaces a barepython -m unittest: it (1) bootstraps the GI versions the Gramps GUI launcher pins (Pango/PangoCairo/Gtk) before anygramps.guiimport, (2) runs each module in a subprocess with a wall-clock timeout so a hung test can't stall the job, and (3) FAILs a wholly-skipped module unless the addon's declared system deps are genuinely unavailable on the platform — so a silent skip can't read as a pass..github/scripts/gi_bootstrap/sitecustomize.py— the same GI version pin as asitecustomizeshim, placed onPYTHONPATHfor the discover-/subprocess-loading steps (e.g. plugin registration) so every spawned interpreter inherits the bootstrap.Shared test harness
tests/gramps_test_env.py—GrampsTestCase/GrampsDbTestCasebase classes (stdlibunittest, mirroring upstream Gramps' own test style).tests/test_plugin_registration.py— registers every addon in a subprocess (crash-safe), verifiesgramps_target_version=6.0plus valid id/name/version, smoke-tests import/export entry functions.tests/__init__.py.Docs
CONTRIBUTING.md— a short note that PRs now run automated CI checks, and that a green check on an unreleasedmaintenance/grampsNNbranch means the addon works against that branch tip (a SHA-pinned snapshot), not against a tagged release..github/CI-MAINTAINER.md— operational runbook for agramps-project/addons-sourcemaintainer: one-time setup when this PR merges, and how to stand up a new maintenance branch. The pipeline is otherwise self-driving.Gate policy
PR-blocking checks should only fire on issues the PR's own diff can cause. Every test here runs gramps or addon code, so the policy is:
continue-on-error: false): Lint, Compile Check, Unit Tests (Linux), Unit Tests (Windows), Integration Tests, Build.continue-on-error: true): Addon Structure — the only advisory job.Lint and both Unit-test jobs were advisory while their pre-existing backlog was being cleared by the companion PRs; that backlog is now cleared on
maintenance/gramps60, so they were flipped to blocking (lint ind265612, both unit-test jobs in0dd3f1b). The single remaining advisory is Addon Structure, which stays non-blocking until the four addons missingpo/template.potare fixed in a follow-up PR — flip it off in that PR.Commits
774a9acc6aa10erequires_modin.gpr.py.8d2654adbffrom image/env — installed via auto-derive now.28febdcshell: bashon unit-test-linux + integration-test steps (fixes DashBad substitution).715e71dtest_linux_*/test_windows_*/test_integration_*).dd0fd38gir1.2-gexiv2-0.10typelib to image (companion to #878, #880).205b21c[ \t]to PCRE. The previous BRE pattern matched any line ending int/\/[/]/space (~3 000 false positives across ~430 files); PCRE (-P) recognises\tas a tab — real count 597 lines across 21 files.f5907af.gpr.pydeclaresinclude_in_listing=False.ff215acrequires_modnames againstfind_spec(Pillow/PIL trap).d265612maintenance/gramps60.0dd3f1b9a91d89make.pyargument from the branch ref.abefdbe9927626GRAMPS_SERIESbuild arg.894e48473d75ed3b2a947.github/CI-MAINTAINER.md).ad2815cdepends_on.bc8df217f8a839146649106b95bcLocal reproduction
This PR can't be exercised against
gramps-project/addons-sourcedirectly (workflows live in the PR), but the per-job commands are reproducible standalone:A complete end-to-end test is also available by forking to a personal repo and pushing a branch — the workflows fire because the workflow files come along with the fork. See the comment thread for the step-by-step walkthrough.
Companion PRs
Code/lint/structure work that was originally bundled here is now submitted as one-PR-per-addon. Tracker:
displayer, JSON: log repr(obj) on unrecognized object instead of undefined data #871 JSONdata, LifeLineChartView: fix tuple-in-if outer condition for tooltip cache #872 LifeLineChartView tuple-in-if, libaccess: add missing value parameter to Person.gramps_id setter lambda #873 libaccess lambda (thanks to @GaryGriffin for catching that the first version of libaccess: add missing value parameter to Person.gramps_id setter lambda #873 was missing the actual code edit), lxml: drop self.uistate.window from module-level ErrorDialog calls #874 lxml module-levelself.webreport/common.pyno-ICU fallback) — both required to unblock plugin-registration smoke tests in this PR.As of HEAD the lint and unit-test backlogs are cleared and those gates are blocking; once the remaining
po/template.potgaps land, Addon Structure can be promoted to blocking too.🤖 Generated with Claude Code