Skip to content

TMGimporter: real-DB import tests skipped on Windows (conda lane can't run matching gramps) #32

@eduralph

Description

@eduralph

Summary

TMGimporter/tests/test_libtmg.py's real-DB import tests (the 13 classes that
drive a real in-memory Gramps database via make_database + db.load) are
skipped on the Windows CI lane. They run and pass on Linux. This issue
records why and what it would take to close the gap, so we can revisit when
the blocker clears.

This is a tracking issue, not a bug in TMGimporter itself — the tests are
correct (157 pass on Linux). The gap is environmental.

Root cause

The CI is branch-neutral and the two lanes get gramps differently:

  • Linux pulls the gramps-ci:<suffix> image, whose Dockerfile does
    PyPI-first, then a git build of gramps-project/gramps@maintenance/grampsNN.
    6.1 isn't on PyPI, so it builds real gramps 6.1. The tests pass.
  • Windows uses conda-forge (.github/environment.yml, pinned
    gramps>=6.0,<6.1). conda-forge has no gramps 6.1, so it resolves to the
    newest in range — gramps 6.0.8.

So on maintenance/gramps61 the Windows lane validates gramps61 addon code
against gramps 6.0.8. TMGimporter is the only addon that exercises a real
Gramps DB (every other addon test mocks DbTxn), so it's the only one to expose
the mismatch — its import_* path hangs against the mismatched 6.0.8 DB API.

Why we can't just git-build gramps 6.1 on the conda-Windows lane

Mirroring the Linux image's git-fallback was attempted and does not work on
conda-Windows. With gettext-tools (msgfmt) installed, the wheel build gets all
the way through compiling the 43 .po files, then aborts in gramps' build hook:

msgfmt: cannot locate ITS rules for ...\data\org.gramps_project.Gramps.xml.in
ERROR: ...\build\share\mime\packages\org.gramps_project.Gramps.xml was not merged

build_intl calls msgfmt --xml to translate the mime/metainfo XML, which needs
the ITS rules shipped by shared-mime-info/appstream — present on Linux,
absent in the conda env. gramps aborts the whole wheel when that merge fails,
even though those GNOME-desktop files are irrelevant to running unit tests.

Deeper reason: gramps' own Windows build targets MSYS2 UCRT64, not conda
(see gramps' windows-aio.yml). conda-forge Windows simply isn't gramps' build
environment, and conda-forge has no prebuilt 6.1 to fall back on.

Current mitigation (landed locally, pending PRs)

  1. Addon (maintenance/gramps61): raise unittest.SkipTest from the shared
    _make_db() helper on sys.platform == "win32". All 13 DB-backed classes
    skip in one place; the 8 pure-function classes (code stripping, date
    conversion, name parsing) still run on Windows.
  2. Pipeline (PR Add CI/CD pipeline with container-based testing (Feature Request ID 9393 in Mantis) gramps-project/addons-source#820): replace the (non-working) git-build attempt with a
    non-failing "Report gramps-vs-branch series" step that surfaces the caveat
    in every gramps61+ Windows run instead of hiding it.

Validated green on a gramps61 overlay run: both lanes succeed, TMGimporter
reports 157 tests, 95 skipped, 0 broke in ~5 min (vs the prior ~9-min hang),
and the caveat warning prints. The per-module timeout in run_addon_tests.py
remains as a backstop so any future hang stays bounded.

What's lost

The 95 real-DB import tests get no Windows coverage while this gap is open.
They are exercised on the Linux lane against the matching gramps, so the import
logic is still tested — just not on Windows.

Paths to close the gap

  • Wait for gramps 6.1 on conda-forge (lowest effort): once it's published,
    environment.yml's pin picks it up automatically, the caveat step goes quiet,
    and the skipIf(win32) in _make_db() can be removed. Likely the right
    default — revisit when 6.1 ships to conda-forge.
  • Switch the Windows lane to MSYS2 UCRT64 (gramps' actual Windows toolchain),
    where gramps 6.1 builds from git. Larger re-architecture of the Windows job;
    would also enable proper 6.1 coverage before conda-forge catches up.
  • Confirm the failure mode: the hang was observed against mismatched gramps
    (6.0.8 vs a 6.1 addon). When 6.1 is available on Windows, re-enable the tests
    and verify they pass — if they still hang against the matching 6.1 on
    Windows, that's a separate OS-specific gramps DB issue worth its own report.

Refinement note

The skip keys on sys.platform == "win32" (broad), so a future MSYS2-UCRT64
Windows lane running the matching gramps would also skip these — the guard should
be narrowed (e.g. gate on the gramps version actually present) when that lane
exists.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions