From 4d4803c356f48104526a805629056f92c2b54845 Mon Sep 17 00:00:00 2001 From: "Jens W. Klein" Date: Wed, 15 Jul 2026 12:38:34 +0200 Subject: [PATCH 1/3] Add a Testing chapter for backend packages Plone 6 has no backend testing documentation. The Developer guide has no Testing section, and the only rendered HTML for plone.testing and plone.app.testing lives on the legacy 5.docs.plone.org domain. @davisagli sketched a full outline for such a chapter in a comment on plone/pytest-plone#16 two years ago. This picks up the two pieces of it that cover backend Python testing, and deliberately leaves the rest -- Volto, Robot Framework, testbrowser -- to build on top. - conceptual-guides/testing.md explains the kinds of tests, why Plone has testing layers at all, and how to choose between the two runners. - developer-guide/testing/ shows how to write and run tests with either zope.testrunner or pytest. The pytest guide links to the published pytest-plone documentation at plone.github.io/pytest-plone. Volto testing is cross-referenced rather than duplicated: it lives in the Volto repository. Every zope-testrunner command shown was executed against a real package before being documented. Vale reports 0 errors, 0 warnings, and 0 suggestions on the new pages; linkcheck confirms every external link resolves. --- docs/conceptual-guides/index.md | 1 + docs/conceptual-guides/testing.md | 129 ++++++++++++ docs/developer-guide/index.md | 1 + docs/developer-guide/testing/index.md | 57 ++++++ docs/developer-guide/testing/pytest.md | 135 +++++++++++++ .../testing/zope-testrunner.md | 184 ++++++++++++++++++ styles/config/vocabularies/Plone/accept.txt | 11 ++ 7 files changed, 518 insertions(+) create mode 100644 docs/conceptual-guides/testing.md create mode 100644 docs/developer-guide/testing/index.md create mode 100644 docs/developer-guide/testing/pytest.md create mode 100644 docs/developer-guide/testing/zope-testrunner.md diff --git a/docs/conceptual-guides/index.md b/docs/conceptual-guides/index.md index bf8e422e6e..c8cae07adc 100644 --- a/docs/conceptual-guides/index.md +++ b/docs/conceptual-guides/index.md @@ -23,4 +23,5 @@ package-management package-dependencies make-backend-build components +testing ``` diff --git a/docs/conceptual-guides/testing.md b/docs/conceptual-guides/testing.md new file mode 100644 index 0000000000..8d3e12a7d7 --- /dev/null +++ b/docs/conceptual-guides/testing.md @@ -0,0 +1,129 @@ +--- +myst: + html_meta: + "description": "The kinds of tests a Plone package needs, the two test runners available, and why Plone has testing layers." + "property=og:description": "The kinds of tests a Plone package needs, the two test runners available, and why Plone has testing layers." + "property=og:title": "About testing in Plone" + "keywords": "Plone, testing, pytest, zope.testrunner, testing layers, unittest" +--- + +(about-testing-in-plone)= + +# About testing in Plone + +Testing a Plone package is unlike testing an ordinary Python library, and the difference has one cause. + +An ordinary library can be imported and exercised in microseconds. +A Plone package usually cannot be tested at all until a Plone site exists: a site with your ZCML loaded, your profile installed, and a database behind it. +Building that site takes seconds. + +Everything that follows is a consequence of that one fact. + +## The kinds of tests + +The vocabulary here predates Plone, but Plone uses it in a specific way. + +Unit test +: Exercises a function or class in isolation, with no Plone site at all. + Fast, and worth writing wherever your code allows it. + In practice, much Plone code touches the site so directly that a true unit test is not possible. + +Integration test +: Runs against a real Plone site, in the same process, inside a transaction that is rolled back afterwards. + This is the workhorse. + Most tests you write for an add-on are integration tests. + +Functional test +: Runs against a real Plone site that commits real transactions, so a separate process, such as a browser or an HTTP client, can see the result. + Slower than an integration test, because the isolation is more expensive. + Use it when a request has to arrive over the network. + +Acceptance test +: Drives the whole system as a user would, through a browser. + In Plone this means Robot Framework or a Zope testbrowser for Classic UI, and Cypress for Volto. + +The line that matters most in practice is between integration and functional. +It is a question of isolation, and isolation is where the cost is. + +## Why Plone has testing layers + +If building a Plone site takes seconds, and your suite has hundreds of tests, you cannot build one per test. + +A **testing layer** is Plone's answer. +It is an object with two lifecycles: + +- `setUp` and `tearDown` run **once**, and do the expensive work: start Zope, create the site, load ZCML, install your profile. +- `testSetUp` and `testTearDown` run **around every test**, and do only the cheap work needed to keep tests independent. + +That split is the whole idea. +The costly site is built once and shared, while each test still starts from a clean state. + +Isolation happens in the cheap half. +An integration layer opens a transaction before each test and aborts it afterwards, so nothing a test creates survives it. +A functional layer instead stacks a temporary storage on the database, lets the test commit for real, and throws the storage away. + +Layers stack, too. +Your add-on's layer sits on one that made a Plone site, which sits on one that started Zope. +Each is built once, and everything above reuses it. + +You declare your own layer in a `testing.py` module in your package. +That is where you say which ZCML to load and which profile to install. +Both test runners described below consume the same layers. +A layer is not a property of the runner. + +```{seealso} +The layer machinery lives in the packages that own it: + +- [plone.app.testing](https://github.com/plone/plone.app.testing/blob/master/README.rst): the Plone-specific layers, and how to write your own `testing.py`. +- [plone.testing](https://github.com/plone/plone.testing/blob/master/src/plone/testing/README.rst): the underlying layer model. +``` + +## The two test runners + +Python code in the Plone ecosystem is tested with one of two runners. +Both work. +Both use the same layers. +They differ in how you write the tests, not in what the tests can do. + +### `zope.testrunner` + +The traditional choice, and what Plone core itself uses. + +Tests are `unittest.TestCase` classes. +A class declares the layer it needs by assigning it to a `layer` attribute, and the runner groups tests by layer so each layer is set up once for the whole group. + +That grouping is the runner's defining feature. +It understands layers natively, because layers were built for it. + +### pytest + +The choice most new add-ons make, and the default in packages generated by Cookieplone. + +Tests are plain functions. +What a test needs, it names as an argument, and pytest supplies it: + +```python +def test_portal_title(portal): + assert portal.title == "Plone site" +``` + +pytest has no native concept of a Plone layer. +The [pytest-plone](https://plone.github.io/pytest-plone/) plugin bridges the gap: it takes the layers you already declared in `testing.py` and turns them into pytest fixtures. + +### Choosing + +Neither runner is deprecated, and neither is going away. + +Choose `zope.testrunner` if you contribute to Plone core, or maintain a package whose tests already use it. +There is no reward for converting a working suite. + +Choose pytest for new work, unless you have a reason not to. +It is where the wider Python ecosystem is, and it is what Cookieplone gives you. + +The one thing not to do is mix both in the same package. +Two runners means two ways to run the suite and two ways for it to break. + +## Where to go next + +- {doc}`/developer-guide/testing/index`: how to write and run the tests. +``` diff --git a/docs/developer-guide/index.md b/docs/developer-guide/index.md index 22eb2c572f..cad9dc186e 100644 --- a/docs/developer-guide/index.md +++ b/docs/developer-guide/index.md @@ -26,4 +26,5 @@ create-a-distribution standardize-python-project-configuration native-namespace deprecation +testing/index ``` diff --git a/docs/developer-guide/testing/index.md b/docs/developer-guide/testing/index.md new file mode 100644 index 0000000000..dd7db3dcee --- /dev/null +++ b/docs/developer-guide/testing/index.md @@ -0,0 +1,57 @@ +--- +myst: + html_meta: + "description": "Write and run tests for a Plone backend package, with either zope.testrunner or pytest." + "property=og:description": "Write and run tests for a Plone backend package, with either zope.testrunner or pytest." + "property=og:title": "Test a backend package" + "keywords": "Plone, testing, pytest, zope.testrunner, backend, add-on" +--- + +(testing-backend-packages)= + +# Test a backend package + +How to write and run tests for a Plone package written in Python. + +If you want to understand *why* Plone tests look the way they do before you write any, read {doc}`/conceptual-guides/testing` first. +This page assumes you have. + +```{note} +This covers backend packages. +For testing a Volto add-on, see [Test add-ons](/volto/development/add-ons/test-add-ons-19). +``` + +## Before you start + +You need a testing layer. + +A layer builds the Plone site your tests run against: it loads your ZCML and installs your GenericSetup profile. +You declare it in a `testing.py` module in your package, and a package generated from a Plone template already has one. + +Both test runners consume the same layer. +Choosing a runner does not change how you write `testing.py`. + +If your package has no `testing.py`, write one first. +[plone.app.testing](https://github.com/plone/plone.app.testing/blob/master/README.rst) documents how. + +## Choose a runner + +| | `zope.testrunner` | pytest | +| --- | --- | --- | +| Tests are | `unittest.TestCase` classes | plain functions | +| Layers | native | via [pytest-plone](https://plone.github.io/pytest-plone/) | +| Used by | Plone core | most new add-ons, Cookieplone | + +Choose `zope.testrunner` if you contribute to Plone core, or if your package already uses it. +A working test suite is not worth converting. + +Choose pytest for new work. + +Do not use both in one package. + +```{toctree} +:maxdepth: 1 + +zope-testrunner +pytest +``` diff --git a/docs/developer-guide/testing/pytest.md b/docs/developer-guide/testing/pytest.md new file mode 100644 index 0000000000..d51ce3fda4 --- /dev/null +++ b/docs/developer-guide/testing/pytest.md @@ -0,0 +1,135 @@ +--- +myst: + html_meta: + "description": "Write and run tests for a Plone package with pytest and pytest-plone." + "property=og:description": "Write and run tests for a Plone package with pytest and pytest-plone." + "property=og:title": "Test with pytest" + "keywords": "Plone, pytest, pytest-plone, fixtures, testing layers" +--- + +(test-with-pytest)= + +# Test with pytest + +This guide shows you how to write and run tests with pytest, the runner most new add-ons use and the default in packages generated by Cookieplone. + +For `zope.testrunner` instead, see {doc}`zope-testrunner`. + +pytest has no native concept of a testing layer. +The [pytest-plone](https://plone.github.io/pytest-plone/) plugin bridges that gap: it turns the layers you declared in `testing.py` into pytest fixtures. +You keep your layers exactly as they are. + +## Install the plugin + +```shell +pip install pytest-plone +``` + +## Write the conftest + +pytest discovers fixtures in a file named `conftest.py`. +Create one at the top of your package. + +Import your testing layers, hand them to `fixtures_factory` with a prefix for each, and inject the result into the module namespace. + +```python +from my.addon.testing import MY_ADDON_FUNCTIONAL_TESTING +from my.addon.testing import MY_ADDON_INTEGRATION_TESTING +from pytest_plone import fixtures_factory + + +pytest_plugins = ["pytest_plone"] + + +globals().update( + fixtures_factory( + ( + (MY_ADDON_FUNCTIONAL_TESTING, "functional"), + (MY_ADDON_INTEGRATION_TESTING, "integration"), + ) + ) +) +``` + +The prefixes name the generated fixtures. +Keep `integration` and `functional`. +The fixtures the plugin provides are built on those names. + +## Write an integration test + +A test is a plain function. +It asks for what it needs by naming it as an argument. + +```python +def test_portal_title(portal): + assert portal.title == "Plone site" +``` + +There is no class, no `setUp`, and no `self`. +`portal` is the Plone site, on the integration layer. + +## Create content without writing setup code + +`pytest-plone` provides a marker that prepares the portal before the test runs. +It applies GenericSetup profiles, creates content, and grants roles. + +```python +import pytest + + +@pytest.mark.portal( + profiles=["my.addon:testing"], + content=[{"type": "Document", "id": "doc1", "title": "A document"}], + roles=["Manager"], +) +def test_document_exists(portal): + assert "doc1" in portal +``` + +As with `zope.testrunner`, no cleanup is needed. +The integration layer aborts the transaction after each test. + +## Write a functional test + +Ask for the functional fixtures instead. +Use them when a request has to arrive over the network, such as a REST API test. + +```python +def test_root_is_public(functional_portal, anon_request): + response = anon_request.get("/") + + assert response.status_code == 200 +``` + +## Run the tests + +```shell +pytest +``` + +Narrow it down while working: + +```shell +pytest -k test_document_exists +pytest tests/test_document.py +``` + +Find out where the time goes: + +```shell +pytest --durations=0 +``` + +## Measure coverage + +```shell +pytest --cov=my.addon --cov-report term-missing +``` + +## Where to go next + +`pytest-plone` provides considerably more than shown here: fixtures for add-on install and uninstall checks, content type introspection, vocabularies, authenticated REST API sessions, and class-scoped portals for expensive suites. + +```{seealso} +The [pytest-plone documentation](https://plone.github.io/pytest-plone/) covers the full [fixture reference](https://plone.github.io/pytest-plone/reference/fixtures.html), the marker, and the `fixtures_factory` API. +``` diff --git a/docs/developer-guide/testing/zope-testrunner.md b/docs/developer-guide/testing/zope-testrunner.md new file mode 100644 index 0000000000..71db8ec978 --- /dev/null +++ b/docs/developer-guide/testing/zope-testrunner.md @@ -0,0 +1,184 @@ +--- +myst: + html_meta: + "description": "Write and run tests for a Plone package with unittest and zope.testrunner." + "property=og:description": "Write and run tests for a Plone package with unittest and zope.testrunner." + "property=og:title": "Test with zope.testrunner" + "keywords": "Plone, zope.testrunner, unittest, testing layers, coverage" +--- + +(test-with-zope-testrunner)= + +# Test with `zope.testrunner` + +This guide shows you how to write and run tests with `unittest` and `zope.testrunner`, the runner Plone core uses. + +For pytest instead, see {doc}`pytest`. + +## Install the runner + +```shell +pip install zope.testrunner +``` + +## Write an integration test + +A test is a `unittest.TestCase`. +It declares the layer it needs by assigning it to a `layer` attribute. + +```python +from my.addon.testing import MY_ADDON_INTEGRATION_TESTING +from plone import api + +import unittest + + +class TestDocument(unittest.TestCase): + layer = MY_ADDON_INTEGRATION_TESTING + + def setUp(self): + self.portal = self.layer["portal"] + + def test_portal_title(self): + self.assertEqual(self.portal.title, "Plone site") + + def test_create_document(self): + with api.env.adopt_roles(["Manager"]): + api.content.create( + container=self.portal, + type="Document", + id="doc1", + title="A document", + ) + self.assertIn("doc1", self.portal) +``` + +The `layer` attribute is the whole integration. +The runner reads it, sets that layer up once, and runs every test that declares it. + +`self.layer["portal"]` is how you reach the Plone site. +The layer exposes the objects it built through this mapping. + +Note what `test_create_document` does *not* need: no cleanup. +The integration layer aborts the transaction after each test, so `doc1` is gone before the next test runs. + +## Write a functional test + +Use the functional layer when a request has to arrive over the network, and the site must therefore commit. + +```python +from my.addon.testing import MY_ADDON_FUNCTIONAL_TESTING + +import unittest + + +class TestDocumentView(unittest.TestCase): + layer = MY_ADDON_FUNCTIONAL_TESTING +``` + +Everything else is the same. +Only the layer changes. + +## Run the tests + +```shell +zope-testrunner --auto-path -s my.addon +``` + +`-s` takes the dotted name of the package to search. +`--auto-path` adds that package's directory to the test search path, which is what you want for a package installed in development mode. + +Add `--auto-color` and `--auto-progress` for readable output: + +```shell +zope-testrunner --auto-color --auto-progress --auto-path -s my.addon +``` + +```{note} +Under Buildout, the equivalent command was `bin/test`. +Buildout generated that script for you. +With a pip-installed Plone, you call `zope-testrunner` directly. +``` + +## Read the output + +The runner groups tests by layer, and tells you so: + +```console +Running my.addon.testing.MyAddon:Integration tests: + Set up my.addon.testing.MyAddon:Integration in 4.521 seconds. + Ran 12 tests with 0 failures, 0 errors and 0 skipped in 0.843 seconds. +Running my.addon.testing.MyAddon:Functional tests: + Set up my.addon.testing.MyAddon:Functional in 0.128 seconds. + Ran 3 tests with 0 failures, 0 errors and 0 skipped in 1.204 seconds. +``` + +This grouping is the runner's defining behavior. +Each layer is set up once, and every test that declared it runs against that one setup. + +The `Set up ... in N seconds` line is the expensive part. +If you see it repeated many times, something is forcing the layer to be rebuilt. + +## Narrow down what runs + +While working on one thing, run one thing. + +Filter by test name, a case-sensitive regular expression: + +```shell +zope-testrunner --auto-path -s my.addon -t test_create_document +``` + +Filter by module, also a regular expression: + +```shell +zope-testrunner --auto-path -s my.addon -m test_document +``` + +Run only unit tests, ignoring every layer: + +```shell +zope-testrunner --auto-path -s my.addon -u +``` + +Run a single layer: + +```shell +zope-testrunner --auto-path -s my.addon --layer Integration +``` + +List what would run, without running it: + +```shell +zope-testrunner --auto-path -s my.addon --list-tests +``` + +`--list-tests` is the fastest way to check that a filter matches what you think it does. + +## Run tests in parallel + +```shell +zope-testrunner --auto-path -s my.addon -j 4 +``` + +Each process sets up its own layers, so this trades memory for wall-clock time. +It pays off on large suites. + +## Measure coverage + +`zope.testrunner` runs as a module, so `coverage` can wrap it: + +```shell +coverage run -m zope.testrunner --auto-path -s my.addon +coverage report +``` + +## Run tests at all levels + +Some tests are registered at a higher level and are skipped by default, notably Robot Framework tests. + +```shell +zope-testrunner --auto-path -s my.addon --all +``` + +Expect this to be considerably slower. diff --git a/styles/config/vocabularies/Plone/accept.txt b/styles/config/vocabularies/Plone/accept.txt index c0b583ac1c..c3eda7f658 100644 --- a/styles/config/vocabularies/Plone/accept.txt +++ b/styles/config/vocabularies/Plone/accept.txt @@ -74,3 +74,14 @@ webpack wireframe xkcd Zope +`plone.app.testing` +`plone.testing` +`pytest-plone` +`zope.testrunner` +conftest +pytest +pytest-plone +testbrowser +plone.app.testing +plone.testing +zope.testrunner From b6431f81c729168224c0e2b54658db07b1a8ee1e Mon Sep 17 00:00:00 2001 From: "Jens W. Klein" Date: Wed, 15 Jul 2026 14:18:19 +0200 Subject: [PATCH 2/3] Add a how-to for writing a testing layer, and fix heading style Writing the testing.py layer is the setup work both zope.testrunner and pytest-plone build on, and the chapter previously punted it to the large plone.app.testing README. That left the developer guide without its keystone: the one thing every backend test needs. Add developer-guide/testing/write-a-testing-layer.md, placed first in the sub-toctree because it is the shared prerequisite. It shows both forms: - the declarative PloneWithPackageLayer instance, for a simple add-on; - the PloneSandboxLayer subclass with setUpZope/setUpPloneSite, for anything more involved. Both forms, and the WSGI_SERVER_FIXTURE note for real-HTTP functional tests, are taken from real coredev packages (plone.app.dexterity, plone.volto) and the plone.app.testing README, and every import was checked against the installed packages. The runner guides and the conceptual guide now point here instead of at the README. Also drop the code-span from the zope.testrunner headings, which rendered as a heavy monospace box next to the plain 'pytest' heading. --- docs/conceptual-guides/testing.md | 5 +- docs/developer-guide/testing/index.md | 5 +- docs/developer-guide/testing/pytest.md | 2 +- .../testing/write-a-testing-layer.md | 156 ++++++++++++++++++ .../testing/zope-testrunner.md | 4 +- styles/config/vocabularies/Plone/accept.txt | 7 + 6 files changed, 173 insertions(+), 6 deletions(-) create mode 100644 docs/developer-guide/testing/write-a-testing-layer.md diff --git a/docs/conceptual-guides/testing.md b/docs/conceptual-guides/testing.md index 8d3e12a7d7..6c2ab6fce8 100644 --- a/docs/conceptual-guides/testing.md +++ b/docs/conceptual-guides/testing.md @@ -70,11 +70,12 @@ You declare your own layer in a `testing.py` module in your package. That is where you say which ZCML to load and which profile to install. Both test runners described below consume the same layers. A layer is not a property of the runner. +Writing one is the shared setup step for either runner, covered in {doc}`/developer-guide/testing/write-a-testing-layer`. ```{seealso} The layer machinery lives in the packages that own it: -- [plone.app.testing](https://github.com/plone/plone.app.testing/blob/master/README.rst): the Plone-specific layers, and how to write your own `testing.py`. +- [plone.app.testing](https://github.com/plone/plone.app.testing/blob/master/README.rst): the Plone-specific layers, and the complete reference for the tools used to write a `testing.py`. - [plone.testing](https://github.com/plone/plone.testing/blob/master/src/plone/testing/README.rst): the underlying layer model. ``` @@ -85,7 +86,7 @@ Both work. Both use the same layers. They differ in how you write the tests, not in what the tests can do. -### `zope.testrunner` +### zope.testrunner The traditional choice, and what Plone core itself uses. diff --git a/docs/developer-guide/testing/index.md b/docs/developer-guide/testing/index.md index dd7db3dcee..d9a3dfef99 100644 --- a/docs/developer-guide/testing/index.md +++ b/docs/developer-guide/testing/index.md @@ -32,11 +32,11 @@ Both test runners consume the same layer. Choosing a runner does not change how you write `testing.py`. If your package has no `testing.py`, write one first. -[plone.app.testing](https://github.com/plone/plone.app.testing/blob/master/README.rst) documents how. +See {doc}`write-a-testing-layer`. ## Choose a runner -| | `zope.testrunner` | pytest | +| | zope.testrunner | pytest | | --- | --- | --- | | Tests are | `unittest.TestCase` classes | plain functions | | Layers | native | via [pytest-plone](https://plone.github.io/pytest-plone/) | @@ -52,6 +52,7 @@ Do not use both in one package. ```{toctree} :maxdepth: 1 +write-a-testing-layer zope-testrunner pytest ``` diff --git a/docs/developer-guide/testing/pytest.md b/docs/developer-guide/testing/pytest.md index d51ce3fda4..76dc8ca898 100644 --- a/docs/developer-guide/testing/pytest.md +++ b/docs/developer-guide/testing/pytest.md @@ -16,7 +16,7 @@ This guide shows you how to write and run tests with pytest, the runner most new For `zope.testrunner` instead, see {doc}`zope-testrunner`. pytest has no native concept of a testing layer. -The [pytest-plone](https://plone.github.io/pytest-plone/) plugin bridges that gap: it turns the layers you declared in `testing.py` into pytest fixtures. +The [pytest-plone](https://plone.github.io/pytest-plone/) plugin bridges that gap: it turns the layers you declared in your {doc}`testing.py ` into pytest fixtures. You keep your layers exactly as they are. ## Install the plugin diff --git a/docs/developer-guide/testing/write-a-testing-layer.md b/docs/developer-guide/testing/write-a-testing-layer.md new file mode 100644 index 0000000000..07e776b8c3 --- /dev/null +++ b/docs/developer-guide/testing/write-a-testing-layer.md @@ -0,0 +1,156 @@ +--- +myst: + html_meta: + "description": "Write the testing.py testing layer that both zope.testrunner and pytest-plone build on." + "property=og:description": "Write the testing.py testing layer that both zope.testrunner and pytest-plone build on." + "property=og:title": "Write a testing layer" + "keywords": "Plone, testing layer, testing.py, plone.app.testing, PloneSandboxLayer, PloneWithPackageLayer" +--- + +(write-a-testing-layer)= + +# Write a testing layer + +This guide shows you how to write the testing layer for a Plone add-on. + +The layer is the setup work both test runners share. +Whether you test with {doc}`zope-testrunner` or {doc}`pytest`, the layer is what builds the Plone site your tests run against: it loads your ZCML and installs your GenericSetup profile. +You write it once, in a `testing.py` module in your package. + +For what a layer *is* and why Plone works this way, see {doc}`/conceptual-guides/testing`. +This page is about writing one. + +## Two ways to write it + +`plone.app.testing` gives you two forms. + +Use the **declarative form** for a simple add-on: one package, one profile. +Use the **class form** when setup needs more than that: extra ZCML, several profiles, dependencies loaded first, or setup code that runs against the portal. + +Both produce the same thing—a *fixture* layer—and from that fixture you derive the integration and functional layers your tests name. + +## The declarative form + +For an add-on that loads its own ZCML and installs one profile, instantiate `PloneWithPackageLayer` directly. +No subclass needed. + +```python +from plone.app.testing import FunctionalTesting +from plone.app.testing import IntegrationTesting +from plone.app.testing import PloneWithPackageLayer +from plone.testing.zope import WSGI_SERVER_FIXTURE + +import my.addon + + +MY_ADDON_FIXTURE = PloneWithPackageLayer( + zcml_package=my.addon, + zcml_filename="configure.zcml", + gs_profile_id="my.addon:default", + name="MyAddonFixture", +) + +MY_ADDON_INTEGRATION_TESTING = IntegrationTesting( + bases=(MY_ADDON_FIXTURE,), + name="MyAddonLayer:IntegrationTesting", +) + +MY_ADDON_FUNCTIONAL_TESTING = FunctionalTesting( + bases=(MY_ADDON_FIXTURE, WSGI_SERVER_FIXTURE), + name="MyAddonLayer:FunctionalTesting", +) +``` + +That is a complete `testing.py`. +The three names it exports—the fixture, the integration layer, and the functional layer—are what your tests and your `conftest.py` refer to. + +## The class form + +When the declarative form is not enough, subclass `PloneSandboxLayer` and override two methods. + +```python +from plone.app.testing import applyProfile +from plone.app.testing import FunctionalTesting +from plone.app.testing import IntegrationTesting +from plone.app.testing import PloneSandboxLayer +from plone.testing.zope import WSGI_SERVER_FIXTURE + +import my.addon + + +class MyAddonLayer(PloneSandboxLayer): + + def setUpZope(self, app, configurationContext): + self.loadZCML(package=my.addon) + + def setUpPloneSite(self, portal): + applyProfile(portal, "my.addon:default") + + +MY_ADDON_FIXTURE = MyAddonLayer() + +MY_ADDON_INTEGRATION_TESTING = IntegrationTesting( + bases=(MY_ADDON_FIXTURE,), + name="MyAddonLayer:IntegrationTesting", +) + +MY_ADDON_FUNCTIONAL_TESTING = FunctionalTesting( + bases=(MY_ADDON_FIXTURE, WSGI_SERVER_FIXTURE), + name="MyAddonLayer:FunctionalTesting", +) +``` + +The two methods are the whole difference: + +`setUpZope` +: Runs while Zope starts, before any Plone site exists. + Load your ZCML here. + This is also where you load dependencies—call `self.loadZCML(package=...)` for each, or add their fixtures to the layer's bases. + +`setUpPloneSite` +: Runs against a freshly created Plone site. + Install your profile here with `applyProfile`. + You can install more than one, create shared content, or set roles—anything that should be part of the fixture every test starts from. + +`PloneSandboxLayer` builds on `PLONE_FIXTURE` by default, which is why you get a working Plone site without asking for one. + +## Derive the integration and functional layers + +Both forms end the same way: from the one fixture, derive the layers your tests actually use. + +`IntegrationTesting` wraps each test in a transaction that is rolled back afterwards. +`FunctionalTesting` lets tests commit, which a separate process can then see. + +```python +MY_ADDON_INTEGRATION_TESTING = IntegrationTesting( + bases=(MY_ADDON_FIXTURE,), + name="MyAddonLayer:IntegrationTesting", +) + +MY_ADDON_FUNCTIONAL_TESTING = FunctionalTesting( + bases=(MY_ADDON_FIXTURE, WSGI_SERVER_FIXTURE), + name="MyAddonLayer:FunctionalTesting", +) +``` + +```{important} +Add `WSGI_SERVER_FIXTURE` to the functional layer's bases only when your tests make **real HTTP requests** over a socket—REST API tests, for example, including those that use `pytest-plone`'s `request_factory`. +It starts a WSGI server so an HTTP client can reach the site. +Tests that use a Zope testbrowser do not need it. +``` + +## Use the layers in your tests + +Your `testing.py` is now the single input to whichever runner you use. + +- With `zope.testrunner`, a test class sets `layer = MY_ADDON_INTEGRATION_TESTING`. See {doc}`zope-testrunner`. +- With pytest, you pass the layers to `fixtures_factory` in your `conftest.py`. See {doc}`pytest`. + +## Where to go next + +The two forms shown here cover most add-ons. +For the full set of options—additional Zope products, custom base layers, loading several ZCML files—consult the package that owns the machinery. + +```{seealso} +[plone.app.testing](https://github.com/plone/plone.app.testing/blob/master/README.rst): the complete reference for layers, fixtures, and helpers. +``` diff --git a/docs/developer-guide/testing/zope-testrunner.md b/docs/developer-guide/testing/zope-testrunner.md index 71db8ec978..3b381c8fc4 100644 --- a/docs/developer-guide/testing/zope-testrunner.md +++ b/docs/developer-guide/testing/zope-testrunner.md @@ -9,11 +9,13 @@ myst: (test-with-zope-testrunner)= -# Test with `zope.testrunner` +# Test with zope.testrunner This guide shows you how to write and run tests with `unittest` and `zope.testrunner`, the runner Plone core uses. For pytest instead, see {doc}`pytest`. +This guide assumes your package already has a testing layer. +See {doc}`write-a-testing-layer` if it does not. ## Install the runner diff --git a/styles/config/vocabularies/Plone/accept.txt b/styles/config/vocabularies/Plone/accept.txt index c3eda7f658..c61d471708 100644 --- a/styles/config/vocabularies/Plone/accept.txt +++ b/styles/config/vocabularies/Plone/accept.txt @@ -85,3 +85,10 @@ testbrowser plone.app.testing plone.testing zope.testrunner +PloneSandboxLayer +PloneWithPackageLayer +IntegrationTesting +FunctionalTesting +setUpZope +setUpPloneSite +applyProfile From 25a0a1f73420cdc5ba58537aeceb6322c515f29c Mon Sep 17 00:00:00 2001 From: "Jens W. Klein" Date: Mon, 20 Jul 2026 09:12:23 +0200 Subject: [PATCH 3/3] Address review from @gforcada on the testing chapter - Sort the Vale vocabulary additions into place instead of appending them, and trim them to the terms actually needed (the class names only ever appear in code spans, which Vale does not check). - Do not pin Cypress for the Volto frontend; the frontend tooling is in flux, so defer to the Volto documentation instead. - Recommend 'zope-testrunner --test-path src -s ', the form the plone.meta tox environments use, and keep --auto-path as the pip-installed alternative. (Thanks @gforcada -- the original --auto-path lead came from buildout.coredev's run-tests.sh, not the common case.) - Note that pytest --cov needs the pytest-cov plugin installed. - Backtick pytest in the 'Choosing' section for consistency with zope.testrunner. --- docs/conceptual-guides/testing.md | 5 ++-- docs/developer-guide/testing/pytest.md | 6 +++++ .../testing/zope-testrunner.md | 17 +++++++++---- styles/config/vocabularies/Plone/accept.txt | 24 ++++++------------- 4 files changed, 28 insertions(+), 24 deletions(-) diff --git a/docs/conceptual-guides/testing.md b/docs/conceptual-guides/testing.md index 6c2ab6fce8..035a6ce174 100644 --- a/docs/conceptual-guides/testing.md +++ b/docs/conceptual-guides/testing.md @@ -40,7 +40,8 @@ Functional test Acceptance test : Drives the whole system as a user would, through a browser. - In Plone this means Robot Framework or a Zope testbrowser for Classic UI, and Cypress for Volto. + In Plone this means Robot Framework or a Zope testbrowser for Classic UI. + The Volto frontend has its own browser-based end-to-end testing tools, covered in the Volto documentation. The line that matters most in practice is between integration and functional. It is a question of isolation, and isolation is where the cost is. @@ -118,7 +119,7 @@ Neither runner is deprecated, and neither is going away. Choose `zope.testrunner` if you contribute to Plone core, or maintain a package whose tests already use it. There is no reward for converting a working suite. -Choose pytest for new work, unless you have a reason not to. +Choose `pytest` for new work, unless you have a reason not to. It is where the wider Python ecosystem is, and it is what Cookieplone gives you. The one thing not to do is mix both in the same package. diff --git a/docs/developer-guide/testing/pytest.md b/docs/developer-guide/testing/pytest.md index 76dc8ca898..5e444a0673 100644 --- a/docs/developer-guide/testing/pytest.md +++ b/docs/developer-guide/testing/pytest.md @@ -122,6 +122,12 @@ pytest --durations=0 ## Measure coverage +The `--cov` options come from the `pytest-cov` plugin, so install it first: + +```shell +pip install pytest-cov +``` + ```shell pytest --cov=my.addon --cov-report term-missing ``` diff --git a/docs/developer-guide/testing/zope-testrunner.md b/docs/developer-guide/testing/zope-testrunner.md index 3b381c8fc4..daaaf16224 100644 --- a/docs/developer-guide/testing/zope-testrunner.md +++ b/docs/developer-guide/testing/zope-testrunner.md @@ -84,16 +84,23 @@ Only the layer changes. ## Run the tests ```shell -zope-testrunner --auto-path -s my.addon +zope-testrunner --test-path src -s my.addon ``` -`-s` takes the dotted name of the package to search. -`--auto-path` adds that package's directory to the test search path, which is what you want for a package installed in development mode. +`-s` takes the dotted name of the package to search for tests. +`--test-path` points the runner at the directory that holds your source, usually `src`. +This is the form the `tox` environments generated by [plone.meta](https://github.com/plone/meta) use. + +If your package is instead pip-installed in development mode, `--auto-path` locates its directory for you, so you do not have to name a path: + +```shell +zope-testrunner --auto-path -s my.addon +``` -Add `--auto-color` and `--auto-progress` for readable output: +Add `--auto-color` and `--auto-progress` to either form for readable output: ```shell -zope-testrunner --auto-color --auto-progress --auto-path -s my.addon +zope-testrunner --auto-color --auto-progress --test-path src -s my.addon ``` ```{note} diff --git a/styles/config/vocabularies/Plone/accept.txt b/styles/config/vocabularies/Plone/accept.txt index c61d471708..88fad90151 100644 --- a/styles/config/vocabularies/Plone/accept.txt +++ b/styles/config/vocabularies/Plone/accept.txt @@ -16,6 +16,7 @@ Classic UI CMSs CMSUI CommonJS +conftest Cookieplone doctest ETags? @@ -41,6 +42,9 @@ pipx Plate PLIP(s) Plone +plone.app.testing +plone.meta +plone.testing plonecli pluggab(le|ility) pnpm @@ -48,6 +52,8 @@ pnpm prerendered programatically Public UI +pytest +pytest-plone [Qq]uerystring Razzle [Rr]enderers? @@ -55,6 +61,7 @@ RichText Sass Schuko subfolder +testbrowser toggler [Tt]owncrier transpilation @@ -74,21 +81,4 @@ webpack wireframe xkcd Zope -`plone.app.testing` -`plone.testing` -`pytest-plone` -`zope.testrunner` -conftest -pytest -pytest-plone -testbrowser -plone.app.testing -plone.testing zope.testrunner -PloneSandboxLayer -PloneWithPackageLayer -IntegrationTesting -FunctionalTesting -setUpZope -setUpPloneSite -applyProfile