Lint and test examples in `examples` and docstrings automatically

[leandro-lucarella-frequenz]

Projects normally have examples in the examples/ directory and in docstring documentation. The ones in examples/ are currently being linted, but never ran. The ones in docstrings are never linted nor ran. All of this very subject to break when there are updates.

In docstrings there are mainly 2 types of examples in documentation:

• doctest: Usually any block starting with >>> in a docstring
• Markdown examples. Could be in:
  • docstrings (Python files)
  • Other high-level documentation in docs/.

We mainly use markdown examples, so we should focus on that first.

There are several tools to actually test those examples to make sure they work, so we should use one:

• https://docs.python.org/3/library/doctest.html

• https://docs.pytest.org/en/7.1.x/how-to/doctest.html

• https://stackoverflow.com/questions/23250068/python-doctest-my-github-markdown-files

  • https://pypi.org/project/phmdoctest/ (+)
  • https://github.com/ryneeverett/mkcodes (-)

[leandro-lucarella-frequenz]

There is work being done in the SDK that eventually should be incorporated here when the path is more clear:

• tracking: Test and lint code examples frequenz-sdk-python#379

[leandro-lucarella-frequenz]

There is the issue on how to deal with extra dependencies to run examples. For examples see this discussion in a PR:

• Validate docstring examples frequenz-sdk-python#384 (comment)

[Marenz]

Sybil, currently used for the examples, has support for directives and invisible code blocks even. I didn't look into it enough yet, but it def. sounds like aspects we can use

[llucax]

We already have a working version of this in the SDK and channels repo:

• https://github.com/frequenz-floss/frequenz-sdk-python/blob/v0.x.x/src/conftest.py

When we add this to repo-config we need to also filter out the conftest.py file from the generated docs.