Idea
Add a testing feature where the library attaches :telemetry handlers for all events defined in the user's documentation structure, accumulates what gets emitted during the test suite, and then validates that emitted events match the documented schema.
How it would work
- At test suite start (
test_helper.exs): user calls something like TelemetryDocs.Testing.attach(events) with the same sections data structure used for documentation.
- This registers telemetry handlers for all documented events, storing received event names, measurement keys, and metadata keys in ETS (needs to work across test processes).
- In
ExUnit.after_suite/1 (registered automatically by attach/1): compare accumulated data against the documented schema.
What it could validate
- For each documented event that was emitted at least once: do the measurement/metadata keys match what's documented? (no missing, no undocumented extras)
- Optionally warn about documented events that were never emitted (missing test coverage or dead code)
User-facing API
# test_helper.exs
{events, _} = Code.eval_file("pages/telemetry_events.exs")
TelemetryDocs.Testing.attach(events)
That's it — one call.
Open questions
- Failure reporting:
after_suite can't add test failures. Options: IO.warn, Logger.warning, or System.at_exit/1 with a non-zero exit code to actually fail CI. Maybe make this configurable.
- Event name parsing: event names are strings like
"[:app, :connected]" — need to convert to atom lists for :telemetry.attach/4. Code.eval_string/1 works fine in test context.
- Partial coverage: should it be an error if a documented event is never emitted? Probably opt-in, since not all test suites exercise everything.
Idea
Add a testing feature where the library attaches
:telemetryhandlers for all events defined in the user's documentation structure, accumulates what gets emitted during the test suite, and then validates that emitted events match the documented schema.How it would work
test_helper.exs): user calls something likeTelemetryDocs.Testing.attach(events)with the same sections data structure used for documentation.ExUnit.after_suite/1(registered automatically byattach/1): compare accumulated data against the documented schema.What it could validate
User-facing API
That's it — one call.
Open questions
after_suitecan't add test failures. Options:IO.warn,Logger.warning, orSystem.at_exit/1with a non-zero exit code to actually fail CI. Maybe make this configurable."[:app, :connected]"— need to convert to atom lists for:telemetry.attach/4.Code.eval_string/1works fine in test context.