Skip to content

Testing helpers: validate emitted telemetry against documented schema #1

Description

@whatyouhide

Idea

Add a testing feature where the library attaches :telemetry handlers for all events defined in the user's documentation structure, accumulates what gets emitted during the test suite, and then validates that emitted events match the documented schema.

How it would work

  1. At test suite start (test_helper.exs): user calls something like TelemetryDocs.Testing.attach(events) with the same sections data structure used for documentation.
  2. This registers telemetry handlers for all documented events, storing received event names, measurement keys, and metadata keys in ETS (needs to work across test processes).
  3. In ExUnit.after_suite/1 (registered automatically by attach/1): compare accumulated data against the documented schema.

What it could validate

  • For each documented event that was emitted at least once: do the measurement/metadata keys match what's documented? (no missing, no undocumented extras)
  • Optionally warn about documented events that were never emitted (missing test coverage or dead code)

User-facing API

# test_helper.exs
{events, _} = Code.eval_file("pages/telemetry_events.exs")
TelemetryDocs.Testing.attach(events)

That's it — one call.

Open questions

  • Failure reporting: after_suite can't add test failures. Options: IO.warn, Logger.warning, or System.at_exit/1 with a non-zero exit code to actually fail CI. Maybe make this configurable.
  • Event name parsing: event names are strings like "[:app, :connected]" — need to convert to atom lists for :telemetry.attach/4. Code.eval_string/1 works fine in test context.
  • Partial coverage: should it be an error if a documented event is never emitted? Probably opt-in, since not all test suites exercise everything.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions