OpenNV Validation Packs

This repository provides repository-driven network health checks for OpenNV. Each validation targets a normalized result path published by an OutputFSM command and includes deterministic passing and failing observations.

The initial catalog contains 160 validations and 320 executed fixtures—20 validations for each of eight platforms:

Platform	Canonical ID	Validations
Cisco IOS	cisco_ios	20
Cisco IOS XE	cisco_iosxe	20
Cisco NX-OS	cisco_nxos	20
Cisco IOS XR	cisco_iosxr	20
Arista EOS	arista_eos	20
Juniper Junos	juniper_junos	20
F5 BIG-IP TMOS	f5_tmos	20
Citrix ADC / NetScaler	citrix_adc	20

All observations and values are synthetic. They are designed to test OpenNV's evaluation behavior, not to state vendor defaults or recommended production thresholds. Operators must review thresholds for their environment.

Validation model

Definitions reference an exact OutputFSM platform and command_id. Conditions support nested all, any, and not groups across current, exact, baseline, deviation, and bounded custom leaves. Omitting spec.mode remains backward compatible and is equivalent to mode: current.

apiVersion: opennv.io/v1alpha1
kind: Validation
metadata:
  id: cisco_ios_show_version_health
  severity: warning
spec:
  platform: cisco_ios
  command_id: show_version
  command: show version
  condition:
    all:
      - assertion:
          path: system.oper_state
          operator: exists
          expected: true
      - assertion:
          path: system.oper_state
          operator: eq
          expected: up

Supported leaf operators are eq, ne, gt, gte, lt, lte, exists, contains, not_contains, in, not_in, and regex.

Evaluation modes

spec.mode	Leaf	Purpose
current	assertion	Existing current-observation checks; this is the default.
exact	exact	Standalone typed equality against a literal.
diff	diff	Before/after comparison using unchanged, changed, increased, decreased, not_increased, or not_decreased.
deviation	deviation	Percentage change from baseline with within, outside, increase_lte, or decrease_lte.
custom	custom	Bounded Starlark-like boolean expression over declared inputs.
composite	any leaf	Mixed leaf types under nested all, any, and not.

Baseline-aware checks use fixture/runtime evidence shaped as baseline plus the current observation. baseline_path is optional and defaults to path:

mode: deviation
condition:
  deviation:
    path: resources.memory_used_pct
    operator: increase_lte
    tolerance_percent: 20
    message: Memory growth must remain within 20 percent of baseline.
fixtures:
  - name: stable
    observation: {resources: {memory_used_pct: 77}}
    baseline: {resources: {memory_used_pct: 70}}
    expected_state: passed
  - name: regression
    observation: {resources: {memory_used_pct: 91}}
    baseline: {resources: {memory_used_pct: 70}}
    expected_state: failed

Missing baseline/evidence returns unknown, never passed; malformed policies and invalid numeric comparisons return error. Both states are fail-closed. Legacy fixtures may keep expected_pass; new fixtures may use expected_state with passed, failed, unknown, or error.

Custom evaluation is data-only and does not run Python. It requires an explicit input allowlist and exactly one result = <boolean expression> assignment:

mode: custom
condition:
  custom:
    inputs: [resources.cpu_pct, resources.memory_used_pct]
    script: >-
      result = current["resources"]["cpu_pct"] <= 80 and current["resources"]["memory_used_pct"] <= 85
    message: CPU and memory must remain within policy.

The runtime projects only declared inputs, limits script/evidence size, AST nodes, nesting, and execution steps, and rejects loads, calls, loops, comprehensions, functions, mutation, attributes, collection literals, slices, and multiplication. Definitions are loaded with yaml.safe_load and capped at 1 MB.

Repository layout

catalog/index.yaml                       full validation catalog
contracts/outputfsm-catalog.yaml         pinned OutputFSM compatibility snapshot
validations/<platform>/*.yaml            one validation per command
schema/validation-v1alpha1.schema.json   JSON Schema contract
src/opennv_validations/                  safe evaluator and linter
scripts/generate.py                      deterministic generator
scripts/validate.py                      reference/link/fixture validator
tests/                                   standard-library unit tests

The pinned OutputFSM catalog makes this repository independently testable. The generator refreshes it from a sibling outputfsm-packs checkout. Production sync automation should pin a reviewed OutputFSM tag or commit, update the snapshot in a pull request, and run the same tests.

Validate locally

The development checks require Python 3.11+, PyYAML 6, and jsonschema 4.

python3 -m pip install -r requirements-dev.txt
python3 scripts/validate.py
python3 -m unittest discover -s tests -v

The validator proves that every assertion path is produced by its referenced command and executes all pass/fail fixtures. It also checks IDs, command coverage, platform counts, condition structure, and operator validity.

For a guided evaluation and cross-repository check, see docs/quickstart.md. The architecture guide documents policy boundaries, compatibility, and release coupling.

With both repositories checked out as siblings, verify reproducibility:

python3 scripts/generate.py
python3 scripts/validate.py
python3 scripts/integration_check.py
git diff --exit-code

See CONTRIBUTING.md and development context before changing policies or thresholds. Security reports follow SECURITY.md.

License

Apache-2.0. See LICENSE.