diff --git a/CHANGELOG.md b/CHANGELOG.md index c207854..a269621 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,8 +10,14 @@ schemas and the `paramify` CLI — not the internal code. ## [Unreleased] +## [0.3.0-beta] - 2026-07-23 + ### Added +- 13 Datadog fetchers (new category): Cloud SIEM detection rules & signals, SIEM + operational configuration, log pipelines / indexes / archives, host & container + inventory, agent check results, the APM service catalog, and incident records with + timelines. Credential setup in [`fetchers/datadog/README.md`](fetchers/datadog/README.md). - `paramify scripts sync` — push each fetcher's entry script (`fetcher.py` / `fetcher.sh`) to Paramify and CONNECT it to that fetcher's evidence set, so the tenant records *how* each piece of evidence is generated. A provisioning step @@ -108,6 +114,7 @@ change before 1.0 (see [`docs/versioning.md`](docs/versioning.md)). - TUI restyled — border titles, status pills, denser controls, and hatched empty states. -[Unreleased]: https://github.com/paramify/paramify-fetchers/compare/v0.2.1-beta...HEAD +[Unreleased]: https://github.com/paramify/paramify-fetchers/compare/v0.3.0-beta...HEAD +[0.3.0-beta]: https://github.com/paramify/paramify-fetchers/compare/v0.2.1-beta...v0.3.0-beta [0.2.1-beta]: https://github.com/paramify/paramify-fetchers/compare/v0.2.0-beta...v0.2.1-beta [0.2.0-beta]: https://github.com/paramify/paramify-fetchers/releases/tag/v0.2.0-beta diff --git a/README.md b/README.md index 9de471d..70491eb 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ [![CI](https://github.com/paramify/paramify-fetchers/actions/workflows/ci.yml/badge.svg)](https://github.com/paramify/paramify-fetchers/actions/workflows/ci.yml) [![License: GPLv3](https://img.shields.io/badge/License-GPLv3-1467ff.svg)](LICENSE) [![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-1467ff.svg)](pyproject.toml) -[![Version](https://img.shields.io/badge/version-0.2.0-1467ff.svg)](CHANGELOG.md) +[![Version](https://img.shields.io/badge/version-0.3.0--beta-1467ff.svg)](CHANGELOG.md) [![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/paramify/paramify-fetchers) Fetchers are small scripts that collect compliance evidence from your infrastructure and write it to disk as JSON. A separate uploader stage pushes that evidence to Paramify. This repo contains the fetchers, the runner that executes them, and the uploader — the fetchers themselves never talk to Paramify directly. @@ -21,6 +21,7 @@ Fetchers are small scripts that collect compliance evidence from your infrastruc
AWS +Datadog Okta SentinelOne KnowBe4 @@ -33,7 +34,8 @@ Fetchers are small scripts that collect compliance evidence from your infrastruc | Category | Fetchers | What it collects | Status | |---|---:|---|---| -| **AWS** | 79 | Encryption at rest, IAM, high availability, logging, network segmentation — across the AWS service surface | ✅ complete | +| **AWS** | 80 | Encryption at rest, IAM, high availability, logging, network segmentation — across the AWS service surface | ✅ complete | +| **Datadog** | 13 | Cloud SIEM detection rules & signals, log pipelines/indexes/archives, host & container inventory, agent checks, APM services, and incidents with timelines | broad coverage | | **Okta** | 8 | Phishing-resistant MFA, authenticators, least privilege, just-in-time access, account management | starter set | | **SentinelOne** | 5 | Agents, activities, cloud detection rules, XDR assets, user config | starter set | | **KnowBe4** | 4 | Security-awareness, high-risk, developer, and module-based training summaries | starter set | @@ -50,7 +52,6 @@ More integrations are in progress. To request a fetcher or upvote what should be SSL Labs Wiz -Datadog CrowdStrike ServiceNow diff --git a/docs/design.md b/docs/design.md index 2b8a704..703b9e7 100644 --- a/docs/design.md +++ b/docs/design.md @@ -283,7 +283,7 @@ Benefits of separation: - The Wiz-style case (writing issues back to Paramify, not just evidence) becomes a different uploader, not a hack inside the fetcher The evidence uploader is built (and exposed as `paramify upload`): -`uploaders/paramify_evidence/` reads an enveloped run directory, gets-or-creates an evidence set by `reference_id` (Paramify REST v0), multipart-uploads the artifact, and is idempotent within a run. It supports `--dry-run`, `--config`, an https-only token guard, customer `reference_id` overrides, and auth via `PARAMIFY_UPLOAD_API_TOKEN` (+ optional `PARAMIFY_API_BASE_URL` / `--config base_url`). The Wiz-style issues uploader (`uploaders/paramify_issues/`) is still an empty stub. +`uploaders/paramify_evidence/` reads an enveloped run directory, gets-or-creates an evidence set by `reference_id` (Paramify REST v0), multipart-uploads the artifact, and is idempotent within a run. It supports `--dry-run`, `--config`, an https-only token guard, customer `reference_id` overrides, and auth via `PARAMIFY_UPLOAD_API_TOKEN` (+ optional `PARAMIFY_API_BASE_URL` / `--config base_url`). A second uploader, `uploaders/paramify_scripts/` (exposed as `paramify scripts sync`), is also built — a provisioning step that pushes each fetcher's entry script to Paramify and CONNECTs it to that fetcher's evidence set, manifest-scoped by default (`--all` for the whole catalog). See [`uploader_design.md`](uploader_design.md) for both. The Wiz-style issues uploader (`uploaders/paramify_issues/`) is still an empty stub. Orchestration that chains collect → upload is customer-owned, not built into the runner. `run_and_upload.sh` at the repo root is example glue. @@ -323,25 +323,28 @@ paramify-fetchers/ │ ├── run_manifest_schema.json │ └── envelope_schema.json │ -├── fetchers/ # 107 fetchers across 8 categories +├── fetchers/ # 122 fetchers across 10 categories │ ├── _categories/ # platform-wide config + auth per category │ │ ├── okta.yaml │ │ ├── aws.yaml │ │ └── ... # (+ azure/ssllabs/wiz stubs — no ported fetchers) │ ├── _template/ # starter directory for new fetchers +│ ├── aws/ # 80 bash (largest category; fanout per region/profile) +│ ├── datadog/ # 13 Python (SIEM, logs, infra, APM, incidents) │ ├── okta/ # 8 (7 Python KSI wrappers + 1 bash); _shared/okta_iam_core.py -│ ├── aws/ # 79 bash (largest category; fanout per region/profile) │ ├── sentinelone/ # 5 single-target Python │ ├── knowbe4/ # 4 bash │ ├── k8s/ # 3 bash (aws-cli + kubectl) │ ├── rippling/ # 3 single-target Python │ ├── gitlab/ # 3 fanout-capable Python (e.g. ci_cd_pipeline_config) -│ └── checkov/ # 2 bash IaC scanners (terraform + kubernetes) +│ ├── checkov/ # 2 bash IaC scanners (terraform + kubernetes) +│ └── demo/ # 1 credential-free synthetic demo (demo_hello) │ ├── comparators/ # scaffold only (_template/); no comparator ported, runner doesn't honor depends_on │ ├── uploaders/ │ ├── paramify_evidence/ # BUILT — get-or-create evidence set + multipart upload +│ ├── paramify_scripts/ # BUILT — sync entry scripts + associate to evidence sets │ └── paramify_issues/ # empty stub (Wiz-style issues; not built) │ ├── deploy/ # containerized bundle — the MVP deployment @@ -411,10 +414,10 @@ The current approach parses JSON output with regex to determine pass/fail. Most ## Current state of the work -**This section, together with `CLAUDE.md`, is the kept-current account of -what's ported and what's in progress.** Snapshot: 107 fetchers across -8 categories (okta, aws, sentinelone, knowbe4, gitlab, k8s, rippling, checkov); the -AWS port is complete (79/79). The pieces that make this run: +**This section is the kept-current account of what's ported and what's in +progress.** Snapshot: 122 fetchers across 10 categories (aws, datadog, okta, +sentinelone, knowbe4, gitlab, k8s, rippling, checkov, demo); the AWS port is +complete (80/80). The pieces that make this run: - **Facade + three front-ends** (`framework/api.py`) — all discovery, manifest editing, validate, and run go through one facade; the human CLI, the `--json` AI CLI, and the Textual TUI (`paramify tui`) all call only the facade - **Fetcher schema** (`framework/schemas/fetcher_schema.json`) — supports fanout: `supports_targets`, `target_schema`, `per_target` secrets, `output.aggregation`. Extended additively from the original minimal version. @@ -433,6 +436,7 @@ Done since the last revision: - ~~**Config injection**~~ — DONE. Category defaults ← platform config ← per-fetcher config, injected as env vars via `config_schema` `env` mappings; `auth.passthrough_env` opens the whitelist for ambient cloud vars. - ~~**Evidence-set identity**~~ — DONE. Every `fetcher.yaml` carries an `evidence_set` block (reference_id / name / instructions), backfilled from the upstream catalog; it flows into the envelope and drives uploader get-or-create. - ~~**Evidence uploader**~~ — DONE. `uploaders/paramify_evidence/` ships (get-or-create by reference_id, multipart upload, idempotent, `--dry-run`, https-only token guard). +- ~~**Scripts uploader**~~ — DONE. `uploaders/paramify_scripts/` (`paramify scripts sync`) provisions each fetcher's entry script and associates it to the fetcher's evidence set — GitOps reconcile (marker identity, version signal, sha256 drift guard), manifest-scoped by default with `--all`. See [`uploader_design.md`](uploader_design.md). What's deferred: diff --git a/docs/envelope_design.md b/docs/envelope_design.md index 0b72aeb..4ec1f00 100644 --- a/docs/envelope_design.md +++ b/docs/envelope_design.md @@ -77,7 +77,7 @@ fetcher writes it. Fetchers do not change.** Rationale: - The runner already knows every metadata field — it computes the same values for `_run_metadata.json` (name, version, target, run_id, timestamps, exit_code). -- **Zero fetcher changes.** Wrapping in each of the 107 fetchers would be 107 edits +- **Zero fetcher changes.** Wrapping in each of the 122 fetchers would be 122 edits and would grow with every new port. One implementation point instead. - Keeps the v0.x interim clause true: fetchers still write raw evidence dicts; the framework adds the envelope. A fetcher can later emit its own envelope and diff --git a/pyproject.toml b/pyproject.toml index eb5b938..1a4cbf0 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "paramify-fetchers" -version = "0.2.1" +version = "0.3.0" description = "Paramify evidence fetcher framework — discover, build, run, and inspect evidence fetchers." readme = "README.md" requires-python = ">=3.10"