Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,14 @@ schemas and the `paramify` CLI — not the internal code.

## [Unreleased]

## [0.3.0-beta] - 2026-07-23

### Added

- 13 Datadog fetchers (new category): Cloud SIEM detection rules & signals, SIEM
operational configuration, log pipelines / indexes / archives, host & container
inventory, agent check results, the APM service catalog, and incident records with
timelines. Credential setup in [`fetchers/datadog/README.md`](fetchers/datadog/README.md).
- `paramify scripts sync` — push each fetcher's entry script (`fetcher.py` /
`fetcher.sh`) to Paramify and CONNECT it to that fetcher's evidence set, so the
tenant records *how* each piece of evidence is generated. A provisioning step
Expand Down Expand Up @@ -108,6 +114,7 @@ change before 1.0 (see [`docs/versioning.md`](docs/versioning.md)).
- TUI restyled — border titles, status pills, denser controls, and hatched empty
states.

[Unreleased]: https://github.com/paramify/paramify-fetchers/compare/v0.2.1-beta...HEAD
[Unreleased]: https://github.com/paramify/paramify-fetchers/compare/v0.3.0-beta...HEAD
[0.3.0-beta]: https://github.com/paramify/paramify-fetchers/compare/v0.2.1-beta...v0.3.0-beta
[0.2.1-beta]: https://github.com/paramify/paramify-fetchers/compare/v0.2.0-beta...v0.2.1-beta
[0.2.0-beta]: https://github.com/paramify/paramify-fetchers/releases/tag/v0.2.0-beta
7 changes: 4 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
[![CI](https://github.com/paramify/paramify-fetchers/actions/workflows/ci.yml/badge.svg)](https://github.com/paramify/paramify-fetchers/actions/workflows/ci.yml)
[![License: GPLv3](https://img.shields.io/badge/License-GPLv3-1467ff.svg)](LICENSE)
[![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-1467ff.svg)](pyproject.toml)
[![Version](https://img.shields.io/badge/version-0.2.0-1467ff.svg)](CHANGELOG.md)
[![Version](https://img.shields.io/badge/version-0.3.0--beta-1467ff.svg)](CHANGELOG.md)
[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/paramify/paramify-fetchers)

Fetchers are small scripts that collect compliance evidence from your infrastructure and write it to disk as JSON. A separate uploader stage pushes that evidence to Paramify. This repo contains the fetchers, the runner that executes them, and the uploader — the fetchers themselves never talk to Paramify directly.
Expand All @@ -21,6 +21,7 @@ Fetchers are small scripts that collect compliance evidence from your infrastruc
<div align="center">

<a href="fetchers/aws/"><img src="fetchers/logos/aws.svg" alt="AWS" width="56" height="56" style="margin: 20px;"></a>
<a href="fetchers/datadog/"><img src="fetchers/logos/datadog.png" alt="Datadog" width="56" height="56" style="margin: 20px;"></a>
<a href="fetchers/okta/"><img src="fetchers/logos/okta.svg" alt="Okta" width="56" height="56" style="margin: 20px;"></a>
<a href="fetchers/sentinelone/"><img src="fetchers/logos/sentinelone.svg" alt="SentinelOne" width="56" height="56" style="margin: 20px;"></a>
<a href="fetchers/knowbe4/"><img src="fetchers/logos/knowbe4.svg" alt="KnowBe4" width="56" height="56" style="margin: 20px;"></a>
Expand All @@ -33,7 +34,8 @@ Fetchers are small scripts that collect compliance evidence from your infrastruc

| Category | Fetchers | What it collects | Status |
|---|---:|---|---|
| **AWS** | 79 | Encryption at rest, IAM, high availability, logging, network segmentation — across the AWS service surface | ✅ complete |
| **AWS** | 80 | Encryption at rest, IAM, high availability, logging, network segmentation — across the AWS service surface | ✅ complete |
| **Datadog** | 13 | Cloud SIEM detection rules & signals, log pipelines/indexes/archives, host & container inventory, agent checks, APM services, and incidents with timelines | broad coverage |
| **Okta** | 8 | Phishing-resistant MFA, authenticators, least privilege, just-in-time access, account management | starter set |
| **SentinelOne** | 5 | Agents, activities, cloud detection rules, XDR assets, user config | starter set |
| **KnowBe4** | 4 | Security-awareness, high-risk, developer, and module-based training summaries | starter set |
Expand All @@ -50,7 +52,6 @@ More integrations are in progress. To request a fetcher or upvote what should be

<img src="fetchers/logos/qualys.svg" alt="SSL Labs" width="56" height="56" style="margin: 20px;">
<img src="fetchers/logos/wiz.jpeg" alt="Wiz" width="56" height="56" style="margin: 20px;">
<img src="fetchers/logos/datadog.png" alt="Datadog" width="56" height="56" style="margin: 20px;">
<img src="fetchers/logos/crowdstrike.svg" alt="CrowdStrike" width="56" height="56" style="margin: 20px;">
<img src="fetchers/logos/servicenow.svg" alt="ServiceNow" width="56" height="56" style="margin: 20px;">

Expand Down
20 changes: 12 additions & 8 deletions docs/design.md
Original file line number Diff line number Diff line change
Expand Up @@ -283,7 +283,7 @@ Benefits of separation:
- The Wiz-style case (writing issues back to Paramify, not just evidence) becomes a different uploader, not a hack inside the fetcher

The evidence uploader is built (and exposed as `paramify upload`):
`uploaders/paramify_evidence/` reads an enveloped run directory, gets-or-creates an evidence set by `reference_id` (Paramify REST v0), multipart-uploads the artifact, and is idempotent within a run. It supports `--dry-run`, `--config`, an https-only token guard, customer `reference_id` overrides, and auth via `PARAMIFY_UPLOAD_API_TOKEN` (+ optional `PARAMIFY_API_BASE_URL` / `--config base_url`). The Wiz-style issues uploader (`uploaders/paramify_issues/`) is still an empty stub.
`uploaders/paramify_evidence/` reads an enveloped run directory, gets-or-creates an evidence set by `reference_id` (Paramify REST v0), multipart-uploads the artifact, and is idempotent within a run. It supports `--dry-run`, `--config`, an https-only token guard, customer `reference_id` overrides, and auth via `PARAMIFY_UPLOAD_API_TOKEN` (+ optional `PARAMIFY_API_BASE_URL` / `--config base_url`). A second uploader, `uploaders/paramify_scripts/` (exposed as `paramify scripts sync`), is also built — a provisioning step that pushes each fetcher's entry script to Paramify and CONNECTs it to that fetcher's evidence set, manifest-scoped by default (`--all` for the whole catalog). See [`uploader_design.md`](uploader_design.md) for both. The Wiz-style issues uploader (`uploaders/paramify_issues/`) is still an empty stub.

Orchestration that chains collect → upload is customer-owned, not built into the runner. `run_and_upload.sh` at the repo root is example glue.

Expand Down Expand Up @@ -323,25 +323,28 @@ paramify-fetchers/
│ ├── run_manifest_schema.json
│ └── envelope_schema.json
├── fetchers/ # 107 fetchers across 8 categories
├── fetchers/ # 122 fetchers across 10 categories
│ ├── _categories/ # platform-wide config + auth per category
│ │ ├── okta.yaml
│ │ ├── aws.yaml
│ │ └── ... # (+ azure/ssllabs/wiz stubs — no ported fetchers)
│ ├── _template/ # starter directory for new fetchers
│ ├── aws/ # 80 bash (largest category; fanout per region/profile)
│ ├── datadog/ # 13 Python (SIEM, logs, infra, APM, incidents)
│ ├── okta/ # 8 (7 Python KSI wrappers + 1 bash); _shared/okta_iam_core.py
│ ├── aws/ # 79 bash (largest category; fanout per region/profile)
│ ├── sentinelone/ # 5 single-target Python
│ ├── knowbe4/ # 4 bash
│ ├── k8s/ # 3 bash (aws-cli + kubectl)
│ ├── rippling/ # 3 single-target Python
│ ├── gitlab/ # 3 fanout-capable Python (e.g. ci_cd_pipeline_config)
│ └── checkov/ # 2 bash IaC scanners (terraform + kubernetes)
│ ├── checkov/ # 2 bash IaC scanners (terraform + kubernetes)
│ └── demo/ # 1 credential-free synthetic demo (demo_hello)
├── comparators/ # scaffold only (_template/); no comparator ported, runner doesn't honor depends_on
├── uploaders/
│ ├── paramify_evidence/ # BUILT — get-or-create evidence set + multipart upload
│ ├── paramify_scripts/ # BUILT — sync entry scripts + associate to evidence sets
│ └── paramify_issues/ # empty stub (Wiz-style issues; not built)
├── deploy/ # containerized bundle — the MVP deployment
Expand Down Expand Up @@ -411,10 +414,10 @@ The current approach parses JSON output with regex to determine pass/fail. Most

## Current state of the work

**This section, together with `CLAUDE.md`, is the kept-current account of
what's ported and what's in progress.** Snapshot: 107 fetchers across
8 categories (okta, aws, sentinelone, knowbe4, gitlab, k8s, rippling, checkov); the
AWS port is complete (79/79). The pieces that make this run:
**This section is the kept-current account of what's ported and what's in
progress.** Snapshot: 122 fetchers across 10 categories (aws, datadog, okta,
sentinelone, knowbe4, gitlab, k8s, rippling, checkov, demo); the AWS port is
complete (80/80). The pieces that make this run:

- **Facade + three front-ends** (`framework/api.py`) — all discovery, manifest editing, validate, and run go through one facade; the human CLI, the `--json` AI CLI, and the Textual TUI (`paramify tui`) all call only the facade
- **Fetcher schema** (`framework/schemas/fetcher_schema.json`) — supports fanout: `supports_targets`, `target_schema`, `per_target` secrets, `output.aggregation`. Extended additively from the original minimal version.
Expand All @@ -433,6 +436,7 @@ Done since the last revision:
- ~~**Config injection**~~ — DONE. Category defaults ← platform config ← per-fetcher config, injected as env vars via `config_schema` `env` mappings; `auth.passthrough_env` opens the whitelist for ambient cloud vars.
- ~~**Evidence-set identity**~~ — DONE. Every `fetcher.yaml` carries an `evidence_set` block (reference_id / name / instructions), backfilled from the upstream catalog; it flows into the envelope and drives uploader get-or-create.
- ~~**Evidence uploader**~~ — DONE. `uploaders/paramify_evidence/` ships (get-or-create by reference_id, multipart upload, idempotent, `--dry-run`, https-only token guard).
- ~~**Scripts uploader**~~ — DONE. `uploaders/paramify_scripts/` (`paramify scripts sync`) provisions each fetcher's entry script and associates it to the fetcher's evidence set — GitOps reconcile (marker identity, version signal, sha256 drift guard), manifest-scoped by default with `--all`. See [`uploader_design.md`](uploader_design.md).

What's deferred:

Expand Down
2 changes: 1 addition & 1 deletion docs/envelope_design.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ fetcher writes it. Fetchers do not change.**
Rationale:
- The runner already knows every metadata field — it computes the same values for
`_run_metadata.json` (name, version, target, run_id, timestamps, exit_code).
- **Zero fetcher changes.** Wrapping in each of the 107 fetchers would be 107 edits
- **Zero fetcher changes.** Wrapping in each of the 122 fetchers would be 122 edits
and would grow with every new port. One implementation point instead.
- Keeps the v0.x interim clause true: fetchers still write raw evidence dicts;
the framework adds the envelope. A fetcher can later emit its own envelope and
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"

[project]
name = "paramify-fetchers"
version = "0.2.1"
version = "0.3.0"
description = "Paramify evidence fetcher framework — discover, build, run, and inspect evidence fetchers."
readme = "README.md"
requires-python = ">=3.10"
Expand Down
Loading