[stormshield] Add outcome, identity and HTTP mappings and extend the metadata allow-list#20085
[stormshield] Add outcome, identity and HTTP mappings and extend the metadata allow-list#20085arenard wants to merge 3 commits into
Conversation
Fix the copy-pasted descriptions of the default and count ingest pipelines, remove field definitions that never appear in events (the pipeline consumes tz into event.timezone, converts duration into event.duration, renames msg to message, and drops origdst/origdstport after converting them to destination fields), and correct the documentation: the agent syslog processor is configured for RFC5424, so the README no longer presents the Legacy format as supported, and the compatibility section now covers SNS 4.x and 5.x, which share the same WELF key-value audit log format. Also document how to collect SNS IPFIX flow records with the NetFlow integration, update vendor links to the SNS v5 documentation, and use arrows for menu paths per the documentation style guide.
d7aa3c0 to
44bf0a4
Compare
✅ Elastic Docs Style Checker (Vale)No issues found on modified lines! The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale. |
🚀 Benchmarks reportTo see the full report comment with |
|
Pinging @elastic/integration-experience (Team:Integration-Experience) |
| | connection | event | network, session | connection, end | | ||
| | count, filterstat, ipsecstat, routerstat | metric | network | info | | ||
| | date | event | host | change | | ||
| | dmrouting, routing | event | network | change | |
There was a problem hiding this comment.
Severity: 🔵 Low confidence: high path: packages/stormshield/_dev/build/docs/README.md:222
The categorization table lists event.category network for dmrouting/routing, but the pipeline sets network, host; update the table row to match.
Details
The ECS categorization table documents the dmrouting, routing families as event.category: network. The pipeline's categorization script (data_stream/log/elasticsearch/ingest_pipeline/default.yml) maps both dmrouting and routing to category: [network, host] (kind event, type change). Every other row in the table matches the pipeline; this row is the only discrepancy, so a reader relying on the docs to build queries/filters on event.category for routing events would get an incomplete picture. The generated docs/README.md (line 285) carries the same error; fixing the template here regenerates it.
Recommendation:
Update the table row so the documented category matches the pipeline (category: [network, host]):
| dmrouting, routing | event | network, host | change |🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
There was a problem hiding this comment.
Right — the pipeline was changed to network, host for these two families during an ECS expected-combination review, and the table row was missed. Fixed at the source: the row now reads network, host (amended in #20031, which owns this content; this branch has been rebased on it).
| changes: | ||
| - description: Set ECS event categorization fields (`event.kind`, `event.category`, `event.type`) for every log family, including `event.kind` `metric` for the periodic statistics families. | ||
| type: enhancement | ||
| link: https://github.com/elastic/integrations/pull/20031 |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/stormshield/changelog.yml:14
Changelog links a different PR number
Details
This changelog entry's link: points at pull/20031, but it was added in PR #20085. It is likely a leftover template placeholder or a copy from another PR.
Recommendation:
Point the changelog link at this PR:
link: https://github.com/elastic/integrations/pull/20085🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
There was a problem hiding this comment.
This entry belongs to a release introduced by a parent PR of this stacked branch (#20031 for this line); it only appears in this diff because the branch is stacked. The link is correct and this resolves once the parent PRs merge and this branch is rebased.
| link: https://github.com/elastic/integrations/pull/20031 | ||
| - description: Map `rcvd` to `destination.bytes`, sum `network.bytes`, map interface fields to `observer.ingress` and `observer.egress`, compute `network.direction`, and populate `related.hosts`. | ||
| type: enhancement | ||
| link: https://github.com/elastic/integrations/pull/20031 |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/stormshield/changelog.yml:17
Changelog links a different PR number
Details
This changelog entry's link: points at pull/20031, but it was added in PR #20085. It is likely a leftover template placeholder or a copy from another PR.
Recommendation:
Point the changelog link at this PR:
link: https://github.com/elastic/integrations/pull/20085🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
There was a problem hiding this comment.
This entry belongs to a release introduced by a parent PR of this stacked branch (#20031 for this line); it only appears in this diff because the branch is stacked. The link is correct and this resolves once the parent PRs merge and this branch is rebased.
| link: https://github.com/elastic/integrations/pull/20031 | ||
| - description: Map alarm identifiers and levels to `event.code`, `event.risk_score` and `event.severity`, and HTTP proxy fields to `url.original` and `url.domain`. | ||
| type: enhancement | ||
| link: https://github.com/elastic/integrations/pull/20031 |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/stormshield/changelog.yml:20
Changelog links a different PR number
Details
This changelog entry's link: points at pull/20031, but it was added in PR #20085. It is likely a leftover template placeholder or a copy from another PR.
Recommendation:
Point the changelog link at this PR:
link: https://github.com/elastic/integrations/pull/20085🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
There was a problem hiding this comment.
This entry belongs to a release introduced by a parent PR of this stacked branch (#20031 for this line); it only appears in this diff because the branch is stacked. The link is correct and this resolves once the parent PRs merge and this branch is rebased.
| link: https://github.com/elastic/integrations/pull/20031 | ||
| - description: Fix the count sub-pipeline, which failed with a Painless runtime error on every count log, and skip malformed counters instead of failing the whole document. | ||
| type: bugfix | ||
| link: https://github.com/elastic/integrations/pull/20031 |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/stormshield/changelog.yml:23
Changelog links a different PR number
Details
This changelog entry's link: points at pull/20031, but it was added in PR #20085. It is likely a leftover template placeholder or a copy from another PR.
Recommendation:
Point the changelog link at this PR:
link: https://github.com/elastic/integrations/pull/20085🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
There was a problem hiding this comment.
This entry belongs to a release introduced by a parent PR of this stacked branch (#20031 for this line); it only appears in this diff because the branch is stacked. The link is correct and this resolves once the parent PRs merge and this branch is rebased.
| link: https://github.com/elastic/integrations/pull/20031 | ||
| - description: Add an Alarms dashboard and pipeline test samples for the alarm, auth, web, vpn, xvpn, count and filter log families. | ||
| type: enhancement | ||
| link: https://github.com/elastic/integrations/pull/20031 |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/stormshield/changelog.yml:26
Changelog links a different PR number
Details
This changelog entry's link: points at pull/20031, but it was added in PR #20085. It is likely a leftover template placeholder or a copy from another PR.
Recommendation:
Point the changelog link at this PR:
link: https://github.com/elastic/integrations/pull/20085🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
There was a problem hiding this comment.
This entry belongs to a release introduced by a parent PR of this stacked branch (#20031 for this line); it only appears in this diff because the branch is stacked. The link is correct and this resolves once the parent PRs merge and this branch is rebased.
| changes: | ||
| - description: Fix ingest pipeline descriptions and remove field definitions that never appear in events (`stormshield.tz`, `stormshield.duration`, `stormshield.msg`, `stormshield.origdst`, `stormshield.origdstport`). | ||
| type: bugfix | ||
| link: https://github.com/elastic/integrations/pull/20030 |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/stormshield/changelog.yml:31
Changelog links a different PR number
Details
This changelog entry's link: points at pull/20030, but it was added in PR #20085. It is likely a leftover template placeholder or a copy from another PR.
Recommendation:
Point the changelog link at this PR:
link: https://github.com/elastic/integrations/pull/20085🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
There was a problem hiding this comment.
This entry belongs to a release introduced by a parent PR of this stacked branch (#20030 for this line); it only appears in this diff because the branch is stacked. The link is correct and this resolves once the parent PRs merge and this branch is rebased.
| link: https://github.com/elastic/integrations/pull/20030 | ||
| - description: Correct the syslog format and SNS version compatibility statements in the documentation, and document IPFIX flow collection through the NetFlow integration. | ||
| type: bugfix | ||
| link: https://github.com/elastic/integrations/pull/20030 |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/stormshield/changelog.yml:34
Changelog links a different PR number
Details
This changelog entry's link: points at pull/20030, but it was added in PR #20085. It is likely a leftover template placeholder or a copy from another PR.
Recommendation:
Point the changelog link at this PR:
link: https://github.com/elastic/integrations/pull/20085🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
There was a problem hiding this comment.
This entry belongs to a release introduced by a parent PR of this stacked branch (#20030 for this line); it only appears in this diff because the branch is stacked. The link is correct and this resolves once the parent PRs merge and this branch is rebased.
Set event.kind, event.category and event.type for every SNS log family through a table-driven Painless script keyed on logtype, with allowed/denied event.type modifiers derived case-insensitively from the rule action. The periodic statistics families (monitor, filterstat, count, routerstat, authstat, ipsecstat) are tagged event.kind: metric so health telemetry can be excluded from detection scopes with a single filter. Unknown families fall back to event/network/info, covered by a dedicated test sample. The script only runs on documents without an error, so the pipeline_error kind set by sub-pipeline failure handlers is preserved. Add the ECS network mappings tracked in elastic#10114: rcvd to destination.bytes (mirroring the existing sent mapping), network.bytes as the sum of both, srcif/srcifname/dstif/dstifname copied to observer.ingress and observer.egress interface fields, and network.direction computed with private ranges as internal networks. Promote alarm fields (alarmid to event.code, risk to event.risk_score, pri to event.severity on the alarm scale where 1 is major and 4 is minor), HTTP proxy fields (arg to url.original, url.domain from destination.domain), and populate related.hosts. All mappings are copies; no stormshield.* field moves or changes location. Guard the ASN geoip processors with null checks, matching the geo processors. Fix the count sub-pipeline, which failed on every count log since creation: the script called splitByToken, which does not exist in Painless, and collapsed all rules into a single map. Rule counters are now a sorted array under stormshield.metadata.rule_stats. Extend the metadata move list with the documented alarm, proxy, vpn and auth fields so they land under stormshield.metadata instead of being left unmapped. Add pipeline test samples for the alarm, auth, web, vpn, xvpn, count and filter families, derived from the log format examples in the official Stormshield documentation with sanitized addresses, including uppercase and empty action values. Add an Alarms (IPS) dashboard keyed on the new categorization fields. Document the categorization table in the README.
…metadata allow-list Derive `event.outcome` on the auth, server and xvpn families from the vendor `error` return code (`0` maps to success, other codes to failure). The official field documentation lists exactly these three families for `error`; on vpn logs the same key is a severity level and is left untouched. The vendor documentation pairs `0`-first numeric examples with `Success`-first display labels without stating the convention outright, so the mapping follows the strong implication of the field tables. Map `domain` to `user.domain` and `usergroup` to `user.group.name`, splitting the comma-separated multi-valued form that SNS 5.x emits. On web logs, map `op` to `http.request.method` and `result` to `http.response.status_code`; on other families these keys carry protocol-specific codes (FTP, Modbus) and are not promoted. Map the module name in `service` to `process.name` on the system, sandboxing and routing families only — on pvm logs the same key names the vulnerable product detected on a scanned host. Set `observer.product`. All mappings are copies; the vendor keys remain available under `stormshield.metadata`. Extend the `stormshield.metadata` allow-list with the documented vendor keys that previously leaked as undeclared `stormshield.*` fields: host and IP reputation (`srchostrep`, `dsthostrep`, `srciprep`, `dstiprep`), alarm packet captures (`pktlen`, `pktdump`, `pktdumplen`), Ethernet protocol (`etherproto`), SSO agent (`agentid`), IKE peer name (`peername`), and the industrial-protocol plugin fields (`unitid`/`unit_id`, `cipservicecode`, `cipclassid`, `requestmode`, `responsemode`, `UI`, `error_class`, `error_code`, `format`). The NAT reading of the pipeline was verified against the official field tables while preparing this change: `origdst`/`origdstport` are documented as the destination before translation and map to `destination.ip`/`destination.port`; `dst` is the translated destination and maps to `destination.nat.ip` when it differs. A connection sample with the documented example values now locks this in the pipeline tests, alongside new samples for the alarm reputation and packet-capture fields, a Modbus plugin line (whose `result` stays under metadata and does not produce `http.*`), a server line, a system line, and multi- and single-valued `usergroup` forms.
44bf0a4 to
8dc4174
Compare
|
Changelog link mismatch — expected
Tip If expected, add the |
💔 Build Failed
Failed CI StepsHistory
|
| changes: | ||
| - description: Set ECS event categorization fields (`event.kind`, `event.category`, `event.type`) for every log family, including `event.kind` `metric` for the periodic statistics families. | ||
| type: enhancement | ||
| link: https://github.com/elastic/integrations/pull/20031 |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/stormshield/changelog.yml:14
Changelog links a different PR number
Details
This changelog entry's link: points at pull/20031, but it was added in PR #20085. It is likely a leftover template placeholder or a copy from another PR.
Recommendation:
Point the changelog link at this PR:
link: https://github.com/elastic/integrations/pull/20085🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| link: https://github.com/elastic/integrations/pull/20031 | ||
| - description: Map `rcvd` to `destination.bytes`, sum `network.bytes`, map interface fields to `observer.ingress` and `observer.egress`, compute `network.direction`, and populate `related.hosts`. | ||
| type: enhancement | ||
| link: https://github.com/elastic/integrations/pull/20031 |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/stormshield/changelog.yml:17
Changelog links a different PR number
Details
This changelog entry's link: points at pull/20031, but it was added in PR #20085. It is likely a leftover template placeholder or a copy from another PR.
Recommendation:
Point the changelog link at this PR:
link: https://github.com/elastic/integrations/pull/20085🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| link: https://github.com/elastic/integrations/pull/20031 | ||
| - description: Map alarm identifiers and levels to `event.code`, `event.risk_score` and `event.severity`, and HTTP proxy fields to `url.original` and `url.domain`. | ||
| type: enhancement | ||
| link: https://github.com/elastic/integrations/pull/20031 |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/stormshield/changelog.yml:20
Changelog links a different PR number
Details
This changelog entry's link: points at pull/20031, but it was added in PR #20085. It is likely a leftover template placeholder or a copy from another PR.
Recommendation:
Point the changelog link at this PR:
link: https://github.com/elastic/integrations/pull/20085🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| link: https://github.com/elastic/integrations/pull/20031 | ||
| - description: Fix the count sub-pipeline, which failed with a Painless runtime error on every count log, and skip malformed counters instead of failing the whole document. | ||
| type: bugfix | ||
| link: https://github.com/elastic/integrations/pull/20031 |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/stormshield/changelog.yml:23
Changelog links a different PR number
Details
This changelog entry's link: points at pull/20031, but it was added in PR #20085. It is likely a leftover template placeholder or a copy from another PR.
Recommendation:
Point the changelog link at this PR:
link: https://github.com/elastic/integrations/pull/20085🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| link: https://github.com/elastic/integrations/pull/20031 | ||
| - description: Add an Alarms dashboard and pipeline test samples for the alarm, auth, web, vpn, xvpn, count and filter log families. | ||
| type: enhancement | ||
| link: https://github.com/elastic/integrations/pull/20031 |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/stormshield/changelog.yml:26
Changelog links a different PR number
Details
This changelog entry's link: points at pull/20031, but it was added in PR #20085. It is likely a leftover template placeholder or a copy from another PR.
Recommendation:
Point the changelog link at this PR:
link: https://github.com/elastic/integrations/pull/20085🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| changes: | ||
| - description: Fix ingest pipeline descriptions and remove field definitions that never appear in events (`stormshield.tz`, `stormshield.duration`, `stormshield.msg`, `stormshield.origdst`, `stormshield.origdstport`). | ||
| type: bugfix | ||
| link: https://github.com/elastic/integrations/pull/20030 |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/stormshield/changelog.yml:31
Changelog links a different PR number
Details
This changelog entry's link: points at pull/20030, but it was added in PR #20085. It is likely a leftover template placeholder or a copy from another PR.
Recommendation:
Point the changelog link at this PR:
link: https://github.com/elastic/integrations/pull/20085🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| link: https://github.com/elastic/integrations/pull/20030 | ||
| - description: Correct the syslog format and SNS version compatibility statements in the documentation, and document IPFIX flow collection through the NetFlow integration. | ||
| type: bugfix | ||
| link: https://github.com/elastic/integrations/pull/20030 |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: high path: packages/stormshield/changelog.yml:34
Changelog links a different PR number
Details
This changelog entry's link: points at pull/20030, but it was added in PR #20085. It is likely a leftover template placeholder or a copy from another PR.
Recommendation:
Point the changelog link at this PR:
link: https://github.com/elastic/integrations/pull/20085🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
Review summaryIssues found across the latest commits 4ce4236, 8dc4174 — 7 medium
Issues found across earlier commits 44bf0a4 — 7 medium, 1 low
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
|
Proposed commit message
Derive
event.outcomeon the auth, server and xvpn families from the vendorerrorreturn code (0maps to success, other codes to failure). The official field documentation lists exactly these three families forerror; on vpn logs the same key is a severity level and is left untouched. The vendor documentation pairs0-first numeric examples withSuccess-first display labels without stating the convention outright, so the mapping follows the strong implication of the field tables.Map
domaintouser.domainandusergrouptouser.group.name, splitting the comma-separated multi-valued form that SNS 5.x emits. On web logs, mapoptohttp.request.methodandresulttohttp.response.status_code; on other families these keys carry protocol-specific codes (FTP, Modbus) and are not promoted. Map the module name inservicetoprocess.nameon the system, sandboxing and routing families only — on pvm logs the same key names the vulnerable product detected on a scanned host. Setobserver.product. All mappings are copies; the vendor keys remain available understormshield.metadata.Extend the
stormshield.metadataallow-list with the documented vendor keys that previously leaked as undeclaredstormshield.*fields: host and IP reputation (srchostrep,dsthostrep,srciprep,dstiprep), alarm packet captures (pktlen,pktdump,pktdumplen), Ethernet protocol (etherproto), SSO agent (agentid), IKE peer name (peername), and the industrial-protocol plugin fields (unitid/unit_id,cipservicecode,cipclassid,requestmode,responsemode,UI,error_class,error_code,format).The NAT reading of the pipeline was verified against the official field tables while preparing this change:
origdst/origdstportare documented as the destination before translation and map todestination.ip/destination.port;dstis the translated destination and maps todestination.nat.ipwhen it differs. A connection sample with the documented example values now locks this in the pipeline tests, alongside new samples for the alarm reputation and packet-capture fields, a Modbus plugin line (whoseresultstays under metadata and does not producehttp.*), a server line, a system line, and multi- and single-valuedusergroupforms.Design notes
Affected logslists drive the family guards). The alarmclasssub-categorization was deliberately left out: the documentation gives example values without an exhaustive list or definitions.event.outcomeis only set where the log reports a return code, never inferred from message text.stormshield.*field changes location or type, so existing dashboards and saved queries are unaffected.Checklist
changelog.ymlfile.Author's Checklist
elastic-package check,test pipeline,test staticandtest systempass against stack 9.4.3result,service,error)How to test this PR locally
Related issues