[Azure frontdoor] Add Health Probe dataset by herrBez · Pull Request #20016 · elastic/integrations

herrBez · 2026-07-07T11:41:33Z

Proposed commit message

Add first support for Frontdoor health probe logs

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

Double check the documentation
Decide whether to activate the dataset by default or not (currently it's active by default)

How to test this PR locally

Configure an Azure Frontdoor and export the logs to an Eventhub
Configure the integration
Verify that Health Check logs are collected

Related issues

[Azure Frontdoor]: Add support for health probe log #13729

Screenshots

github-actions · 2026-07-07T11:43:52Z

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

infra-vault-gh-plugin-prod · 2026-07-07T12:02:13Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

elastic-vault-github-plugin-prod · 2026-07-07T12:20:01Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

efd6

Can you update the README?

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

vera-review-bot · 2026-07-08T06:18:46Z

@@ -0,0 +1,40 @@
+title: "FrontDoor Health Probe"


Severity: 🟡 Medium confidence: high path: packages/azure_frontdoor/data_stream/health_probe/manifest.yml:1

New health_probe data stream is added but the package README was not updated — it still says Health Probe logs are unsupported and lists only two data streams. Update the docs template to cover the new stream.

Details

This PR adds the health_probe data stream, but no documentation was changed (the README is not in the changed-file set). The rendered docs are built from _dev/build/docs/README.md, which currently contradicts the new functionality: it states "Currently, the integration does not support Activity Logs or Health Probe logs.", says "This integration collects two types of data streams: access log, waf logs", and has no field-reference section for the new stream. After this merges, users will read that health probe collection is unsupported while the integration ships it. (Anchored on the new data stream manifest because the README file itself is unchanged and cannot receive an inline comment.)

Recommendation:

Update packages/azure_frontdoor/_dev/build/docs/README.md: remove the "does not support ... Health Probe logs" statement, list the new stream, and add its generated field reference. For example:

## Data streams This integration collects the following data streams: - access log - waf logs - health probe logs ## Health Probe Logs {{fields "health_probe"}}

Also update the introductory sentence that currently reads "Currently, the integration does not support Activity Logs or Health Probe logs." so it no longer excludes Health Probe logs.

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

navnit-elastic · 2026-07-08T06:25:59Z

+      field: azure.frontdoor.health_probe.properties.originName
+      target_field: azure.frontdoor.health_probe.origin_name
+      ignore_missing: true
+  - rename:


Azure documents originIP as IP:port, so values with a port suffix are invalid for ECS ip and break GeoIP/ASN enrichment. Should we split IP and port in such cases?

- grok: field: destination.ip patterns: - '%{IP:destination.ip}:%{NUMBER:destination.port}' ignore_missing: true if: ctx.destination?.ip != null && ctx.destination.ip.contains(':') on_failure: - append: tag: append_error_message_e18f0b9e field: error.message value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'

Hey, thanks for the feedback, this documeantion https://learn.microsoft.com/en-us/azure/frontdoor/monitor-front-door?pivots=front-door-standard-premium#health-probe-log and my tests seem to confirm that originIP is an IP.

May I ask you where you found the information?

Sorry, I cannot even read the examples of the doc 🤣 (to be fair the table and the examples don't report the same information xD) Let me fix that

ctx.destination?.ip != null && ctx.destination.ip.contains(':')

Is it not a risky condition? What if the IP is IPv6?

Good catch. Yes, the condition is too broad and risky for IPv6.
A safer approach is to use convert first, then fall back to grok for host:port forms.

OK, I tried my best :) Can you double check my proposal?

Take a look at

integrations/packages/cisco_duo/data_stream/auth/elasticsearch/ingest_pipeline/default.yml

Lines 89 to 101 in 49864db

- grok:

field: json.access_device.ip

patterns:

- "^%{IPV4:json.access_device.ip}:%{PORT:json.access_device.port}$"

- "^\\[%{IPV6:json.access_device.ip}\\]:%{PORT:json.access_device.port}$"

- "^%{IPV6NOCOMPRESS:json.access_device.ip}:%{PORT:json.access_device.port}$"

- "^%{IPV6:json.access_device.ip}%{IPV6PORTSEP}%{PORT:json.access_device.port}$"

pattern_definitions:

IPV6NOCOMPRESS: '([0-9A-Fa-f]{1,4}:){7}[0-9A-Fa-f]{1,4}'

IPV6PORTSEP: '(?: port |[p#.])'

PORT: '[0-9]+'

ignore_missing: true

ignore_failure: true

Thanks @efd6.

@herrBez, Sorry for the back and forth. Rather than using grok inside the convert on_failure handler, we can:

Run grok with explicit anchored patterns (IPv4, IPv4:port, [IPv6]:port, etc.)

Validate with convert after extraction (remove the field on failure)

That avoids the need for contains(':'), while handling IP:port.

vera-review-bot · 2026-07-08T13:02:25Z


 - access log
 - waf logs
+- health probe logs


Severity: 🔵 Low confidence: high path: packages/azure_frontdoor/_dev/build/docs/README.md:24

The README now lists three data streams but the sentence above still says the integration collects "two types". Update the count to "three".

Details

This PR adds 'health probe logs' to the data streams list under the '## Data streams' section, so the list now contains three entries (access, waf, health probe). The introductory sentence immediately above the list still reads 'This integration collects two types of data streams:', which is now factually incorrect. Because the built docs/README.md is generated from this template, the same stale wording ships in the rendered README.

Recommendation:

Update the count word in the '## Data streams' section:

## Data streams This integration collects three types of data streams: - access log - waf logs - health probe logs

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

vera-review-bot · 2026-07-08T13:02:25Z

+        "type": "logs"
+    },
+    "destination": {
+        "ip": "4.246.53.113"


Severity: 🔵 Low confidence: medium path: packages/azure_frontdoor/data_stream/health_probe/sample_event.json:35

sample_event.json appears hand-authored and inconsistent with the pipeline output — destination.ip has no geo/as enrichment or destination.address. Regenerate it from a system test.

Details

The health_probe pipeline always adds destination.geo, destination.as and (via the originIP handling) destination.address whenever destination.ip is present. This sample_event.json contains destination.ip (4.246.53.113 — an address that appears in none of the pipeline test fixtures) with related.ip populated, but no destination.geo, no destination.as and no destination.address. A system-test-generated sample would carry those enrichments. Combined with the old agent version (8.6.0), this indicates the file was hand-authored rather than produced by elastic-package test system, so it does not accurately represent an ingested document in the Fleet UI.

Recommendation:

Regenerate the sample from a system test so it reflects real pipeline output (geo/ASN enrichment, destination.address):

cd packages/azure_frontdoor elastic-package test system -v --data-streams health_probe --generate

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

elastic-vault-github-plugin-prod · 2026-07-08T16:25:15Z

✅ All changelog entries have the correct PR link.

vera-review-bot · 2026-07-08T16:41:41Z

Reviewed the latest commits 409da34 — nothing new beyond already posted comments.

Review summary

Issues found across earlier commits a15ee51 — 2 low

🔵 The README now lists three data streams but the sentence above still says the integration collects "two types". Update the count to "three". (link) (Unresolved)
🔵 sample_event.json appears hand-authored and inconsistent with the pipeline output — destination.ip has no geo/as enrichment or destination.address. Regenerate it from a system test. (link) (Unresolved)

Issues found across earlier commits 9669a03 — 1 medium

🟡 New health_probe data stream is added but the package README was not updated — it still says Health Probe logs are unsupported and lists only two data streams. Update the docs template to cover the new stream. (link) (Unresolved)

A new commit triggers another review — at most once every 15 minutes. I skip the PR while it's approved or has merge conflicts.

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

infra-vault-gh-plugin-prod · 2026-07-08T16:41:47Z

💚 Build Succeeded

Buildkite Build
Commit: 409da34

History

💚 Build #45719 succeeded a15ee51
💚 Build #45688 succeeded fdcc472
💚 Build #45633 succeeded ca61077

herrBez added 3 commits July 7, 2026 12:32

Bump the version

160ba03

Add health probe dataset

43c1e78

Make sure to deal with lowercase n/a

8d58b09

herrBez requested review from a team as code owners July 7, 2026 11:41

Update PR number

b8898a1

Replace latinism with plain english

ca61077

andrewkroh added Integration:azure_frontdoor Azure Frontdoor (Community supported) Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels Jul 7, 2026

efd6 reviewed Jul 8, 2026

View reviewed changes

herrBez and others added 5 commits July 8, 2026 08:01

Apply suggestions from code review

9669a03

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

Remove event.ingested from test-common-config.yml

3147929

Add if condition to geoip processors as per review

1fd94fc

Update README

c0439c5

Update README.md

fdcc472

vera-review-bot Bot reviewed Jul 8, 2026

View reviewed changes

navnit-elastic reviewed Jul 8, 2026

View reviewed changes

andrewkroh added the documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. label Jul 8, 2026

Add support for IP:PORT format in originIP. Add test-case

a15ee51

vera-review-bot Bot reviewed Jul 8, 2026

View reviewed changes

Fix automatic check complaints

409da34

	- grok:
	field: json.access_device.ip
	patterns:
	- "^%{IPV4:json.access_device.ip}:%{PORT:json.access_device.port}$"
	- "^\\[%{IPV6:json.access_device.ip}\\]:%{PORT:json.access_device.port}$"
	- "^%{IPV6NOCOMPRESS:json.access_device.ip}:%{PORT:json.access_device.port}$"
	- "^%{IPV6:json.access_device.ip}%{IPV6PORTSEP}%{PORT:json.access_device.port}$"
	pattern_definitions:
	IPV6NOCOMPRESS: '([0-9A-Fa-f]{1,4}:){7}[0-9A-Fa-f]{1,4}'
	IPV6PORTSEP: '(?: port \|[p#.])'
	PORT: '[0-9]+'
	ignore_missing: true
	ignore_failure: true

Uh oh!

Conversation

herrBez commented Jul 7, 2026

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

github-actions Bot commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Elastic Docs Style Checker (Vale)

Uh oh!

infra-vault-gh-plugin-prod Bot commented Jul 7, 2026

Uh oh!

elastic-vault-github-plugin-prod Bot commented Jul 7, 2026

🚀 Benchmarks report

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vera-review-bot Bot Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

navnit-elastic Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

herrBez Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

herrBez Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

herrBez Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

navnit-elastic Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

herrBez Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

efd6 Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

navnit-elastic Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

vera-review-bot Bot Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

vera-review-bot Bot Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

elastic-vault-github-plugin-prod Bot commented Jul 8, 2026

Uh oh!

vera-review-bot Bot commented Jul 8, 2026

Review summary

Uh oh!

infra-vault-gh-plugin-prod Bot commented Jul 8, 2026

💚 Build Succeeded

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

github-actions Bot commented Jul 7, 2026 •

edited

Loading

navnit-elastic Jul 8, 2026 •

edited

Loading

herrBez Jul 8, 2026 •

edited

Loading