Skip to content

[IIS] Add grok pattern for s-ip + cookie + referer + split-bytes log layout#20048

Open
brian-mckinney wants to merge 3 commits into
elastic:mainfrom
brian-mckinney:fix/iis-grok-pattern-cookie-bytes-split
Open

[IIS] Add grok pattern for s-ip + cookie + referer + split-bytes log layout#20048
brian-mckinney wants to merge 3 commits into
elastic:mainfrom
brian-mckinney:fix/iis-grok-pattern-cookie-bytes-split

Conversation

@brian-mckinney

Copy link
Copy Markdown
Contributor

Summary

Fixes ingestion failures for IIS access logs using the field layout documented at packages/iis/docs/README.md:69:

date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(cookie) cs(Referer) sc-status sc-substatus sc-win32-status sc-bytes, cs-bytes time-taken

None of the 10 grok patterns shipped in packages/iis/data_stream/access/elasticsearch/ingest_pipeline/default.yml (v1.24.6) implement this exact combination (s-ip without s-sitename, plus cs(cookie), cs(Referer), and sc-bytes/cs-bytes as two separate fields). Every pattern fails to match, ingestion falls through to on_failure, and a stray numeric-conversion attempt on a desynced token surfaces as error.message: "For input string: ..." instead of a clean parse.

Reported in SDH-beats#7343 — customer configured IIS with exactly this documented layout and every event failed ingestion. Root cause and field-by-field trace are in that issue.

  • Adds the missing grok pattern (mirrors the existing s-ip-based pattern, adding the iis.access.cookie capture between user_agent.original and http.request.referrer)
  • Bumps package version 1.24.6 → 1.24.7
  • Adds a changelog entry
  • Adds a pipeline test log fixture reproducing the customer's field layout (values anonymized)

Test plan

  • Verified by hand that the new pattern's 17 capture groups align 1:1 with the 17 space-delimited tokens in the added test log (timestamp consumes the date time pair as a single TIMESTAMP_ISO8601 match)
  • elastic-package test pipeline --generate --data-streams access still needs to run against a live stack to generate test-iis-access-cookie-bytes-split.log-expected.json — I don't have a stack running locally, so this is left for CI/a reviewer with elastic-package stack up available. Marking draft until that's filled in.
  • Confirm the new pattern doesn't shadow/regress any of the other 9 patterns for their respective documented layouts (existing test fixtures should still pass unchanged since pattern order for those layouts is untouched)

…layout

The documented IIS access log field layout at packages/iis/docs/README.md:69
(s-ip, cs(cookie), cs(Referer), sc-bytes, cs-bytes as separate fields) has no
matching grok pattern in the ingest pipeline, so any log using this exact
layout fails ingestion with a grok/number-conversion error instead of parsing
cleanly. Adds the missing pattern (SDH-beats#7343).
@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Elastic Docs Style Checker (Vale)

Summary: 2 suggestions found

💡 Suggestions (2): Optional style improvements. Apply when helpful.
File Line Rule Message
packages/iis/changelog.yml 4 Elastic.Ellipses In general, don't use an ellipsis.
packages/iis/changelog.yml 4 Elastic.Ellipses In general, don't use an ellipsis.

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

@github-actions

This comment has been minimized.

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

TL;DR

Check integrations iis failed after this PR added a new IIS pipeline .log fixture without the matching generated *-expected.json file. Add packages/iis/data_stream/access/_dev/test/pipeline/test-iis-access-cookie-bytes-split.log-expected.json and rerun the IIS pipeline/package tests.

Remediation

  • Generate and commit the expected output for the new fixture: packages/iis/data_stream/access/_dev/test/pipeline/test-iis-access-cookie-bytes-split.log-expected.json.
  • Validate with elastic-package test pipeline --generate --data-streams access from packages/iis, then rerun without --generate or rerun the Buildkite Check integrations iis step.
Investigation details

Root Cause

This is a test fixture completeness failure in the IIS pipeline test suite. The PR adds packages/iis/data_stream/access/_dev/test/pipeline/test-iis-access-cookie-bytes-split.log, but the PR branch directory listing contains no corresponding test-iis-access-cookie-bytes-split.log-expected.json. Existing IIS pipeline fixtures in the same directory all follow the paired naming pattern, for example test-iis-access.log plus test-iis-access.log-expected.json.

The Buildkite command was .buildkite/scripts/test_one_package.sh packages/iis origin/main ff63a013e036da07c8791ee817afb6c113276c9f; that helper calls process_package, and .buildkite/scripts/common.sh runs package tests including elastic-package test pipeline for the package.

Evidence

--- [iis] failed
🚨 Error: The command exited with status 1
build/test-results/iis-pipeline-1783548526084994555.xml
  • PR diff adds only the new input fixture at packages/iis/data_stream/access/_dev/test/pipeline/test-iis-access-cookie-bytes-split.log; get_files shows no added expected JSON for that fixture.

Verification

  • Confirmed from PR metadata and the PR branch directory listing that the new .log file exists and the matching .log-expected.json file is absent.
  • Did not rerun elastic-package test pipeline locally in this detective workflow; the checked-out workspace is main, while the failing fixture exists only on the PR branch.

Follow-up

If the generated expected document shows the new grok pattern maps fields incorrectly, adjust packages/iis/data_stream/access/elasticsearch/ingest_pipeline/default.yml before committing the expected JSON.


What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

Adds the missing test-iis-access-cookie-bytes-split.log-expected.json
fixture generated via elastic-package test pipeline --generate, which
resolves the failing IIS pipeline test.
@marc-gr marc-gr marked this pull request as ready for review July 9, 2026 09:48
@marc-gr marc-gr requested a review from a team as a code owner July 9, 2026 09:48
@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

@andrewkroh andrewkroh added Integration:iis IIS Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] labels Jul 9, 2026
@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

✅ All changelog entries have the correct PR link.

@infra-vault-gh-plugin-prod

Copy link
Copy Markdown

💚 Build Succeeded

History

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Integration:iis IIS Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations]

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants