Skip to content

[cloudflare_logpush] ingest pipeline improments #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
brijesh-elastic wants to merge 14 commits into
base: main
Choose a base branch
from
Open

[cloudflare_logpush] ingest pipeline improments #3

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
a2a2f55
update ECS version to 9.3.0 and format_version to 3.3.2
brijesh-elastic Apr 9, 2026
d580393
sort fields.yml alphabetically, add new fields, and fix field descrip…
brijesh-elastic Apr 9, 2026
1e0d902
fix email_security_alerts timestamp normalization
brijesh-elastic Apr 9, 2026
fc4e9d6
fix spectrum_event broken grok guard condition and remove processor
brijesh-elastic Apr 9, 2026
376886c
fix network_analytics split processor case-sensitivity for TCPSackBlocks
brijesh-elastic Apr 9, 2026
1c4f6bf
fix audit rename condition to use PascalCase for Interface field
brijesh-elastic Apr 9, 2026
7ca6dd7
fix integer-to-keyword type casting by replacing rename with convert
brijesh-elastic Apr 9, 2026
b2aaa2b
remove ignore_failure from initial JSON processor
brijesh-elastic Apr 9, 2026
0a44ac2
convert grok processors to dissect for improved performance
brijesh-elastic Apr 9, 2026
5c8d688
consolidate timestamp handling, add convert processors, implement null
brijesh-elastic Apr 9, 2026
30d8b0e
implement dynamic DNS mapping and pipeline improvements for dns and
brijesh-elastic Apr 9, 2026
e42d8b6
add ignore_failure to community_id processor and apply pipeline
brijesh-elastic Apr 9, 2026
ccd6d97
add changelog entry for cloudflare_logpush pipeline improvements
brijesh-elastic Apr 9, 2026
5754cd3
run pipeline and system tests
brijesh-elastic Apr 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion packages/cloudflare_logpush/_dev/build/build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
dependencies:
ecs:
reference: "git@v8.11.0"
reference: 'git@v9.3.0'
6 changes: 6 additions & 0 deletions packages/cloudflare_logpush/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,10 @@
# newer versions go on top
- version: "1.44.0"
changes:
- description: |
Add support for new fields, improve error handling, and ECS field mappings across all data streams.
type: enhancement
link: https://github.com/elastic/integrations/pull/18257
- version: "1.43.3"
changes:
- description: Use triple-brace Mustache templating when referencing variables in ingest pipelines.
Expand Down
2 changes: 1 addition & 1 deletion ...re_logpush/data_stream/access_request/_dev/test/pipeline/test-pipeline-access-request.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
{"Action":"login","Allowed":true,"AppDomain":"partner-zt-logs.cloudflareaccess.com/warp","AppUUID":"123e4567-e89b-12d3-a456-426614174000","Connection":"onetimepin","Country":"us","CreatedAt":"2023-05-24T02:48:33+09:30","Email":"user@example.com","IPAddress":"67.43.156.93","PurposeJustificationPrompt":"Please provide your reason for accessing the application.","PurposeJustificationResponse":"I need to access the application for work purposes.","RayID":"00c0ffeeabc12345","TemporaryAccessApprovers":["approver1@example.com","approver2@example.com"],"TemporaryAccessDuration":7200,"UserUID":"166befbb-00e3-5e20-bd6e-27245333949f"}
{"Action":"login","Allowed":true,"AppDomain":"partner-zt-logs.cloudflareaccess.com/warp","AppUUID":"123e4567-e89b-12d3-a456-426614174000","Connection":"onetimepin","Country":"us","CreatedAt":1684862313,"Email":"user@example.com","IPAddress":"67.43.156.93","PurposeJustificationPrompt":"Please provide your reason for accessing the application.","PurposeJustificationResponse":"I need to access the application for work purposes.","RayID":"00c0ffeeabc12345","TemporaryAccessApprovers":["approver1@example.com","approver2@example.com"],"TemporaryAccessDuration":7200,"UserUID":"166befbb-00e3-5e20-bd6e-27245333949f"}
{"Action":"login","Allowed":true,"AppDomain":"partner-zt-logs.cloudflareaccess.com/warp","AppUUID":"123e4567-e89b-12d3-a456-426614174000","Connection":"onetimepin","Country":"us","CreatedAt":1684862313000000000,"Email":"user@example.com","IPAddress":"67.43.156.93","PurposeJustificationPrompt":"Please provide your reason for accessing the application.","PurposeJustificationResponse":"I need to access the application for work purposes.","RayID":"00c0ffeeabc12345","TemporaryAccessApprovers":["approver1@example.com","approver2@example.com"],"TemporaryAccessDuration":7200,"UserUID":"166befbb-00e3-5e20-bd6e-27245333949f"}
{"Action":"login","Allowed":true,"AppDomain":"partner-zt-logs.cloudflareaccess.com/warp","AppUUID":"123e4567-e89b-12d3-a456-426614174000","Connection":"onetimepin","Country":"us","CreatedAt":1684862313000000000,"Email":"user@example.com","IPAddress":"67.43.156.93","PurposeJustificationPrompt":"Please provide your reason for accessing the application.","PurposeJustificationResponse":"I need to access the application for work purposes.","RayID":"00c0ffeeabc12345","TemporaryAccessApprovers":["approver1@example.com","approver2@example.com"],"TemporaryAccessDuration":7200,"UserUID":"166befbb-00e3-5e20-bd6e-27245333949f"}
6 changes: 3 additions & 3 deletions ...a_stream/access_request/_dev/test/pipeline/test-pipeline-access-request.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@
}
},
"ecs": {
"version": "8.11.0"
"version": "9.3.0"
},
"event": {
"action": "login",
Expand Down Expand Up @@ -142,7 +142,7 @@
}
},
"ecs": {
"version": "8.11.0"
"version": "9.3.0"
},
"event": {
"action": "login",
Expand Down Expand Up @@ -232,7 +232,7 @@
}
},
"ecs": {
"version": "8.11.0"
"version": "9.3.0"
},
"event": {
"action": "login",
Expand Down
Loading
Loading