[Rule Tuning] Add Corelight support for existing rules

[eric-forte-elastic]

Pull Request

Issue link(s):

Summary - What I changed

Small update to add Corelight coverage to existing rules that already carry Zeek support. Also updates the integrations manifest to register the corelight package.

Note

Corelight data does not land in logs-zeek.* despite Corelight sensors running Zeek. The Elastic corelight integration package is dashboards-only — normalization is handled by the customer-installed corelight/ecs-templates + corelight/ecs-mapping pipelines, which write to logs-corelight.* with data_stream.dataset:corelight.<stream>. Rules with only logs-zeek.* in their index list return zero results on Corelight data.

For the five flow-level rules in this PR, no query changes were required. Corelight populates event.category:network and the standard ECS 5-tuple fields; the existing event.category:(network or network_traffic) branch in each rule fires on Corelight data once logs-corelight.* is added to the index list.

Because the corelight package ships no ingest pipelines or field schema, corelight was added to NON_DATASET_PACKAGES in definitions.py so the integration tag validator accepts the metadata tag on rules whose queries do not reference a data_stream.dataset:corelight.* clause directly.

How To Test

RTAs in elastic/cortado#34

And unit testing.

Checklist

• Added a label for the type of pr: bug, enhancement, schema, maintenance, Rule: New, Rule: Deprecation, Rule: Tuning, Hunt: New, or Hunt: Tuning so guidelines can be generated
• Added the meta:rapid-merge label if planning to merge within 24 hours
• Secret and sensitive material has been managed correctly
• Automated testing was updated or added to match the most common scenarios
• Documentation and comments were added for features that require explanation

Contributor checklist

• Have you signed the contributor license agreement?
• Have you followed the contributor guidelines?

[github-actions]

Enhancement - Guidelines

These guidelines serve as a reminder set of considerations when addressing adding a new schema feature to the code.

Documentation and Context

• Describe the feature enhancement in detail (alternative solutions, description of the solution, etc.) if not already documented in an issue.
• Include additional context or screenshots.
• Ensure the enhancement includes necessary updates to the documentation and versioning.

Code Standards and Practices

• Code follows established design patterns within the repo and avoids duplication.
• Ensure that the code is modular and reusable where applicable.

Testing

• New unit tests have been added to cover the enhancement.
• Existing unit tests have been updated to reflect the changes.
• Provide evidence of testing and validating the enhancement (e.g., test logs, screenshots).
• Validate that any rules affected by the enhancement are correctly updated.
• Ensure that performance is not negatively impacted by the changes.
• Verify that any release artifacts are properly generated and tested.
• Conducted system testing, including fleet, import, and create APIs (e.g., run make test-cli, make test-remote-cli, make test-hunting-cli)

Additional Schema Related Checks

• Verify that the enhancement works across all relevant environments (e.g., different OS versions).
• Link to the relevant Kibana PR or issue provided
• Test export/import flow:
  • Exported detection rule(s) from Kibana to showcase the feature(s)
  • Converted the exported ndjson file(s) to toml in the detection-rules repo
  • Re-exported the toml rule(s) to ndjson and re-imported into Kibana
• Updated necessary unit tests to accommodate the feature
• Incorporated a comprehensive test rule in unit tests for full schema coverage
• Applied min_compat restrictions to limit the feature to a specified minimum stack version
• Executed all unit tests locally with a test toml rule to confirm passing
• Included Kibana PR implementer as an optional reviewer for insights on the feature
• Implemented requisite downgrade functionality
• Cross-referenced the feature with product documentation for consistency
• Confirm that the proper version label is applied to the PR patch, minor, major.

[tradebot-elastic]

⛔️ Test failed

Results

• ❌ RPC (Remote Procedure Call) from the Internet (kuery)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ RPC (Remote Procedure Call) to the Internet (kuery)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ Accepted Default Telnet Port Connection (kuery)
  • stack_validation_failed: no_alerts - 0 alerts
• ❌ VNC (Virtual Network Computing) to the Internet (kuery)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ VNC (Virtual Network Computing) from the Internet (kuery)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ RDP (Remote Desktop Protocol) from the Internet (kuery)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ IPSEC NAT Traversal Port Activity (kuery)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ SMB (Windows File Sharing) Activity to the Internet (kuery)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ SMTP on Port 26/TCP (kuery)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta

[tradebot-elastic]

⛔️ Test failed

Results

• ❌ RPC (Remote Procedure Call) from the Internet (kuery)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ RPC (Remote Procedure Call) to the Internet (kuery)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ RDP (Remote Desktop Protocol) from the Internet (kuery)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ SMB (Windows File Sharing) Activity to the Internet (kuery)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ SMTP on Port 26/TCP (kuery)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta

[Copilot]

Pull request overview

This PR tunes several existing network flow rules to recognize Corelight-ingested Zeek/ECS network telemetry by adding corelight metadata and logs-corelight.* index coverage, and updates the rules tooling to accept corelight as a non-dataset integration package.

Changes:

• Add corelight to the integration metadata and logs-corelight.* to index for five existing network rules, plus a Data Source: Corelight tag.
• Add corelight to NON_DATASET_PACKAGES to relax dataset-based validation expectations for the package.
• Bump the Python package version from 1.6.51 to 1.6.52.

Reviewed changes

Copilot reviewed 7 out of 9 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
rules/network/initial_access_smb_windows_file_sharing_activity_to_the_internet.toml	Adds Corelight integration metadata, index pattern, and tag for SMB-to-Internet detection.
rules/network/initial_access_rpc_remote_procedure_call_to_the_internet.toml	Adds Corelight integration metadata, index pattern, and tag for RPC-to-Internet detection.
rules/network/initial_access_rpc_remote_procedure_call_from_the_internet.toml	Adds Corelight integration metadata, index pattern, and tag for RPC-from-Internet detection.
rules/network/command_and_control_rdp_remote_desktop_protocol_from_the_internet.toml	Adds Corelight integration metadata, index pattern, and tag for RDP-from-Internet detection.
rules/network/command_and_control_port_26_activity.toml	Adds Corelight integration metadata, index pattern, and tag for SMTP-on-26 detection.
detection_rules/schemas/definitions.py	Adds corelight to NON_DATASET_PACKAGES to support validation/packaging behavior.
pyproject.toml	Increments project version.

[Mikaayenson]

@eric-forte-elastic what about lateral_movement_dns_server_overflow.toml? otherwise lgtm

[eric-forte-elastic]

@eric-forte-elastic what about lateral_movement_dns_server_overflow.toml? otherwise lgtm

Good question, I should have called this one out in the description. It is not directly compatible without a more involved query modification. At the time I was going to have it in a separate tuning PR. The short version is that for that rule Corelight behaves more like a flow based source rather than a packet based source depending on what data_stream is populating the events. With this, it has a decent risk of introducing FPs for long DNS connections (rather than just large, since the destination.bytes can in certain cases be calculated on aggregate instead of at the connection level).

It is now tuned in this PR in be46253

[tradebot-elastic]

⛔️ Test failed

Results

• ❌ Abnormally Large DNS Response (kuery)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ RPC (Remote Procedure Call) from the Internet (kuery)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ RPC (Remote Procedure Call) to the Internet (kuery)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ RDP (Remote Desktop Protocol) from the Internet (kuery)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ SMB (Windows File Sharing) Activity to the Internet (kuery)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta
• ❌ SMTP on Port 26/TCP (kuery)
  • coverage_issue: no_rta
  • stack_validation_failed: no_rta