[Security AI Prompts] Update Entity Highlights prompt#19999
Open
CAWilson94 wants to merge 2 commits into
Open
Conversation
Update the entityDetailsHighlights prompt content: add MITRE ATT&CK / Lockheed Martin Kill Chain mapping for anomalies, anti-fabrication guardrails, synthesise-don't-list guidance, and response size limits. Bump package version 1.0.13 -> 1.0.14. Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
Elastic Docs Style Checker (Vale)Summary: 1 suggestion found 💡 Suggestions (1): Optional style improvements. Apply when helpful.
The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale. |
This comment has been minimized.
This comment has been minimized.
Contributor
TL;DRBoth failed Buildkite steps are caused by the new Remediation
Investigation detailsRoot CauseThe PR diff adds a Evidence
Verification
What is this? | From workflow: PR Buildkite Detective Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not. |
andrew-goldstein
approved these changes
Jul 6, 2026
Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
|
✅ All changelog entries have the correct PR link. |
💚 Build Succeeded
History
|
Contributor
|
Tick the box to add this pull request to the merge queue (same as
|
Author
|
Waiting to merge in kibana PR first: elastic/kibana#276038 |
Author
|
@andrewkroh @andrew-goldstein added this to merge queue prior to the kibana related PR merging in due to timing. Just an FYI |
CAWilson94
added a commit
to elastic/kibana
that referenced
this pull request
Jul 7, 2026
…regenerate nudge (#276038) ## Summary Persists the **Entity AI Summary** so it no longer regenerates on every flyout open/click-away. Instead of writing to the entity-store `entity.attributes.summary` field (POC #269344), the summary is now stored as an **append-only document in the entity metadata datastream** (`.entities.v2.metadata.security_*`, base #271806), per [security-team#17968](elastic/security-team#17968). ## What changed - **Write** — generation persists a single immutable `ai_summary.*` doc via `asInternalUser` (server-derived `generated_by`; no per-user metadata write privilege needed). Structural caps applied (max 4 highlights, 3 recommended actions). - **Read** — gated `GET` (`asCurrentUser`): with metadata read access the persisted summary is shown; without it, the flyout gracefully falls back to on-demand generation (nothing persisted). - **Removed** the never-merged `entity.attributes.summary` field + `includeSummary` listing param. - **Staleness nudge** — if the entity's risk score drifts from the snapshot taken at generation time, a warning callout prompts regeneration. - **Prompt** — promoted the improved entity-highlights prompt (MITRE / Kill Chain mapping, synthesize-don't-list, anti-fabrication) as the single default. - **Tests/docs** — Scout API permissions matrix + `check_privileges` coverage, unit tests, and README privilege docs. Updated Security AI prompts Integration SO: elastic/integrations#19999 <details> <summary> <strong>Prompt and Summary Content</strong> </summary> - [Customer feedback flagged that the summary was **too long to be a summary**.](elastic/security-team#16484 (comment)) Rather than just trimming length, I reworked the prompt to change the *shape* of the output while keeping , and focussing, its value - helping an analyst quickly see **what needs their attention and how to move forward with an investigation**, not handing them a raw data dump to interpret themselves. - **Risk score** stays high-level: describe the dominant threat pattern, and only name a specific rule/alert when it clearly accounts for the majority of the score - instead of listing every contributing rule. - **Anomalies** are summarized as the single most significant pattern rather than listed, and mapped to a MITRE ATT&CK tactic or Lockheed Martin Kill Chain phase only when the ML results clearly correspond to a known technique (the mapping is omitted when anomalies are ambiguous or benign, to avoid guessing). - **Criticality / vulnerabilities** state the level / most critical finding and *why it matters*, not raw values. - **Reinforced with hard guardrails:** 1 sentence per highlight (2 only when an anomaly needs the extra clause for a MITRE/Kill Chain mapping), a concise highlights budget, up to 3 prioritized recommendations, no fabricated values, and consistent formatting (backticks for technical values, 2-decimal rounding). - The result is shorter *because* it's focused - not truncated. This directly addresses design/product feedback that the risk score and anomalies should be summarized at a high level rather than enumerated. Prompt quality is inherently subjective and hard to unit-test, so I validated it by desk-testing against seeded rich data (risk, criticality, anomalies, vulnerabilities); the telemetry below is there to keep tuning it from real usage post-merge. </details> <details> <summary> <strong>Telemetry</strong> </summary> - Generating an entity AI summary emits an `entity_ai_summary_persisted` EBT event with `entityType`, `spaceId`, and `highlightsCount` / `recommendedActionsCount`. The two counts are the **raw number of items the model produced, captured client-side before structural capping** — not the number persisted (the doc is always capped to `MAX_ENTITY_SUMMARY_HIGHLIGHTS` / `MAX_ENTITY_SUMMARY_RECOMMENDED_ACTIONS`). - **Why it's useful:** comparing these raw counts against the caps tells us how often, and by how much, the prompt overshoots the intended summary length — a data-driven feedback loop for tuning the prompt (e.g. if the model routinely returns 7 highlights against a cap of 4, the prompt is asking for too much). The counts are captured on the client because the server only ever receives the already-capped content and so cannot observe overshoot itself. No summary text or entity identifiers are included in the event. - **How it connects to the prompt change:** the prompt was reworked to be shorter by being more focused (per the customer feedback that it was too long); this telemetry closes the loop by letting us track overshoot over time and keep tuning the prompt and caps from real usage rather than guesswork. </details> ## Demo Latest Demo! 🔥 https://github.com/user-attachments/assets/471bb53e-ce6d-46dd-838b-4bd2e2e63c26 ## Notes for reviewers - **Generation is currently client-side** (the client POSTs the generated summary). This is transitional — it reuses the assistant's client-side anonymization/replacements context. Moving generation fully server-side is a tracked follow-up (also required to move staleness computation server-side so Agent Builder / external clients share it). - **The summary body (highlights + recommended actions) is English-only.** This matches all AI-generated content in the Security solution (no assistant prompt does locale-based generation) and is the same as pre-persistence behavior — **not a regression**. All UI chrome (headers, buttons, staleness callout + message) is i18n'd. **- Entity AI Summary runtime helpers:** (staleness detection + caps) are exported from a dedicated `@kbn/entity-store/common/entity_summary` subpath, not the top-level common barrel. The barrel is the page-load entry (always-loaded primitives only); this logic is used only by the lazy flyout and server, so keeping it in the barrel pushed the entityStore page-load bundle over budget. The subpath code-splits it out - fixing the size regression structurally rather than raising the limit - mirroring the existing euid_helpers / entity_maintainers modules. Types stay in the barrel (zero page-load cost). ## Testing **1. Prereqs:** Entity Store v2 enabled, an Enterprise license, and a configured AI connector. **2. Seed data** (uses generator PR [security-documents-generator#417](elastic/security-documents-generator#417) for entity-correlated vulnerabilities): ```bash # Entities + risk score + asset criticality + alerts yarn start risk-score-v2 --entity-kinds host,idp_user --hosts 20 --no-setup # Layer anomalies + vulnerabilities + misconfigurations, correlated to those entities yarn start generate-entity-ai-insights --correlate-with-entity-store -h 20 -u 20 -a 20 ``` **3. Verify persistence:** - Open a host/user entity flyout → AI summary → **Generate**. - Close and reopen the flyout → the **same** summary shows, with no regeneration, plus a "Generated by … on …" line. - As a second user (with metadata read) → the same summary + original generated-by is shown. **4. Verify staleness nudge:** - For an entity with an already generated summary - change the asset criticality, then re-open the flyout for that summary and verify → a warning callout appears ("Entity risk has changed…") → click **Regenerate summary** → callout clears. **5. Verify gated-read fallback:** - As a user **without** read on `.entities.v2.metadata.security_*` → the flyout falls back to on-demand generation and does **not** persist. Mappings - Using `GET .entities.v2.metadata.security_default/_mapping` from dev tools, after some reviews the latest mappings (Tuesday July 7th): ``` "Ai_summary": { "properties": { "anomaly_job_ids": { "type": "keyword" }, "generated_at": { "type": "date", "format": "epoch_millis" }, "generated_by": { "type": "keyword" }, "highlights": { "type": "object", "enabled": false }, "recommended_actions": { "type": "keyword", "index": false, "doc_values": false }, "staleness": { "properties": { "enabled_signals": { "type": "keyword" }, "snapshot": { "properties": { "risk_score": { "type": "float" } } } } }, "variant_id": { "type": "keyword" } } }, ``` --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Alex Prozorov <alex.prozorov@elastic.co>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Update the
entityDetailsHighlightsprompt content: add MITRE ATT&CK / Lockheed Martin Kill Chain mapping for anomalies, anti-fabrication guardrails, synthesise-don't-list guidance, and response size limits. Bump package version1.0.13→1.0.14.Proposed commit message
Update the
entityDetailsHighlightsSecurity AI prompt (aiForEntityDetailsgroup) used by the Entity Analytics "Entity summary" flyout in the Security Solution.WHAT:
entityDetailsHighlightssaved object; the saved-objectid,type, and filename are unchanged.1.0.13→1.0.14(patch, content-only) with a matchingchangelog.ymlentry.WHY:
Checklist
changelog.ymlfile.Author's Checklist
promptId(entityDetailsHighlights) andpromptGroupId(aiForEntityDetails) are unchanged — onlyprompt.defaultcontent changed.How to test this PR locally
cd packages/security_ai_prompts elastic-package lint elastic-package build elastic-package stack up -d -v --services package-registry