Relationships store types

[ereteog]

Summary

Add source_type and target_type fields to relationship documents for efficient filtering in bundle/export API.

Related https://cisco-sbg.atlassian.net/browse/XDR-45534

Problem

• Bundle export API filters relationships by entity type using source_type/target_type params
• Current query: source_ref:*malware* (wildcard = inefficient, doesn't use ES index)
• Needed query: source_type:malware (exact match = uses ES index efficiently)

Solution

1. Add source_type/target_type fields extracted from URIs on write
2. Backward-compatible query during migration: ((source_type:X) OR (source_ref:*X*))
   • New docs: fast path via exact match
   • Legacy docs: slow path via wildcard fallback
3. ES migration script using _update_by_query to backfill existing documents

Files Changed

• src/ctia/entity/relationship/es_store.clj - ES mapping and transforms
• src/ctia/bundle/core.clj - Backward-compatible query generation
• src/ctia/entity/relationship.clj - Use new es-store
• scripts/migrate_relationship_types.sh - ES migration script
• scripts/check_relationship_migration.sh - Migration verification script
• test/ctia/entity/relationship/es_store_test.clj - Unit tests for transforms
• test/ctia/bundle/core_test.clj - Updated query tests

---

Deployment Steps

Phase 1: Deploy this PR

New documents will automatically get source_type/target_type populated.
Queries use backward-compatible OR clause (works for both new and old docs).

Phase 2: Update ES mapping

Add the new field definitions to the existing index:

java -cp ctia.jar clojure.main -m ctia.task.update-index-state \
  -Dctia.store.es.relationship.update-mappings=true

Phase 3: Run migration script

Backfill existing documents with source_type/target_type values:

# Dry run first
./scripts/migrate_relationship_types.sh <ES_HOST> <INDEX> --dry-run

# Execute with throttling (recommended for production)
./scripts/migrate_relationship_types.sh <ES_HOST> <INDEX> --throttle 500 --async

# Monitor async task
curl -X GET "http://<ES_HOST>/_tasks/<TASK_ID>"

Phase 4: Verify migration

Check migration status and validate data:

./scripts/check_relationship_migration.sh <ES_HOST> <INDEX>

Output includes:

• Migration statistics (total, migrated, remaining)
• Progress percentage
• Breakdown by source_type and target_type
• Sample validation (ensures extracted types match refs)

Phase 5: Refresh mappings

Reindex documents so new fields are properly searchable:

java -cp ctia.jar clojure.main -m ctia.task.update-index-state \
  -Dctia.store.es.relationship.refresh-mappings=true

Phase 6: Follow-up PR

Once migration is complete in all environments, deploy follow-up PR to remove the wildcard fallback from queries.

---

Script Options

migrate_relationship_types.sh

Option	Description
--dry-run	Show count and sample doc without making changes
--throttle <N>	Limit to N documents per second
--async	Run in background, returns task ID for monitoring

check_relationship_migration.sh

Exit Code	Meaning
0	Migration complete and validated
1	Migration complete but validation errors
2	Migration incomplete

---

QA Checklist

• Verify new relationships have source_type/target_type populated
• Verify bundle/export with source_type filter works for both new and old docs
• Run migration script in INT with --dry-run first
• Run migration in INT with --throttle 500 --async
• Run check_relationship_migration.sh to verify
• Run refresh-mappings after migration completes
• Verify queries use new fields efficiently (check ES slow logs)

---

Related

• Jira: https://cisco-sbg.atlassian.net/browse/XDR-45534
• Follow-up: PR to remove wildcard fallback after migration

🤖 Generated with Claude Code

[yogsototh]

Verdict: NEEDS DISCUSSION (comment-only) -- 0 CRITICAL, 1 HIGH (latent), 4 MEDIUM. Current delivered behavior is correct and tested; the HIGH is a landmine for the planned follow-up, not a runtime break in this PR.

Design is sound: the new query is a strict superset of master (((source_type:X) OR (source_ref:*X*))), so there is no wrong-results window during the staged migration, and the read paths strip source_type/target_type so API responses are unchanged. Verified: the moved RelationshipStore/relationship-mapping symbols have no external consumers, the node-filters arity change has no stale caller, and the check_es_stores.clj override to ESStoredRelationship is required (real migrated docs would fail strict coercion against plain StoredRelationship).

Findings

• HIGH [correctness] -- Hyphenated types break the exact-match clause once the wildcard fallback is removed: node-filters emits source_type:attack-pattern (unquoted) into a raw query_string. In the classic parser the - is the prohibit operator, so for every hyphenated CTIA type (attack-pattern, data-table, identity-assertion, ...) the new exact-match term does not match the keyword token. Today this is masked by the OR (source_ref:*attack-pattern*) fallback (current results are correct), but the PR's stated Phase 6 removes that fallback -- at which point hyphenated-type filtering silently returns empty/wrong results. No test covers a hyphenated type (core_test.clj only uses malware/sighting/vulnerability/incident). Fix before the follow-up: quote/escape the term (source_type:"attack-pattern") or use a structured term filter for the exact clause; add a hyphenated-type test. (src/ctia/bundle/core.clj:422)

• MEDIUM [correctness/ops] -- Migration-completeness gate can never reach 0: any relationship whose ref lacks a parseable /ctia/<type>/ segment (legacy/imported/malformed) never gets the field set, so it permanently matches the "needs migration" query and check_relationship_migration.sh exits 2 forever. Consider restricting the needs-migration query to docs whose refs match */ctia/*, or marking unextractable docs as excluded. (scripts/check_relationship_migration.sh:51)

• MEDIUM [security] -- ES credentials exposed (cleartext HTTP + process table): both scripts hardcode http://${ES_HOST} while passing -u ${ES_USER}:${ES_PASS} on the curl argv (visible in ps//proc/<pid>/cmdline), and the docs steer operators at prod hosts. Use --netrc-file / a bash array, and make the scheme configurable (ES_SCHEME/detect https://). (scripts/migrate_relationship_types.sh:79)

• MEDIUM [test/quality] -- Loose return annotation defeats the schema fixture: es-stored-relationship->stored-relationship is annotated :- ESStoredRelationship (optional type keys) but exists to strip those keys and return a StoredRelationship. A regression that failed to dissoc would still validate, so validate-schemas gives zero protection for the exact bug this fn prevents. Tighten the annotation to rs/StoredRelationship. (src/ctia/entity/relationship/es_store.clj:51)

• MEDIUM [cross-cutting] -- Three divergent "extract type from ref" implementations, none cross-tested: Clojure long-id->id (strict regex, returns nil on malformed), Painless indexOf('/ctia/')+substring (no validation), and sed in the checker all differ on malformed refs (e.g. …/ctia/malware/not-a-uuid -> nil in Clojure, malware in Painless/sed). Acceptable for one-shot tooling, but document the intentional difference or add a divergence test. (scripts/migrate_relationship_types.sh:172)

PASS: Security (no injection -- entity types are enum-validated; Painless is pure string ops, no eval), Cross-cutting contracts (no moved-symbol/arity break), Deployment ordering (no wrong-results window).

Also left 7 inline comments.

Non-blocking follow-ups (LOW)

• Remove unused require ctia.entity.event.schemas :as es (src/ctia/entity/relationship.clj:19).
• Remove dead defs ESPartialStoredRelationshipList / PartialStoredRelationshipList (src/ctia/entity/relationship/es_store.clj:37-38).
• grep -o ... || echo 0 swallows curl/network/auth failures and can misreport a failed _update_by_query as a no-op success (scripts/migrate_relationship_types.sh:108,233); check curl exit code + HTTP status.
• Add a behavioral test for the legacy wildcard fallback: insert a relationship doc WITHOUT the type fields and assert bundle export by source_type still returns it via source_ref:*X*. core_test.clj asserts only the query string.
• Add an assertion in bundle-export-graph-test that exported relationships omit source_type/target_type.

-- auto-review-team (BOT_APPROVABLE gate)

[yogsototh]

HIGH -- Hyphenated types break the exact-match clause once the wildcard fallback is removed: this emits source_type:attack-pattern (unquoted) into a raw query_string. In the classic Lucene parser an unescaped - is the prohibit operator, so for every hyphenated CTIA type (attack-pattern, data-table, identity-assertion, ...) the exact term does not match the keyword token. Today the OR (source_ref:*attack-pattern*) fallback masks this (results are correct), but Phase 6 of the deployment plan removes that fallback -- after which hyphenated-type filtering silently returns empty/wrong results. Suggest quoting the term (source_type:"attack-pattern") or using a structured term filter, and adding a hyphenated-type case to core_test.clj.

[yogsototh]

MEDIUM -- Return annotation defeats the schema fixture: this fn exists to strip :source_type/:target_type and return a StoredRelationship, but it is annotated :- ESStoredRelationship (which has those keys as optional). A regression that failed to dissoc would still validate against ESStoredRelationship, so the :once validate-schemas fixture gives zero protection for the exact bug this fn prevents. The mirrored sighting transform (es-stored-sighting->stored-sighting) correctly returns the narrowed schema. Suggest :- rs/StoredRelationship.

[yogsototh]

LOW -- Unused require: the es alias for ctia.entity.event.schemas is never referenced in this namespace. Remove it to avoid an unnecessary dependency edge and linter noise.

[yogsototh]

LOW -- Dead code: ESPartialStoredRelationshipList (and PartialStoredRelationshipList on the next line) are defined but referenced nowhere in src/ or test/. Remove, or wire them into the route/swagger schemas if they were intended for a response type.

[yogsototh]

MEDIUM -- Completeness gate can never reach 0: any relationship whose source_ref/target_ref lacks a parseable /ctia/<type>/ segment (legacy/imported/malformed) never gets the field populated by the Painless migration, so it permanently matches this "needs migration" query and the script exits 2 (line 181) forever. Consider restricting this query to docs whose refs match */ctia/*, or otherwise excluding unextractable docs so the migration can be declared complete.

[yogsototh]

MEDIUM -- Credential exposure: -u ${ES_USER}:${ES_PASS} is expanded onto the curl argv (lines 91/125/195), so the cleartext user:password is visible in ps -ef / /proc/<pid>/cmdline to any local user; combined with the hardcoded http:// scheme (and the prod-host example in the sibling script), the secret is also exposed on the wire. Suggest curl --netrc-file (or a bash array AUTH_OPTS=(-u "$ES_USER:$ES_PASS")) and a configurable ES_SCHEME/https:// detection.

[yogsototh]

LOW -- || echo 0 masks curl/network/auth failures: if curl fails entirely (network error, auth rejected, ES error JSON), this swallows it and reports DOC_COUNT=0 -> "No documents need migration. Done." The same idiom at line 233 (UPDATED) can misreport a failed _update_by_query as a no-op success. Suggest checking curl's exit code and the HTTP status separately before parsing the count.