Skip to content

Add size constraints to CTIM collection fields (XDR-46972) #483

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ereteog wants to merge 4 commits into from
Closed

Add size constraints to CTIM collection fields (XDR-46972) #483

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
a927b2e
add size constraints to collection fields (XDR-46972)
ereteog Mar 12, 2026
28787a9
replace unbounded f/any-str with typed string constraints
ereteog Mar 16, 2026
bc08b36
raise limits based on production data and add relation_info.actions c…
ereteog Mar 16, 2026
8237a23
remove unused pred require from bundle.cljc
ereteog Mar 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion project.clj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
[com.google.protobuf/protobuf-java "4.32.1"] ;clj-momo > org.clojure/clojurescript
[threatgrid/clj-momo "0.4.1"]
[org.mozilla/rhino "1.8.0"] ;threatgrid/flanders > kovacnica/clojure.network.ip
[threatgrid/flanders "1.1.0"]
[threatgrid/flanders "1.1.1-SNAPSHOT"]

@yogsototh yogsototh Jun 5, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CRITICAL -- SNAPSHOT dependency on unreleased flanders: 1.1.1-SNAPSHOT has no flanders git tag, no GitHub release, and Clojars 404s for it. The :spec support on SequenceOfType/SetOfType/MapType that this entire PR depends on exists only on flanders master, not in 1.1.0. Merging this to master makes the build non-reproducible (a SNAPSHOT is mutable and can be GC'd), blocks any stable ctim release, and leaves these constraints inert until flanders 1.1.1 ships. Release flanders 1.1.1 to Clojars and pin the released version before merge.

[metosin/ring-swagger "1.0.0"]
[org.clojure/test.check "1.1.1"]
[com.gfredericks/test.chuck "0.2.15"]
Expand Down
6 changes: 4 additions & 2 deletions src/ctim/lib/predicates.cljc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
(ns ctim.lib.predicates)

(defn max-len [len]
(fn [test]
(>= len (count test))))
(with-meta
(fn [test]
(>= len (count test)))
{:max-len len}))

(defn max-non-neg-int [max-num]
(fn [test]
Expand Down
4 changes: 2 additions & 2 deletions src/ctim/schemas/asset_properties.cljc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,12 @@
(f/def-map-type AssetProperty
(concat
(f/required-entries
(f/entry :name f/any-str
(f/entry :name c/ShortString
:description "The properties are an open vocabulary."
:comment (str "The properties are an open vocabulary, meaning that there is "
"a defined set of values, but users may add their own values."
"See: https://github.com/threatgrid/ctim/blob/master/src/ctim/schemas/identity_assertion.cljc#L11"))
(f/entry :value f/any-str))))
(f/entry :value c/MedString))))

(def-entity-type AssetProperties
{:description properties-desc
Expand Down
4 changes: 2 additions & 2 deletions src/ctim/schemas/casebook.cljc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,8 @@

(def-map-type Text
(f/required-entries
(f/entry :type f/any-str)
(f/entry :text f/any-str)))
(f/entry :type c/ShortString)
(f/entry :text c/LongString)))

(def-entity-type Casebook
{:description casebook-desc
Expand Down
49 changes: 31 additions & 18 deletions src/ctim/schemas/common.cljc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,12 @@

(def ctim-schema-version (ctim-version))

;; Default size limit for collection fields (strings, references, etc.)
(def default-collection-max-len 500)

;; Max number of actions allowed in a single relation's relation_info
(def max-relation-actions 1000)

(def-eq CTIMSchemaVersion ctim-schema-version)

(cs/def ::ctim-schema-version
Expand Down Expand Up @@ -175,9 +181,9 @@
(f/entry :description Markdown)
(f/entry :url URI
:description "A URL reference to an external resource.")
(f/entry :hashes f/any-string-seq
(f/entry :hashes (f/seq-of ShortString)
:description "Specifies a dictionary of hashes for the contents of the url.")
(f/entry :external_id f/any-str
(f/entry :external_id ShortString
:description "An identifier for the external reference content.")))
:description (str "External references are used to describe pointers to information "
"represented outside of CTIM. For example, a Malware object could "
Expand All @@ -191,13 +197,14 @@
(f/required-entries
(f/entry :id ID
:description "Globally unique URI identifying this object.")
(f/entry :type f/any-str)
(f/entry :type ShortString)
(f/entry :schema_version SchemaVersion
:description "CTIM schema version for this entity."))
(f/optional-entries
(f/entry :revision PosInt
:description "A monotonically increasing revision, incremented each time the object is changed.")
(f/entry :external_ids f/any-string-seq
(f/entry :external_ids (f/seq-of ShortString
:spec (pred/max-len default-collection-max-len))
:description (str "It is used to store a list of external identifiers that can be linked to the "
"incident, providing a reliable and manageable way to correlate and group related "
"events across multiple data sources. It is especially useful in larger "
Expand All @@ -209,7 +216,8 @@
"information can be shared among incident management systems. It can be used to "
"cross-reference with other external tools such as threat intelligence feeds and "
"vulnerability scanners."))
(f/entry :external_references [ExternalReference]
(f/entry :external_references (f/seq-of ExternalReference
:spec (pred/max-len default-collection-max-len))
:description (str "Specifies a list of external references which refers to non-CTIM "
"information.\n\n"
"Similar to `external_ids` field with major differences:\n\n"
Expand Down Expand Up @@ -263,7 +271,7 @@
base-entity-entries
(f/optional-entries
(f/entry :id ID)
(f/entry :type f/any-str
(f/entry :type ShortString
:description "A valid entity type identifer")
(f/entry :schema_version CTIMSchemaVersion
:description "CTIM schema version for this entity."))))
Expand Down Expand Up @@ -322,20 +330,20 @@

(def-map-type Contributor
(f/optional-entries
(f/entry :role f/any-str
(f/entry :role ShortString
:description "Role played by this contributor.")
(f/entry :name f/any-str
(f/entry :name ShortString
:description "Name of this contributor.")
(f/entry :email f/any-str
(f/entry :email ShortString
:description "Email of this contributor.")
(f/entry :phone f/any-str
(f/entry :phone ShortString
:description "Telephone number of this contributor.")
(f/entry :organization f/any-str
(f/entry :organization ShortString
:description "Organization name of this contributor.")
(f/entry :date Time
:description (str "Description (bounding) of the timing of this "
"contributor's involvement."))
(f/entry :contribution_location f/any-str
(f/entry :contribution_location ShortString
:description (str "information describing the location at which the "
"contributory activity occured")))
:description "Person who contributed cyber observation data."
Expand All @@ -350,10 +358,10 @@
(f/entry :confidence v/HighMedLow
:description (str "Specifies the level of confidence in the assertion "
"of the relationship between the two objects."))
(f/entry :information_source f/any-str
(f/entry :information_source ShortString
:description (str "Specifies the source of the information about "
"the relationship between the two components."))
(f/entry :relationship f/any-str)))
(f/entry :relationship ShortString)))
:description "Describes a related Identity"
:reference "[RelatedIdentityType](http://stixproject.github.io/data-model/1.2/stixCommon/RelatedIdentityType/)")

Expand Down Expand Up @@ -618,12 +626,17 @@
:gen (cs/gen relation-types))

(def-map-type ObservedRelation
[(f/entry :origin f/any-str)
[(f/entry :origin ShortString)
(f/entry :origin_uri URI
:required? false)
(f/entry :relation ObservableRelationType)
(f/entry :relation_info (f/map
[(f/entry f/any-keyword f/any)])
[(f/entry f/any-keyword f/any)]
:spec (fn [m]
(let [actions (:actions m)]
(or (nil? actions)
(not (sequential? actions))

@yogsototh yogsototh Jun 5, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MEDIUM -- Actions cap is bypassable: the limit only applies when :actions is sequential?. A non-sequential value -- a JSON object (decodes to a map) or a string -- makes (not (sequential? actions)) return true, so it passes unbounded. Only the literal :actions key is inspected; the rest of this free-form f/any map is uncapped. If the goal is to bound oversized relation_info, consider rejecting non-sequential :actions rather than waving them through, and confirm :actions is the real key used in production payloads.

((pred/max-len max-relation-actions) actions)))))
:required? false)
(f/entry :source Observable)
(f/entry :related Observable)]
Expand All @@ -642,12 +655,12 @@
(f/entry :observables [Observable])
(f/entry :observed_time ObservedTime))
(f/optional-entries
(f/entry :os f/any-str)))
(f/entry :os ShortString)))
:description "Describes the target of the sighting and contains identifying observables for the target.")

(def scalar
(f/conditional
#(string? %) f/any-str
#(string? %) MedString
#(number? %) f/any-num
#(inst? %) f/any-inst
#(keyword? %) f/any-keyword
Expand Down
4 changes: 2 additions & 2 deletions src/ctim/schemas/data_table.cljc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,13 +29,13 @@
(def-map-type ColumnDefinition
(concat
(f/required-entries
(f/entry :name f/any-str)
(f/entry :name c/ShortString)
(f/entry :type ColumnType))
(f/optional-entries
(f/entry :description c/Markdown)
(f/entry :required f/any-bool
:description "If `true`, the row entries for this column cannot contain `nulls`. Defaults to `true`.")
(f/entry :short_description f/any-str))))
(f/entry :short_description c/ShortString))))

(def Datum
"A generic data object, this is really limited to the types
Expand Down
2 changes: 1 addition & 1 deletion src/ctim/schemas/feedback.cljc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
(f/entry :type FeedbackTypeIdentifier)
(f/entry :entity_id c/Reference)
(f/entry :feedback #{-1 0 1})
(f/entry :reason f/any-str)))
(f/entry :reason c/ShortString)))

(def-entity-type NewFeedback
"Schema for submitting new Feedback."
Expand Down
2 changes: 1 addition & 1 deletion src/ctim/schemas/identity_assertion.cljc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@
(def-map-type Assertion
(f/required-entries
(f/entry :name AssertionType)
(f/entry :value f/any-str)))
(f/entry :value c/ShortString)))

(def-map-type IdentityCoordinates
(f/required-entries
Expand Down
13 changes: 9 additions & 4 deletions src/ctim/schemas/incident.cljc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
(:require [ctim.schemas.common :as c]
[ctim.schemas.relationship :as rel]
[ctim.schemas.vocabularies :as v]
[ctim.lib.predicates :as pred]
#?(:clj [flanders.core :as f :refer [def-entity-type def-map-type def-eq]]
:cljs [flanders.core :as f :refer-macros [def-entity-type def-map-type def-eq]])
#?(:clj [clojure.test.check.generators :as gen])))
Expand Down Expand Up @@ -127,13 +128,15 @@
:description "metadata associated to the incident.")
(f/entry :scores IncidentScores
:description "Used to indicate the severity or impact score of the threat represented by the incident.")
(f/entry :categories [v/IncidentCategory]
(f/entry :categories (f/seq-of v/IncidentCategory
:spec (pred/max-len c/default-collection-max-len))
:description "A set of categories for this incident.")
(f/entry :discovery_method v/DiscoveryMethod
:description "Identifies how the incident was discovered.")
(f/entry :intended_effect v/IntendedEffect
:description "Specifies the suspected intended effect of this incident")
(f/entry :assignees [c/ShortString]
(f/entry :assignees (f/seq-of c/ShortString
:spec (pred/max-len c/default-collection-max-len))
:description "A set of owners assigned to this incident.")
(f/entry :detection_sources [c/MedString]

@yogsototh yogsototh Jun 5, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MEDIUM -- Uncapped sibling: :detection_sources kept the plain [c/MedString] form while its neighbours :categories, :assignees, :tactics, and :techniques all received (f/seq-of ... :spec (pred/max-len default-collection-max-len)) in this PR. If the size cap is meant to cover incident collection fields, this one was missed; if it is intentionally unbounded, a one-line note would make that explicit. (Same asymmetry exists between the bounded embedded SightingDataTable and the unbounded standalone DataTable entity in data_table.cljc.)

:description "A set of sources that contributed threat detections to the incident.")
Expand Down Expand Up @@ -164,13 +167,15 @@
"interpreted differently by different organizations or analysts. Therefore, it "
"should be used in conjunction with other intelligence attributes, such as the "
"`confidence` field, to provide a more comprehensive view of the incident."))
(f/entry :tactics [c/ShortString]
(f/entry :tactics (f/seq-of c/ShortString
:spec (pred/max-len c/default-collection-max-len))
:description (str "Represents the offensive techniques, approaches, or procedures that an adversary "
"may use to achieve their objectives during an attack. It helps in understanding "
"the intent and capabilities of the adversary and can be used to identify "
"indicators of attack (IoAs) or indicators of compromise (IoCs) that are "
"associated with the adversary's tactics."))
(f/entry :techniques [c/ShortString]
(f/entry :techniques (f/seq-of c/ShortString
:spec (pred/max-len c/default-collection-max-len))
:description (str "Represents the specific methods or actions used by an attacker "
"to carry out an offensive maneuver or achieve their goals."))
(f/entry :short_id c/ShortString
Expand Down
10 changes: 5 additions & 5 deletions src/ctim/schemas/indicator.cljc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,33 +28,33 @@

(def-map-type ThreatBrainSpecification
[(f/entry :type ThreatBrainSpecificationType)
(f/entry :query f/any-str
(f/entry :query c/MedString
:required? false)
(f/entry :variables f/any-str-seq)]
(f/entry :variables (f/seq-of c/ShortString))]
:description "An indicator which runs in threatbrain...")

(def-eq SnortSpecificationType "Snort")

(def-map-type SnortSpecification
(f/required-entries
(f/entry :type SnortSpecificationType)
(f/entry :snort_sig f/any-str))
(f/entry :snort_sig c/MedString))
:description "An indicator which runs in snort...")

(def-eq SIOCSpecificationType "SIOC")

(def-map-type SIOCSpecification
(f/required-entries
(f/entry :type SIOCSpecificationType)
(f/entry :SIOC f/any-str))
(f/entry :SIOC c/MedString))
:description "An indicator which runs in snort...")

(def-eq OpenIOCSpecificationType "OpenIOC")

(def-map-type OpenIOCSpecification
(f/required-entries
(f/entry :type OpenIOCSpecificationType)
(f/entry :open_IOC f/any-str))
(f/entry :open_IOC c/MedString))

@yogsototh yogsototh Jun 5, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HIGH -- Bound likely too small for documented content: this field's docstring describes it as "an XML blob of an openIOC indicator," and OpenIOC XML documents routinely exceed 2048 chars, so MedString will reject most real open_IOC values. SIOC (line 49) and snort_sig (line 41) share the same 2048 cap and the same risk. More broadly, replacing unbounded f/any-str with a hard cap rejects documents that validated before, with no migration path. Recommend auditing existing PROD data against these bounds and confirming the values are intended for these blob fields.

:description "An indicator which contains an XML blob of an openIOC indicator.")

(def-enum-type BooleanOperator
Expand Down
4 changes: 2 additions & 2 deletions src/ctim/schemas/note.cljc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@

(def-map-type NoteRelatedEntity
(f/required-entries
(f/entry :entity_type f/any-str)
(f/entry :entity_type c/ShortString)
(f/entry :entity_id c/Reference)))

(def-entity-type Note
Expand All @@ -27,7 +27,7 @@ For example, an analyst may add a Note to a Campaign object created by another o
(f/entry :content c/Markdown)
(f/entry :related_entities (f/seq-of NoteRelatedEntity)))
(f/optional-entries
(f/entry :author f/any-str)))
(f/entry :author c/ShortString)))

(def-entity-type NewNote
"Schema for submitting Notes"
Expand Down
Loading
Loading