Main 3d

.gitignore

@@ -5,6 +5,9 @@ modules.zip
 # macOS system files
 .DS_Store
 
+# Binary pickle files (never commit cached data)
+*.pkl
+
 # Python cache and environment
 __pycache__/
 *.py[cod]

modules/accelerators/contextualization/cdf_three_dimension/README.md

@@ -0,0 +1,149 @@
+# 3D Contextualization (CDF Toolkit)
+
+This CDF Toolkit module deploys a **data modeling (DM)–first** pipeline that contextualizes 3D CAD nodes with Cognite Asset instances: manual and rule-based input, ML matching, quality review, and RAW outputs for iterative tuning.
+
+It is parameterized by **`default_location`** and **`source_name`** (for example `site-a` / `navisworks`), so resource external IDs follow the pattern `…_{{default_location}}_{{source_name}}` after build.
+
+---
+
+## Documentation
+
+### [docs/3d_contextualization_dm_only_guide.md](docs/3d_contextualization_dm_only_guide.md)
+
+Step-by-step setup guide for running this module in a **DM-only or hybrid** CDF project.
+
+| Section | What it covers |
+|---------|---------------|
+| **1. Confirm Project Type** | Check whether classic 3D asset mappings are available or if the project is DM-only |
+| **2. Full DM 3D Chain** | Required 5-node chain: `Asset → Cognite3DObject → CogniteCADNode → CogniteCADRevision → CogniteCADModel` with correct space assignment rules |
+| **3. Required Data Model Views** | Full list of 18 views (including edge sources) that must be added to the data model to avoid `_UnknownType` GraphQL errors in Industrial Tools |
+| **4. Create DM 3D Instances** | Use the dedicated `/3d/contextualization/cad` endpoint (DM-only projects) or the fallback manual script |
+| **5. Create Scene Configuration** | Create the `SceneConfiguration` + `Cdf3dModel` + `RevisionProperties` edge required for Industrial Tools 3D preview |
+| **6. Space Assignment Rules** | Which instance space to use for each node type (assets, 3D objects, CAD nodes, scene) |
+| **7. Industrial Tools Location Setup** | One-time browser step to configure the IT Location (cannot be done via API) |
+| **8. Verification Checklist** | End-to-end Python verification script that checks every link in the chain |
+| **9. Common Mistakes** | Node ID vs tree index confusion, wrong spaces, missing views, missing scene config |
+
+---
+
+### [docs/CONFIGURATION_VERIFICATION.md](docs/CONFIGURATION_VERIFICATION.md)
+
+Reference guide for verifying that pipeline config and instance/view spaces are correctly deployed before running functions.
+
+| Section | What it covers |
+|---------|---------------|
+| **1. Concepts** | Instance space vs view space vs asset DM space vs CAD node space — when they differ and why |
+| **2. Where config is read** | Each function reads runtime settings from the deployed CDF extraction pipeline config (not from local YAML); env var fallbacks where available |
+| **3. Main annotation pipeline** | Required `parameters` to verify in the deployed `ctx_3d_*_annotation` pipeline config (`assetDmSpace`, `assetView*`, `cadNodeDmSpace`, `rawdb`, etc.) |
+| **4. Upload asset hierarchy pipeline** | Required parameters for `ctx_3d_*_upload_asset_hierarchy` (`assetInstanceSpace`, `assetView*`) |
+| **5. Verification checklist** | Build → deploy → inspect live config → check view versions → check spaces → smoke test |
+| **6. Common failures** | Unresolved `${VAR}` in config, 0 assets found, ValueError on upload, permission errors |
+| **7. Changing location/source_name** | Impact on resource naming and what needs manual migration |
+
+---
+
+## Prerequisites
+
+- A **DM-only or hybrid** CDF project with the 3D chain and views required for Industrial Tools / CAD contextualization. Follow **[docs/3d_contextualization_dm_only_guide.md](docs/3d_contextualization_dm_only_guide.md)** for spaces, `Cognite3DObject` / `CADNode` linking, scene configuration, and common pitfalls.
+- After configuring **`asset_instance_space`**, **view** (`assetView*` / `asset_view_*`), and pipeline IDs for your project, use **[docs/CONFIGURATION_VERIFICATION.md](docs/CONFIGURATION_VERIFICATION.md)** to verify live CDF pipeline config and instance vs view spaces.
+- A **3D model** ingested and processed in CDF (file upload via UI or the file extractor pattern in this module).
+- Toolkit **`variables`** set in your environment config (see `default.config.yaml` and `env.template`).
+
+---
+
+## Processing Workflow
+
+High-level contextualization flow (manual input → ML → good/bad RAW → retune):
+
+![Processing workflow](https://github.com/cognitedata/toolkit/assets/31886431/b29522f8-7f4b-4e23-b06a-f3ffffde103c)
+
+---
+
+## Managed Resources
+
+Names below use toolkit variables; **deployed** external IDs use your configured `default_location` and `source_name`.
+
+### 1. Auth groups
+
+| Pattern | Purpose |
+|--------|---------|
+| `gp_3d_{{default_location}}` | Merged group for 3D extraction, processing, and read access |
+
+Source IDs for the group come from your IdP (`3d_location_group_source_id` in config).
+
+### 2. Data set
+
+| External ID | Role |
+|-------------|------|
+| `ds_3d_{{default_location}}` | Lineage for extraction pipelines, functions, RAW, and files |
+
+### 3. Extraction pipelines
+
+| External ID pattern | Role |
+|---------------------|------|
+| `ep_src_3d_{{default_location}}_{{source_name}}` | File extractor: upload 3D files from local disk, SharePoint, etc. |
+| `ep_ctx_3d_{{default_location}}_{{source_name}}_annotation` | Main **3D ↔ asset** contextualization (CDF Function) |
+| `ep_ctx_3d_{{default_location}}_{{source_name}}_annotation_quality_check` | Quality check on contextualization results |
+| `ep_ctx_3d_{{default_location}}_{{source_name}}_upload_manual_mappings` | Load **CSV** mappings into RAW (`3DId`, `assetId`, …) |
+| `ep_ctx_3d_{{default_location}}_{{source_name}}_upload_asset_hierarchy` | Load **asset hierarchy** CSV into DM asset instances |
+
+Pipeline documentation and **runtime parameters** (RAW DB, DM asset/CAD spaces, model name, thresholds) live in the corresponding `*.config.Config.yaml` next to each pipeline.
+
+### 4. Functions
+
+| External ID pattern | Role |
+|---------------------|------|
+| `fn_context_3d_{{default_location}}_{{source_name}}_asset` | Main annotation / contextualization |
+| `fn_context_3d_{{default_location}}_{{source_name}}_quality_check` | Post-run quality checks |
+| `fn_context_3d_{{default_location}}_{{source_name}}_upload_manual_mappings` | CSV → `contextualization_manual_input` |
+| `fn_context_3d_{{default_location}}_{{source_name}}_upload_asset_hierarchy` | CSV → DM assets |
+
+The main contextualization function follows the pipeline described on the annotation extraction pipeline: read manual RAW, apply overrides, match assets using DM configuration, write **good** / **bad** tables for workflow and tuning.
+
+### 5. RAW database and tables
+
+| Database | Tables |
+|----------|--------|
+| `3d_{{default_location}}_{{source_name}}` | `contextualization_good`, `contextualization_bad`, `contextualization_manual_input`, `contextualization_rule` (optional rule-based mapping before ML) |
+
+### Illustrations
+
+**3D data pipeline:**
+
+![3D data pipeline](https://github.com/cognitedata/toolkit/assets/31886431/f1129181-bab0-42cb-8366-860e8fb30d7e)
+
+**Contextualization workflow** (good/bad tables, manual and rule modules):
+
+![Contextualization workflow](https://github.com/cognitedata/toolkit/assets/31886431/0e990b47-0c06-4040-b680-7e2dddcdccee)
+
+---
+
+## Variables
+
+Set module variables in **`default.config.yaml`** (merged by the Toolkit). Values such as DM spaces, model names, and `function_space` are read from **environment variables** (see **`env.template`**). Copy `env.template` → `.env` and fill in real values before `cdf build`.
+
+| Variable | Description |
+|----------|-------------|
+| `default_location` | Short location key used in resource names (e.g. `site-a`) |
+| `source_name` | Source system key (e.g. `navisworks`, `fileshare`) |
+| `3d_model_name` | 3D / CAD model identifier in CDF (from `.env`: `THREE_D_MODEL_NAME`) |
+| `3d_dataset` | Data set external ID for 3D resources (typically `ds_3d_<location>`) |
+| `raw_db`, `raw_table_manual` | RAW database and manual-input table |
+| `asset_instance_space`, `cad_node_instance_space` | DM instance spaces (from `.env`: `ASSET_INSTANCE_SPACE`, `CAD_NODE_INSTANCE_SPACE`) |
+| `function_space` | Space for Cognite Function code artifacts (from `.env`: `FUNCTION_SPACE`) |
+| `default_dm_space`, `dm_ext_id`, `dm_version` | Data model reference (from `.env` via `${DM_EXT_ID}` / `${DM_VERSION}`) |
+| `default_cad_space`, `default_scene_space` | CAD and scene configuration spaces |
+| `required_views`, `cad_model_view`, … | Optional overrides; defaults are in `fn_context_3d_cad_asset_contextualization/config.py` (`_DEFAULT_*`) |
+| `file_extractor_watch_path` | Local path for file extractor config (if used) |
+| `3d_location_group_source_id` | IdP object ID for the auth group |
+| `cicd_clientId`, `cicd_clientSecret` | Optional: schedules / automation |
+
+---
+
+## Usage
+
+Copy this module into your project's `custom_modules` (or reference it from `modules/`), adjust **`variables`** for your project and IdP groups, then add it under **`selected_modules_and_packages`** in your `config.<env>.yaml` and deploy with the CDF Toolkit.
+
+See [Using Templates](https://developer.cognite.com/sdks/toolkit/templates).
+
+**Note:** Cognite Functions have **time and memory limits**. Very large asset or node volumes may need batching or a different runtime than a single function invocation.

modules/accelerators/contextualization/cdf_three_dimension/auth/3d.Group.yaml

@@ -0,0 +1,95 @@
+# Single group for all 3D pipeline actors: file extractor, contextualization functions, and read-only users.
+# Merged from 3d.extractor.Group, 3d.processing.Group, 3d.read.Group for simplified dev/small-team setup.
+name: 'gp_3d_{{default_location}}'
+sourceId: '{{3d_location_group_source_id}}'
+metadata:
+  origin: 'cdf-project-templates'
+capabilities:
+  # RAW: read/write (extractor + processing)
+  - rawAcl:
+      actions:
+        - READ
+        - WRITE
+      scope:
+        tableScope:
+          dbsToTables:
+            3d_{{default_location}}_{{source_name}}: []
+
+  # Files: read/write (extractor uploads, processing reads, users view)
+  - filesAcl:
+      actions:
+        - READ
+        - WRITE
+      scope:
+        datasetScope:
+          ids: ['ds_3d_{{default_location}}']
+
+  # 3D models: read (processing + users)
+  - threedAcl:
+      actions:
+        - READ
+      scope:
+        datasetScope:
+          ids: ['ds_3d_{{default_location}}']
+
+  # Data Model instances: read/write (processing writes AssetExtension nodes)
+  - dataModelInstancesAcl:
+      actions:
+        - READ
+        - WRITE
+      scope:
+        spaceIdScope:
+          spaceIds:
+            - '{{data_model_space}}'
+            - cdf_cdm
+
+  # Extraction pipelines: read configs, read/write runs
+  - extractionConfigsAcl:
+      actions:
+        - READ
+        - WRITE
+      scope:
+        datasetScope:
+          ids: ['ds_3d_{{default_location}}']
+  - extractionRunsAcl:
+      actions:
+        - READ
+        - WRITE
+      scope:
+        datasetScope:
+          ids: ['ds_3d_{{default_location}}']
+
+  # Sessions: needed for CDF Functions
+  - sessionsAcl:
+      actions:
+        - LIST
+        - CREATE
+        - DELETE
+      scope:
+        all: {}
+
+  # Entity matching + functions (contextualization processing)
+  - entitymatchingAcl:
+      actions:
+        - READ
+        - WRITE
+      scope:
+        all: {}
+  - functionsAcl:
+      actions:
+        - READ
+        - WRITE
+      scope:
+        all: {}
+
+  # Annotations + labels: read (users)
+  - annotationsAcl:
+      actions:
+        - READ
+      scope:
+        all: {}
+  - labelsAcl:
+      actions:
+        - READ
+      scope:
+        all: {}

modules/accelerators/contextualization/cdf_three_dimension/data_sets/ds_3d_default_location.DataSet.yaml

@@ -0,0 +1,14 @@
+externalId: ds_3d_{{default_location}}
+name:  3d:{{default_location}}
+description: 3D data for {{default_location}} from {{source_name}}
+metadata:
+  consoleSource:
+    names:
+      - '{{source_name}}'
+  rawTables:
+    - databaseName: 3d_{{default_location}}_{{source_name}}
+      tableName: 'contextualization_good'
+    - databaseName: '3d_{{default_location}}_{{source_name}}'
+      tableName: 'contextualization_bad'
+    - databaseName: '3d_{{default_location}}_{{source_name}}'
+      tableName: 'contextualization_manual_input'

modules/accelerators/contextualization/cdf_three_dimension/default.config.yaml

@@ -0,0 +1,46 @@
+environment:
+  name: dev
+  project: <my-project>
+  validation-type: dev
+  selected:
+  - modules/
+
+variables:
+  modules:
+    accelerators:
+      contextualization:
+        cdf_three_dimension:
+          default_location: ${default_location}
+          source_name: ${source_name}
+          module_owner: ${MODULE_OWNER}  # owner string attached to deployed Functions (team/email)
+          raw_db: ${RAW_DB}          # raw database name for manual mappings
+          raw_table_manual: contextualization_manual_input
+          raw_table_asset_staging: asset_hierarchy_staging
+          function_space: ${FUNCTION_SPACE}
+          3d_model_name: ${3d_model_name}   # 3d_model_name is the "name" property of cognite3dmodel
+          cad_model_name: cdf_cdm
+          cad_model_type: CAD
+          default_cad_space: cdf_cdm
+          default_dm_space: cdf_cdm
+          data_model_space: cdf_cdm
+          dm_ext_id: ${DM_EXT_ID}  # data model external id for the project
+          dm_version: ${DM_VERSION}
+          default_scene_space: scene
+          cad_contextualization_batch_size: 100
+          # required_views / cad_*_view: omitted here — defaults are in
+          # functions/fn_context_3d_cad_asset_contextualization/config.py (_DEFAULT_REQUIRED_VIEWS, _DEFAULT_CAD_VIEWS).
+          # Override in your project config.dev.yaml if you use non-cdf_cdm view spaces.
+
+          asset_instance_space: ${ASSET_INSTANCE_SPACE}
+          cad_node_instance_space: ${CAD_NODE_INSTANCE_SPACE}
+          asset_view_space: ${ASSET_VIEW_SPACE}            # view space (often equal to data_model_space)
+          asset_view_ext_id: ${ASSET_VIEW_EXT_ID}          # view where asset are available for contextualization
+          asset_view_version: ${ASSET_VIEW_VERSION}        # version of the view where asset are available for contextualization
+          file_extractor_watch_path: c:/tmp/files
+          3d_location_group_source_id: ${3d_location_group_source_id}   #group source id for the project
+          cicd_clientId: ${cicd_clientId}
+          cicd_clientSecret: ${cicd_clientSecret}
+          cicd_tokenUri: ${IDP_TOKEN_URL}
+          cdf_project_name: ${CDF_PROJECT}
+          cdf_cluster: ${CDF_CLUSTER}
+          external_root_id_asset: ${EXTERNAL_ROOT_ID_ASSET}  #optional: only used for legacy behavior