Feat/storage visibility

.bumpversion.toml

@@ -3,7 +3,7 @@
 # https://peps.python.org/pep-0440/
 
 [tool.bumpversion]
-    current_version = "1.0.2.dev4"
+    current_version = "1.0.2.dev7"
     parse = """(?x)
     (?P<major>0|[1-9]\\d*)\\.
     (?P<minor>0|[1-9]\\d*)\\.

.gitignore

@@ -192,6 +192,7 @@ certs/
 docker-compose.override.yml
 
 CLAUDE.md
+/docker/init_pycharm_helpers.sh
 
 # Claude Code (share .claude/agents + .claude/skills; keep local settings out)
 .claude/settings.local.json

.pre-commit-config.yaml

@@ -53,7 +53,7 @@ repos:
         name: pytest
         # --build avoids stale images; TEST_MARKER env is honored by entrypoint-test.sh
         # (CLI args after the service name are ignored), so set the marker via env.
-        entry: docker compose run --rm -T --build -e "TEST_MARKER=not integration" tests
+        entry: docker compose -f docker-compose.yml run --rm -T --build -e "TEST_MARKER=not integration" tests
         language: system
         pass_filenames: false
         types: [python]

docs/changelog/1.0.2.dev6.md

@@ -0,0 +1,203 @@
+# 1.0.2.dev5 → 1.0.2.dev6 — Record Visibility & Context Scopes (incl. cross-owner USERS / ORGANIZATIONS)
+
+## Summary
+
+This changelog documents the storage work delivered across the **1.0.2.dev5 → 1.0.2.dev6** line. Two related
+capabilities landed:
+
+1. **Record visibility** — every storage record now carries a read-access scope (`Visibility`) that is independent of
+   write ownership. You can tag a record `PUBLIC` / `PRIVATE` / `INTERNAL` on write and filter by it on list.
+2. **Context scopes** — the old string `scope: Literal["mission", "setup"]`
+   argument on the storage service was replaced by a typed `Context`
+   enum, and the enum was extended with two **read-only cross-owner** scopes,
+   `USERS` and `ORGANIZATIONS`. `dev6` completes the client so those scopes are actually emitted on the wire — a kin can
+   now list records shared by other kins of the same user/organization.
+
+The concrete owner id for the cross-owner scopes is **resolved server-side**
+from request metadata; the client only sends the context *kind*.
+
+> **Requirement:** these features need `agentic-mesh-protocol` with the
+> `visibility` fields and the `CONTEXT_USERS` / `CONTEXT_ORGANIZATIONS` enum
+> values (shipped in the proto ≥ `1.0.1.dev4`). It is pulled in transitively by
+> this SDK version.
+
+## What changed
+
+### `Context` — the owner/scope of an operation (replaces `scope`)
+
+```python
+
+from digitalkin.models.services.services import Context
+
+
+class ContextStorage(Enum):
+    UNSPECIFIED = "unspecified"
+    MISSIONS = "missions"  # this mission (default)
+    SETUP_VERSIONS = "setup_versions"  # this setup version (shared across missions)
+    USERS = "users"  # read-only: all kins of the same user
+    ORGANIZATIONS = "organizations"  # read-only: all kins of the same organization
+```
+
+- `MISSIONS` (default) and `SETUP_VERSIONS` are **read/write** owner contexts.
+- `USERS` and `ORGANIZATIONS` are **read-only, list-only** cross-owner scopes. The strategy holds no user/org id; it
+  sends only the kind and the storage service resolves the concrete id from the `x-user-id` / `x-organization-id`
+  request metadata.
+
+Every public storage method now takes `context: ContextStorage` instead of the old `scope: str`:
+
+| Method              | Signature (relevant args)                                                                                                            |
+|---------------------|--------------------------------------------------------------------------------------------------------------------------------------|
+| `store`             | `store(collection, record_id, data, data_type=DataType.OUTPUT, context=ContextStorage.MISSIONS, visibility=Visibility.UNSPECIFIED)`  |
+| `read`              | `read(collection, record_id, context=ContextStorage.MISSIONS)`                                                                       |
+| `update`            | `update(collection, record_id, data, context=ContextStorage.MISSIONS, visibility=Visibility.UNSPECIFIED)`                            |
+| `remove`            | `remove(collection, record_id, context=ContextStorage.MISSIONS)`                                                                     |
+| `list`              | `list(collection, context=ContextStorage.MISSIONS, visibilities=None)`                                                               |
+| `remove_collection` | `remove_collection(collection, context=ContextStorage.MISSIONS)`                                                                     |
+| `upsert`            | `upsert(collection, record_id, data, data_type=DataType.OUTPUT, context=ContextStorage.MISSIONS, visibility=Visibility.UNSPECIFIED)` |
+
+### `Visibility` — read-access scope of a record
+
+```python
+from digitalkin.models.services.storage import Visibility
+
+class Visibility(Enum):
+    UNSPECIFIED = 0   # let the storage service apply its default
+    PUBLIC      = 1
+    PRIVATE     = 2
+    INTERNAL    = 3
+```
+
+- The integer values **mirror the storage proto** exactly.
+- Ownership (who may *edit*) stays keyed on the record's `context`; `Visibility`
+  only governs who may *read* it.
+- `StorageRecord` gained a `visibility: Visibility` field (default
+  `UNSPECIFIED`), populated from the wire on read.
+- `visibility=UNSPECIFIED` is the proto default (`0`) and is wire-identical to not setting it, so the storage service
+  applies its own default.
+
+### Cross-owner wire mapping (completed in dev6)
+
+`GrpcStorage._context_enum` now maps the resolved context to the right wire enum, including the new cross-owner kinds:
+
+- `setup_versions:…` → `CONTEXT_SETUP_VERSIONS`
+- `users:` → `CONTEXT_USERS`
+- `organizations:` → `CONTEXT_ORGANIZATIONS`
+- otherwise → `CONTEXT_MISSIONS`
+
+and `StorageStrategy._resolve_context` returns a kind-only marker (`users:` /
+`organizations:`) for the cross-owner scopes, since the concrete id is resolved server-side.
+
+> **Local `DefaultStorage`** has no cross-owner data model, so listing under
+> `USERS` / `ORGANIZATIONS` returns `[]` in local/dev mode. Cross-owner reads
+> are a remote (`GrpcStorage`) capability.
+
+## How to use
+
+All examples assume you have a storage strategy (e.g. `context.storage` inside a trigger handler).
+
+### Write a record with a visibility
+
+```python
+from digitalkin.models.services.storage import Visibility
+from digitalkin.models.services.services import Context
+
+# Readable by every kin of the same user, owned by this mission
+await storage.store(
+    "reports",
+    "q3-summary",
+    {"title": "Q3", "body": "..."},
+    visibility=Visibility.PUBLIC,
+)
+
+# Persist under the setup version (survives across missions), keep it internal
+await storage.upsert(
+    "shared_config",
+    "defaults",
+    {"lang": "fr"},
+    context=Context.SETUP_VERSIONS,
+    visibility=Visibility.INTERNAL,
+)
+```
+
+### Change a record's visibility later
+
+```python
+# UNSPECIFIED leaves the current visibility unchanged
+await storage.update("reports", "q3-summary", {"title": "Q3", "body": "..."},
+                     visibility=Visibility.PRIVATE)
+```
+
+### List and filter by visibility
+
+```python
+# All readable records in this mission
+records = await storage.list("reports")
+
+# Only PUBLIC + INTERNAL records
+records = await storage.list(
+    "reports",
+    visibilities=[Visibility.PUBLIC, Visibility.INTERNAL],
+)
+
+for r in records:
+    print(r.record_id, r.visibility.name, r.context)
+```
+
+### Cross-owner reads (discover data produced by other kins of the same user)
+
+```python
+# Records other kins of the SAME USER created and shared, subject to visibility.
+# The server resolves the concrete user id from the request metadata.
+records = await storage.list(
+    "reports",
+    context=ContextStorage.USERS,
+    visibilities=[Visibility.PUBLIC],
+)
+
+# Same, but across the whole organization
+records = await storage.list("reports", context=ContextStorage.ORGANIZATIONS)
+```
+
+> Cross-owner scopes are **read-only**: use them with `list` only. `store` /
+> `update` / `remove` always target the owning `MISSIONS` / `SETUP_VERSIONS`
+> context.
+
+## Migration
+
+- **`scope=` → `context=`**: replace every `scope="mission"` /
+  `scope="setup"` string argument with `context=ContextStorage.MISSIONS` /
+  `context=ContextStorage.SETUP_VERSIONS`. The parameter was renamed and retyped from a `str` literal to the
+  `Context` enum, so passing the old string raises `TypeError`.
+- **`data_type`**: pass the `DataType` enum (e.g. `DataType.OUTPUT`), not a string — `data_type="OUTPUT"` no longer
+  works.
+- **New optional args**: `visibility` (on `store`/`update`/`upsert`) and
+  `visibilities` (on `list`) are optional; omit them to keep the previous behaviour (server default visibility, no
+  visibility filter).
+- **No change to `read` / `remove` semantics** beyond the `scope` → `context`
+  rename.
+
+Minimal before/after:
+
+```python
+# before (<= 1.0.0a0)
+await storage.list("reports", scope="setup")
+await storage.store("reports", "r1", data, data_type="OUTPUT")
+
+# after (>= 1.0.2.dev6)
+from digitalkin.models.services.storage import DataType
+from digitalkin.models.services.services import Context
+
+await storage.list("reports", context=Context.SETUP_VERSIONS)
+await storage.store("reports", "r1", data, data_type=DataType.OUTPUT)
+```
+
+## Verification
+
+Storage regression coverage lives in `tests/services/storage/`:
+
+- `test_grpc_storage.py` — round-trips `visibility` on `store`/`update`, the
+  `visibilities` filter on `list`, and
+  `test_list_cross_owner_context_and_visibilities` locks the wire mapping (`USERS → CONTEXT_USERS`,
+  `ORGANIZATIONS → CONTEXT_ORGANIZATIONS`).
+- `test_storage_strategy_locks.py` — per-record lock keys use the resolved context string, so locks are created and
+  cleaned up under the right owner.

docs/changelog/1.0.2.dev7.md

@@ -0,0 +1,115 @@
+# 1.0.2.dev6 → 1.0.2.dev7 — Filesystem Context Scopes (mission/setup/user/organization)
+
+## Summary
+
+This release brings the **filesystem** service in line with the storage context model shipped in `dev6`. The string
+`context: Literal["mission", "setup"]`
+argument is replaced by a typed `Context` enum, extended with the two **read-only cross-owner** scopes `USERS` and
+`ORGANIZATIONS`. A file produced by one kin can now be read by another kin of the same user/organization, subject to
+server-side access control.
+
+A small consistency fix also lands on the **storage** side: `ContextStorage.UNSPECIFIED`
+now maps to the unspecified wire enum instead of being silently treated as MISSIONS, matching the filesystem behaviour.
+
+Only the context *kind* is sent on the wire — no id is transmitted by the client. The concrete owner (mission / setup /
+user / organization) is resolved server-side from the request context.
+
+> **Requirement:** needs `agentic-mesh-protocol` with the filesystem
+> `CONTEXT_USERS` / `CONTEXT_ORGANIZATIONS` enum values. It is pulled in
+> transitively by this SDK version.
+
+## What changed
+
+### `Context` — the owner/scope of a filesystem operation (replaces `scope`/`context` strings)
+
+```python
+
+from digitalkin.models.services.services import Context
+
+
+class ContextFile(Enum):
+  UNSPECIFIED = "unspecified"
+  MISSIONS = "mission"  # this mission (default)
+  SETUP = "setup"  # this setup version
+  USERS = "user"  # read-only: all kins of the same user
+  ORGANIZATIONS = "organization"  # read-only: all kins of the same organization
+```
+
+- `MISSIONS` (default) and `SETUP` are the read/write owner contexts.
+- `USERS` and `ORGANIZATIONS` are **read-only cross-owner** scopes (use them on reads: `get_file` / `get_files`).
+- The enum values are singular strings, so Pydantic still coerces legacy string contexts on `FileFilter` (e.g.
+  `FileFilter(context="setup")`).
+
+Read methods now take `context: ContextFile`:
+
+| Method      | Signature (relevant args)                                                             |
+|-------------|---------------------------------------------------------------------------------------|
+| `get_file`  | `get_file(file_id, context=ContextFile.MISSIONS, *, include_content=False)`           |
+| `get_files` | `get_files(filters, ...)` where `filters.context: ContextFile = ContextFile.MISSIONS` |
+
+`_context_enum` maps every kind to its wire enum, including
+`CONTEXT_USERS` / `CONTEXT_ORGANIZATIONS` / `CONTEXT_UNSPECIFIED`.
+
+### Writes stay owner-scoped
+
+`upload_files` / `update_file` / `delete_files` remain mission-scoped, exactly as before. Cross-owner scopes are
+read-only — you cannot write into another kin's user/organization space.
+
+### Storage: UNSPECIFIED consistency fix
+
+`ContextStorage.UNSPECIFIED` now resolves to the `unspecified:` kind marker and maps to `CONTEXT_UNSPECIFIED` on the
+wire (server applies its default), instead of silently becoming `CONTEXT_MISSIONS`. Public callers are unaffected — the
+default context stays `MISSIONS`.
+
+## How to use
+
+```python
+
+from digitalkin.models.services.services import Context
+from digitalkin.services.filesystem.filesystem_strategy import FileFilter
+
+# Read a file owned by the current mission (default)
+record = await filesystem.get_file(file_id, include_content=True)
+
+# Read a file from the setup-version scope
+record = await filesystem.get_file(file_id, context=Context.SETUP)
+
+# Cross-owner: list files shared by other kins of the same user.
+# The server resolves the concrete user id; no id is sent by the client.
+records, total = await filesystem.get_files(
+  FileFilter(context=Context.USERS, prefix="reports/"),
+)
+
+# Same across the whole organization
+records, total = await filesystem.get_files(FileFilter(context=Context.ORGANIZATIONS))
+```
+
+## Migration
+
+- **`context="mission"` / `context="setup"` → `Context`**: pass
+  `ContextFile.MISSIONS` / `ContextFile.SETUP` to `get_file`. For `FileFilter`, the legacy strings still validate
+  (Pydantic coerces them by value), but prefer the enum for clarity.
+- **No change to write calls** (`upload_files` / `update_file` / `delete_files`).
+- Import the enum from `digitalkin.models.services.filesystem`.
+
+Minimal before/after:
+
+```python
+# before
+await filesystem.get_file(file_id, context="setup")
+await filesystem.get_files(FileFilter(context="mission", prefix="x/"))
+
+# after
+from digitalkin.models.services.services import Context
+
+await filesystem.get_file(file_id, context=Context.SETUP)
+await filesystem.get_files(FileFilter(context=Context.MISSIONS, prefix="x/"))
+```
+
+## Verification
+
+- `tests/services/filesystem/test_grpc_filesystem.py::TestContextScopes` locks the wire mapping for all kinds —
+  `MISSIONS`, `SETUP`, `USERS`, `ORGANIZATIONS`,
+  `UNSPECIFIED` — on both `get_files` and `get_file`.
+- `tests/services/storage/test_grpc_storage.py::TestListData` covers
+  `UNSPECIFIED → CONTEXT_UNSPECIFIED` alongside the cross-owner storage scopes.

pyproject.toml

@@ -37,7 +37,7 @@
         "pydantic>=2.12.4",
         "redis[hiredis]>=7.4.0,<9",
     ]
-    version = "1.0.2.dev4"
+    version = "1.0.2.dev7"
 
     [project.optional-dependencies]
         agno = [ "agno>=2.6" ]

src/digitalkin/__version__.py

@@ -5,4 +5,4 @@
 try:
     __version__ = version("digitalkin")
 except PackageNotFoundError:
-    __version__ = "1.0.2.dev4"
+    __version__ = "1.0.2.dev7"

src/digitalkin/grpc_servers/_base_server.py

@@ -3,6 +3,7 @@
 import abc
 import asyncio
 import os
+import sys
 
 from digitalkin.models.settings.profiling import get_profiling_settings
 
@@ -196,10 +197,16 @@ def _create_server(self) -> GrpcServer:
         try:  # noqa: PLW0717
             grpc_compression = get_server_settings().grpc.compression.to_grpc()
 
-            try:
-                cpu_count = len(os.sched_getaffinity(0))
-                logger.info("vCPU count: %d", cpu_count)
-            except (AttributeError, OSError):
+            # sched_getaffinity is Linux-only; the sys.platform guard lets mypy skip
+            # it as unreachable on other platforms without a type: ignore.
+            if sys.platform == "linux":
+                try:
+                    cpu_count = len(os.sched_getaffinity(0))
+                    logger.info("vCPU count: %d", cpu_count)
+                except OSError:
+                    cpu_count = os.cpu_count() or 1
+                    logger.info("CPU count: %d", cpu_count)
+            else:
                 cpu_count = os.cpu_count() or 1
                 logger.info("CPU count: %d", cpu_count)