Issue: Adding Annotations to the Cosmograph App

[thorwhalen]

Issue: Adding Annotations to the Cosmograph App

Sonnet 3.5

Overview

Annotations enable users to add contextual information and documentation to various aspects of their visualizations, improving communication, knowledge transfer, and reproducibility. This functionality allows teams to maintain institutional knowledge about their data and visualizations without relying on external documentation.

Example use cases:

1. A data analyst marks a cluster of anomalous points with "Q3 2024 system outage - exclude from baseline calculations" to help future analysts understand why certain data points should be filtered out.
2. A team lead adds project-level notes documenting "Data from legacy CRM system (2020-2023), merged with new system data (2024+). Use source_system field to distinguish" to prevent confusion about data inconsistencies.

Design Philosophy

Start Unstructured, Evolve Incrementally

Annotations should begin as freeform text (implicitly markdown-formatted) to minimize friction for both implementation and adoption. As usage patterns emerge, we can progressively add structure where it provides clear value.

Evolution path example:

• Phase 1: Data annotation as a single markdown text field
• Phase 2: Recognize common pattern of field descriptions, split into notes (freeform) + field_descriptions (freeform)
• Phase 3: Structure field_descriptions as Mapping[str, str] (field_name → description)
• Phase 4: Add validation, auto-completion, type inference for field descriptions

Core Abstraction: Key-Value Pairs

At the most general level, an annotation is:

Annotation = (key: Annotatable, value: AnnotationContent)

Where:

• key: Points to what is being annotated (a project, data source, visualization, specific data points, etc.)
• value: What we want to record about it (initially freeform text/markdown)

This abstraction allows us to:

• Apply consistent patterns across different annotation types
• Build reusable tooling (indexing, search, export, inversion)
• Evolve each annotation type independently as needs crystallize

Annotation Types & Use Cases

1. Project-Level Annotations

Purpose: High-level context about the entire project

Key examples:

• Data origins and lineage
• Business purpose and stakeholders
• Access permissions and data sensitivity
• Related projects or dashboards

Key structure: project_id → markdown_text

2. Data Source Annotations

Purpose: Document the raw data being visualized

Key examples:

• Source URL or database connection
• Data preparation steps and transformations
• Field descriptions and data dictionary
• Update frequency and freshness

Key structure (initial): data_source_id → markdown_text

Evolution path: Extract field descriptions into structured mapping

3. Visualization Annotations

Purpose: Document the visualization configuration itself

Key examples:

• Why specific visual encodings were chosen
• Known limitations or caveats in the visualization
• Instructions for interpreting the view
• Change log for visualization config

Key structure: (data_source_id, viz_config_hash) → markdown_text

This treats the visualization as data + viz_mapping, allowing annotations on specific visualization instances.

4. Data Point/Subset Annotations

Purpose: Mark and explain specific observations or patterns

Key examples:

• Annotate individual outlier points
• Label clusters or groups discovered in exploration
• Document filtered subsets (e.g., "training set", "validation set")
• Mark known data quality issues

Key structure:

• Point: (data_source_id, point_id) → markdown_text
• Group: (data_source_id, point_ids: Collection) → markdown_text
• Filter: (data_source_id, filter_function: Callable) → markdown_text

UI/UX Considerations

1. In-line annotation editor: Click any element (project name, data source, viz, point) to add/edit annotation
2. Annotation panel: Side panel showing all annotations relevant to current view
3. Search: Global annotation search across all types
4. Export: Download all annotations as a markdown documentation bundle
5. Visual indicators: Badge/icon showing which elements have annotations

Migration Path

1. v1: Implement project and data source annotations (simplest, highest value)
2. v2: Add visualization annotations
3. v3: Add point/subset annotations with basic retrieval
4. v4: Introduce structured field descriptions for data sources
5. v5: Add enhanced querying for point annotations (filters, groups)
6. v6: Implement search, indexing, and export tooling

Open Questions

1. Permissions: Should annotations have separate access controls from the underlying data?
2. Versioning: Should we track annotation history/changes?
3. Collaboration: How do multiple users annotate the same element? Merge, separate, thread?
4. Formatting: Start with markdown, but allow configuration? Rich text editor?
5. Size limits: Should individual annotations have size constraints?

[thorwhalen]

Python Implementation Ideas

Sonnet 3.5 w/ i2i instructions

Storage Design

Use a MutableMapping interface for annotation stores:

from typing import MutableMapping, Any, Protocol

class AnnotationKey(Protocol):
    """Protocol for objects that can be annotation keys"""
    def to_hashable(self) -> Any:
        """Convert to a hashable representation for storage"""
        ...

class AnnotationStore(MutableMapping[AnnotationKey, str]):
    """
    Base store for annotations with markdown content.
    
    Example:
        >>> store = AnnotationStore()
        >>> store[ProjectKey("proj_123")] = "# My Project\\nData from..."
        >>> print(store[ProjectKey("proj_123")])
        # My Project
        Data from...
    """
    def __init__(self, backend: MutableMapping[str, str]):
        self._backend = backend
    
    def __setitem__(self, key: AnnotationKey, value: str) -> None:
        self._backend[key.to_hashable()] = value
    
    def __getitem__(self, key: AnnotationKey) -> str:
        return self._backend[key.to_hashable()]
    
    def __delitem__(self, key: AnnotationKey) -> None:
        del self._backend[key.to_hashable()]
    
    def __iter__(self):
        return iter(self._backend)
    
    def __len__(self) -> int:
        return len(self._backend)

"Mall" Pattern for Multiple Annotation Types

Group related annotation stores:

annotation_mall = {
    'project': AnnotationStore(project_backend),
    'data_source': AnnotationStore(data_source_backend),
    'visualization': AnnotationStore(viz_backend),
    'data_points': AnnotationStore(points_backend),
}

# Usage:
annotation_mall['project'][project_key] = "# Project Notes\n..."
annotation_mall['data_points'][point_key] = "Outlier due to sensor malfunction"

Enhanced Retrieval for Data Point Annotations

Extend __getitem__ to support rich querying:

class DataPointAnnotationStore(AnnotationStore):
    """
    Enhanced store supporting multiple retrieval patterns.
    
    Examples:
        >>> store[point_id]  # Single point
        >>> store[[p1, p2, p3]]  # Multiple points
        >>> store[lambda p: p.x > 100]  # Filter function
        >>> store[cluster_id::]  # Slice notation for ranges
    """
    def __getitem__(self, key):
        if callable(key):
            # Filter function: return all annotations matching predicate
            return {k: v for k, v in self.items() if key(k)}
        elif isinstance(key, (list, set)):
            # Multiple keys: return mapping
            return {k: self._backend[k] for k in key if k in self._backend}
        elif isinstance(key, slice):
            # Slice: range-based retrieval
            return self._get_slice(key)
        else:
            # Single key: normal retrieval
            return super().__getitem__(key)

Tooling & Utilities

Build annotation-agnostic utilities that work across all types:

Search/Index

def search_annotations(
    store: Mapping[Any, str], 
    query: str
) -> 'Iterable[tuple[Any, str]]':
    """
    Search annotation content for query string.
    
    Yields:
        (key, value) pairs where value contains query
    """
    for key, value in store.items():
        if query.lower() in value.lower():
            yield key, value

Inversion

def invert_annotations(
    store: Mapping[Any, str]
) -> 'Mapping[str, list[Any]]':
    """
    Create inverted index: annotation content → keys.
    
    Useful for finding "all things annotated with X"
    """
    from collections import defaultdict
    inverted = defaultdict(list)
    for key, value in store.items():
        inverted[value].append(key)
    return dict(inverted)

Export

def export_annotations(
    store: Mapping[Any, str], 
    format: str = 'markdown'
) -> str:
    """
    Export all annotations in specified format.
    
    Supports: 'markdown', 'json', 'yaml'
    """
    if format == 'markdown':
        return '\n\n'.join(
            f"## {key}\n\n{value}" 
            for key, value in store.items()
        )
    # ... other formats