[Feature] Find semantically similar and duplicated code across the codebase

[mytecor]

Problem

cocoindex-code already indexes AST-based code chunks and allows finding semantically similar chunks for a given query.

However, detecting duplicated or highly similar code across an entire repository currently requires an external loop:

1. enumerate every indexed chunk;
2. use each chunk as a search query;
3. request a sufficiently large top-k;
4. filter results by similarity score;
5. deduplicate pairs;
6. build groups of related chunks.

A fixed top-k is inconvenient for this use case. One chunk may have no similar implementations, while another may have dozens. A low limit can miss matches, while a high limit returns many irrelevant candidates.

Proposed feature

Add a command that scans the existing index and reports groups of semantically similar code chunks.

For example:

ccc duplicates \
  --threshold 0.88 \
  --min-lines 5 \
  --lang typescript \
  --format text

Possible output:

Group 1: 3 similar chunks

  0.94  src/users/get-user.ts:12-28
  0.92  src/admin/load-user.ts:41-59
  0.89  src/legacy/user-query.ts:8-25

Group 2: 2 similar chunks

  0.91  src/orders/validate.ts:17-34
  0.91  src/refunds/validate-order.ts:22-40

Suggested options:

--threshold <score>
--min-lines <number>
--lang <language>
--path <glob>
--exclude-path <glob>
--exclude-same-file
--exclude-overlapping
--format text|json|sarif

Search semantics

The operation should find all indexed chunk pairs whose similarity is above a threshold, rather than requiring the caller to choose a fixed number of results.

Conceptually:

similarity(chunkA, chunkB) >= threshold

The implementation can still use an approximate nearest-neighbor index. The request is about threshold-based result semantics, not necessarily exact brute-force comparison.

To avoid duplicate work and output, only one canonical pair should be processed:

chunkA.id < chunkB.id

Related pairs could then be grouped using connected components or union-find:

A similar to B
B similar to C
=> group [A, B, C]

Lower-level API

Even without a complete ccc duplicates command, exposing a lower-level API would make this possible for external tools.

For example:

find_similar_chunks(
    chunk_id: str,
    score_threshold: float,
    limit: int | None = None,
    languages: list[str] | None = None,
    paths: list[str] | None = None,
) -> list[SearchResult]

It would also be useful to expose an iterator over indexed chunks:

iter_chunks(
    languages: list[str] | None = None,
    paths: list[str] | None = None,
) -> Iterator[Chunk]

This would allow users to implement custom duplicate detection, clustering and reporting without accessing CocoIndex Code's internal SQLite tables.

Incremental behavior

A later optimization could reuse CocoIndex's incremental indexing model:

• when a chunk is added or changed, compare only that chunk against the index;
• remove similarity relationships associated with deleted or changed chunks;
• preserve relationships between unchanged chunks.

This would make duplicate-code analysis suitable for CI and large repositories.

Notes

Embedding similarity should be treated as candidate generation rather than definitive proof that two implementations are equivalent.

Potential future verification stages could include:

• normalized token similarity;
• AST similarity;
• exclusion of generated code;
• exclusion of overlapping chunks;
• optional reranking model.

For an initial version, embedding similarity groups with configurable thresholds and JSON output would already be valuable.

Use cases

• finding duplicated business logic implemented under different names;
• identifying refactoring opportunities;
• detecting similar validation and mapping functions;
• finding duplicated code across packages in a monorepo;
• generating CI reports for newly introduced duplication;
• providing coding agents with existing implementations before they create another one.