From aaabcb9db34e6971282e72d23882e6ad14782d26 Mon Sep 17 00:00:00 2001 From: Christopher Tso Date: Tue, 30 Jun 2026 01:50:11 +0200 Subject: [PATCH 1/4] docs: record case conversion boundary decision --- ...11-consolidate-case-conversion-boundary.md | 118 ++++++++++++++++++ 1 file changed, 118 insertions(+) create mode 100644 docs/adr/0011-consolidate-case-conversion-boundary.md diff --git a/docs/adr/0011-consolidate-case-conversion-boundary.md b/docs/adr/0011-consolidate-case-conversion-boundary.md new file mode 100644 index 000000000..4817e5afc --- /dev/null +++ b/docs/adr/0011-consolidate-case-conversion-boundary.md @@ -0,0 +1,118 @@ +# 11. Consolidate case conversion at the artifact boundary + +Date: 2026-06-30 + +## Status + +Accepted + +## Context + +AgentV normalizes many provider and harness shapes into one internal evaluation +model. The internal TypeScript contract is camelCase, while persisted +YAML/JSONL artifacts use snake_case for Python, shell, jq, and JSONL +portability. This split is already documented on the trace model and matches +AgentV's role as a multi-provider abstraction layer. + +The alternative of making TypeScript types snake_case-native fits +single-provider SDKs whose public types mirror one HTTP API. It does not fit +AgentV because AgentV's core model must normalize Claude, Codex, Copilot, Pi, +VS Code, CLI, function, replay, mock, and future providers into a single +provider-neutral shape. + +Before this decision, conversion logic was duplicated in four places: + +- `packages/core/src/evaluation/case-conversion.ts` +- `packages/sdk/src/case-conversion.ts` +- `apps/cli/src/utils/case-conversion.ts` +- an inline `toCamelCaseDeep()` copy in + `packages/core/src/evaluation/run-artifacts.ts` + +Those implementations had already drifted. The inline result-artifact parser +converted only `/_([a-z])/g`, while the canonical core converter handles digits +and guards proper-noun keys that start with uppercase letters. + +Current conversion call sites include: + +- core result parsing and artifact writing in + `packages/core/src/evaluation/run-artifacts.ts` +- core grader and prompt-template stdin payloads in + `packages/core/src/evaluation/graders/code-grader.ts` and + `packages/core/src/evaluation/graders/prompt-resolution.ts` +- SDK stdin boundaries in `packages/sdk/src/runtime.ts`, + `packages/sdk/src/assertion.ts`, and `packages/sdk/src/prompt-template.ts` +- CLI JSON/YAML output and manifest reading in `apps/cli/src/commands/eval`, + `apps/cli/src/commands/inspect`, `apps/cli/src/commands/compare`, + `apps/cli/src/commands/trend`, `apps/cli/src/commands/trim`, and + `apps/cli/src/commands/results` + +## Decision + +Keep camelCase TypeScript internals and snake_case persisted wire artifacts. +Consolidate deep case conversion into +`packages/core/src/evaluation/case-conversion.ts` and export it from +`@agentv/core` for SDK and CLI reuse. Delete local SDK, CLI, and inline +duplicates. + +Use a Zod-backed boundary serializer for AgentV-owned result wire shapes. The +serializer validates canonical camelCase objects, converts to or from snake_case +at the boundary, and uses `.passthrough()` so unknown additive keys are +preserved instead of silently dropped. + +The canonical edge-case semantics are: + +- keys that start with uppercase letters are preserved unchanged, which keeps + proper-noun tool names such as `Read` and `Edit` stable; +- camelCase to snake_case lowers uppercase letters after the first character, + so `topP` becomes `top_p`; +- snake_case to camelCase converts underscore-plus-lowercase or digit, so + `top_p` becomes `topP` and `top_2` becomes `top2`; +- acronym keys that start with uppercase letters, such as `HTTPStatus`, are + treated as proper nouns and preserved. + +Where existing converters disagree, treat the divergence as a latent defect and +converge on the canonical core behavior. Baseline diffs caused by digit +handling or uppercase proper-noun guarding are accepted bug fixes, not +bug-for-bug compatibility requirements. Any baseline diff not explained by this +known drift is a regression and must be fixed before release. + +If artifact or JSONL baselines change because of the accepted convergence, add a +CHANGELOG or migration note describing the casing fix. + +## Consequences + +Positive: + +- SDK, CLI, and core behavior use one tested conversion implementation. +- AgentV keeps the documented multi-provider architecture: provider-neutral + camelCase internals with snake_case wire boundaries. +- Zod validation protects the artifact boundary without dropping unknown + additive keys. +- Future conversion edge cases are covered in one shared test suite. + +Negative: + +- Rows or artifacts that previously passed through the drifted inline converter + can change for keys such as `top_2` and `Foo_bar`. +- SDK and CLI packages now depend on the exported core converter instead of + carrying local copies. + +## Alternatives Considered + +- **Make TypeScript types snake_case-native.** Rejected. That pattern is useful + for single-provider SDKs with generated API types, but AgentV is a + provider-neutral normalization layer. +- **Keep duplicate converters and add tests to each.** Rejected. That preserves + the drift risk and makes future boundary fixes harder to reason about. +- **Preserve the inline converter's bug-for-bug behavior.** Rejected. The drift + is a latent artifact bug, and the canonical behavior already exists in core. +- **Use strict Zod schemas at the wire boundary.** Rejected. Strict parsing + would drop or reject unknown additive keys and create an unintended breaking + change for portable artifacts. + +## Non-Goals + +- Changing grader-author-facing SDK input casing. +- Changing YAML authoring conventions. +- Rewriting provider-specific opaque payloads that intentionally preserve their + native key casing. From f90fd754565fa839e4b56e7c57d9a87b3cbac576 Mon Sep 17 00:00:00 2001 From: Christopher Tso Date: Tue, 30 Jun 2026 01:50:18 +0200 Subject: [PATCH 2/4] refactor(core): centralize artifact case conversion --- .../core/src/evaluation/case-conversion.ts | 92 ++++++++++++ .../src/evaluation/graders/code-grader.ts | 4 +- .../evaluation/graders/prompt-resolution.ts | 4 +- packages/core/src/evaluation/run-artifacts.ts | 24 +-- packages/core/src/index.ts | 18 ++- .../test/evaluation/case-conversion.test.ts | 140 ++++++++++++++++++ 6 files changed, 255 insertions(+), 27 deletions(-) create mode 100644 packages/core/test/evaluation/case-conversion.test.ts diff --git a/packages/core/src/evaluation/case-conversion.ts b/packages/core/src/evaluation/case-conversion.ts index 5cad8d3fa..c2deb09f5 100644 --- a/packages/core/src/evaluation/case-conversion.ts +++ b/packages/core/src/evaluation/case-conversion.ts @@ -1,3 +1,63 @@ +import { z } from 'zod'; +import type { Trace, TraceSummary } from './trace.js'; +import type { EvaluationResult } from './types.js'; + +/** + * Shared case-conversion and boundary serialization helpers. + * + * AgentV internals use camelCase TypeScript objects. Persisted JSON/JSONL and + * process-boundary payloads use snake_case for portability. This module is the + * single conversion implementation used by core, SDK, and CLI code. + */ + +const JsonObjectSchema = z.object({}).passthrough(); +const TokenUsageBoundarySchema = z + .object({ + input: z.number(), + output: z.number(), + cached: z.number().optional(), + reasoning: z.number().optional(), + }) + .passthrough(); + +export const TraceSummaryBoundarySchema = z + .object({ + eventCount: z.number().int().nonnegative(), + toolCalls: z.record(z.string(), z.number()), + errorCount: z.number().int().nonnegative(), + toolDurations: z.record(z.string(), z.array(z.number())).optional(), + llmCallCount: z.number().int().nonnegative().optional(), + }) + .passthrough(); + +export const TraceBoundarySchema = TraceSummaryBoundarySchema.extend({ + messages: z.array(z.unknown()), + events: z.array(z.unknown()), + tokenUsage: TokenUsageBoundarySchema.optional(), + costUsd: z.number().optional(), + durationMs: z.number().optional(), + startTime: z.string().optional(), + endTime: z.string().optional(), + metadata: z.record(z.string(), z.unknown()).optional(), +}).passthrough(); + +export const EvaluationResultBoundarySchema = z + .object({ + timestamp: z.string(), + testId: z.string(), + score: z.number(), + assertions: z.array(z.unknown()), + target: z.string(), + output: z.string(), + trace: TraceBoundarySchema, + executionStatus: z.enum(['ok', 'quality_failure', 'execution_error']).optional(), + }) + .passthrough(); + +export type TraceSummaryWire = Record; +export type TraceWire = Record; +export type EvaluationResultWire = Record; + /** * Converts a camelCase string to snake_case. * Examples: @@ -86,3 +146,35 @@ export function toCamelCaseDeep(obj: unknown): unknown { return obj; } + +export function serializeTraceSummaryWire(summary: TraceSummary): TraceSummaryWire { + return toSnakeCaseDeep(TraceSummaryBoundarySchema.parse(summary)) as TraceSummaryWire; +} + +export function parseTraceSummaryBoundary(value: unknown): TraceSummary { + return TraceSummaryBoundarySchema.parse(value) as TraceSummary; +} + +export function serializeTraceWire(trace: Trace): TraceWire { + return toSnakeCaseDeep(TraceBoundarySchema.parse(trace)) as TraceWire; +} + +export function parseTraceBoundary(value: unknown): Trace { + return TraceBoundarySchema.parse(value) as Trace; +} + +export function serializeEvaluationResultWire(result: EvaluationResult): EvaluationResultWire { + return toSnakeCaseDeep(EvaluationResultBoundarySchema.parse(result)) as EvaluationResultWire; +} + +export function parseEvaluationResultBoundary(value: unknown): EvaluationResult { + return EvaluationResultBoundarySchema.parse(value) as EvaluationResult; +} + +/** + * Serialize a generic object-shaped process-boundary payload. Use focused + * serializers above when the payload is an AgentV-owned result or trace model. + */ +export function serializeSnakeCaseBoundaryPayload(value: unknown): unknown { + return toSnakeCaseDeep(JsonObjectSchema.parse(value)); +} diff --git a/packages/core/src/evaluation/graders/code-grader.ts b/packages/core/src/evaluation/graders/code-grader.ts index 330830c86..b5a6c6e06 100644 --- a/packages/core/src/evaluation/graders/code-grader.ts +++ b/packages/core/src/evaluation/graders/code-grader.ts @@ -8,7 +8,7 @@ import { type TargetProxyUsageMetadata, createTargetProxy, } from '../../runtime/target-proxy.js'; -import { toSnakeCaseDeep } from '../case-conversion.js'; +import { serializeSnakeCaseBoundaryPayload } from '../case-conversion.js'; import { type ContentImage, isContentArray } from '../content.js'; import type { AssertionEntry, JsonObject, TargetAccessConfig } from '../types.js'; import { getRepoCheckoutTargets } from '../workspace/repo-checkout.js'; @@ -199,7 +199,7 @@ export class CodeGrader implements Grader { config: this.config ?? null, }; - const inputPayload = JSON.stringify(toSnakeCaseDeep(payload), null, 2); + const inputPayload = JSON.stringify(serializeSnakeCaseBoundaryPayload(payload), null, 2); // Set up target proxy if configured and grader provider is available let proxyEnv: Record | undefined; diff --git a/packages/core/src/evaluation/graders/prompt-resolution.ts b/packages/core/src/evaluation/graders/prompt-resolution.ts index 2306c4b56..d7dfd9173 100644 --- a/packages/core/src/evaluation/graders/prompt-resolution.ts +++ b/packages/core/src/evaluation/graders/prompt-resolution.ts @@ -13,7 +13,7 @@ import path from 'node:path'; -import { toSnakeCaseDeep } from '../case-conversion.js'; +import { serializeSnakeCaseBoundaryPayload } from '../case-conversion.js'; import { readTextFile } from '../file-utils.js'; import type { Message } from '../providers/types.js'; import { VALID_TEMPLATE_VARIABLES } from '../template-variables.js'; @@ -114,7 +114,7 @@ async function executePromptTemplate( config: config ?? context.config ?? null, }; - const inputJson = JSON.stringify(toSnakeCaseDeep(payload), null, 2); + const inputJson = JSON.stringify(serializeSnakeCaseBoundaryPayload(payload), null, 2); const scriptPath = script[script.length - 1]; const cwd = path.dirname(scriptPath); diff --git a/packages/core/src/evaluation/run-artifacts.ts b/packages/core/src/evaluation/run-artifacts.ts index 5bd0609bc..f0e8dfd46 100644 --- a/packages/core/src/evaluation/run-artifacts.ts +++ b/packages/core/src/evaluation/run-artifacts.ts @@ -16,6 +16,7 @@ import { traceEnvelopeToNormalizedTranscriptJsonLines, traceEnvelopeToTranscriptJsonLines, } from '../import/types.js'; +import { parseEvaluationResultBoundary, toCamelCaseDeep } from './case-conversion.js'; import type { ExperimentArtifactMetadata } from './experiment.js'; import { type ExternalTraceMetadataWire, @@ -2012,27 +2013,6 @@ async function rewriteExistingIndexRecords( await writeJsonlFile(indexPath, records); } -function toCamelCase(str: string): string { - return str.replace(/_([a-z])/g, (_, letter) => letter.toUpperCase()); -} - -function toCamelCaseDeep(obj: unknown): unknown { - if (obj === null || obj === undefined) { - return obj; - } - if (Array.isArray(obj)) { - return obj.map((item) => toCamelCaseDeep(item)); - } - if (typeof obj === 'object') { - const result: Record = {}; - for (const [key, value] of Object.entries(obj)) { - result[toCamelCase(key)] = toCamelCaseDeep(value); - } - return result; - } - return obj; -} - type ParsedEvaluationResult = Record & { timestamp: string; testId: string; @@ -2151,7 +2131,7 @@ export function parseJsonlResults(content: string): EvaluationResult[] { const camelCased = toCamelCaseDeep(canonicalRow); const normalized = normalizeParsedResult(camelCased); if (normalized) { - results.push(normalized); + results.push(parseEvaluationResultBoundary(normalized)); } } return results; diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts index 8c58dca1e..ada1cbb8e 100644 --- a/packages/core/src/index.ts +++ b/packages/core/src/index.ts @@ -107,7 +107,23 @@ export { shouldEnableCache, shouldSkipCacheForTemperature, } from './evaluation/cache/response-cache.js'; -export { toSnakeCaseDeep, toCamelCaseDeep } from './evaluation/case-conversion.js'; +export { + EvaluationResultBoundarySchema, + TraceBoundarySchema, + TraceSummaryBoundarySchema, + parseEvaluationResultBoundary, + parseTraceBoundary, + parseTraceSummaryBoundary, + serializeEvaluationResultWire, + serializeSnakeCaseBoundaryPayload, + serializeTraceSummaryWire, + serializeTraceWire, + toCamelCaseDeep, + toSnakeCaseDeep, + type EvaluationResultWire, + type TraceSummaryWire, + type TraceWire, +} from './evaluation/case-conversion.js'; export { ensureResultsRepoClone, syncResultsRepo, diff --git a/packages/core/test/evaluation/case-conversion.test.ts b/packages/core/test/evaluation/case-conversion.test.ts new file mode 100644 index 000000000..479f394c1 --- /dev/null +++ b/packages/core/test/evaluation/case-conversion.test.ts @@ -0,0 +1,140 @@ +import { describe, expect, it } from 'bun:test'; + +import { + type EvaluationResult, + parseEvaluationResultBoundary, + serializeEvaluationResultWire, + toCamelCaseDeep, + toSnakeCaseDeep, +} from '../../src/index.js'; + +describe('case conversion', () => { + it('converts camelCase keys to snake_case recursively', () => { + expect( + toSnakeCaseDeep({ + testId: 'test-001', + outputText: 'hello world', + conversationId: 'conv-123', + trace: { + toolCalls: 5, + totalSteps: 10, + }, + scores: [ + { evalName: 'test1', hitCount: 3 }, + { evalName: 'test2', hitCount: 5 }, + ], + }), + ).toEqual({ + test_id: 'test-001', + output_text: 'hello world', + conversation_id: 'conv-123', + trace: { + tool_calls: 5, + total_steps: 10, + }, + scores: [ + { eval_name: 'test1', hit_count: 3 }, + { eval_name: 'test2', hit_count: 5 }, + ], + }); + }); + + it('preserves primitive values and arrays of primitives', () => { + expect(toSnakeCaseDeep('hello')).toBe('hello'); + expect(toSnakeCaseDeep(42)).toBe(42); + expect(toSnakeCaseDeep(true)).toBe(true); + expect(toSnakeCaseDeep(null)).toBe(null); + expect(toSnakeCaseDeep(undefined)).toBe(undefined); + expect(toSnakeCaseDeep([1, 2, 3, 'test'])).toEqual([1, 2, 3, 'test']); + }); + + it('keeps acronym and proper-noun keys while converting their nested payloads', () => { + expect( + toSnakeCaseDeep({ + HTTPStatus: 200, + Read: { filePath: 'src/index.ts' }, + Edit: { targetFile: 'src/index.ts' }, + topP: 0.8, + }), + ).toEqual({ + HTTPStatus: 200, + Read: { file_path: 'src/index.ts' }, + Edit: { target_file: 'src/index.ts' }, + top_p: 0.8, + }); + }); + + it('converts snake_case keys to camelCase with digit boundaries', () => { + expect( + toCamelCaseDeep({ + HTTPStatus: 200, + Read: { file_path: 'src/index.ts' }, + Edit: { target_file: 'src/index.ts' }, + top_p: 0.8, + top_2: true, + }), + ).toEqual({ + HTTPStatus: 200, + Read: { filePath: 'src/index.ts' }, + Edit: { targetFile: 'src/index.ts' }, + topP: 0.8, + top2: true, + }); + }); +}); + +describe('evaluation result boundary serializer', () => { + const result = { + timestamp: '2026-06-30T00:00:00.000Z', + testId: 'case-1', + score: 1, + assertions: [], + target: 'mock', + output: 'done', + trace: { + eventCount: 0, + toolCalls: {}, + errorCount: 0, + messages: [], + events: [], + extraTraceField: { topP: 0.5 }, + }, + executionStatus: 'ok', + experimentalField: { + topP: 0.8, + top2: true, + Read: { filePath: 'src/index.ts' }, + }, + } as EvaluationResult & { + readonly experimentalField: Record; + }; + + it('validates camelCase internals and serializes snake_case wire fields', () => { + const wire = serializeEvaluationResultWire(result); + + expect(wire).toMatchObject({ + test_id: 'case-1', + execution_status: 'ok', + trace: { + event_count: 0, + tool_calls: {}, + error_count: 0, + extra_trace_field: { top_p: 0.5 }, + }, + experimental_field: { + top_p: 0.8, + top2: true, + Read: { file_path: 'src/index.ts' }, + }, + }); + }); + + it('keeps unknown camelCase keys when parsing boundary-normalized internals', () => { + const parsed = parseEvaluationResultBoundary(result) as EvaluationResult & { + readonly experimentalField: Record; + }; + + expect(parsed.experimentalField).toEqual(result.experimentalField); + expect((parsed.trace as Record).extraTraceField).toEqual({ topP: 0.5 }); + }); +}); From 9af025bfe35434a74a2f9af830c50bb282b849af Mon Sep 17 00:00:00 2001 From: Christopher Tso Date: Tue, 30 Jun 2026 01:50:26 +0200 Subject: [PATCH 3/4] refactor: reuse core case conversion in sdk and cli --- apps/cli/src/commands/compare/index.ts | 2 +- apps/cli/src/commands/eval/jsonl-writer.ts | 8 +- apps/cli/src/commands/eval/task-bundle.ts | 3 +- apps/cli/src/commands/inspect/list.ts | 2 +- apps/cli/src/commands/inspect/search.ts | 2 +- apps/cli/src/commands/inspect/stats.ts | 2 +- apps/cli/src/commands/trend/index.ts | 2 +- apps/cli/src/utils/case-conversion.ts | 52 ------- apps/cli/test/commands/eval/aggregate.test.ts | 2 +- .../commands/eval/artifact-writer.test.ts | 2 +- apps/cli/test/unit/case-conversion.test.ts | 140 ------------------ packages/sdk/src/assertion.ts | 2 +- packages/sdk/src/case-conversion.ts | 40 ----- packages/sdk/src/prompt-template.ts | 2 +- packages/sdk/src/runtime.ts | 2 +- 15 files changed, 14 insertions(+), 249 deletions(-) delete mode 100644 apps/cli/src/utils/case-conversion.ts delete mode 100644 apps/cli/test/unit/case-conversion.test.ts delete mode 100644 packages/sdk/src/case-conversion.ts diff --git a/apps/cli/src/commands/compare/index.ts b/apps/cli/src/commands/compare/index.ts index 20f1eba31..f857920a7 100644 --- a/apps/cli/src/commands/compare/index.ts +++ b/apps/cli/src/commands/compare/index.ts @@ -11,7 +11,7 @@ import { string, } from 'cmd-ts'; -import { toSnakeCaseDeep } from '../../utils/case-conversion.js'; +import { toSnakeCaseDeep } from '@agentv/core'; import { loadLightweightResults, resolveResultSourcePath } from '../results/manifest.js'; // ANSI color codes (no dependency needed) diff --git a/apps/cli/src/commands/eval/jsonl-writer.ts b/apps/cli/src/commands/eval/jsonl-writer.ts index 827bb9648..003077351 100644 --- a/apps/cli/src/commands/eval/jsonl-writer.ts +++ b/apps/cli/src/commands/eval/jsonl-writer.ts @@ -2,10 +2,9 @@ import { createWriteStream } from 'node:fs'; import { mkdir } from 'node:fs/promises'; import path from 'node:path'; import { finished } from 'node:stream/promises'; +import { type EvaluationResult, serializeEvaluationResultWire } from '@agentv/core'; import { Mutex } from 'async-mutex'; -import { toSnakeCaseDeep } from '../../utils/case-conversion.js'; - export class JsonlWriter { private readonly stream: ReturnType; private readonly mutex = new Mutex(); @@ -22,13 +21,12 @@ export class JsonlWriter { return new JsonlWriter(stream); } - async append(record: unknown): Promise { + async append(record: EvaluationResult): Promise { await this.mutex.runExclusive(async () => { if (this.closed) { throw new Error('Cannot write to closed JSONL writer'); } - // Convert camelCase keys to snake_case for Python ecosystem compatibility - const snakeCaseRecord = toSnakeCaseDeep(record); + const snakeCaseRecord = serializeEvaluationResultWire(record); const line = `${JSON.stringify(snakeCaseRecord)}\n`; if (!this.stream.write(line)) { await new Promise((resolve, reject) => { diff --git a/apps/cli/src/commands/eval/task-bundle.ts b/apps/cli/src/commands/eval/task-bundle.ts index f6e263d2a..526a2c313 100644 --- a/apps/cli/src/commands/eval/task-bundle.ts +++ b/apps/cli/src/commands/eval/task-bundle.ts @@ -10,11 +10,10 @@ import { type TestMessage, type WorkspaceConfig, parseYamlValue, + toSnakeCaseDeep, } from '@agentv/core'; import { stringify as stringifyYaml } from 'yaml'; -import { toSnakeCaseDeep } from '../../utils/case-conversion.js'; - const TEST_BUNDLE_DIRNAME = 'test'; const TASK_EVAL_FILENAME = 'EVAL.yaml'; const TASK_TARGETS_FILENAME = 'targets.yaml'; diff --git a/apps/cli/src/commands/inspect/list.ts b/apps/cli/src/commands/inspect/list.ts index 719c08c8c..f48010406 100644 --- a/apps/cli/src/commands/inspect/list.ts +++ b/apps/cli/src/commands/inspect/list.ts @@ -1,5 +1,5 @@ +import { toSnakeCaseDeep } from '@agentv/core'; import { command, number, oneOf, option, optional, string } from 'cmd-ts'; -import { toSnakeCaseDeep } from '../../utils/case-conversion.js'; import { type ResultFileMeta, c, diff --git a/apps/cli/src/commands/inspect/search.ts b/apps/cli/src/commands/inspect/search.ts index 76b9c9286..79012f3ab 100644 --- a/apps/cli/src/commands/inspect/search.ts +++ b/apps/cli/src/commands/inspect/search.ts @@ -15,8 +15,8 @@ import { existsSync, readFileSync, readdirSync, statSync } from 'node:fs'; import path from 'node:path'; +import { toSnakeCaseDeep } from '@agentv/core'; import { command, oneOf, option, optional, positional, string } from 'cmd-ts'; -import { toSnakeCaseDeep } from '../../utils/case-conversion.js'; import { isReservedResultsNamespace } from '../eval/result-layout.js'; import { c, padRight } from './utils.js'; diff --git a/apps/cli/src/commands/inspect/stats.ts b/apps/cli/src/commands/inspect/stats.ts index cf3df312c..e494f596b 100644 --- a/apps/cli/src/commands/inspect/stats.ts +++ b/apps/cli/src/commands/inspect/stats.ts @@ -1,5 +1,5 @@ +import { toSnakeCaseDeep } from '@agentv/core'; import { command, oneOf, option, optional, positional, string } from 'cmd-ts'; -import { toSnakeCaseDeep } from '../../utils/case-conversion.js'; import { type RawResult, c, diff --git a/apps/cli/src/commands/trend/index.ts b/apps/cli/src/commands/trend/index.ts index 01ce70792..7573c2765 100644 --- a/apps/cli/src/commands/trend/index.ts +++ b/apps/cli/src/commands/trend/index.ts @@ -1,8 +1,8 @@ import path from 'node:path'; +import { toSnakeCaseDeep } from '@agentv/core'; import { command, flag, number, oneOf, option, optional, restPositionals, string } from 'cmd-ts'; -import { toSnakeCaseDeep } from '../../utils/case-conversion.js'; import { RESULT_INDEX_FILENAME, isRunManifestPath } from '../eval/result-layout.js'; import { listResultFiles } from '../inspect/utils.js'; import { diff --git a/apps/cli/src/utils/case-conversion.ts b/apps/cli/src/utils/case-conversion.ts deleted file mode 100644 index 52cd5aaf3..000000000 --- a/apps/cli/src/utils/case-conversion.ts +++ /dev/null @@ -1,52 +0,0 @@ -/** - * Converts a camelCase string to snake_case. - * Examples: - * testId -> test_id - * outputText -> output_text - * conversationId -> conversation_id - * - * Note: Keys that start with an uppercase letter are treated as proper nouns - * and returned unchanged (e.g., "Read", "Edit" for tool names). - */ -function toSnakeCase(str: string): string { - // Don't convert keys that start with uppercase (proper nouns/tool names) - if (/^[A-Z]/.test(str)) { - return str; - } - return str.replace(/[A-Z]/g, (letter) => `_${letter.toLowerCase()}`); -} - -/** - * Recursively converts all keys in an object from camelCase to snake_case. - * This is used to convert TypeScript internal representations to snake_case - * for Python ecosystem compatibility in JSONL output files. - * - * Conversion rules: - * - Object keys: camelCase -> snake_case - * - Array elements: recursively converted - * - Primitives: returned unchanged - * - null/undefined: returned unchanged - * - * @param obj - The object to convert (can be any JSON-serializable value) - * @returns A new object with all keys converted to snake_case - */ -export function toSnakeCaseDeep(obj: unknown): unknown { - if (obj === null || obj === undefined) { - return obj; - } - - if (Array.isArray(obj)) { - return obj.map((item) => toSnakeCaseDeep(item)); - } - - if (typeof obj === 'object') { - const result: Record = {}; - for (const [key, value] of Object.entries(obj)) { - const snakeKey = toSnakeCase(key); - result[snakeKey] = toSnakeCaseDeep(value); - } - return result; - } - - return obj; -} diff --git a/apps/cli/test/commands/eval/aggregate.test.ts b/apps/cli/test/commands/eval/aggregate.test.ts index 165e43980..b6a7763a9 100644 --- a/apps/cli/test/commands/eval/aggregate.test.ts +++ b/apps/cli/test/commands/eval/aggregate.test.ts @@ -12,7 +12,7 @@ import { tmpdir } from 'node:os'; import path from 'node:path'; import { type EvaluationResult, buildTraceFromMessages } from '@agentv/core'; -import { toSnakeCaseDeep } from '../../../src/utils/case-conversion.js'; +import { toSnakeCaseDeep } from '@agentv/core'; import { RESULT_INDEX_FILENAME, diff --git a/apps/cli/test/commands/eval/artifact-writer.test.ts b/apps/cli/test/commands/eval/artifact-writer.test.ts index 2bc53aa26..de42a1f67 100644 --- a/apps/cli/test/commands/eval/artifact-writer.test.ts +++ b/apps/cli/test/commands/eval/artifact-writer.test.ts @@ -14,6 +14,7 @@ import { buildResultIndexArtifact, buildTraceFromMessages, parseYamlValue, + toSnakeCaseDeep, } from '@agentv/core'; import { @@ -33,7 +34,6 @@ import { writeArtifactsFromResults, } from '../../../src/commands/eval/artifact-writer.js'; import { prepareResultForJsonl } from '../../../src/commands/eval/run-eval.js'; -import { toSnakeCaseDeep } from '../../../src/utils/case-conversion.js'; function makeResult(overrides: Partial = {}): EvaluationResult { const result = { diff --git a/apps/cli/test/unit/case-conversion.test.ts b/apps/cli/test/unit/case-conversion.test.ts deleted file mode 100644 index 3749563d0..000000000 --- a/apps/cli/test/unit/case-conversion.test.ts +++ /dev/null @@ -1,140 +0,0 @@ -import { describe, expect, test } from 'vitest'; - -import { toSnakeCaseDeep } from '../../src/utils/case-conversion.js'; - -describe('toSnakeCaseDeep', () => { - test('converts simple camelCase keys to snake_case', () => { - const input = { - testId: 'test-001', - outputText: 'hello world', - conversationId: 'conv-123', - }; - - const result = toSnakeCaseDeep(input); - - expect(result).toEqual({ - test_id: 'test-001', - output_text: 'hello world', - conversation_id: 'conv-123', - }); - }); - - test('converts nested objects recursively', () => { - const input = { - testId: 'test-001', - trace: { - toolCalls: 5, - totalSteps: 10, - }, - }; - - const result = toSnakeCaseDeep(input); - - expect(result).toEqual({ - test_id: 'test-001', - trace: { - tool_calls: 5, - total_steps: 10, - }, - }); - }); - - test('converts arrays of objects', () => { - const input = { - scores: [ - { evalName: 'test1', hitCount: 3 }, - { evalName: 'test2', hitCount: 5 }, - ], - }; - - const result = toSnakeCaseDeep(input); - - expect(result).toEqual({ - scores: [ - { eval_name: 'test1', hit_count: 3 }, - { eval_name: 'test2', hit_count: 5 }, - ], - }); - }); - - test('preserves primitive values', () => { - expect(toSnakeCaseDeep('hello')).toBe('hello'); - expect(toSnakeCaseDeep(42)).toBe(42); - expect(toSnakeCaseDeep(true)).toBe(true); - expect(toSnakeCaseDeep(null)).toBe(null); - expect(toSnakeCaseDeep(undefined)).toBe(undefined); - }); - - test('handles arrays of primitives', () => { - const input = [1, 2, 3, 'test']; - expect(toSnakeCaseDeep(input)).toEqual([1, 2, 3, 'test']); - }); - - test('handles complex nested structures', () => { - const input = { - timestamp: '2025-01-02T00:00:00Z', - testId: 'test-001', - requests: { - agent: { - modelName: 'gpt-4', - maxTokens: 1000, - }, - }, - scores: [ - { - evaluatorName: 'code-grader', - rawRequest: { - outputText: 'code', - expectedOutcome: 'correct', - }, - }, - ], - assertions: [ - { text: 'check1', passed: true }, - { text: 'check2', passed: true }, - ], - score: 0.85, - }; - - const result = toSnakeCaseDeep(input); - - expect(result).toEqual({ - timestamp: '2025-01-02T00:00:00Z', - test_id: 'test-001', - requests: { - agent: { - model_name: 'gpt-4', - max_tokens: 1000, - }, - }, - scores: [ - { - evaluator_name: 'code-grader', - raw_request: { - output_text: 'code', - expected_outcome: 'correct', - }, - }, - ], - assertions: [ - { text: 'check1', passed: true }, - { text: 'check2', passed: true }, - ], - score: 0.85, - }); - }); - - test('handles keys that are already snake_case', () => { - const input = { - test_id: 'test-001', - answer: 'hello', - }; - - const result = toSnakeCaseDeep(input); - - expect(result).toEqual({ - test_id: 'test-001', - answer: 'hello', - }); - }); -}); diff --git a/packages/sdk/src/assertion.ts b/packages/sdk/src/assertion.ts index 1d654f329..3e326a103 100644 --- a/packages/sdk/src/assertion.ts +++ b/packages/sdk/src/assertion.ts @@ -6,8 +6,8 @@ * contract focused on pass/fail with optional score granularity. */ import { readFileSync } from 'node:fs'; +import { toCamelCaseDeep } from '@agentv/core'; -import { toCamelCaseDeep } from './case-conversion.js'; import { enrichInput } from './deprecation.js'; import { type CodeGraderInput, diff --git a/packages/sdk/src/case-conversion.ts b/packages/sdk/src/case-conversion.ts deleted file mode 100644 index 2fdaf188d..000000000 --- a/packages/sdk/src/case-conversion.ts +++ /dev/null @@ -1,40 +0,0 @@ -/** - * Case conversion utilities for JSON payloads. - * Converts between snake_case (wire format) and camelCase (TypeScript). - */ - -function toCamelCase(str: string): string { - // Don't convert keys that start with uppercase (proper nouns/tool names) - if (/^[A-Z]/.test(str)) { - return str; - } - return str.replace(/_([a-z0-9])/g, (_, letter) => letter.toUpperCase()); -} - -/** - * Recursively converts all keys in an object from snake_case to camelCase. - * Used to map wire payloads into TypeScript-friendly shapes. - * - * @param obj - The object to convert (can be any JSON-serializable value) - * @returns A new object with all keys converted to camelCase - */ -export function toCamelCaseDeep(obj: unknown): unknown { - if (obj === null || obj === undefined) { - return obj; - } - - if (Array.isArray(obj)) { - return obj.map((item) => toCamelCaseDeep(item)); - } - - if (typeof obj === 'object') { - const result: Record = {}; - for (const [key, value] of Object.entries(obj)) { - const camelKey = toCamelCase(key); - result[camelKey] = toCamelCaseDeep(value); - } - return result; - } - - return obj; -} diff --git a/packages/sdk/src/prompt-template.ts b/packages/sdk/src/prompt-template.ts index 841240bb4..322edc325 100644 --- a/packages/sdk/src/prompt-template.ts +++ b/packages/sdk/src/prompt-template.ts @@ -3,8 +3,8 @@ * Handles stdin parsing, validation, error handling, and string output. */ import { readFileSync } from 'node:fs'; +import { toCamelCaseDeep } from '@agentv/core'; -import { toCamelCaseDeep } from './case-conversion.js'; import { enrichInput } from './deprecation.js'; import { type CodeGraderInput, PromptTemplateInputSchema } from './schemas.js'; diff --git a/packages/sdk/src/runtime.ts b/packages/sdk/src/runtime.ts index 4a404008f..9d9e011c5 100644 --- a/packages/sdk/src/runtime.ts +++ b/packages/sdk/src/runtime.ts @@ -3,8 +3,8 @@ * Handles stdin parsing, validation, error handling, and output formatting. */ import { readFileSync } from 'node:fs'; +import { toCamelCaseDeep } from '@agentv/core'; -import { toCamelCaseDeep } from './case-conversion.js'; import { enrichInput } from './deprecation.js'; import { type CodeGraderInput, From f80949e40255f83bf16bf4b6aafd60d0a42fd2a5 Mon Sep 17 00:00:00 2001 From: Christopher Tso Date: Tue, 30 Jun 2026 03:19:31 +0200 Subject: [PATCH 4/4] docs(adr): record camelcase-keys rejection and explicit-mapping follow-up Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- ...11-consolidate-case-conversion-boundary.md | 28 +++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/docs/adr/0011-consolidate-case-conversion-boundary.md b/docs/adr/0011-consolidate-case-conversion-boundary.md index 4817e5afc..aa97162a1 100644 --- a/docs/adr/0011-consolidate-case-conversion-boundary.md +++ b/docs/adr/0011-consolidate-case-conversion-boundary.md @@ -109,6 +109,34 @@ Negative: - **Use strict Zod schemas at the wire boundary.** Rejected. Strict parsing would drop or reject unknown additive keys and create an unintended breaking change for portable artifacts. +- **Adopt a third-party case library (`camelcase-keys` / `snakecase-keys`).** + Rejected. (1) `camelcase-keys` is one-directional, so the snake_case wire + direction would require a second library (`snakecase-keys`, built on + `change-case`) with a different edge-case engine. (2) Those libraries have + different acronym/number semantics (e.g. `preserveConsecutiveUppercase`, + `top2` vs `top_2`), which would maximize baseline diffs rather than minimize + them — the opposite of the byte-compatibility goal. (3) They lack AgentV's + proper-noun guard, so tool names such as `Read`/`Edit` would be transformed; + reconstructing the guard via `exclude: [/^[A-Z]/]` on both libraries is + fragile. (4) `snakecase-keys ∘ camelcase-keys` is not provably identity for + AgentV keys, while the in-house pair is. (5) AgentV keys are its own finite + schema, not arbitrary user input, so the library's main value (robust handling + of arbitrary keys) does not apply. Revisit only if AgentV must ingest + arbitrary external/user kebab+snake keys or drops byte-compatibility. An + optional test may cross-check the in-house converter against `camelcase-keys` + on the known key set to document divergences, without a runtime dependency. + +## Future Work + +The strictly-better long-term boundary is an explicit per-field casing mapping +encoded in the Zod schemas (no deep stringly-typed key walk), which also mirrors +how the Vercel AI SDK does explicit per-provider mapping. That is a larger, +separate effort and is intentionally out of scope for this consolidation. Note +that kebab-case is the same conversion problem as snake_case (only the separator +differs: `[-_]`), but the camelCase-to-kebab reverse is lossy/ambiguous and +would need an explicit per-field rule rather than a blanket reverse — another +reason to prefer explicit schema-level mapping if a kebab dialect is ever +required. ## Non-Goals