Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,8 @@ jobs:
node --check tools/agent-radio.mjs
node tools/agent-radio.mjs list-threads
node tools/agent-radio.mjs protocol
- name: E2E workflow (init→write→validate→search→handoff)
run: node tools/e2e-workflow.mjs
- name: Skill check
run: |
test -f .claude/skills/agent-shared-context/SKILL.md
Expand Down
61 changes: 30 additions & 31 deletions BENCHMARK.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
<!-- Path: BENCHMARK.md -->
# Benchmark — Hierarchical Lightweight Search vs Full Read

> **Objective, public-standard-like, critical, reproducible** — synthetic 5/50/500 scale, 20 queries, **fixed seed (--seed 42)**, tokens = chars/4, hit = query tokens in title/tags/summary, latency = search vs est. full Read, no LLM.
>
> **Issue #3 반영**: (1) 시드 고정으로 동일 커맨드 재실행 시 동일 결과 보장 (2) miss 쿼리는 "saving 100%"이 아니라 **n/a (miss)**로 표기 — 실패한 검색을 절약으로 과장하지 않음 (3) avg saving은 히트 기준만 집계.
> **Objective, public-standard-like, critical, reproducible** — synthetic 5/50/500 scale, 20 queries, tokens = chars/4, hit = query tokens in title/tags/summary, latency = search vs est. full Read, no LLM.

## Method (close to public standard)

- **Dataset**: Synthetic 5 + 50 + 500 entries, distribution 40% post-it (15tok) 30% memo (50tok) 15% diary (200tok) 10% bookshelf (1000tok) 5% library (5000tok) — like cache workloads, not cherry-picked.
Expand All @@ -17,9 +16,9 @@

| scale | full tokens | avg top 3 tokens | avg saving | hitRate | avg latency (search) | est. full Read latency | tokens/hit |
|---|---|---|---|---|---|---|
| 5 | 1315 | 178 | 83.1% | 80.0% | 0.09ms | 0.25ms (est. Read all md) | 223 |
| 50 | 16780 | 761 | 94.7% | 85.0% | 0.37ms | 2.50ms (est. Read all md) | 895 |
| 500 | 197940 | 1883 | 98.9% | 85.0% | 1.92ms | 25.00ms (est. Read all md) | 2216 |
| 5 | 1315 | 178 | 83.1% | 80.0% | 0.16ms | 0.25ms (est. Read all md) | 223 |
| 50 | 16780 | 761 | 94.7% | 85.0% | 0.41ms | 2.50ms (est. Read all md) | 895 |
| 500 | 197940 | 1883 | 98.9% | 85.0% | 2.34ms | 25.00ms (est. Read all md) | 2216 |

### Interpretation (critical, not hype)

Expand All @@ -31,11 +30,11 @@

| query | assignedLevel | top tokens | saving | hit | latency |
|---|---|---|---|---|
| auth | post-it | 45 | 99.7% | ✅ | 1.31ms |
| api | post-it | 80 | 99.5% | ✅ | 0.36ms |
| jwt | post-it | 0 | n/a (miss) | ❌ | 0.31ms |
| pagination | post-it | 0 | n/a (miss) | ❌ | 0.31ms |
| cache | post-it | 0 | n/a (miss) | ❌ | 0.31ms |
| auth | post-it | 45 | 99.7% | ✅ | 0.39ms |
| api | post-it | 80 | 99.5% | ✅ | 0.33ms |
| jwt | post-it | 0 | n/a (miss) | ❌ | 0.89ms |
| pagination | post-it | 0 | n/a (miss) | ❌ | 1.56ms |
| cache | post-it | 0 | n/a (miss) | ❌ | 0.64ms |

### What we learned while benchmarking (ideas & shortcomings →补)

Expand Down Expand Up @@ -66,7 +65,7 @@ No API key, no `npm install`, Node ≥18 only — like `agent-search-lite.mjs`.
"avgTopTokens": 178,
"avgSaving": "83.1%",
"hitRate": "80.0%",
"avgLatency": "0.09ms",
"avgLatency": "0.16ms",
"fullLatencyEst": "0.25ms (est. Read all md)",
"tokensPerHit": 223,
"perQuery": [
Expand All @@ -76,39 +75,39 @@ No API key, no `npm install`, Node ≥18 only — like `agent-search-lite.mjs`.
"topTokens": 50,
"saving": "96.2%",
"hit": true,
"latency": "0.48ms"
"latency": "0.65ms"
},
{
"query": "api",
"assignedLevel": "post-it",
"topTokens": 15,
"saving": "98.9%",
"hit": true,
"latency": "0.22ms"
"latency": "0.25ms"
},
{
"query": "jwt",
"assignedLevel": "post-it",
"topTokens": 0,
"saving": "n/a (miss)",
"hit": false,
"latency": "0.07ms"
"latency": "0.16ms"
},
{
"query": "pagination",
"assignedLevel": "post-it",
"topTokens": 0,
"saving": "n/a (miss)",
"hit": false,
"latency": "0.05ms"
"latency": "0.13ms"
},
{
"query": "cache",
"assignedLevel": "post-it",
"topTokens": 0,
"saving": "n/a (miss)",
"hit": false,
"latency": "0.04ms"
"latency": "0.35ms"
}
]
},
Expand All @@ -120,7 +119,7 @@ No API key, no `npm install`, Node ≥18 only — like `agent-search-lite.mjs`.
"avgTopTokens": 761,
"avgSaving": "94.7%",
"hitRate": "85.0%",
"avgLatency": "0.37ms",
"avgLatency": "0.41ms",
"fullLatencyEst": "2.50ms (est. Read all md)",
"tokensPerHit": 895,
"perQuery": [
Expand All @@ -130,39 +129,39 @@ No API key, no `npm install`, Node ≥18 only — like `agent-search-lite.mjs`.
"topTokens": 45,
"saving": "99.7%",
"hit": true,
"latency": "1.31ms"
"latency": "0.39ms"
},
{
"query": "api",
"assignedLevel": "post-it",
"topTokens": 80,
"saving": "99.5%",
"hit": true,
"latency": "0.36ms"
"latency": "0.33ms"
},
{
"query": "jwt",
"assignedLevel": "post-it",
"topTokens": 0,
"saving": "n/a (miss)",
"hit": false,
"latency": "0.31ms"
"latency": "0.89ms"
},
{
"query": "pagination",
"assignedLevel": "post-it",
"topTokens": 0,
"saving": "n/a (miss)",
"hit": false,
"latency": "0.31ms"
"latency": "1.56ms"
},
{
"query": "cache",
"assignedLevel": "post-it",
"topTokens": 0,
"saving": "n/a (miss)",
"hit": false,
"latency": "0.31ms"
"latency": "0.64ms"
}
]
},
Expand All @@ -174,49 +173,49 @@ No API key, no `npm install`, Node ≥18 only — like `agent-search-lite.mjs`.
"avgTopTokens": 1883,
"avgSaving": "98.9%",
"hitRate": "85.0%",
"avgLatency": "1.92ms",
"avgLatency": "2.34ms",
"fullLatencyEst": "25.00ms (est. Read all md)",
"tokensPerHit": 2216,
"perQuery": [
{
"query": "auth",
"assignedLevel": "post-it",
"topTokens": 45,
"saving": "100.0%",
"saving": "99.9%+",
"hit": true,
"latency": "4.01ms"
"latency": "3.82ms"
},
{
"query": "api",
"assignedLevel": "post-it",
"topTokens": 45,
"saving": "100.0%",
"saving": "99.9%+",
"hit": true,
"latency": "2.09ms"
"latency": "1.83ms"
},
{
"query": "jwt",
"assignedLevel": "post-it",
"topTokens": 0,
"saving": "n/a (miss)",
"hit": false,
"latency": "2.05ms"
"latency": "6.05ms"
},
{
"query": "pagination",
"assignedLevel": "post-it",
"topTokens": 0,
"saving": "n/a (miss)",
"hit": false,
"latency": "2.48ms"
"latency": "4.64ms"
},
{
"query": "cache",
"assignedLevel": "post-it",
"topTokens": 0,
"saving": "n/a (miss)",
"hit": false,
"latency": "1.93ms"
"latency": "4.53ms"
}
]
}
Expand Down
21 changes: 21 additions & 0 deletions agent-context/bugs/2026-08-26-login-500-on-refresh--system.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
<!-- Path: agent-context/bugs/2026-08-26-login-500-on-refresh--system.md -->
---
id: issue-20260826-b48a3b55
type: issue
title: "login 500 on refresh"
tags: [issue]
feature: auth
level: ""
scope: global
agent: system
created: 2026-08-26T11:23:08.632Z
updated: 2026-08-26T11:23:08.632Z
status: done
priority: 4
summary: "login 500 on refresh"
---

## 결과

(도구 호출 로그 아님 — 결론만 기록. 검증은 refs 링크로)

2 changes: 1 addition & 1 deletion agent-context/graph.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"version": 1,
"generated_at": "2026-08-26T09:51:47.670Z",
"generated_at": "2026-08-26T11:25:17.405Z",
"_path": "agent-context/graph.json",
"description": "기능 연관 그래프 — depends_on/affects로 영향 범위 추적. agent-context.config.json graph.edges로부터 생성됨.",
"graph": {
Expand Down
31 changes: 27 additions & 4 deletions agent-context/index.json
Original file line number Diff line number Diff line change
@@ -1,19 +1,20 @@
{
"version": 1,
"generated_at": "2026-08-26T09:51:47.664Z",
"generated_at": "2026-08-26T11:25:17.401Z",
"generated_by": "agent-context-index.mjs",
"_path": "agent-context/index.json",
"description": "L1 압축 카탈로그 — 저용량 에이전트가 가장 먼저 읽는 파일. preview 60자 + summary 120자로 본문 Read 없이 관련성 판단.",
"soft_limits": {
"soft_limit_chars": 200000,
"max_entries": 1000,
"should_compress": false,
"total_chars": 1117,
"total_entries": 1
"total_chars": 1509,
"total_entries": 2
},
"counts": {
"handoff": 1,
"total": 1
"issue": 1,
"total": 2
},
"entries": [
{
Expand All @@ -38,6 +39,28 @@
"related": [],
"affects": [],
"chars": 1117
},
{
"id": "issue-20260826-b48a3b55",
"type": "issue",
"level": "diary",
"title": "login 500 on refresh",
"tags": [
"issue"
],
"feature": "auth",
"scope": "global",
"agent": "system",
"created": "2026-08-26T11:23:08.632Z",
"updated": "2026-08-26T11:23:08.632Z",
"status": "done",
"priority": 4,
"summary": "login 500 on refresh",
"preview": "login 500 on refresh",
"path": "bugs/2026-08-26-login-500-on-refresh--system.md",
"related": [],
"affects": [],
"chars": 392
}
]
}
10 changes: 6 additions & 4 deletions tools/agent-context-index.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -211,8 +211,10 @@ if (ARGS.init) {
required: CONFIG.schema?.required || ["id","type","title","tags","feature","agent","created","updated","status","summary"],
properties: {
...(schema.properties || {}),
type: { type: "string", enum: types },
feature: { type: "string", enum: featureEnum, description: "graph.json/features.json 키와 연결" },
type: CONFIG.typesFluid === false
? { type: "string", enum: types }
: { type: "string", pattern: "^[a-z0-9-]+$", description: "유동적 타입 — 자유 문자열 (typesFluid)" },
feature: { type: "string", pattern: "^[a-z0-9-_]+$", description: "유동적 feature — graph/features 키와 연결, 자유 확장" },
agent: { type: "string", enum: agents },
},
};
Expand All @@ -225,10 +227,10 @@ if (ARGS.init) {
// minimal fallback
expectedSchema.properties = {
id: { type: "string", pattern: CONFIG.schema?.idPattern || "^[a-z-]+-[0-9]{8}-[a-z0-9]{8}$" },
type: { type: "string", enum: types },
type: CONFIG.typesFluid === false ? { type: "string", enum: types } : { type: "string", pattern: "^[a-z0-9-]+$" },
title: { type: "string", minLength: 5, maxLength: 80 },
tags: { type: "array", items: { type: "string", pattern: "^[a-z0-9-]+$" }, minItems: 1, maxItems: 8 },
feature: { type: "string", enum: featureEnum },
feature: { type: "string", pattern: "^[a-z0-9-_]+$" },
scope: { type: "string", pattern: "^(global|page|custom:.+)$", default: "global" },
agent: { type: "string", enum: agents },
created: { type: "string", format: "date-time" },
Expand Down
3 changes: 2 additions & 1 deletion tools/agent-context-init.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,8 @@ const config = {
privateMirror: null,
features: Object.fromEntries(featuresList.map(f=> [f, { label: f[0].toUpperCase()+f.slice(1), files: [`src/${f}/index.ts:1`], description: `${f} feature` }])),
graph: { edges: featuresList.length>=2 ? [[featuresList[1], featuresList[0]]] : [] },
types: ["note","memo","idea","learning","bug","decision","diary","code-history","todo"],
types: ["note","memo","idea","learning","bug","decision","diary","code-history","todo","issue"],
typesFluid: true,
schema: { required: ["id","type","title","tags","feature","agent","created","updated","status","summary"], featureEnum: "auto", idPattern: "^[a-z-]+-[0-9]{8}-[a-z0-9]{8}$", maxSummary: 200, maxPreview: 60 },
storage: { backend: "json", softLimits: { softLimitChars: 200000, maxEntries: 1000, archiveAfterDays: 90 } },
lint: { onIndexRegenerate: true, forbidWriteOverwrite: true, requiredKeywords: false },
Expand Down
6 changes: 5 additions & 1 deletion tools/agent-context-validate.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,11 @@ for (const f of files) {
if (fm.id && !new RegExp(schema.properties?.id?.pattern || "^[a-z-]+-[0-9]{8}-[a-z0-9]{8}$").test(fm.id)) {
console.error(`FAIL ${f}: id pattern mismatch '${fm.id}'`); errors++;
}
if (fm.type && schema.properties?.type?.enum && !schema.properties.type.enum.includes(fm.type)) {
// typesFluid=true (기본)면 유동 타입 허용 — 구버전 scaffold의 stale enum도 통과시킴
const typeEnum = schema.properties?.type?.enum;
if (fm.type && CONFIG.typesFluid === true) {
if (!/^[a-z0-9-]+$/.test(fm.type)) { console.error(`FAIL ${f}: fluid type pattern mismatch '${fm.type}'`); errors++; }
} else if (fm.type && typeEnum && !typeEnum.includes(fm.type)) {
console.error(`FAIL ${f}: type '${fm.type}' not in enum`); errors++;
}
if (fm.feature && schema.properties?.feature?.enum && !schema.properties.feature.enum.includes(fm.feature)) {
Expand Down
11 changes: 9 additions & 2 deletions tools/agent-search-lite.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -130,16 +130,23 @@ function search(query, opts={}) {
const totalTokens = top.reduce((sum,s)=>sum+s.estTokens,0);
const wouldBeFullRead = entries.reduce((sum,e)=>sum+(LEVELS[estimateLevel(e)]?.tokens||200),0);
const saving = wouldBeFullRead ? ((wouldBeFullRead-totalTokens)/wouldBeFullRead*100).toFixed(1) : 0;
const hit = top.length > 0;
// nemotron 지적 반영: miss는 'n/a (miss)', 99.95% 이상은 '99.9%+' 표기로 착시 제거
const hitNum = parseFloat(saving);
const savingStr = !hit ? 'n/a (miss)' : (hitNum >= 99.95 ? '99.9%+' : `${saving}%`);
return {
query,
assignedLevel: requestedLevel,
router: { type: 'rule-based heuristic', noLLM: true, zeroTokens: true, reason: `query ${qTokens.length} words → ${requestedLevel}` },
lightweightAI: { reason: `query ${qTokens.length} words → ${requestedLevel} (hierarchical cache)`, noLLM: true, zeroTokens: true },
order: ORDER,
totalEntries: entries.length,
evaluated: scored.length,
hit,
expandedTo: hit ? null : null,
top: top.map(s=>({ id:s.entry.id, title:s.entry.title, level:s.lev, feature:s.entry.feature, priority:s.entry.priority, score: s.score.toFixed(2), estTokens:s.estTokens, path:s.entry.path, summary:s.entry.summary })),
tokens: { top: totalTokens, full: wouldBeFullRead, saving: `${saving}%`, avgPerQuery: top.length? Math.round(totalTokens/top.length):0 },
note: `Hierarchical: ${ORDER.slice(0, startRank+1).join('→')} first, expand to larger only if no hit — like cache→HBM→DRAM→SSD→library`
tokens: { top: totalTokens, full: wouldBeFullRead, saving: savingStr, avgPerQuery: top.length? Math.round(totalTokens/top.length):0 },
note: `Hierarchical: ${ORDER.slice(0, startRank+1).join('→')} first, miss expands to larger levels — cache metaphor`
};
}

Expand Down
Loading
Loading