Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
124 changes: 124 additions & 0 deletions docs/playbooks/VOLUME_AUTO_SCAN_PLAYBOOK.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
# VOLUME AUTO-SCAN PLAYBOOK (Performance-aware)

## 목표
볼륨 등록/활성화 시 기존 파일을 자동 스캔하여 파일 트리에 반영한다.
성능 저하를 막기 위해 단계적/비동기 방식으로 설계하고, 기존 UI/API 계약을 깨지 않는다.

---

## 범위
- 대상 레포: `~/Bento`
- 백엔드 중심 구현 (`scripts/dev_server.mjs` + 관련 API/DB 로직)
- UI는 상태 노출(스캔 진행/완료/오류) 최소 반영
- SSOT 준수: `openapi/openapi.yaml`, `docs/ui/IA_NAV_SSOT.md`, `docs/ui/COPY_KEYS_SSOT.md`

---

## 핵심 요구사항
1. 볼륨 등록/활성화 시 자동 스캔 트리거
2. `/mnt/storage` 기존 파일/디렉토리 트리 반영
3. 대용량 디렉토리에서도 서버 응답성 유지
4. 재실행 안전(idempotent) + 중복 노드 방지
5. 실패 복구 가능(중간 실패 후 재개)

---

## 설계 원칙 (성능/안정성)
- API 요청 스레드에서 전체 스캔 수행 금지 (비동기 잡 큐)
- 스캔 단위 분할(배치 처리): 디렉토리 단위/파일 N개 단위
- DB upsert 기반으로 중복 방지
- 경로 해시/mtime/size 기반 변경 탐지(최소 재처리)
- 스캔 중에도 `/files` 읽기 가능 유지
- 스캔 상태(job progress) 노출

---

## 구현 단계 (Playbook)

### Phase 1 — 데이터 모델/계약 정렬
- [ ] OpenAPI에 자동 스캔 동작/상태 필드 명시
- [ ] 볼륨/잡 스키마에 scan state(queued/running/succeeded/failed) 정의
- [ ] DB 컬럼/인덱스 필요한 경우 추가

완료 기준:
- API/스키마 정의가 코드 구현과 일치

### Phase 2 — 스캔 엔진 MVP
- [ ] 볼륨 경로 walk 구현 (심볼릭 링크/권한 에러 처리 포함)
- [ ] 디렉토리/파일을 nodes에 upsert (path 기준)
- [ ] root children에 반영되도록 parent-path 매핑
- [ ] dry-run 모드(로그만) 지원

완료 기준:
- `/mnt/storage` 파일이 `/nodes/{root}/children`에 노출

### Phase 3 — 비동기 잡/성능 제어
- [ ] 볼륨 activate/create 시 scan job enqueue
- [ ] 스캔 워커 분리(요청-응답과 분리)
- [ ] 배치 처리 + 주기적 progress 업데이트
- [ ] 시간 제한/중단/재시도 정책 추가

완료 기준:
- 대용량에서도 API 응답 지연 과도 증가 없음

### Phase 4 — 증분 스캔/재실행 안전성
- [ ] 이전 스캔 결과와 비교해 변경분만 반영
- [ ] 삭제/이동 파일 처리 정책 정의(soft-delete 또는 re-link)
- [ ] 중복/충돌(name/path) 처리 일관화

완료 기준:
- 재스캔 시 중복 노드 없이 일관성 유지

### Phase 5 — UI/운영 가시성
- [ ] Admin Storage에 스캔 상태/진행률 표시
- [ ] 실패 원인 메시지/재시도 버튼 제공
- [ ] 최초 스캔 전 사용자 안내(시간 소요/영향)

완료 기준:
- 관리자 UI에서 상태 파악/재시도 가능

### Phase 6 — 검증/릴리즈
- [ ] 기능 검증: 등록→활성화→자동스캔→목록 노출
- [ ] 회귀 검증: 업로드/다운로드/CRUD 기존 기능 유지
- [ ] `bash scripts/run_evidence.sh --scope ui_light` PASS
- [ ] 관련 evidence task PASS

완료 기준:
- CI PASS + 리뷰 반영 + 머지

---

## 운영 규칙
- 1 Phase = 1 PR
- CI PASS 전 머지 금지
- 리뷰 발생 시 실질 수정 + 답글 + thread resolve
- 마지막 커밋 후 5분 신규 리뷰 없을 때 머지
- 머지 후 브랜치/dirty 정리

---

## 금지 사항
- 민감 산출물 커밋 금지 (`evidence/**/actual/**`, logs, raw json 등)
- 토큰/세션 실값 커밋 금지
- destructive 명령 금지 (`reset --hard`, `rm -rf`, force push)
- 테스트 스킵/우회 금지

---

## 최종 보고 포맷
STATUS: PASS|FAIL
PHASE: <current>
ROOT_CAUSE:
- <...>
CHANGED_FILES:
- <...>
VALIDATION:
- <command>: <result>
CI:
- <url/status>
REVIEW:
- unresolved threads: <n>
MERGE:
- <pr url>
NEXT:
- <next phase>
120 changes: 84 additions & 36 deletions packages/ui/src/api/uploads.ts
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ export type UploadChunkParams = {
uploadId: string;
chunkIndex: number;
chunk: Blob;
chunkHash: string;
chunkHash?: string | null;
idempotencyKey?: string;
signal?: AbortSignal;
};
Expand All @@ -50,6 +50,20 @@ export type UploadsApiOptions = {
export type UploadsApi = ReturnType<typeof createUploadsApi>;

export const createUploadsApi = ({ client }: UploadsApiOptions) => {
const withFallback = async <T,>(primary: () => Promise<T>, fallback: () => Promise<T>): Promise<T> => {
try {
return await primary();
} catch (error: unknown) {
const status = typeof error === "object" && error !== null && "status" in error
? Number((error as { status?: unknown }).status)
: null;
if (status === null || [404, 405, 502, 503, 504].includes(status)) {
return fallback();
}
throw error;
}
};

return {
createUpload: async ({
parentId,
Expand All @@ -61,20 +75,30 @@ export const createUploadsApi = ({ client }: UploadsApiOptions) => {
idempotencyKey,
signal,
}: CreateUploadParams): Promise<CreateUploadResponse> => {
return client.request<CreateUploadResponse>({
path: "/uploads",
method: "POST",
body: {
parent_id: parentId,
filename,
size_bytes: sizeBytes,
...(sha256 ? { sha256 } : {}),
...(mimeType !== undefined ? { mime_type: mimeType } : {}),
...(modifiedAt !== undefined ? { modified_at: modifiedAt } : {}),
} satisfies CreateUploadRequest,
headers: idempotencyKey ? { "Idempotency-Key": idempotencyKey } : undefined,
signal,
});
const payload = {
parent_id: parentId,
filename,
size_bytes: sizeBytes,
...(sha256 ? { sha256 } : {}),
...(mimeType !== undefined ? { mime_type: mimeType } : {}),
...(modifiedAt !== undefined ? { modified_at: modifiedAt } : {}),
} satisfies CreateUploadRequest;
return withFallback(
() => client.request<CreateUploadResponse>({
path: "/nodes/uploads",
method: "POST",
body: payload,
headers: idempotencyKey ? { "Idempotency-Key": idempotencyKey } : undefined,
signal,
}),
() => client.request<CreateUploadResponse>({
path: "/uploads",
method: "POST",
body: payload,
headers: idempotencyKey ? { "Idempotency-Key": idempotencyKey } : undefined,
signal,
}),
);
},
uploadChunk: async ({
uploadId,
Expand All @@ -84,31 +108,55 @@ export const createUploadsApi = ({ client }: UploadsApiOptions) => {
idempotencyKey,
signal,
}: UploadChunkParams): Promise<UploadSession> => {
return client.request<UploadSession>({
path: `/uploads/${uploadId}/chunks/${chunkIndex}`,
method: "PUT",
rawBody: chunk,
headers: {
"Content-Type": "application/octet-stream",
"X-Chunk-SHA256": chunkHash,
...(idempotencyKey ? { "Idempotency-Key": idempotencyKey } : {}),
},
signal,
});
const headers = {
"Content-Type": "application/octet-stream",
...(chunkHash ? { "X-Chunk-SHA256": chunkHash } : {}),
...(idempotencyKey ? { "Idempotency-Key": idempotencyKey } : {}),
};
return withFallback(
() => client.request<UploadSession>({
path: `/nodes/uploads/${uploadId}/chunks/${chunkIndex}`,
method: "PUT",
rawBody: chunk,
headers,
signal,
}),
() => client.request<UploadSession>({
path: `/uploads/${uploadId}/chunks/${chunkIndex}`,
method: "PUT",
rawBody: chunk,
headers,
signal,
}),
);
},
completeUpload: async ({ uploadId, idempotencyKey }: CompleteUploadParams): Promise<CompleteUploadResponse> => {
return client.request<CompleteUploadResponse>({
path: `/uploads/${uploadId}/complete`,
method: "POST",
headers: idempotencyKey ? { "Idempotency-Key": idempotencyKey } : undefined,
});
return withFallback(
() => client.request<CompleteUploadResponse>({
path: `/nodes/uploads/${uploadId}/complete`,
method: "POST",
headers: idempotencyKey ? { "Idempotency-Key": idempotencyKey } : undefined,
}),
() => client.request<CompleteUploadResponse>({
path: `/uploads/${uploadId}/complete`,
method: "POST",
headers: idempotencyKey ? { "Idempotency-Key": idempotencyKey } : undefined,
}),
);
},
abortUpload: async ({ uploadId, idempotencyKey }: AbortUploadParams): Promise<SuccessResponse> => {
return client.request<SuccessResponse>({
path: `/uploads/${uploadId}`,
method: "DELETE",
headers: idempotencyKey ? { "Idempotency-Key": idempotencyKey } : undefined,
});
return withFallback(
() => client.request<SuccessResponse>({
path: `/nodes/uploads/${uploadId}`,
method: "DELETE",
headers: idempotencyKey ? { "Idempotency-Key": idempotencyKey } : undefined,
}),
() => client.request<SuccessResponse>({
path: `/uploads/${uploadId}`,
method: "DELETE",
headers: idempotencyKey ? { "Idempotency-Key": idempotencyKey } : undefined,
}),
);
},
};
};
12 changes: 12 additions & 0 deletions packages/ui/src/api/volumes.ts
Original file line number Diff line number Diff line change
Expand Up @@ -49,5 +49,17 @@ export const createVolumesApi = (client: ReturnType<typeof createApiClient>) =>
body: payload && Object.keys(payload).length > 0 ? payload : undefined,
});
},
deactivateVolume: async (volumeId: string): Promise<Volume> => {
return client.request<Volume>({
path: `/admin/volumes/${volumeId}/deactivate`,
method: "POST",
});
},
deleteVolume: async (volumeId: string): Promise<{ ok: boolean }> => {
return client.request<{ ok: boolean }>({
path: `/admin/volumes/${volumeId}`,
method: "DELETE",
});
},
};
};
2 changes: 1 addition & 1 deletion packages/ui/src/app/AdminMigrationPage.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -195,7 +195,7 @@ export default function AdminMigrationPage() {
label={t("field.targetVolumeId")}
value={targetVolumeId}
onChange={(event) => setTargetVolumeId(event.target.value)}
placeholder="volume-id"
placeholder={t("placeholder.volumeId")}
/>
<label className="admin-migration__checkbox">
<input
Expand Down
45 changes: 36 additions & 9 deletions packages/ui/src/app/AdminStoragePage.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,8 @@ export default function AdminStoragePage() {

const [activateErrorKey, setActivateErrorKey] = useState<I18nKey | null>(null);
const [activating, setActivating] = useState(false);
const [deactivating, setDeactivating] = useState(false);
const [deleting, setDeleting] = useState(false);
const [activated, setActivated] = useState(false);

const [scanDeleteFiles, setScanDeleteFiles] = useState(false);
Expand Down Expand Up @@ -137,15 +139,8 @@ export default function AdminStoragePage() {
void fetchScanJob(activeVolume.scan_job_id);
}, [activeVolume?.scan_job_id, fetchScanJob]);

useEffect(() => {
if (!activeVolume) return;
if (activeVolume.scan_state !== "queued" && activeVolume.scan_state !== "running") return;

const timer = window.setTimeout(() => {
void loadVolumes();
}, 3000);
return () => window.clearTimeout(timer);
}, [activeVolume, loadVolumes]);
// Auto polling disabled: it made the storage page feel like constant refresh.
// Users can refresh manually via toolbar or action buttons.

const handleValidate = async () => {
if (!validatePath || validating) return;
Expand Down Expand Up @@ -202,6 +197,34 @@ export default function AdminStoragePage() {
}
};

const handleDeactivate = async () => {
if (!selectedVolume || !selectedVolume.is_active || deactivating) return;
setDeactivating(true);
setActivateErrorKey(null);
try {
await volumesApi.deactivateVolume(selectedVolume.id);
await loadVolumes();
} catch (error) {
setActivateErrorKey(error instanceof ApiError ? error.key : "err.network");
} finally {
setDeactivating(false);
}
};

const handleDeleteVolume = async () => {
if (!selectedVolume || selectedVolume.is_active || deleting) return;
setDeleting(true);
setActivateErrorKey(null);
try {
await volumesApi.deleteVolume(selectedVolume.id);
await loadVolumes();
} catch (error) {
setActivateErrorKey(error instanceof ApiError ? error.key : "err.network");
} finally {
setDeleting(false);
}
};

const handleStartScan = async () => {
if (scanSubmitting) return;

Expand Down Expand Up @@ -364,7 +387,11 @@ export default function AdminStoragePage() {
activateErrorKey={activateErrorKey}
activated={activated}
activating={activating}
deactivating={deactivating}
deleting={deleting}
onActivate={handleActivate}
onDeactivate={handleDeactivate}
onDelete={handleDeleteVolume}
/>

<ScanCleanupSection
Expand Down
Loading
Loading