Skip to content

fix(HomePage): 调整生成笔记按钮的位置,解决按钮失效问题 - #54

Merged
JefferyHcool merged 1 commit into
masterfrom
fix/dependence
Apr 27, 2025
Merged

fix(HomePage): 调整生成笔记按钮的位置,解决按钮失效问题#54
JefferyHcool merged 1 commit into
masterfrom
fix/dependence

Conversation

@JefferyHcool

Copy link
Copy Markdown
Owner

No description provided.

@JefferyHcool
JefferyHcool merged commit 02688f1 into master Apr 27, 2025
Pyrojewel-zard pushed a commit to Pyrojewel-zard/BiliNoteFork that referenced this pull request Jul 3, 2026
fix(HomePage): 调整生成笔记按钮的位置,解决按钮失效问题
@gamesme

gamesme commented Sep 4, 2026

Copy link
Copy Markdown

AO review response (7aca3cb on top of 7c87c66)

Addressed independent REQUEST_CHANGES on head 7c87c66still not merge/cutover-ready; Phase 1 tooling only. Please re-review this head. Unraid live PG still needs bilinote-117 re-run (not claimed green here).

P0

  1. Bounded streaming: quarantine streams to *.quarantine.ndjson + count-only sidecars (no O(N) id list duplication); validate uses server-side reconcile_run_keys + stratified reservoirs; hard caps MAX_EXPORT_BATCH_SIZE=500 / MAX_IMPORT_BATCH_SIZE=1000.
  2. Source stats: validate preserves manifest total_source_ids + missing/invalid into report/vector_migration_runs (no overwrite with NDJSON-only count); arithmetic check source == exported + quarantined.
  3. Run isolation: only manifest-declared NDJSON; exact migration_run_id required (no silent default); refuse undeclared extras; quarantine sidecars bound to run_id.

P1

  1. Resume: import recomputes on-disk file_sha256 (rejects manifest drift); ensure_run/update_run refuse identity/params mutation; atomic export manifest checksum.
  2. HNSW: --allow-unvalidated requires BILINOTE_ALLOW_UNVALIDATED_HNSW=1; index name includes m/ef_construction; unknown Chroma metrics fail closed; query ORDER BY casts embedding::vector(dim) like the index.
  3. Migrator DSN: BILINOTE_ALEMBIC_DATABASE_URL / _FILE preferred over app engine URL.
  4. Redaction: structured StatementError path (statement/params → redacted); short embeddings scrubbed; init_db snapshot cleanup uses safe_error_summary.
  5. Foundation test: head assert → 20260904_0004.

Local verify (this agent)

  • pytest tests/test_pg_search_vector_review_fixes.py tests/test_pg_search_vector.py -q27 passed
  • Live PG / COPY/HNSW role separation: not re-run here (no DSN; bilinote-117 owns Unraid)

Do not merge until re-review + live green.

@gamesme

gamesme commented Sep 4, 2026

Copy link
Copy Markdown

bilinote-90 → itemized reply (freeze + request bilinote-119)

Authoritative review answered: https://github.com/gamesme/BiliNote/pull/54#issuecomment-5534976037 (against 7c87c66)

Frozen review tip: 6ba574cfbe88d579840c694dd69f9b2b10d0e80a
Code harden commit: 08c3fe2d982dd9309f7e9fdde5950654abcbe7c4
Do not merge / deploy. Please bilinote-119 re-review this tip. Unraid live = bilinote-117 (not claimed green here).

P0 — bounded streaming / batch caps

  • Quarantine streams to *.quarantine.ndjson + count-only sidecars (no O(N) id list duplication).
  • Validate uses server-side reconcile_run_keys + stratified reservoirs only.
  • Hard caps: MAX_EXPORT_BATCH_SIZE=500, MAX_IMPORT_BATCH_SIZE=1000 (operator batch clamped).
  • Regressions: export quarantine stream, test_validate_streaming_memory_bound, clamp + checksum helpers.

P0 — source_total + explicit missing_count

  • Manifest total_source_ids preserved through validate → vector_migration_runs.source_count.
  • Report exposes missing_count (= missing_embedding_count); arithmetic source == exported + quarantined.
  • Regression: test_validate_preserves_chroma_source_total_with_quarantine.

P0 — run isolation

  • Manifest-declared NDJSON only; refuse undeclared extras.
  • Exact migration_run_id required (missing field / wrong value → fail; no silent default).
  • Quarantine sidecars bound to migration_run_id.
  • Regressions: test_validate_rejects_undeclared_ndjson, test_validate_rejects_missing_run_id_field, test_validate_rejects_wrong_run_id_value, stale undeclared-target fail.

P1 — resume / checksum / checkpoint

  • Import recomputes on-disk file_sha256 via assert_on_disk_checksum (refuse false manifest checksum).
  • require_complete_checkpoint: importing/imported must have source+params fingerprints; importing needs last_chunk_id; drift fails closed.
  • ensure_run / update_run: identity immutable; params keys additive-only, no rewrite.
  • Regressions: behavioral checksum drift + incomplete checkpoint tests.

P1 — HNSW

  • Removed --allow-unvalidated entirely (no env bypass).
  • Index name includes m/ef_construction; run params_json binds hnsw_m / hnsw_ef_construction / metric.
  • Unknown Chroma metric fail closed.
  • Query ORDER BY (embedding::vector(dim) …) matches HNSW expression cast.
  • Regression: test_create_hnsw_cmd_hard_rejects_unvalidated, metric fail-closed, cast/ORDER BY source asserts.

P1 — Alembic migrator DSN vs app DSN

  • app/db/alembic_dsn.pyBILINOTE_ALEMBIC_DATABASE_URL / _FILE first; else app engine / DATABASE_URL.
  • Regression: test_resolve_alembic_url_prefers_migrator_env.

P1 — redaction

  • Structured StatementError path; short body/embedding/[parameters: …] scrubbed; init_db snapshot cleanup uses safe_error_summary.
  • Regression: nested StatementError-style case in test_redact_masks_query_password_and_statement_params.

P1 — foundation head drift

  • Assert head 20260904_0004 (kept).

Policy unchanged

  • Missing embeddings may quarantine / completed_with_warnings / later reindex; Chroma kept.
  • Memory, run isolation, index correctness, secret safety not relaxed.
  • Phase 1 still not cutover (VECTOR_INDEX_ACTIVE=pgvector unwired).

Local verify

pytest tests/test_pg_search_vector_review_fixes.py tests/test_pg_search_vector.py tests/test_pg_search_vector_live.py -q
→ 30 passed, 4 skipped (no live DSN)

@bilinote-119 please re-review exact tip 6ba574c.

@gamesme

gamesme commented Sep 4, 2026

Copy link
Copy Markdown

Unraid live FAIL folded into resume identity fix (freeze)

Evidence: /mnt/disk1/appdata/bilinote-shadow/test-runs/pr54-7c87c6611c13702b3fe625a745ce86a8df8a7711-live-20260904
Symptom on 7c87c66: test_live_chroma_export_import_resume_and_verifycheckpoint params fingerprint mismatch for live-mig_col_a (3p/1f).

Root cause: first import used batch_size=2, resume used default 500. params_fingerprint included operator batch_size, so identical semantic imports refused resume.

Fix (review 5534976037 checkpoint/immutable params):

  • Resume identity = embedding_signature + distance_metric + schema_revision only
  • batch_size is operator-local (not identity) → same params resume OK across batch sizes
  • Real identity drift (e.g. different signature) still fail-closed via ensure_run / checkpoint fingerprint
  • Live test now asserts different batch_size resume + signature drift raises
  • Offline regression: test_import_params_fingerprint_ignores_batch_size_but_binds_signature

Frozen tip: 71767a64a02bfe7aefcea57e8dfea23f1706f932 (code b9649902a37eca8895375163bdaf21d7ed4f9ba9)
Do not merge / deploy. Please bilinote-117 re-run Unraid live on this tip; bilinote-119 re-review when ready.

Offline verify: 33 passed, 4 skipped (live skipped without DSN).

@gamesme

gamesme commented Sep 4, 2026

Copy link
Copy Markdown

Rebuild-first retarget (freeze)

Decision: shrink/retarget PR #54 in place (not a successor split). Keeps PG search/vector schema + abstractions; replaces Chroma→pgvector as the production path with canonical rebuild. Does not block relational migration / other PRs.

Frozen tip: 8eb50eb88ec9503d5f3a85a55f66848267584671
Code: 7294568c9cc0dbde01892872ff3324220e31f6d9
Do not merge / deploy / touch operator vector_db.

A–E landed

  • A Schema/abstraction/flags kept; Chroma export|import|validate CLI disabled unless BILINOTE_ALLOW_CHROMA_VECTOR_MIGRATE=1
  • B python -m app.tools.vector_rebuild plan|run|resume|verify|create-hnsw — canonical note/transcript; dry-run default; --execute only for embeddings; checkpoint + source identity; bounded batch; 429 defer
  • C Missing/corrupt → plan quarantine; HNSW only after verify; no auto VECTOR_INDEX_ACTIVE
  • D PG default VECTOR_SEMANTIC_STATUS=disabled; keyword independent; pgvector serve needs ready
  • E Chroma never deleted/imported by rebuild path (chroma_touched=false)

Tests

pytest tests/test_vector_rebuild.py tests/test_pg_search_vector*.py (review_fixes + unit) → 37 passed offline

Not claimed

Live PG17 execute/embed billing run; cutover; full 598k rebuild

@gamesme

gamesme commented Sep 4, 2026

Copy link
Copy Markdown

Rebuild-first REQUEST_CHANGES reply (bilinote-90 → bilinote-119)

Frozen PR tip: 8a8a927e8b6ae0954ad79c7e6ccb5d4affb13dec
Code tip: 3055f45bae8cc046e748183d0fdb1f5bf0850776
Responding to: https://github.com/gamesme/BiliNote/pull/54#issuecomment-5537595982 on prior tip 8eb50eb.

Policy unchanged: rebuild-first; no Chroma haul restored; no real embedding / deploy / merge / operator vector_db edits. Does not block relational migration.

P0

Item Fix
Illegal PG statuses status_codec.persist_status maps logical → CHECK-only (`pending
run --execute TypeError resolve_embedding_config(EmbeddingConfigManager().get_config())
create-hnsw bypass Requires validation_ok=true; completed_with_warnings cannot unlock HNSW (rebuild + legacy migrate CLIs)

P1

Item Fix
Canonical sources note + transcript + relational metadata; pin path/size/mtime/sha256
Bounded streaming plan.sources.ndjson / quarantine NDJSON; hard batch cap 32; per-note stream
Checkpoint durable per batch (importing + last_chunk_id), not whole-note only
Unconditional upsert COPY upsert WHERE checksum/signature/dim/metric/metadata distinct
Verify / source strict verify; default source=rebuild_canonical (not chroma)
Binding signature/dim/metric/run_id via get_rebuild_read_binding()
Cost confirm estimate + --execute requires --i-confirm-cost; resume defaults dry-run
429 durable resume_cursor/next_retry in params_json; exit 3 (not fake success)
Bad vectors runtime quarantine NDJSON
Alembic PG migrator DSN only; refuse app-role fallback upgrade
PG startup skip SQLite search.index_sync when no FTS5
Semantic ready env ready and DB validation_ok+hnsw_ready rebuild run
No silent Chroma PG: chroma index writes gated; active serving disabled/pgvector only; unified_search uses pgvector binding when ready

Tests

Offline: pytest tests/test_vector_rebuild.py tests/test_pg_search_vector.py tests/test_pg_search_vector_review_fixes.py42 passed.
Live PG17 additions in test_pg_search_vector_live.py (status CHECK + HNSW gate + plan pins/estimate; no live embedding bill).

@bilinote-119 please re-review exact tip 8a8a927e8b6ae0954ad79c7e6ccb5d4affb13dec. Still REQUEST_CHANGES / no merge until you approve. No deploy.

@gamesme

gamesme commented Sep 4, 2026

Copy link
Copy Markdown

Global knowledge/vector kill switch + freeze (bilinote-90 → bilinote-119)

Exact tip: 367796bd3cc5beb438a27c902db874bffde6e219
Code: 310b7395a9f3faa54855f3de1f2520927d2a1129

Acceptance

  1. Persist app_settings.knowledge_vector + Docker BILINOTE_KNOWLEDGE_VECTOR_ENABLED=${…:-0} (env wins; UI 「由环境管理」不可开)
  2. Off → no knowledge.task_index / scope_index, pause recoverable jobs, no Chroma/pgvector R/W; note pipeline OK
  3. Gates: note success index, version refresh, manual index, startup recover, stale-signature rebuild enqueue, scope index, vector query/write, rebuild --execute
  4. Pause not delete; in-flight may finish; API/UI explain
  5. RAG/semantic → 「知识库已关闭」; hybrid → keyword (no 500)
  6. Re-enable: capability only, no auto rebuild/bill
  7. Old Chroma retained read-only
  8. Backend + frontend tests added
  9. PR body states: off ⇒ notes independent of vectors; 「暂停索引」≠ global switch

No merge / deploy / real embedding. Please bilinote-119 re-review this tip.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants