fix(curate): improve Vertex AI reliability for large curation extractions#79
Open
mpeter wants to merge 1 commit into
Open
fix(curate): improve Vertex AI reliability for large curation extractions#79mpeter wants to merge 1 commit into
mpeter wants to merge 1 commit into
Conversation
…ions Three bugs causing curation failures on real-world vaults with moderate content density (9-42 documents per store): Bug 1 — VertexSynthesizer MaxTokens hardcoded at 4096 (unbound-force#76) Real curation extractions produce 4K-16K output tokens. Claude truncates mid-JSON at the 4096 limit → ParseExtractionResponse fails with 'unexpected end of JSON input'. Raised to 16000. llm/vertex.go: MaxTokens 4096 → 16000 Bug 2 — VertexSynthesizer HTTP timeout too short for large prompts (unbound-force#77) Vertex AI with Claude Sonnet takes 120-260s for 30K-40K token prompts. The 120s HTTP client timeout consistently cuts off large extractions. Raised to 300s (Vertex has different latency than local Ollama). llm/vertex.go: Timeout 120s → 300s Bug 3 — ParseExtractionResponse strict unmarshal fails on quality_flags string (unbound-force#78) LLMs non-deterministically output 'quality_flags': 'none' (string) instead of 'quality_flags': [] (array). Strict json.Unmarshal into []KnowledgeFile rejects the entire response. Changed to parse []json.RawMessage first, then unmarshal each item individually with a permissive fallback that accepts any type for quality_flags and discards non-array values. curate/curate.go: strict unmarshal → per-item with fallback Also improved the extraction prompt to request concise output (3-8 items per document, short excerpts, terse content) which reduces token consumption and makes truncation less likely even if the max_tokens limit is hit. Tested against 3 real accounts (42, 10, 9 documents) on Vertex AI with claude-sonnet-4-6@default. All stores now curate successfully.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes three bugs that cause
dewey curateto fail consistently on real-world vaults with moderate content density.Bugs Fixed
#76 — MaxTokens hardcoded at 4096 in VertexSynthesizer
Real curation extractions produce 4K-16K output tokens. Claude truncates the JSON array at the 4096 token limit, causing
ParseExtractionResponseto fail withunexpected end of JSON input.#77 — HTTP timeout too short for Vertex AI
Vertex AI with Claude Sonnet takes 120-260s for 30K-40K token prompts. The 120s HTTP client timeout cuts off large extractions mid-stream. Vertex has fundamentally different latency characteristics than local Ollama inference.
#78 — quality_flags string type crashes ParseExtractionResponse
LLMs non-deterministically output
"quality_flags": "none"(string) instead of"quality_flags": [](array). Strictjson.Unmarshalinto[]KnowledgeFilerejects the entire response for a type mismatch on a single field.Changed to parse
[]json.RawMessagefirst, then unmarshal each item individually. Items that fail strict unmarshal fall back to a permissive struct that accepts any JSON type forquality_flagsand discards non-array values. Valid items are preserved; unparseable items are logged and skipped.Additional: prompt conciseness
The extraction prompt now requests concise output (3-8 items per document maximum, short excerpts ≤20 words, content ≤30 words per item). This reduces output token consumption, making truncation less likely even near the
max_tokenslimit, and produces more scannable curated files.Testing
Tested against 3 knowledge stores with varying content density (9, 10, and 42 documents respectively). All previously failed with one or more of the three bugs above and now produce knowledge files successfully.
Closes #76, #77, #78