Skip to content

fix(curate): improve Vertex AI reliability for large curation extractions#79

Open
mpeter wants to merge 1 commit into
unbound-force:mainfrom
mpeter:fix/curation-vertex-reliability
Open

fix(curate): improve Vertex AI reliability for large curation extractions#79
mpeter wants to merge 1 commit into
unbound-force:mainfrom
mpeter:fix/curation-vertex-reliability

Conversation

@mpeter

@mpeter mpeter commented Jul 11, 2026

Copy link
Copy Markdown

Fixes three bugs that cause dewey curate to fail consistently on real-world vaults with moderate content density.

Bugs Fixed

#76 — MaxTokens hardcoded at 4096 in VertexSynthesizer

Real curation extractions produce 4K-16K output tokens. Claude truncates the JSON array at the 4096 token limit, causing ParseExtractionResponse to fail with unexpected end of JSON input.

// Before
MaxTokens: 4096,

// After
MaxTokens: 16000,

#77 — HTTP timeout too short for Vertex AI

Vertex AI with Claude Sonnet takes 120-260s for 30K-40K token prompts. The 120s HTTP client timeout cuts off large extractions mid-stream. Vertex has fundamentally different latency characteristics than local Ollama inference.

// Before
Timeout: 120 * time.Second,

// After
Timeout: 300 * time.Second,

#78 — quality_flags string type crashes ParseExtractionResponse

LLMs non-deterministically output "quality_flags": "none" (string) instead of "quality_flags": [] (array). Strict json.Unmarshal into []KnowledgeFile rejects the entire response for a type mismatch on a single field.

Changed to parse []json.RawMessage first, then unmarshal each item individually. Items that fail strict unmarshal fall back to a permissive struct that accepts any JSON type for quality_flags and discards non-array values. Valid items are preserved; unparseable items are logged and skipped.

Additional: prompt conciseness

The extraction prompt now requests concise output (3-8 items per document maximum, short excerpts ≤20 words, content ≤30 words per item). This reduces output token consumption, making truncation less likely even near the max_tokens limit, and produces more scannable curated files.

Testing

Tested against 3 knowledge stores with varying content density (9, 10, and 42 documents respectively). All previously failed with one or more of the three bugs above and now produce knowledge files successfully.

Closes #76, #77, #78

…ions

Three bugs causing curation failures on real-world vaults with moderate
content density (9-42 documents per store):

Bug 1 — VertexSynthesizer MaxTokens hardcoded at 4096 (unbound-force#76)
  Real curation extractions produce 4K-16K output tokens. Claude truncates
  mid-JSON at the 4096 limit → ParseExtractionResponse fails with
  'unexpected end of JSON input'. Raised to 16000.

  llm/vertex.go: MaxTokens 4096 → 16000

Bug 2 — VertexSynthesizer HTTP timeout too short for large prompts (unbound-force#77)
  Vertex AI with Claude Sonnet takes 120-260s for 30K-40K token prompts.
  The 120s HTTP client timeout consistently cuts off large extractions.
  Raised to 300s (Vertex has different latency than local Ollama).

  llm/vertex.go: Timeout 120s → 300s

Bug 3 — ParseExtractionResponse strict unmarshal fails on quality_flags string (unbound-force#78)
  LLMs non-deterministically output 'quality_flags': 'none' (string) instead
  of 'quality_flags': [] (array). Strict json.Unmarshal into []KnowledgeFile
  rejects the entire response. Changed to parse []json.RawMessage first, then
  unmarshal each item individually with a permissive fallback that accepts any
  type for quality_flags and discards non-array values.

  curate/curate.go: strict unmarshal → per-item with fallback

Also improved the extraction prompt to request concise output (3-8 items per
document, short excerpts, terse content) which reduces token consumption and
makes truncation less likely even if the max_tokens limit is hit.

Tested against 3 real accounts (42, 10, 9 documents) on Vertex AI with
claude-sonnet-4-6@default. All stores now curate successfully.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: curation fails with 'unexpected end of JSON input' — VertexSynthesizer MaxTokens hardcoded at 4096

1 participant