feat: add optional you.com search integration - #5
Open
mouse-value-add wants to merge 1 commit into
Open
Conversation
Adds youcom as a third search provider behind the existing Provider interface. You.com's Search API can return full page content with each result (extraction_mode=full_page), which feeds the same fetch-skipping path Tavily uses, and it filters domains server-side, which Brave cannot. Activates only when search.provider is set to youcom.
Author
|
Brought this back up to date with the current |
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds You.com as a third search provider behind the existing
Providerinterface, alongside Brave and Tavily. It only activates whensearch.provideris set toyoucom— nothing changes for existing configurations.Why this fits mole
Two properties of the You.com Search API map directly onto things the search package already cares about:
extraction_mode: "full_page"and returns each result with acontents.markdownfield carrying the page text. That feeds the same fetch-skipping path Tavily uses —Result.Content+HasUsableContent()— so the WebActor can ground claims without a subsequent fetch (§10.4). The provider is chosen for exactly this inIncludeContentmode.include_domains/exclude_domainsare request parameters, so both options travel to the provider instead of Brave's query-operator / client-side workaround.Auth is
X-API-Key, which is already in the cassette redaction list, so keys stay out of cassettes without any change to the recorder.What changed
internal/tools/search/youcom.go— the provider. POST tohttps://ydc-index.io/v1/search, mapsresults.web(url, title, description, snippets,page_age,contents.markdown), drops empty-URL rows, leaves unparseable dates nil rather than guessing (same rationale asparseBraveAge). Default cost 5,000 micros ($5/1000 queries, entry tier); the full-page extraction surcharge is documented in a comment rather than silently folded into the default.internal/tools/search/search.go—KindYoucomin the registry,newYoucomin the factory, package doc updated.internal/config/config.go—search.youcom-keyfield (secret),ActiveKey()case, env overrideMOLE_YOUCOM_API_KEY/YDC_API_KEY, provider validation acceptsyoucom.cmd/mole/main.go—mole doctorreports youcom with the same "returns page content, skips fetches" note Tavily gets, since it changes how many fetches a session makes.cmd/mole/research.go— the "no search provider selected" hint lists youcom.README.md— config example.Kinds()-iterating tests (error classification, selectability) pick the new provider up automatically.Setup
Env-only works too:
MOLE_SEARCH_PROVIDER=youcom+YDC_API_KEY.Validation
go build ./...,go vet ./...— cleango test ./...— 33 packages pass (new:TestYoucomParsesResults,TestYoucomDomainFiltersAreSentServerSide,TestYoucomSnippetsFallbackForDescription;TestErrorsAreClassifiedForTheRetryPolicyandTestProviderIsSelectablecover youcom viaKinds())go test -race ./internal/tools/search/ ./internal/config/ ./cmd/mole/— cleanHappy to adjust the shape if you'd rather see the extraction surcharge priced into
queryCost(), or the provider named differently.