Skip to content

fix(search): read truncation from SMW's query-continue-offset - #171

Open
LukasGold wants to merge 3 commits into
mainfrom
fix/semantic-search-truncation-state
Open

fix(search): read truncation from SMW's query-continue-offset#171
LukasGold wants to merge 3 commits into
mainfrom
fix/semantic-search-truncation-state

Conversation

@LukasGold

@LukasGold LukasGold commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Fixes #162.

Changes

  • semantic_search reads truncation from the top-level query-continue-offset key of the ask response instead of comparing the result count against the limit.
  • New SearchParam.return_meta. When True, semantic_search returns one SemanticSearchResult per query instead of a flat list of titles.
  • New SemanticSearchResult model with query, titles, count, truncated and next_offset.
  • The return_json=True path is unchanged. return_json takes precedence when both flags are set.
  • WtSite.modify_search_results raises ValueError for a SearchParam that sets return_json or return_meta.

Rationale

The old check was if limit and n >= limit. It was wrong in both directions.

It missed truncation. With no limit in force it could never fire, and it could not see SMW's own $smwgQMaxLimit cap (default 10000, https://www.semantic-mediawiki.org/wiki/Help:$smwgQMaxLimit). A query asking for 50000 and cut at 10000 gives 10000 >= 50000, which is false.

It also invented truncation. A complete result set of exactly limit results was reported as truncated. Measured against https://www.semantic-mediawiki.org with mwclient, using [[Has type::Date]], which has 18 results in total:

query count returned query-continue-offset
[[Has type::Date]]|limit=18 18 absent
[[Has type::Date]]|limit=17 17 17

The key is present exactly when the result set was cut short, so it replaces the count comparison rather than supplementing it. No fallback is kept, because on a wiki that sends the key the count comparison adds only false positives.

Issue #162 proposed reading query.meta.hasFurtherResults. That key does not exist in the ask response. See the comment on the issue for the measurement.

Returning the state, not only warning

#133 adds osw.service.ops.search, whose search_entities calls semantic_search with the default return_json=False and receives a flat list of titles. The truncation signal was not reachable from there. return_meta makes it reachable without changing the default return shape.

Wiring osw.service.ops.search to use it stays with #133, since src/osw/service/ does not exist on main.

The modify_search_results guard

WtSite.modify_search_results looks up and edits one page per search result, so it needs page titles. Its param.query accepts a SearchParam, and return_json already made it hand back raw dicts instead, failing later inside GetPageParam with a type error that names neither the flag nor the method. return_meta added a second way to reach that state, so both are now rejected up front with a message that names the offending flag.

The check covers both search modes, since prefix_search honours return_json as well. No working code can be affected: return_meta is new in this PR, and a return_json query already failed on the first result.

Tests

44 tests in tests/test_wiki_tools.py, 9 in tests/test_wtsite_modify_search_results.py, 215 in the full unit suite, all passing.

New in tests/test_wiki_tools.py: truncation detected from the offset below the limit, detected with limit=None, the raw return_json dict keeping the offset, return_meta reporting a truncated and a complete result, one result per query, order preserved on the parallel path, count including pages dropped as non-existing, and return_json winning over return_meta.

New in tests/test_wtsite_modify_search_results.py: both flags rejected in both search modes, and a plain SearchParam still accepted in both. All four rejection cases were confirmed to fail with the guard disabled.

Two existing tests changed because they encoded the removed heuristic:

  • test_semantic_search_truncation_warning now sends an offset in the mocked response.
  • test_semantic_search_truncation_warning_uses_the_query_limit became ..._names_the_query_as_sent, since the warning now quotes the query and the offset rather than the requested limit.

A new test asserts the false positive is gone: a complete set of exactly limit results does not warn.

Not addressed

search_titles and search_content in #133 use MediaWiki's prefixsearch and search, not SMW's ask. They report continuation differently and need separate handling in that PR.

- replace the count-versus-limit check, blind to $smwgQMaxLimit
- stop reporting a complete set of exactly 'limit' results as truncated
- add SearchParam.return_meta yielding SemanticSearchResult per query
- report query, titles, count, truncated and next_offset to callers
- leave return_json untouched, its raw dict already holds the offset
@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Release preview

Merging this PR would release v2.3.4 (current: v2.3.3).

Changelog preview (truncated)
## v2.3.4 (2026-09-08)

Preview via python-semantic-release and conventional commits.

- return_json and return_meta yield dicts and objects, not page titles
- raise ValueError instead of failing later inside GetPageParam
- covers both search modes, since prefix_search honours return_json too
Resolves an adjacency conflict in tests/test_wiki_tools.py: main added the
read_domains_from_credentials_file tests directly above _ask_result, which
this branch gave a continue_offset argument. Both sides are kept.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

semantic_search: truncation warning misses SMW's server-side result cap

1 participant