fix(search): read truncation from SMW's query-continue-offset - #171
Open
LukasGold wants to merge 3 commits into
Open
fix(search): read truncation from SMW's query-continue-offset#171LukasGold wants to merge 3 commits into
LukasGold wants to merge 3 commits into
Conversation
- replace the count-versus-limit check, blind to $smwgQMaxLimit - stop reporting a complete set of exactly 'limit' results as truncated - add SearchParam.return_meta yielding SemanticSearchResult per query - report query, titles, count, truncated and next_offset to callers - leave return_json untouched, its raw dict already holds the offset
Contributor
Release previewMerging this PR would release v2.3.4 (current: Changelog preview (truncated)## v2.3.4 (2026-09-08)
Preview via python-semantic-release and conventional commits. |
- return_json and return_meta yield dicts and objects, not page titles - raise ValueError instead of failing later inside GetPageParam - covers both search modes, since prefix_search honours return_json too
Resolves an adjacency conflict in tests/test_wiki_tools.py: main added the read_domains_from_credentials_file tests directly above _ask_result, which this branch gave a continue_offset argument. Both sides are kept.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #162.
Changes
semantic_searchreads truncation from the top-levelquery-continue-offsetkey of theaskresponse instead of comparing the result count against the limit.SearchParam.return_meta. When True,semantic_searchreturns oneSemanticSearchResultper query instead of a flat list of titles.SemanticSearchResultmodel withquery,titles,count,truncatedandnext_offset.return_json=Truepath is unchanged.return_jsontakes precedence when both flags are set.WtSite.modify_search_resultsraisesValueErrorfor aSearchParamthat setsreturn_jsonorreturn_meta.Rationale
The old check was
if limit and n >= limit. It was wrong in both directions.It missed truncation. With no limit in force it could never fire, and it could not see SMW's own
$smwgQMaxLimitcap (default 10000, https://www.semantic-mediawiki.org/wiki/Help:$smwgQMaxLimit). A query asking for 50000 and cut at 10000 gives10000 >= 50000, which is false.It also invented truncation. A complete result set of exactly
limitresults was reported as truncated. Measured against https://www.semantic-mediawiki.org withmwclient, using[[Has type::Date]], which has 18 results in total:query-continue-offset[[Has type::Date]]|limit=18[[Has type::Date]]|limit=17The key is present exactly when the result set was cut short, so it replaces the count comparison rather than supplementing it. No fallback is kept, because on a wiki that sends the key the count comparison adds only false positives.
Issue #162 proposed reading
query.meta.hasFurtherResults. That key does not exist in theaskresponse. See the comment on the issue for the measurement.Returning the state, not only warning
#133 adds
osw.service.ops.search, whosesearch_entitiescallssemantic_searchwith the defaultreturn_json=Falseand receives a flat list of titles. The truncation signal was not reachable from there.return_metamakes it reachable without changing the default return shape.Wiring
osw.service.ops.searchto use it stays with #133, sincesrc/osw/service/does not exist on main.The modify_search_results guard
WtSite.modify_search_resultslooks up and edits one page per search result, so it needs page titles. Itsparam.queryaccepts aSearchParam, andreturn_jsonalready made it hand back raw dicts instead, failing later insideGetPageParamwith a type error that names neither the flag nor the method.return_metaadded a second way to reach that state, so both are now rejected up front with a message that names the offending flag.The check covers both search modes, since
prefix_searchhonoursreturn_jsonas well. No working code can be affected:return_metais new in this PR, and areturn_jsonquery already failed on the first result.Tests
44 tests in
tests/test_wiki_tools.py, 9 intests/test_wtsite_modify_search_results.py, 215 in the full unit suite, all passing.New in
tests/test_wiki_tools.py: truncation detected from the offset below the limit, detected withlimit=None, the rawreturn_jsondict keeping the offset,return_metareporting a truncated and a complete result, one result per query, order preserved on the parallel path,countincluding pages dropped as non-existing, andreturn_jsonwinning overreturn_meta.New in
tests/test_wtsite_modify_search_results.py: both flags rejected in both search modes, and a plainSearchParamstill accepted in both. All four rejection cases were confirmed to fail with the guard disabled.Two existing tests changed because they encoded the removed heuristic:
test_semantic_search_truncation_warningnow sends an offset in the mocked response.test_semantic_search_truncation_warning_uses_the_query_limitbecame..._names_the_query_as_sent, since the warning now quotes the query and the offset rather than the requested limit.A new test asserts the false positive is gone: a complete set of exactly
limitresults does not warn.Not addressed
search_titlesandsearch_contentin #133 use MediaWiki'sprefixsearchandsearch, not SMW'sask. They report continuation differently and need separate handling in that PR.