Skip to content

api-reference.md

Latest commit

 

History

History
460 lines (386 loc) · 19.1 KB

api-reference.md

File metadata and controls

460 lines (386 loc) · 19.1 KB
Raw

API Reference

II-42 has one index lifecycle and one overloaded ii42_query(...) product family. Scalar overloads compose semantic ranking with ordinary table SQL; overloads with an explicit k return hit rows. Index options select exact BM25 or semantic-enabled unified posting internally.

Application API

Create

CREATE INDEX index_name
ON table_name USING ii42 (column_name [, ...])
WITH (...);

sae = false is the default. sae = true selects the semantic-enabled contract and is eventual-only. See Index Parameters.

Single-column indexes support int4[], text[], varchar[], text, or varchar. Multicolumn indexes require homogeneous text[], varchar[], text, or varchar columns. The default multicolumn shape fuses columns into one logical document. field_aware = true preserves field identity for BM25; it can also be combined with sae = true to preserve both lexical and semantic field identity in one unified posting index.

SAE indexes may declare ordinary table columns with PostgreSQL INCLUDE. Included columns are non-scoring scope dimensions: they do not enter BM25, semantic encoding, field weights, or document length. On a converged root, exact scalar eq/in/range and array overlap predicates can resolve through same-root scope postings; string columns also support ILIKE. A compatible published scope remains usable while linked-L0 and sealed delta work converges. Returned candidates are rechecked against current heap rows, while post-baseline matches may be temporarily absent. Unsupported predicate shapes fall back to the current SQL predicate resolver. When every structured predicate is scope-backed, the bounded serving-scope route does not materialize the complete current matching universe; after current-row recheck it may return fewer than k until background convergence publishes a newer scope.

See Getting Started for the canonical first-use flow and Supported Input Types for every supported index shape.

Planner-Native Semantic Search

ii42_query(index_name regclass, query_text text) RETURNS real

ii42_query(
    index_name regclass,
    query_text text,
    field_names text[],
    field_weights real[]
) RETURNS real

The two- and four-argument scalar ii42_query(...) overloads are planner markers, not row-local scoring functions. They must appear as the only descending sort key over one base table with a bounded LIMIT. PostgreSQL owns final WHERE evaluation under the statement snapshot. For predicates without an eligible same-root scope, PostgreSQL supplies the complete visible TID subset and the custom scan ranks inside it. Eligible INCLUDE predicates may instead use a compatible published scope baseline. Every returned row is rechecked against the current snapshot, but post-baseline matches may be absent when the bounded probe fills the limit. If recheck cannot fill the requested limit, planner-native execution discards the probe and falls back to the complete current subset.

SELECT source.*,
       ii42_query('docs_search_idx'::regclass, 'graph retrieval') AS score
FROM docs AS source
WHERE source.publish_date >= DATE '2026-01-01'
  AND source.categories && ARRAY['cs.LG']
ORDER BY score DESC
LIMIT 20;

The current path rejects joins, row-dependent marker arguments, ascending or secondary ordering, unbounded ranking, row locking, WITH TIES, RLS, and partitioned-parent global ranking. The marker fails closed if PostgreSQL cannot use the II42 custom executor.

Explicit Hit Search

ii42_query(
    index_name regclass,
    query_text text,
    k int4,
    weight_mask real[] DEFAULT NULL,
    lowercase boolean DEFAULT NULL,
    stopwords text[] DEFAULT NULL,
    stem_english boolean DEFAULT NULL,
    fold_diacritics boolean DEFAULT NULL
)
RETURNS SETOF ii42_result_hit

Field-aware indexes also expose an overload through the same product name:

ii42_query(
    index_name regclass,
    query_text text,
    field_names text[],
    field_weights real[],
    k int4
)
RETURNS SETOF ii42_result_hit

SAE-enabled indexes expose predicate-defined subset-ranking overloads:

ii42_query(
    index_name regclass,
    query_text text,
    filters jsonb,
    k int4 DEFAULT 10
)

ii42_query(
    index_name regclass,
    query_text text,
    field_names text[],
    field_weights real[],
    filters jsonb,
    k int4 DEFAULT 10
)

ii42_query(
    index_name regclass,
    query_text text,
    allowed_tids tid[],
    k int4 DEFAULT 10
)

ii42_query(
    index_name regclass,
    query_text text,
    field_names text[],
    field_weights real[],
    allowed_tids tid[],
    k int4 DEFAULT 10
)

Each predicate-defined overload returns SETOF ii42_result_hit.

The JSON overload is the explicit structured-predicate API. Filters are ANDed by column. Each column specifies exactly one operation: eq, in, overlap, ilike, ilike_any, or range; range accepts gt, gte, lt, and lte bounds. ilike accepts one PostgreSQL pattern and ilike_any accepts an OR-list of patterns; on a string-array column they match individual elements.

An eligible same-root scope may rank a bounded published baseline and recheck every returned row against the active snapshot. Returned membership is current, but the matching universe may omit post-baseline rows and a fully scope-backed request may return fewer than k. Without an eligible scope, II42 may use a bounded ranked-prefix probe; if that probe is insufficient, PostgreSQL resolves the complete current predicate set and can use ordinary B-tree, GIN, BRIN, or suitable trigram/expression indexes. For a nonempty overlap operand, the SQL resolver includes the equivalent cardinality(column) > 0 condition so a matching partial GIN index remains usable. All routes select top-k inside their selected candidate universe. Scope and SQL-subset routes are predicate-first; the bounded prefix route is explicitly a global overfetch probe followed by current predicate recheck, not an exact resumable filtered iterator. Collection operands are limited to 4,096 values; use a selective table predicate rather than transporting a large application-owned ID list.

The tid[] overload remains a low-level exact-membership boundary: no result can come from outside the supplied set. Ranking still uses the index's selected exact or bounded-approximate route, so a stale accelerator may omit an allowed post-baseline row. Generate TIDs in the same statement and do not persist them across table rewrites. Null and empty sets return no rows. By default filter metadata remains PostgreSQL-owned. An SAE index can opt frequently used exact dimensions into its existing root with INCLUDE; this adds no generation, worker, compaction, or fold lifecycle. Predicate-defined filtered top-k is reserved for SAE-enabled unified indexes; pure BM25 continues to use PostgreSQL's ordinary predicate path.

The default overload searches all fields with weight 1.0. The field-aware overload searches the selected unique fields and applies each weight to the field's complete unified contribution: sum(weight * (BM25 + SAE)). Weights must be finite and non-negative.

ii42_result_hit contains:

Field Meaning
ctid Physical row identity for joining to the indexed table.
doc_id Index-local document slot. Do not persist it as row identity.
score Query-time score produced by the selected index contract.

Example:

SELECT source.id, source.body, hit.score
FROM ii42_query(
    'docs_body_idx'::regclass,
    'postgres index maintenance',
    20
) AS hit
JOIN docs AS source ON source.ctid = hit.ctid
ORDER BY hit.score DESC, source.id;

For BM25, omitted normalization arguments inherit the named index options. For sae = true, all BM25-only overrides and weight_mask are rejected; normalization belongs to the model checkout. Field weights remain valid on a semantic-enabled field-aware index because they scale complete field-local lexical and semantic evidence after model encoding.

weight_mask is an exact-BM25 diagnostic surface and is inherently a document-slot-sized operation. It is therefore admitted only when the physical index and all other active fallback snapshots fit the finite positive per-backend ii42.workspace_cache_bytes budget. Ordinary queries should omit it and use the shared resident-fold/page-native route.

The caller needs SELECT on the indexed table. ii42_query(...) rejects row-level-security tables and partitioned parent indexes. An explicit TID set provides predicate-defined subset ranking, but it is not an RLS policy boundary and cannot combine independently ranked child corpora into one global top-k.

Inspect

ii42_index_options(index_name regclass) RETURNS jsonb
ii42_index_status(index_name regclass) RETURNS jsonb
ii42_index_audit(index_name regclass) RETURNS jsonb
ii42_index_details(index_name regclass) RETURNS TABLE (
    index_name regclass,
    source_type text,
    docs int8,
    index_bytes int8,
    pages int8,
    stale bool,
    consistency text,
    rebuilds int8,
    pending_writes int8,
    pending_deletes int8,
    delta_records int8,
    delta_bytes int8
)
  • ii42_index_options(...) reports effective type, source shape, reloptions, and semantic configuration. Semantic indexes report semantic_impact_precision, semantic_alpha_mass, and an exact or approximate semantic_accuracy_profile; defaults are f32 and 1.0. The packed semantic authority always uses 64-document blocks; precision is selectable per index, but block geometry is not.
  • ii42_index_status(...) is the application readiness surface. Check query_ready and blocker rather than interpreting internal counters. query_usable and query_ready mean the exact fallback remains correct. For semantic indexes, performance_ready additionally requires either a serving semantic accelerator or a current exact-root resident fold. A serving accelerator may report state=ready_baseline_delta and baseline_current=false: the immutable baseline remains authoritative while bounded overfetch and current-row validation reject stale candidates. Newer rows may be temporarily omitted under the declared approximate profile. This is an expected online state, not a fallback or an invalid accelerator, and it has no maximum serving age. Small debt is still scheduled periodically by ii42.maintenance_low_debt_interval_ms; record and byte high-water marks bypass that interval. Status exposes periodic_refresh_eligible separately from immediate refresh_due. Failed publication or a concurrent builder is retried no sooner than ii42.maintenance_timer_interval_ms; that internal cooldown applies only to accelerator construction and never invalidates the serving baseline. It becomes state=ready and baseline_current=true after sealing and derived publication catch up. A compatible manifest seal does not clear query_metadata_warm; that marker follows the unchanged accelerator directory and baseline sequence while exact manifest projections converge in the background. A false marker in this state therefore indicates startup, shared-runtime admission pressure, or a genuine serving-authority change, not ordinary baseline drift. This keeps the established query-readiness contract while preventing a relation-sized semantic fallback from passing performance qualification. performance_blocker identifies outstanding convergence work even when the existing baseline remains usable. If auto_preload > 0, it also reports query_metadata_not_warm until accelerator metadata is warm or an exact resident fold is current. Its generation projection is bounded to root metadata even on relation-sized indexes; diagnostics_complete=false and null reachability/reclaim fields mean that no full storage walk was requested. A semantic accelerator reports authenticated fixed-header metadata and directory_bytes; aggregate artifact bytes remains null because calculating it requires walking every child reference.
  • ii42_index_audit(...) is the explicit heavy integrity surface. It validates the complete generation closure and SHA-256 hashes every SAE model artifact. Do not call it from readiness polling or request paths.
  • ii42_index_details(...) exposes operator-oriented root, mutation, maintenance, and builder details.

These functions enforce access to the indexed table. Status validates bounded manifest/runtime identity but deliberately reports model_artifacts_valid as null. Audit validates server-local model bytes under the guarded extension-owner boundary; callers cannot supply arbitrary paths.

Maintain

ii42_index_refresh(index_name regclass)
ii42_index_maintain(index_name regclass)
ii42_index_try_maintain(index_name regclass)
ii42_index_maintain_due(max_indexes integer DEFAULT 1)
  • ii42_index_maintain(...) may wait and performs one needed bounded action or returns a no-op.
  • ii42_index_try_maintain(...) avoids waiting on a busy publication boundary and returns retryable no-op results when necessary. This is non-blocking lock admission, not a deadline on the maintenance action once admitted.
  • ii42_index_maintain_due(...) uses the same native selector for a bounded number of automatic-policy indexes. It is revoked from PUBLIC and is meant for a trusted maintenance role.
  • ii42_index_refresh(...) is an explicit operator refresh surface. Use REINDEX when options or model contract changed.

Per-index mutation and maintenance require index ownership. Semantic completion, sealing, compaction, fold, and reclamation all act on the same page-native v3 root and linked L0.

Drop

DROP INDEX index_name;

PostgreSQL relation lifecycle is authoritative for both modes. All II-42 payloads are owned by the index relation, so there is no semantic side object or external cleanup step.

Text Utilities

The public value-local helpers are:

  • ii42_tokenize_text(text, ...);
  • ii42_normalize_tokens(text[], ...);
  • ii42_highlight(text[] | text | varchar, query_text, ...);
  • ii42_snippet(text[] | text | varchar, query_text, ...).

They operate on supplied values. They do not perform index retrieval.

BM25 Planner Surface

BM25 indexes support PostgreSQL operator integration:

  • tokens @@ 'query text' for text[] and varchar[] predicates;
  • value @@@ ii42_prepared_query(...) for owner diagnostics and scalar text;
  • ORDER BY value <=> query_tokens ASC LIMIT k for index-ordered retrieval.

@@ is a boolean predicate, not a ranking API. <=> has index ranking semantics only when PostgreSQL chooses an actual ii42 index scan. Application code that needs an explicit hit set can use ii42_query(...); semantic table queries should prefer ii42_query(...).

These operators are BM25 surfaces. They do not dispatch to semantic scoring.

Owner-Only BM25 Diagnostics

The extension owner can use exact BM25 functions for regression, benchmark, and implementation diagnostics:

  • ii42_query_ids(...);
  • ii42_query_tokens(...);
  • ii42_prepared_query(...), ii42_order_tokens(...), and local match/score helpers;
  • single-index token-level field-weight helpers.

These functions are revoked from PUBLIC. Exact BM25 rowset and scoring helpers reject sae = true indexes. None may become an alternate application API. Use PostgreSQL EXPLAIN (FORMAT JSON) directly when validating planner paths. II-42 does not wrap or execute caller-supplied SQL text.

Public Composition APIs

The ii42_fusion_* family combines top-k results from multiple independently maintained ii42 indexes. The ii42_hybrid_* family combines II-42 candidates with externally retrieved candidates such as vector-index distances. These families are granted to PUBLIC; each II-42 source still enforces source-table SELECT through ii42_query(...).

Fusion hit rows are keyed by ctid, so both ii42_fusion_* and ii42_hybrid_* sources must belong to the same base table and SQL snapshot. For different tables or partitions, map hits to stable document IDs and aggregate in application SQL outside these helpers. Both families fuse finite source prefixes, not the complete matching universes; candidate limits can affect recall. See Multi-Index Fusion and Hybrid Fusion Engine.

Composition is a product layer above single-index retrieval. It does not alter an index's unified posting layout, mutation lifecycle, maintenance policy, or native scorer. Use planner-native SQL or ii42_query(...) when one index is sufficient; use fusion or hybrid APIs only when the product intentionally combines independent indexes or retrieval engines.

Runtime And Residency Diagnostics

Runtime and residency surfaces include:

ii42_index_runtime_state(index_name regclass)
ii42_index_runtime_state_json(index_name regclass)
ii42_index_preload(index_name regclass)
ii42_runtime_cache_clear()
ii42_runtime_service_status()

The state functions require SELECT on the indexed table. ii42_index_preload(...) requires index ownership, while cache clearing and runtime-service status are extension-owner diagnostics revoked from PUBLIC. These functions report or control checked-root markers, relation page warming, optional HOT_FOLD and exact-root resident-fold state, bounded workspace, and shared runtime state. They do not expose posting storage or a second mutation authority. ii42_index_runtime_state_json(...) reports resident_fold_current, resident_fold_entries, and resident_fold_bytes. The text and JSON forms use the same C snapshot collector; JSON diagnostics do not parse the human-readable state string.

ii42_index_preload(...) first attempts to publish one pointer-free exact-root fold when the index is converged and the complete image fits ii42.shared_runtime_size. Its result then reports tier=shared_resident_fold, prewarm_scope=exact, and resident_bytes. Otherwise it performs exact relation-page warming within ii42.prewarm_max_bytes or bounded roots-and-payload warming for a larger index. Query readers still validate the checked root; durable authority never moves out of the index relation.

ii42_runtime_cache_clear() is revoked from PUBLIC. Clearing disposable residency may change cold latency but cannot change results.

Privilege Summary

Surface Intended caller
ii42_query Application role with source-table SELECT.
ii42_fusion_*, ii42_hybrid_* Application role; source queries retain their own authorization.
Options/status/details Role allowed to inspect the source relation.
Per-index maintenance Index owner.
PostgreSQL DROP INDEX Index owner under the normal PostgreSQL lifecycle.
ii42_index_maintain_due Trusted maintenance role.
Text utilities Application role.
Runtime state Role allowed to inspect the source relation.
Per-index preload Index owner.
Exact BM25, model, cache, and runtime internals Extension owner/diagnostics.

Internal functions ending in _internal are implementation boundaries. Do not grant them to application roles.

See Also