Feat/sdbql set operations

CHANGELOG.md

@@ -108,6 +108,27 @@
 
 ### Features
 
+* **SDBQL gains set operations, recursive CTEs, `RETURN DISTINCT`, `COLLECT
+  … KEEP`, the `NONE` quantifier, and standalone `OFFSET`.** Query blocks can
+  be combined with `UNION [ALL]`, `INTERSECT`, and `EXCEPT` (`FOR … RETURN …
+  UNION ALL FOR … RETURN …`; either side may be parenthesized; duplicates are
+  removed except for `UNION ALL`). Chains follow SQL precedence — `INTERSECT`
+  binds tighter than `UNION`/`EXCEPT`, which chain left to right — and rows
+  are compared by value, the same equality `UNION()`/`INTERSECTION()` use.
+  `WITH RECURSIVE name AS (<anchor> UNION ALL <step>)` iterates the step until
+  it stops producing rows — inside the step the CTE name is bound to the rows
+  of the *previous* iteration (hierarchies, org charts, transitive closures),
+  with safety caps at 1,000 iterations / 1M rows. `RETURN DISTINCT expr`
+  deduplicates result rows (first occurrence wins). `COLLECT … INTO g KEEP v1,
+  v2` restricts which in-scope variables are stored in the group arrays (an
+  unknown name is an error). `NONE(x IN arr SATISFIES cond)` — and the
+  function form `NONE(arr, x -> cond)` — is true when no element satisfies.
+  `OFFSET n` works alone or as `LIMIT n OFFSET m`. A CTE declared before a set
+  operation binds in every operand, and operands, CTE bodies and subqueries
+  are all executed as full query blocks — their own `WITH`, pre-`FOR` `LET`s,
+  `SORT`/`LIMIT` and nested set operations all apply. Permission checks, query
+  caching, cache invalidation, the long-running-query gate and
+  `has_mutations()` all see through set-operation operands and CTE bodies.
 * **Query-driven auto-indexes (opt-in).** Collections with `autoIndex: true`
   (or unset + `SOLIDB_AUTO_INDEX=1`) create a persistent `_auto_{field}`
   index the first time a `FOR` + `FILTER` equality/range miss would have used

admin/app/views/query/_explain.html.slv

@@ -97,8 +97,13 @@
           </span>
         <% end %>
         <% if !limit_info.nil? %>
+          <%# A standalone OFFSET reports no count %>
+          <% limit_offset = limit_info["offset"] ?? 0 %>
+          <% limit_count = limit_info["count"] %>
+          <% limit_text = limit_count.nil? ? "offset " + str(limit_offset) : "limit " + str(limit_count) %>
+          <% limit_text = limit_text + " offset " + str(limit_offset) if !limit_count.nil? && limit_offset > 0 %>
           <span class="console-label border border-rose-800 px-1.5 py-0.5 text-rose-400">
-            limit <%= limit_info["count"] ?? 0 %><% if (limit_info["offset"] ?? 0) > 0 %> offset <%= limit_info["offset"] %><% end %>
+            <%= limit_text %>
           </span>
         <% end %>
       </div>

doc/app/views/docs/changelog.html.slv

@@ -17,6 +17,7 @@
         Security hardening, SDBQL function completeness, and <strong class="text-white">one product breaking change:</strong> the client-facing job and cron queue is removed. Application background jobs now live in the Soli framework. <strong class="text-white">Triggers keep working</strong>, on the same machinery as before. Also breaking if you relied on them: default bind is loopback, replication TCP requires a keyfile, <code>LENGTH</code> on a string is character count (not a collection count), installing Lua requires Admin, and API keys must declare a role.
       </p>
       <ul class="space-y-3 text-gray-400 text-sm leading-relaxed">
+        <li><strong class="text-white">New SDBQL syntax: set operations, recursive CTEs, <code>RETURN DISTINCT</code>, <code>COLLECT &hellip; KEEP</code>, <code>NONE</code>, standalone <code>OFFSET</code></strong> &mdash; combine query blocks with <code>UNION [ALL]</code>, <code>INTERSECT</code> and <code>EXCEPT</code> (either side may be parenthesized; duplicates removed except for <code>UNION ALL</code>; chains follow SQL precedence, so <code>INTERSECT</code> binds tighter than <code>UNION</code>/<code>EXCEPT</code>). <code>WITH RECURSIVE name AS (&lt;anchor&gt; UNION ALL &lt;step&gt;)</code> iterates until the step stops producing rows, binding the CTE name inside the step to the rows of the previous iteration (hierarchies, org charts, transitive closures), capped at 1,000 iterations / 1M rows. <code>RETURN DISTINCT expr</code> deduplicates rows. <code>COLLECT &hellip; INTO g KEEP v1, v2</code> restricts which variables land in the group arrays. <code>NONE(x IN arr SATISFIES cond)</code> and <code>NONE(arr, x -&gt; cond)</code> are true when nothing satisfies. <code>OFFSET n</code> works alone or as <code>LIMIT n OFFSET m</code>.</li>
         <li><strong class="text-white">Native HTTPS termination</strong> &mdash; new <code class="text-cyan-400">--tls-cert</code> / <code class="text-cyan-400">--tls-key</code> flags terminate TLS in the server itself (rustls, no OpenSSL). On the multiplexed port the listener sniffs for a TLS handshake and terminates only when the client offers one, so HTTP and the native driver protocol run inside the tunnel for TLS-capable clients while the plaintext-only driver SDKs and sync/cluster peers keep connecting; <code class="text-cyan-400">SOLIDB_TLS_REQUIRE=1</code> refuses plaintext there. Both flags are required together.</li>
         <li><strong class="text-white">Per-client API rate limiting</strong> &mdash; only <code>/auth/login</code> was throttled before; now the whole router has a per-client-IP sliding-window limiter (default 600 requests / 60s) answering <code>429</code> with <code>Retry-After</code> before any handler work. Internal cluster traffic and CORS preflights are exempt. Configure with <code class="text-cyan-400">SOLIDB_API_RATE_LIMIT</code> (0 disables) and <code class="text-cyan-400">SOLIDB_API_RATE_WINDOW_SECS</code>.</li>
         <li><strong class="text-white">Driver queries have a timeout</strong> &mdash; HTTP capped execution at 30s but the binary protocol ran queries unbounded on a runtime thread. Driver <code>Query</code> and <code>Explain</code> now run under the same 30s cap, on the same long-running-query gate HTTP uses so point reads stay inline.</li>

doc/app/views/docs/sdbql-aggregations.html.slv

@@ -103,10 +103,33 @@
               </div>
             </div>
 
+            <!-- KEEP restriction -->
+            <div class="bg-gray-800/30 rounded-xl border border-white/5 overflow-hidden">
+              <div class="p-5 border-b border-white/5 bg-white/5">
+                <h3 class="text-lg font-bold text-orange-300 flex items-center">Restricting Group Items with KEEP</h3>
+              </div>
+              <div class="p-5 space-y-4">
+                <p class="text-sm text-gray-400">
+                  With <code>INTO groups</code>, each group array normally stores every variable in
+                  scope. Add <code class="text-orange-300">KEEP v1, v2</code> to store only those
+                  variables — smaller results, no leaked document bodies.
+                </p>
+                <pre><code class="language-sql text-sm">FOR u IN users
+  LET n = u.name
+  COLLECT city = u.city INTO g KEEP n
+  RETURN { city, names: g[*].n }</code></pre>
+                <p class="text-xs text-gray-500">
+                  Without <code>KEEP</code>, each item of <code>g</code> would also carry <code>u</code>.
+                  Every name listed must be a variable in scope at the <code>COLLECT</code> — an
+                  unknown name is an error, not an empty group item.
+                </p>
+              </div>
+            </div>
+
             <!-- Syntax Reference -->
             <div class="bg-gray-950/50 rounded-lg border border-white/5 p-4 mt-8">
               <code class="text-sm text-gray-300 font-mono block whitespace-pre overflow-x-auto">COLLECT <span class="text-orange-400">variable</span> = <span class="text-purple-400">expression</span> [, ...]
-[INTO <span class="text-orange-400">groupVariable</span>]
+[INTO <span class="text-orange-400">groupVariable</span> [KEEP <span class="text-orange-400">var1</span>, <span class="text-orange-400">var2</span>, ...]]
 [WITH COUNT INTO <span class="text-orange-400">countVariable</span>]
 [AGGREGATE <span class="text-orange-400">variable</span> = <span class="text-blue-400">FUNC</span>(<span class="text-purple-400">expression</span>) [, ...]]</code>
             </div>

doc/app/views/docs/sdbql-cte.html.slv

@@ -135,10 +135,50 @@ FOR s IN summary
               </div>
             </div>
 
+            <div class="p-6">
+              <h3 class="text-lg font-bold text-white mb-4">Recursive CTE</h3>
+              <p class="text-gray-400 mb-4">
+                <code class="text-orange-400">WITH RECURSIVE</code> walks hierarchical data
+                (org charts, category trees, bill of materials, transitive closures) in pure
+                SDBQL. The body must have the shape
+                <code class="text-orange-400">&lt;anchor query&gt; UNION ALL &lt;recursive step&gt;</code>.
+                The anchor runs once; then the step runs repeatedly until it stops producing rows.
+              </p>
+              <p class="text-gray-400 mb-4">
+                Inside the step, the CTE name is bound to the rows produced by the
+                <strong>previous iteration</strong> — use it to expand one level at a time.
+                Recursion is capped at 1,000 iterations and 1M rows.
+              </p>
+              <div class="bg-gray-950/50 rounded-lg overflow-hidden border border-white/5 mb-4">
+                <div class="bg-black/20 px-4 py-2 border-b border-white/5">
+                  <span class="text-xs text-gray-400">Example: everyone reporting to alice, any depth</span>
+                </div>
+                <pre><code class="language-sql text-xs bg-gray-950/50 p-3 rounded border border-white/5">WITH RECURSIVE reports AS (
+  -- Anchor: the root of the hierarchy
+  FOR e IN employees
+    FILTER e._key == 'alice'
+    RETURN e._key
+  UNION ALL
+  -- Step: find direct reports of the previous level
+  FOR m IN employees
+    FILTER m.manager IN reports
+    RETURN m._key
+)
+FOR x IN reports
+  RETURN x</code></pre>
+              </div>
+              <ul class="text-gray-400 space-y-1 text-sm">
+                <li><code class="text-orange-400">RECURSIVE</code> applies to every CTE in the WITH list (standard SQL semantics)</li>
+                <li>Steps must be joined with <code class="text-orange-400">UNION ALL</code>; other set operators are rejected on a recursive CTE body</li>
+                <li>The anchor and each step are full query blocks: they may have their own <code>LET</code>s, <code>SORT</code>/<code>LIMIT</code>, and nested <code>WITH</code></li>
+                <li>A step that never stops producing new rows ends with an iteration-limit error rather than hanging. Cyclic data (a → b → a) is exactly that case: filter out rows you have already visited in the step, or carry a depth and stop at it</li>
+              </ul>
+            </div>
+
             <div class="p-6">
               <h3 class="text-lg font-bold text-white mb-4">Syntax</h3>
               <div class="bg-gray-950/50 rounded-lg overflow-hidden border border-white/5">
-                <pre><code class="language-sql text-xs bg-gray-950/50 p-3 rounded border border-white/5">WITH
+                <pre><code class="language-sql text-xs bg-gray-950/50 p-3 rounded border border-white/5">WITH [RECURSIVE]
   cte_name [(col1, col2, ...)] AS ( query ),
   cte_name2 [(col1, col2, ...)] AS ( query )
 query_body</code></pre>

doc/app/views/docs/sdbql-operators.html.slv

@@ -354,7 +354,10 @@ SATISFIES tag != "deprecated"</code></pre>
                   <span class="text-xs text-gray-400">No matches</span>
                 </div>
                 <pre><code class="language-sql text-xs">FILTER NONE user IN doc.admins
-SATISFIES user.status == "inactive"</code></pre>
+SATISFIES user.status == "inactive"
+
+-- Function form
+RETURN NONE(doc.admins, u -> u.status == "inactive")</code></pre>
                 <div class="mt-3 text-xs text-gray-400">
                   True if <strong>zero</strong> elements satisfy the condition.
                 </div>

doc/app/views/docs/sdbql-syntax.html.slv

@@ -44,8 +44,8 @@
 FOR variable IN collection
   FILTER expression
   SORT field [ASC|DESC]
-  LIMIT [offset,] count
-  RETURN expression</code></pre>
+  LIMIT [offset,] count | OFFSET n [LIMIT m]
+  RETURN [DISTINCT] expression</code></pre>
             </div>
 
             <div class="grid grid-cols-1 md:grid-cols-2 gap-6">
@@ -75,13 +75,94 @@ FOR variable IN collection
                   <span class="bg-indigo-500/20 p-1.5 rounded mr-2 text-xs">RETURN</span>
                   Projection
                 </h3>
-                <p class="text-sm text-gray-400">Constructs the result documents.</p>
+                <p class="text-sm text-gray-400">Constructs the result documents. Add <code class="text-indigo-300">DISTINCT</code> to remove duplicate rows.</p>
               </div>
             </div>
           </div>
         </div>
       </section>
 
+      <!-- Set Operations -->
+      <section id="set-operations" class="scroll-mt-24">
+        <div class="bg-gray-800/30 rounded-xl border border-white/5 overflow-hidden">
+          <div class="p-5 border-b border-white/5 bg-white/5">
+            <h3 class="text-lg font-bold text-emerald-300 flex items-center">
+              <span class="bg-emerald-500/20 text-emerald-400 text-xs font-mono px-2 py-1 rounded mr-3">UNION / INTERSECT / EXCEPT</span>
+              Set Operations
+            </h3>
+          </div>
+          <div class="p-5 space-y-4">
+            <p class="text-sm text-gray-400">
+              Combine query blocks with <code class="text-emerald-300">UNION [ALL]</code>,
+              <code class="text-emerald-300">INTERSECT</code>, or
+              <code class="text-emerald-300">EXCEPT</code>. Either side may be parenthesized,
+              and rows are compared by value — <code>1</code> and <code>1.0</code> are the
+              same row, like in <code>UNION()</code> and <code>INTERSECTION()</code>.
+            </p>
+            <p class="text-sm text-gray-400">
+              Chains follow SQL precedence: <code class="text-emerald-300">INTERSECT</code>
+              binds tighter than <code class="text-emerald-300">UNION</code> and
+              <code class="text-emerald-300">EXCEPT</code>, which chain left to right. So
+              <code>a UNION b INTERSECT c</code> is <code>a UNION (b INTERSECT c)</code>, and
+              <code>a EXCEPT b EXCEPT c</code> is <code>(a EXCEPT b) EXCEPT c</code>.
+              Parenthesize to group differently.
+            </p>
+            <div class="grid grid-cols-1 gap-4">
+              <div>
+                <h4 class="text-xs font-bold text-gray-400 uppercase tracking-wider mb-2">UNION — both sides, duplicates removed (UNION ALL keeps them)</h4>
+                <pre><code class="language-sql text-sm">FOR u IN users FILTER u.age < 26 RETURN u._key
+UNION
+FOR u IN users FILTER u.city == 'London' RETURN u._key
+
+FOR i IN 1..3 RETURN i UNION ALL FOR i IN 1..3 RETURN i   -- [1, 2, 3, 1, 2, 3]</code></pre>
+              </div>
+              <div>
+                <h4 class="text-xs font-bold text-gray-400 uppercase tracking-wider mb-2">INTERSECT — rows present in both sides; EXCEPT — left rows absent from the right side</h4>
+                <pre><code class="language-sql text-sm">FOR u IN paris_users RETURN u._key
+INTERSECT
+FOR o IN newsletter RETURN o.user_key
+
+FOR u IN users RETURN u._key
+EXCEPT
+FOR b IN banned RETURN b.user_key
+
+-- Explicit grouping on either side
+(FOR u IN eu_users RETURN u._key UNION FOR u IN us_users RETURN u._key)
+EXCEPT
+FOR b IN banned RETURN b.user_key</code></pre>
+              </div>
+            </div>
+          </div>
+        </div>
+      </section>
+
+      <!-- DISTINCT and OFFSET -->
+      <section id="distinct-offset" class="scroll-mt-24">
+        <div class="bg-gray-800/30 rounded-xl border border-white/5 overflow-hidden">
+          <div class="p-5 border-b border-white/5 bg-white/5">
+            <h3 class="text-lg font-bold text-cyan-300 flex items-center">
+              <span class="bg-cyan-500/20 text-cyan-400 text-xs font-mono px-2 py-1 rounded mr-3">DISTINCT / OFFSET</span>
+              Result Shaping
+            </h3>
+          </div>
+          <div class="p-5 space-y-4">
+            <p class="text-sm text-gray-400">
+              <code class="text-cyan-300">RETURN DISTINCT expr</code> removes duplicate result rows
+              (first occurrence wins, order preserved). <code class="text-cyan-300">OFFSET n</code> skips rows —
+              standalone or after a limit as <code class="text-cyan-300">LIMIT count OFFSET n</code>.
+            </p>
+            <pre><code class="language-sql text-sm">-- Unique cities, no COLLECT needed
+FOR u IN users SORT u.city RETURN DISTINCT u.city
+
+-- Second page of 10, SQL style
+FOR d IN docs SORT d.created_at DESC LIMIT 10 OFFSET 10 RETURN d
+
+-- Skip without limiting
+FOR u IN users SORT u.age OFFSET 5 RETURN u._key</code></pre>
+          </div>
+        </div>
+      </section>
+
       <!-- Range Expressions -->
       <section id="range" class="scroll-mt-24">
         <div class="bg-gray-800/30 rounded-xl border border-white/5 overflow-hidden">

doc/public/js/docs-search.js

@@ -248,6 +248,10 @@ var SDBQL_KEYWORDS = [
 ["RETURN","RETURN expression","Shape and return the result of the query.","/docs/sdbql-syntax#syntax"],
 ["SORT","SORT expr ASC | DESC","Order results by one or more expressions.","/docs/sdbql-syntax#syntax"],
 ["LIMIT","LIMIT offset?, count","Restrict the number of results, with an optional offset.","/docs/sdbql-syntax#syntax"],
+["OFFSET","OFFSET n [LIMIT count]","Skip rows before returning — standalone or after a limit (LIMIT 10 OFFSET 20).","/docs/sdbql-syntax#distinct-offset"],
+["UNION","query1 UNION [ALL] query2","Combine two query blocks; duplicates removed unless UNION ALL.","/docs/sdbql-syntax#set-operations"],
+["INTERSECT","query1 INTERSECT query2","Rows present in both sides of a set operation, deduplicated.","/docs/sdbql-syntax#set-operations"],
+["EXCEPT","query1 EXCEPT query2","Rows of the left side not present in the right side.","/docs/sdbql-syntax#set-operations"],
 ["LET","LET name = expression","Bind a variable or a subquery for reuse.","/docs/sdbql-syntax#let-subqueries"],
 ["COLLECT","COLLECT key = expr AGGREGATE …","Group rows and aggregate — SDBQL's GROUP BY.","/docs/sdbql-aggregations"],
 ["AGGREGATE","AGGREGATE total = SUM(x)","Compute aggregates within a COLLECT group.","/docs/sdbql-aggregations"],
@@ -272,7 +276,10 @@ var SDBQL_KEYWORDS = [
 ["WITH","WITH name AS ( subquery )","Define a named CTE (common table expression).","/docs/sdbql-cte"],
 ["OUTBOUND","FOR v IN OUTBOUND start edges","Traverse graph edges in the outbound direction.","/docs/sdbql-graphs"],
 ["INBOUND","FOR v IN INBOUND start edges","Traverse graph edges in the inbound direction.","/docs/sdbql-graphs"],
-["DISTINCT","RETURN DISTINCT expr","Return only unique results.","/docs/sdbql-syntax#syntax"],
+["DISTINCT","RETURN DISTINCT expr","Return only unique results.","/docs/sdbql-syntax#distinct-offset"],
+["RECURSIVE","WITH RECURSIVE t AS (anchor UNION ALL step)","Walk hierarchies: the step re-runs on the previous iteration's rows until empty.","/docs/sdbql-cte"],
+["NONE","NONE x IN arr SATISFIES cond","True when no element satisfies. Function form: NONE(arr, x -> cond).","/docs/sdbql-operators#op-none"],
+["KEEP","COLLECT … INTO g KEEP v1, v2","Restrict which variables are stored in COLLECT group arrays.","/docs/sdbql-aggregations"],
 ["LIKE","FILTER str LIKE \"%foo%\"","Pattern-match a string with % / _ wildcards.","/docs/sdbql-operators"]
 ];