From 132ca554e1baedda4edb59435c86ea820fed75d8 Mon Sep 17 00:00:00 2001 From: ascottDI Date: Tue, 16 Jun 2026 16:53:20 +0100 Subject: [PATCH 01/14] basic scafolding of feature --- di/serverselect/init.q | 4 + di/serverselect/serverselect.md | 213 ++++++++++++++++++++++++++++++++ di/serverselect/serverselect.q | 167 +++++++++++++++++++++++++ di/serverselect/test.csv | 100 +++++++++++++++ 4 files changed, 484 insertions(+) create mode 100644 di/serverselect/init.q create mode 100644 di/serverselect/serverselect.md create mode 100644 di/serverselect/serverselect.q create mode 100644 di/serverselect/test.csv diff --git a/di/serverselect/init.q b/di/serverselect/init.q new file mode 100644 index 00000000..4323911e --- /dev/null +++ b/di/serverselect/init.q @@ -0,0 +1,4 @@ +/ library for selecting backend servers from a registered pool based on servertype or attribute requirements +\l ::serverselect.q + +export:([init;addserverattr;addserver;setserveractive;getserverids;addserversfromtable;getservers]) diff --git a/di/serverselect/serverselect.md b/di/serverselect/serverselect.md new file mode 100644 index 00000000..64189143 --- /dev/null +++ b/di/serverselect/serverselect.md @@ -0,0 +1,213 @@ +# di.serverselect + +Module for registering backend servers and selecting them by servertype or attribute requirements. Extracted from the `.gw` namespace across TorQ's `gatewaylib.q` and `gateway.q`. Designed as the server-selection layer of a gateway — decoupled from query execution and connection management. + +## Usage + +```q +srvsel:use`di.serverselect + +log:use`di.log +logdep:`info`warn`error!(log.info;log.warn;log.error) +srvsel.init[enlist[`log]!enlist logdep] + +/ register servers as they connect +srvsel.addserver[h;`rdb] +srvsel.addserverattr[h;`hdb;`date`sym!(2024.01.01 2024.01.02;`AAPL`MSFT)] + +/ mark inactive on disconnect +srvsel.setserveractive[h;0b] + +/ select by servertype +srvsel.getserverids[`rdb`hdb] + +/ select by attribute requirements +srvsel.getserverids[enlist[`date]!enlist enlist 2024.01.01] +``` + +### Typical gateway integration + +```q +srvsel:use`di.serverselect +log:use`di.log +logdep:`info`warn`error!(log.info;log.warn;log.error) +srvsel.init[enlist[`log]!enlist logdep] + +/ on open - register the connecting server +.z.po:{[h] srvsel.addserverattr[h;getproctype[h];getattributes[h]]} + +/ on close - mark the server inactive +.z.pc:{[h] srvsel.setserveractive[h;0b]} + +/ on query - select target servers then dispatch +serverids:srvsel.getserverids[`rdb`hdb] +``` + +### Bulk registration from a TorQ connection table + +In a TorQ stack, `.servers.SERVERS` is populated by `trackservers.q`. Pass it directly: + +```q +srvsel.addserversfromtable[`rdb`hdb; .servers.SERVERS] + +/ or register all process types +srvsel.addserversfromtable[`ALL; .servers.SERVERS] +``` + +## API + +### `init[deps]` + +Wire injectable dependencies. Must be called before any other function. + +| Key | Required | Type | Description | +|---|---|---|---| +| `` `log `` | yes | dict | Functions keyed `` `info`warn`error ``, each with signature `{[ctx;msg]}` | + +Errors with prefix `di.serverselect:` if `log` is missing or does not contain all three required keys. + +--- + +### `addserverattr[handle;servertype;attributes]` + +Register a server with a handle, servertype, and attribute dictionary. Sets `active:1b` on registration. + +| Parameter | Type | Description | +|---|---|---| +| `handle` | int | Connection handle | +| `servertype` | symbol | Process type (e.g. `` `rdb ``, `` `hdb ``) | +| `attributes` | dict | Key-value attribute pairs (e.g. `` `date`sym!(2024.01.01;`AAPL) ``) | + +```q +srvsel.addserverattr[5i; `hdb; `date`sym!(2024.01.01 2024.01.02; `AAPL`MSFT)] +``` + +--- + +### `addserver[handle;servertype]` + +Register a server with no attributes. Convenience wrapper over `addserverattr`. + +```q +srvsel.addserver[5i; `rdb] +``` + +--- + +### `setserveractive[handle;active]` + +Mark a registered server active or inactive. Call with `0b` on disconnect, `1b` on reconnect. + +| Parameter | Type | Description | +|---|---|---| +| `handle` | int | Connection handle of the server to update | +| `active` | boolean | `1b` = active, `0b` = inactive | + +```q +srvsel.setserveractive[5i; 0b] / server disconnected +srvsel.setserveractive[5i; 1b] / server reconnected +``` + +--- + +### `getserverids[att]` + +Return server IDs matching the given servertype list or attribute requirement dictionary. + +| Parameter | Type | Description | +|---|---|---| +| `att` | symbol list or dict | Servertype list (11h) — returns IDs grouped by type. Attribute dict (99h) — returns IDs satisfying the requirements. | + +**Symbol list path** — validates that all requested types exist and are active: + +```q +srvsel.getserverids[`rdb] +srvsel.getserverids[`rdb`hdb] +``` + +**Attribute dict path** — matches servers whose attribute dictionaries satisfy the requirements. Supports two matching strategies controlled by the reserved key `` `attributetype ``: + +| `attributetype` | Behaviour | +|---|---| +| `` `cross `` (default) | Every combination of attribute values must be coverable — e.g. every date must be available for every sym | +| `` `independent `` | Each attribute value only needs to be matched by at least one server | + +The reserved key `` `besteffort `` (boolean, default `1b`) controls whether a partial match is acceptable. If `0b`, an error is thrown when the requirements cannot be fully satisfied. + +```q +/ cross match (default): find servers covering the full date × sym cross product +srvsel.getserverids[`date`sym!(2024.01.01 2024.01.02; `AAPL`MSFT)] + +/ independent match: each date and each sym just needs one server somewhere +srvsel.getserverids[`date`sym`attributetype!(2024.01.01 2024.01.02; `AAPL`MSFT; `independent)] + +/ scope to a specific servertype then apply attribute filter +srvsel.getserverids[`servertype`date!(`hdb; enlist 2024.01.01)] + +/ strict mode: error if requirements cannot be fully satisfied +srvsel.getserverids[`date`besteffort!(enlist 2024.01.01; 0b)] +``` + +--- + +### `addserversfromtable[proctypes;conntable]` + +Register servers from a connection table, skipping handles already active in the pool. + +| Parameter | Type | Description | +|---|---|---| +| `proctypes` | symbol or symbol list | Process types to register; pass `` `ALL `` to register all | +| `conntable` | table | Must have columns: `w` (int handle), `proctype` (symbol), `attributes` (dict per row) | + +```q +/ TorQ stack — pass .servers.SERVERS directly +srvsel.addserversfromtable[`rdb`hdb; .servers.SERVERS] + +/ custom connection table +conntab:([] w:1i 2i; proctype:`rdb`hdb; attributes:(()!(); `date!enlist 2024.01.01)) +srvsel.addserversfromtable[`ALL; conntab] +``` + +--- + +### `getservers[]` + +Return the current registered server table. + +```q +srvsel.getservers[] +``` + +Schema: + +| Column | Type | Description | +|---|---|---| +| `serverid` | int (keyed) | Unique auto-assigned server ID | +| `handle` | int | Connection handle | +| `servertype` | symbol | Process type | +| `active` | boolean | Whether the server is currently active | +| `attributes` | any | Attribute dictionary registered with the server | + +## Log dependency contract + +`di.serverselect` requires a log dependency dictionary with keys `` `info`warn`error ``: + +```q +`info`warn`error!({[ctx;msg] ...};{[ctx;msg] ...};{[ctx;msg] ...}) +``` + +`di.log` satisfies this contract: + +```q +log:use`di.log +logdep:`info`warn`error!(log.info;log.warn;log.error) +srvsel.init[enlist[`log]!enlist logdep] +``` + +Context symbols used by `di.serverselect` in log calls: + +| Context | Level | When | +|---|---|---| +| `` `addserverattr `` | info | Server registered | +| `` `setserveractive `` | info | Server active flag changed | +| `` `addserversfromtable `` | info | Bulk registration — count of servers being added | diff --git a/di/serverselect/serverselect.q b/di/serverselect/serverselect.q new file mode 100644 index 00000000..2a2328c1 --- /dev/null +++ b/di/serverselect/serverselect.q @@ -0,0 +1,167 @@ +/ library for selecting backend servers from a registered pool based on servertype or attribute requirements + +/ registered server pool - populated by addserverattr +servers:([serverid:`u#`int$()] handle:`int$(); servertype:`symbol$(); active:`boolean$(); attributes:()); + +/ autoincrement counter for server IDs +serverid:0i; + +init:{[deps] + / wire injectable log dependency (required) + logdict:$[99h=type deps; + $[`log in key deps; + $[99h=type deps`log; deps`log; ()!()]; + ()!()]; + ()!()]; + if[not count logdict; + '"di.serverselect: log dependency is required; pass `info`warn`error functions - ", + "see di.log for a default implementation or refer to confluence documentation"; + ]; + if[not all (`info`warn`error) in key logdict; + '"di.serverselect: log dict must have `info`warn`error keys; got: ",(", " sv string key logdict); + ]; + .z.m.loginfo:logdict`info; + .z.m.logwarn:logdict`warn; + .z.m.logerr:logdict`error; + }; + +/ internal - increment and return the next unique server ID +nextserverid:{ + .z.m.serverid:serverid+1i; + :.z.m.serverid; + }; + +addserverattr:{[h;st;att] + / register a server handle with a servertype and attribute dictionary + .z.m.loginfo[`addserverattr;"registering server: handle=",(string h),", type=",string st]; + .z.m.servers:servers upsert (nextserverid[];h;st;1b;att); + }; + +addserver:{[h;st] + / register a server with no attributes; convenience wrapper over addserverattr + addserverattr[h;st;()!()]; + }; + +setserveractive:{[h;a] + / mark a registered server active (1b) or inactive (0b); called on connect and disconnect + .z.m.loginfo[`setserveractive;"marking handle=",(string h)," active=",string a]; + .z.m.servers:update active:a from servers where handle=h; + }; + +getservers:{[] + / return the current registered server table + :servers; + }; + +addserversfromtable:{[proctypes;conntable] + / register active servers from a connection table filtered by proctype + / conntable must have columns: w (int handle), proctype (symbol), attributes (dict per row) + / pass proctypes:`ALL to register all process types + activehandles:(0i;0Ni),exec handle from servers where active; + rows:select w,proctype,attributes from conntable where + ((proctype in proctypes) or proctypes~`ALL), + not w in activehandles; + .z.m.loginfo[`addserversfromtable;"registering ",(string count rows)," servers from connection table"]; + addserverattr'[rows`w;rows`proctype;rows`attributes]; + }; + +getserverids:{[att] + / return server IDs matching a servertype list or attribute requirement dictionary + / att: symbol list of servertypes, or dict of attribute requirements (optionally keyed on `servertype) + if[99h<>type att; + if[not 11h=abs type att; + '"di.serverselect: servertype must be a symbol list (11h) or attribute dict (99h)"; + ]; + servertype:distinct att,(); + activeservers:exec distinct servertype from servers where active; + allservers:exec distinct servertype from servers; + activeserversmsg:". available servers: ",", " sv string activeservers; + if[any null att; + '"di.serverselect: null servertype passed as argument",activeserversmsg; + ]; + if[count servertype except activeservers; + '"di.serverselect: ", + $[max not servertype in allservers; + "not valid servers: ",", " sv string servertype except allservers; + "requested servers currently inactive: ",", " sv string servertype except activeservers + ],activeserversmsg; + ]; + :(exec serverid by servertype from servers where active)[servertype]; + ]; + serverids:$[`servertype in key att; + raze getserveridstype[delete servertype from att] each (),att`servertype; + getserveridstype[att;`all]]; + if[all 0=count each serverids;'"di.serverselect: no servers match requested attributes"]; + :serverids; + }; + +/ internal - filter active servers by servertype and attribute requirements +getserveridstype:{[att;typ] + / resolve besteffort and attributetype control keys then dispatch to cross or independent matching + besteffort:1b; + attype:`cross; + svrs:$[typ=`all; + exec serverid!attributes from servers where active; + exec serverid!attributes from servers where active,servertype=typ]; + if[`besteffort in key att; + if[-1h=type att`besteffort; besteffort:att`besteffort]; + att:delete besteffort from att; + ]; + if[`attributetype in key att; + if[-11h=type att`attributetype; attype:att`attributetype]; + att:delete attributetype from att; + ]; + res:$[attype=`independent; + getserversindependent[att;svrs;besteffort]; + getserverscross[att;svrs;besteffort]]; + serverids:first value flip $[99h=type res; key res; res]; + if[all 0=count each serverids;'"di.serverselect: no servers match ",string[typ]," requested attributes"]; + :serverids; + }; + +/ internal - build a cross product table from a nested dictionary +buildcross:{(cross/){flip (enlist y)#x}[x] each key x}; + +/ internal - initial filter shared by cross and independent matching +/ drops servers missing any required attribute key, ranks survivors by coverage +getserversinitial:{[req;att] + if[0=count req; :([]serverid:enlist key att)]; + att:(where all each (key req) in/: key each att)#att; + if[not count att;'"di.serverselect: no servers report all requested attributes"]; + s:update serverid:key att from value req in'/: (key req)#/:att; + s:s idesc value min each sum each' `serverid xkey s; + s:`serverid xkey 0!(key req) xgroup s; + :s; + }; + +/ internal - find servers satisfying the cross product of all attribute requirements +/ each attribute combination must be coverable by a single server +getserverscross:{[req;att;besteffort] + if[0=count req; :([]serverid:enlist key att)]; + s:getserversinitial[req;att]; + reqcross:buildcross[req]; + / scan through each server group accumulating which cross-product rows have been covered + util:flip `remaining`found!flip ( + {[x;y;z] (y[0] except found; y[0] inter found:$[0=count y[0];y[0];buildcross x@'where each z])}[req]\ + )[(reqcross;());value s]; + if[(count last util`remaining) and not besteffort; + '"di.serverselect: cannot satisfy query - cross product of all attributes cannot be matched"; + ]; + s:1!(0!s) w:where not 0=count each util`found; + :(key s)!distinct each' flip each util[w]`found; + }; + +/ internal - find servers satisfying attribute requirements independently +/ each individual requirement only needs to be matched by one server +getserversindependent:{[req;att;besteffort] + if[0=count req; :([]serverid:enlist key att)]; + s:getserversinitial[req;att]; + / mask out server groups whose contribution is already covered by earlier groups + filter:(value s)¬ -1 _ (0b&(value s) enlist 0),maxs value s; + alldone:1+first where all each all each' maxs value s; + if[(null alldone) and not besteffort; + '"di.serverselect: cannot satisfy query - not all attributes can be matched"; + ]; + s:1!(0!s) w:where any each any each' filter; + :(key s)!{(key x)!(value x)@'where each y key x}[req] each value s&filter w; + }; diff --git a/di/serverselect/test.csv b/di/serverselect/test.csv new file mode 100644 index 00000000..46d7e633 --- /dev/null +++ b/di/serverselect/test.csv @@ -0,0 +1,100 @@ +action,ms,bytes,lang,code,repeat,minver,comment +before,0,0,q,srvsel:use`di.serverselect,1,1,load module +before,0,0,q,mylog:`info`warn`error!({[c;m]};{[c;m]};{[c;m]}),1,1,no-op mock logger +before,0,0,q,"caplog:([] fn:`symbol$();ctx:`symbol$();msg:())",1,1,initialise log capture table +before,0,0,q,logcap:`info`warn`error!({[c;m] `caplog upsert(`info;c;m)};{[c;m] `caplog upsert(`warn;c;m)};{[c;m] `caplog upsert(`error;c;m)}),1,1,define capturing mock logger +before,0,0,q,srvsel.init[enlist[`log]!enlist mylog],1,1,init with no-op logger for setup + +/ ---- init: dependency injection validation ---- +comment,,,,,,,init - dependency injection validation +fail,0,0,q,srvsel.init[(::)],1,1,init rejects :: as deps +fail,0,0,q,srvsel.init[enlist[`log]!enlist(::)],1,1,init rejects null log dep +fail,0,0,q,srvsel.init[()!()],1,1,init rejects empty dict +fail,0,0,q,srvsel.init[enlist[`other]!enlist mylog],1,1,init rejects dict missing log key +fail,0,0,q,srvsel.init[enlist[`log]!enlist 42],1,1,init rejects non-dict log value +fail,0,0,q,srvsel.init[enlist[`log]!enlist(`info`warn!(mylog`info;mylog`warn))],1,1,init rejects log dict missing required key +true,0,0,q,"(7#@[srvsel.init;(::);{x}])~""di.serverselect""",1,1,error message is prefixed di.serverselect + +/ ---- getservers: initial state ---- +comment,,,,,,,getservers - initial state before any addserverattr +true,0,0,q,0=count srvsel.getservers[],1,1,servers empty on fresh module load +true,0,0,q,`serverid`handle`servertype`active`attributes~cols srvsel.getservers[],1,1,servers has correct schema + +/ ---- addserverattr / addserver: registration ---- +comment,,,,,,,addserverattr and addserver - server registration +run,0,0,q,srvsel.addserver[1i;`rdb],1,1,register rdb server with no attributes +run,0,0,q,srvsel.addserver[2i;`rdb],1,1,register second rdb server +run,0,0,q,"srvsel.addserverattr[3i;`hdb;`date`sym!(2024.01.01 2024.01.02;`AAPL`MSFT)]",1,1,register hdb server with date and sym attributes +true,0,0,q,3=count srvsel.getservers[],1,1,three servers registered +true,0,0,q,all srvsel.getservers[][`active],1,1,all registered servers are active by default +true,0,0,q,`rdb`rdb`hdb~exec servertype from srvsel.getservers[],1,1,servertypes match registered order +true,0,0,q,(1i;2i;3i)~exec handle from srvsel.getservers[],1,1,handles match registered order +true,0,0,q,1 2 3i~exec serverid from srvsel.getservers[],1,1,server IDs autoincrement from 1 + +/ ---- setserveractive: marking active/inactive ---- +comment,,,,,,,setserveractive - active flag management +run,0,0,q,srvsel.setserveractive[2i;0b],1,1,mark second rdb server inactive +true,0,0,q,not (exec active from srvsel.getservers[] where handle=2i)[0],1,1,handle 2 is now inactive +true,0,0,q,(exec active from srvsel.getservers[] where handle=1i)[0],1,1,handle 1 remains active +run,0,0,q,srvsel.setserveractive[2i;1b],1,1,mark second rdb server active again +true,0,0,q,(exec active from srvsel.getservers[] where handle=2i)[0],1,1,handle 2 is active again + +/ ---- getserverids: symbol list path ---- +comment,,,,,,,getserverids - symbol list (servertype) path +true,0,0,q,0 Date: Thu, 18 Jun 2026 11:35:17 +0100 Subject: [PATCH 02/14] Added more of the required functions, fixed bugs and added more tests --- di/serverselect/init.q | 3 +- di/serverselect/serverselect.md | 6 +- di/serverselect/serverselect.q | 111 +++++++++++++++++++++++--------- di/serverselect/test.csv | 101 +++++++++++++++++++---------- 4 files changed, 154 insertions(+), 67 deletions(-) diff --git a/di/serverselect/init.q b/di/serverselect/init.q index 4323911e..5512c3c2 100644 --- a/di/serverselect/init.q +++ b/di/serverselect/init.q @@ -1,4 +1,3 @@ -/ library for selecting backend servers from a registered pool based on servertype or attribute requirements \l ::serverselect.q -export:([init;addserverattr;addserver;setserveractive;getserverids;addserversfromtable;getservers]) +export:([init;addserverfull;addserverattr;addserver;setserveractive;getserverstable;addserversfromtable;getservers;selector;getserverbytype;gethandlebytype;gethpbytype;getserverids]) diff --git a/di/serverselect/serverselect.md b/di/serverselect/serverselect.md index 64189143..e0f5f6f7 100644 --- a/di/serverselect/serverselect.md +++ b/di/serverselect/serverselect.md @@ -64,7 +64,7 @@ Wire injectable dependencies. Must be called before any other function. |---|---|---|---| | `` `log `` | yes | dict | Functions keyed `` `info`warn`error ``, each with signature `{[ctx;msg]}` | -Errors with prefix `di.serverselect:` if `log` is missing or does not contain all three required keys. +Errors with prefix `di.serverselect:` if `configs` is not a dict, `log` key is missing, or `log` value is not a dict with all three required keys. --- @@ -170,12 +170,12 @@ srvsel.addserversfromtable[`ALL; conntab] --- -### `getservers[]` +### `getserverstable[]` Return the current registered server table. ```q -srvsel.getservers[] +srvsel.getserverstable[] ``` Schema: diff --git a/di/serverselect/serverselect.q b/di/serverselect/serverselect.q index 2a2328c1..07ccf45a 100644 --- a/di/serverselect/serverselect.q +++ b/di/serverselect/serverselect.q @@ -1,28 +1,22 @@ / library for selecting backend servers from a registered pool based on servertype or attribute requirements -/ registered server pool - populated by addserverattr -servers:([serverid:`u#`int$()] handle:`int$(); servertype:`symbol$(); active:`boolean$(); attributes:()); +/ registered server pool - populated by addserverfull / addserverattr / addserver +servers:([serverid:`u#`int$()] handle:`int$(); procname:`symbol$(); servertype:`symbol$(); hpup:`symbol$(); active:`boolean$(); lastp:`timestamp$(); hits:`int$(); attributes:()); / autoincrement counter for server IDs serverid:0i; -init:{[deps] - / wire injectable log dependency (required) - logdict:$[99h=type deps; - $[`log in key deps; - $[99h=type deps`log; deps`log; ()!()]; - ()!()]; - ()!()]; - if[not count logdict; - '"di.serverselect: log dependency is required; pass `info`warn`error functions - ", - "see di.log for a default implementation or refer to confluence documentation"; - ]; - if[not all (`info`warn`error) in key logdict; - '"di.serverselect: log dict must have `info`warn`error keys; got: ",(", " sv string key logdict); - ]; - .z.m.loginfo:logdict`info; - .z.m.logwarn:logdict`warn; - .z.m.logerr:logdict`error; +init:{[configs] + / wire log dependency - required; configs must be a dict with `log key + if[99h<>type configs; + '"di.serverselect: configs must be a dict with `log key"]; + if[not `log in key configs; + '"di.serverselect: log dependency is required; pass `info`warn`error functions keyed on `log"]; + if[99h<>type configs`log; + '"di.serverselect: log value must be a dict; pass `info`warn`error functions"]; + if[not all (`info`warn`error) in key configs`log; + '"di.serverselect: log dict must have `info`warn`error keys; got: ",(", " sv string key configs`log)]; + .z.m.log:configs`log; }; / internal - increment and return the next unique server ID @@ -31,24 +25,34 @@ nextserverid:{ :.z.m.serverid; }; +/ internal - update last-access timestamp and hit count for a server handle +updatestats:{[h] + .z.m.servers:update lastp:.z.p,hits:hits+1i from servers where handle=h; + }; + +addserverfull:{[h;pname;st;hp;att] + / register a server with full details: handle, procname, servertype, hpup and attribute dictionary + .z.m.log[`info][`addserverfull;"registering server: handle=",(string h),", procname=",string[pname],", type=",string st]; + .z.m.servers:servers upsert (nextserverid[];h;pname;st;hp;1b;0Np;0i;att); + }; + addserverattr:{[h;st;att] - / register a server handle with a servertype and attribute dictionary - .z.m.loginfo[`addserverattr;"registering server: handle=",(string h),", type=",string st]; - .z.m.servers:servers upsert (nextserverid[];h;st;1b;att); + / register a server with servertype and attributes; procname and hpup default to null + addserverfull[h;`;st;`;att]; }; addserver:{[h;st] - / register a server with no attributes; convenience wrapper over addserverattr - addserverattr[h;st;()!()]; + / register a server with no attributes; procname and hpup default to null + addserverfull[h;`;st;`;()!()]; }; setserveractive:{[h;a] / mark a registered server active (1b) or inactive (0b); called on connect and disconnect - .z.m.loginfo[`setserveractive;"marking handle=",(string h)," active=",string a]; + .z.m.log[`info][`setserveractive;"marking handle=",(string h)," active=",string a]; .z.m.servers:update active:a from servers where handle=h; }; -getservers:{[] +getserverstable:{[] / return the current registered server table :servers; }; @@ -56,15 +60,64 @@ getservers:{[] addserversfromtable:{[proctypes;conntable] / register active servers from a connection table filtered by proctype / conntable must have columns: w (int handle), proctype (symbol), attributes (dict per row) + / optional columns: procname (symbol), hpup (symbol) - populated from conntable if present / pass proctypes:`ALL to register all process types activehandles:(0i;0Ni),exec handle from servers where active; - rows:select w,proctype,attributes from conntable where + rows:select from conntable where ((proctype in proctypes) or proctypes~`ALL), not w in activehandles; - .z.m.loginfo[`addserversfromtable;"registering ",(string count rows)," servers from connection table"]; - addserverattr'[rows`w;rows`proctype;rows`attributes]; + .z.m.log[`info][`addserversfromtable;"registering ",(string count rows)," servers from connection table"]; + pnames:$[`procname in cols rows; rows`procname; count[rows]#`]; + hpups:$[`hpup in cols rows; rows`hpup; count[rows]#`]; + addserverfull'[rows`w;pnames;rows`proctype;hpups;rows`attributes]; + }; + +attributematch:{[req;avail] + / compute match result for each key in req against what avail advertises + / returns dict of attrname!(complete_match_bool;matched_values) for each required attribute key + / keys present in req but absent in avail return (0b;()) + vals:key[req] inter key avail; + notpresent:noval!(count noval:key[req] except key avail)#enlist(0b;()); + :notpresent,vals!{($[0>type y;x~y;all x in y];(x,()) inter y,())}'[req vals;avail vals]; }; +getservers:{[nameortype;lookups;req] + / look up active servers by servertype or procname with per-attribute match scoring + / nameortype: `servertype or `procname; pass ` as lookups to return all active servers + / req: attribute requirements dict - use ()!() for no attribute filtering + / returns table with attribmatch column showing (complete_bool;matched_values) per attribute key + r:$[`~lookups; + select serverid,procname,servertype,hpup,handle,lastp,attributes from servers where active; + nameortype~`servertype; + select serverid,procname,servertype,hpup,handle,lastp,attributes from servers where active,servertype in lookups; + select serverid,procname,servertype,hpup,handle,lastp,attributes from servers where active,procname in lookups]; + if[0=count r;:update attribmatch:attributes from r]; + am:attributematch[req] each r`attributes; + :update attribmatch:am from r; + }; + +selector:{[servertable;selection] + / pick one row from servertable using the given strategy + / selection: `roundrobin (least recently used), `any (random), `last (most recently used) + :$[selection=`roundrobin; first `lastp xasc servertable; + selection=`any; rand servertable; + selection=`last; last `lastp xasc servertable; + '"di.serverselect: unknown selection strategy: ",string selection]; + }; + +getserverbytype:{[ptype;serverval;selection] + / return a single server attribute value for a servertype using the given selection strategy + / ptype: servertype symbol; serverval: column to return e.g. `handle or `hpup; selection: `roundrobin`any`last + r:getservers[`servertype;ptype;()!()]; + if[not count r;:()]; + r:selector[r;selection]; + updatestats[r`handle]; + :r serverval; + }; + +gethandlebytype:getserverbytype[;`handle;]; +gethpbytype:getserverbytype[;`hpup;]; + getserverids:{[att] / return server IDs matching a servertype list or attribute requirement dictionary / att: symbol list of servertypes, or dict of attribute requirements (optionally keyed on `servertype) diff --git a/di/serverselect/test.csv b/di/serverselect/test.csv index 46d7e633..42e88888 100644 --- a/di/serverselect/test.csv +++ b/di/serverselect/test.csv @@ -7,37 +7,71 @@ before,0,0,q,srvsel.init[enlist[`log]!enlist mylog],1,1,init with no-op logger f / ---- init: dependency injection validation ---- comment,,,,,,,init - dependency injection validation -fail,0,0,q,srvsel.init[(::)],1,1,init rejects :: as deps -fail,0,0,q,srvsel.init[enlist[`log]!enlist(::)],1,1,init rejects null log dep -fail,0,0,q,srvsel.init[()!()],1,1,init rejects empty dict -fail,0,0,q,srvsel.init[enlist[`other]!enlist mylog],1,1,init rejects dict missing log key +run,0,0,q,srvsel.init[enlist[`log]!enlist mylog],1,1,init accepts valid log dependency +fail,0,0,q,srvsel.init[(::)],1,1,init rejects :: (not a dict) +fail,0,0,q,srvsel.init[()!()],1,1,init rejects empty config (missing log key) +fail,0,0,q,srvsel.init[enlist[`other]!enlist mylog],1,1,init rejects config without log key +fail,0,0,q,srvsel.init[enlist[`log]!enlist(::)],1,1,init rejects log dep that is not a dict fail,0,0,q,srvsel.init[enlist[`log]!enlist 42],1,1,init rejects non-dict log value fail,0,0,q,srvsel.init[enlist[`log]!enlist(`info`warn!(mylog`info;mylog`warn))],1,1,init rejects log dict missing required key -true,0,0,q,"(7#@[srvsel.init;(::);{x}])~""di.serverselect""",1,1,error message is prefixed di.serverselect +true,0,0,q,"(15#@[srvsel.init;enlist[`log]!enlist(`info`warn!(mylog`info;mylog`warn));{x}])~""di.serverselect""",1,1,error message is prefixed di.serverselect -/ ---- getservers: initial state ---- -comment,,,,,,,getservers - initial state before any addserverattr -true,0,0,q,0=count srvsel.getservers[],1,1,servers empty on fresh module load -true,0,0,q,`serverid`handle`servertype`active`attributes~cols srvsel.getservers[],1,1,servers has correct schema +/ ---- getserverstable: initial state ---- +comment,,,,,,,getserverstable - initial state before any addserverattr +true,0,0,q,0=count srvsel.getserverstable[],1,1,servers empty on fresh module load +true,0,0,q,`serverid`handle`procname`servertype`hpup`active`lastp`hits`attributes~cols srvsel.getserverstable[],1,1,servers has correct schema -/ ---- addserverattr / addserver: registration ---- +/ ---- addserverattr / addserver / addserverfull: registration ---- comment,,,,,,,addserverattr and addserver - server registration run,0,0,q,srvsel.addserver[1i;`rdb],1,1,register rdb server with no attributes run,0,0,q,srvsel.addserver[2i;`rdb],1,1,register second rdb server run,0,0,q,"srvsel.addserverattr[3i;`hdb;`date`sym!(2024.01.01 2024.01.02;`AAPL`MSFT)]",1,1,register hdb server with date and sym attributes -true,0,0,q,3=count srvsel.getservers[],1,1,three servers registered -true,0,0,q,all srvsel.getservers[][`active],1,1,all registered servers are active by default -true,0,0,q,`rdb`rdb`hdb~exec servertype from srvsel.getservers[],1,1,servertypes match registered order -true,0,0,q,(1i;2i;3i)~exec handle from srvsel.getservers[],1,1,handles match registered order -true,0,0,q,1 2 3i~exec serverid from srvsel.getservers[],1,1,server IDs autoincrement from 1 +true,0,0,q,3=count srvsel.getserverstable[],1,1,three servers registered +true,0,0,q,all exec active from srvsel.getserverstable[],1,1,all registered servers are active by default +true,0,0,q,`rdb`rdb`hdb~exec servertype from srvsel.getserverstable[],1,1,servertypes match registered order +true,0,0,q,(1i;2i;3i)~exec handle from srvsel.getserverstable[],1,1,handles match registered order +true,0,0,q,1 2 3i~exec serverid from srvsel.getserverstable[],1,1,server IDs autoincrement from 1 +true,0,0,q,all null exec procname from srvsel.getserverstable[],1,1,procname is null when registered via addserverattr +true,0,0,q,all null exec hpup from srvsel.getserverstable[],1,1,hpup is null when registered via addserverattr +run,0,0,q,"srvsel.addserverfull[4i;`myproc;`hdb;`:localhost:5555;(enlist`date)!enlist 2024.12.01]",1,1,register server with full details via addserverfull +true,0,0,q,4=count srvsel.getserverstable[],1,1,fourth server registered via addserverfull +true,0,0,q,`myproc~first exec procname from srvsel.getserverstable[] where handle=4i,1,1,procname populated via addserverfull +true,0,0,q,`:localhost:5555~first exec hpup from srvsel.getserverstable[] where handle=4i,1,1,hpup populated via addserverfull / ---- setserveractive: marking active/inactive ---- comment,,,,,,,setserveractive - active flag management run,0,0,q,srvsel.setserveractive[2i;0b],1,1,mark second rdb server inactive -true,0,0,q,not (exec active from srvsel.getservers[] where handle=2i)[0],1,1,handle 2 is now inactive -true,0,0,q,(exec active from srvsel.getservers[] where handle=1i)[0],1,1,handle 1 remains active +true,0,0,q,not (exec active from srvsel.getserverstable[] where handle=2i)[0],1,1,handle 2 is now inactive +true,0,0,q,(exec active from srvsel.getserverstable[] where handle=1i)[0],1,1,handle 1 remains active run,0,0,q,srvsel.setserveractive[2i;1b],1,1,mark second rdb server active again -true,0,0,q,(exec active from srvsel.getservers[] where handle=2i)[0],1,1,handle 2 is active again +true,0,0,q,(exec active from srvsel.getserverstable[] where handle=2i)[0],1,1,handle 2 is active again + + +/ ---- getservers ---- +comment,,,,,,,getservers - server lookup with attribute match scoring +true,0,0,q,`serverid`procname`servertype`hpup`handle`lastp`attributes`attribmatch~cols srvsel.getservers[`;`;()!()],1,1,getservers returns table with correct columns +true,0,0,q,count[select from srvsel.getserverstable[] where active]=count srvsel.getservers[`;`;()!()],1,1,getservers with ` lookups returns all active servers +true,0,0,q,all `hdb=exec servertype from srvsel.getservers[`servertype;`hdb;()!()],1,1,getservers filters by servertype correctly +true,0,0,q,0=count srvsel.getservers[`servertype;`nonexistent;()!()],1,1,getservers returns empty table for unknown servertype +true,0,0,q,0 Date: Thu, 18 Jun 2026 14:20:06 +0100 Subject: [PATCH 03/14] adding more tests and writeup --- di/serverselect/serverselect.md | 256 ++++++++++++++++---------------- di/serverselect/test.csv | 117 +++++++++++++++ 2 files changed, 244 insertions(+), 129 deletions(-) diff --git a/di/serverselect/serverselect.md b/di/serverselect/serverselect.md index e0f5f6f7..a871ae90 100644 --- a/di/serverselect/serverselect.md +++ b/di/serverselect/serverselect.md @@ -1,144 +1,158 @@ -# di.serverselect +# Server Selection -Module for registering backend servers and selecting them by servertype or attribute requirements. Extracted from the `.gw` namespace across TorQ's `gatewaylib.q` and `gateway.q`. Designed as the server-selection layer of a gateway — decoupled from query execution and connection management. +`serverselect.q` maintains a pool of registered backend servers and selects from them by servertype or attribute requirements. Extracted from the `.gw` namespace in TorQ's `gatewaylib.q` and `gateway.q`, it provides the server-selection layer of a gateway — decoupled from query execution and connection management. -## Usage - -```q -srvsel:use`di.serverselect +--- -log:use`di.log -logdep:`info`warn`error!(log.info;log.warn;log.error) -srvsel.init[enlist[`log]!enlist logdep] +## :sparkles: Features -/ register servers as they connect -srvsel.addserver[h;`rdb] -srvsel.addserverattr[h;`hdb;`date`sym!(2024.01.01 2024.01.02;`AAPL`MSFT)] +- Register backend servers with servertype, procname, host/port, and attribute dictionaries +- Track active/inactive state per server, updated on connect and disconnect +- Select servers using round-robin, most-recently-used, or random strategy +- Query servers by servertype list or attribute requirement dictionary +- Attribute matching supports cross-product and independent strategies with configurable best-effort mode +- Bulk registration from a TorQ-compatible connection table +- Injected log dependency — no hard module dependencies -/ mark inactive on disconnect -srvsel.setserveractive[h;0b] +--- -/ select by servertype -srvsel.getserverids[`rdb`hdb] +## :memo: Dependencies -/ select by attribute requirements -srvsel.getserverids[enlist[`date]!enlist enlist 2024.01.01] -``` +| Dependency | Required | Description | +|---|---|---| +| `log` | yes | Dict of `info`warn`error functions, each with signature `{[ctx;msg]}` | -### Typical gateway integration +Pass the log dependency via `init`: ```q -srvsel:use`di.serverselect log:use`di.log logdep:`info`warn`error!(log.info;log.warn;log.error) srvsel.init[enlist[`log]!enlist logdep] - -/ on open - register the connecting server -.z.po:{[h] srvsel.addserverattr[h;getproctype[h];getattributes[h]]} - -/ on close - mark the server inactive -.z.pc:{[h] srvsel.setserveractive[h;0b]} - -/ on query - select target servers then dispatch -serverids:srvsel.getserverids[`rdb`hdb] ``` -### Bulk registration from a TorQ connection table +--- -In a TorQ stack, `.servers.SERVERS` is populated by `trackservers.q`. Pass it directly: +## :label: Server Table Schema -```q -srvsel.addserversfromtable[`rdb`hdb; .servers.SERVERS] +Servers are tracked in the `servers` keyed table (keyed on `serverid`): -/ or register all process types -srvsel.addserversfromtable[`ALL; .servers.SERVERS] -``` +| Column | Type | Description | +|---|---|---| +| `serverid` | `int` (key) | Unique auto-assigned server ID | +| `handle` | `int` | Connection handle | +| `procname` | `symbol` | Process name (null if not provided) | +| `servertype` | `symbol` | Process type e.g. `` `rdb ``, `` `hdb `` | +| `hpup` | `symbol` | Host/port symbol e.g. `` `:host:5010 `` (null if not provided) | +| `active` | `boolean` | Whether the server is currently active | +| `lastp` | `timestamp` | Last time this server was selected | +| `hits` | `int` | Number of times this server has been selected | +| `attributes` | `any` | Attribute dictionary registered with the server | -## API +--- -### `init[deps]` +## :gear: Configuration -Wire injectable dependencies. Must be called before any other function. +`init` wires the required log dependency. It must be called before any other function. -| Key | Required | Type | Description | -|---|---|---|---| -| `` `log `` | yes | dict | Functions keyed `` `info`warn`error ``, each with signature `{[ctx;msg]}` | +```q +srvsel.init[enlist[`log]!enlist logdep] +``` -Errors with prefix `di.serverselect:` if `configs` is not a dict, `log` key is missing, or `log` value is not a dict with all three required keys. +Throws with prefix `di.serverselect:` if: +- `configs` is not a dictionary +- `` `log `` key is missing +- `log` value is not a dictionary with `` `info`warn`error `` keys --- -### `addserverattr[handle;servertype;attributes]` +## :wrench: Functions -Register a server with a handle, servertype, and attribute dictionary. Sets `active:1b` on registration. +### Registration -| Parameter | Type | Description | -|---|---|---| -| `handle` | int | Connection handle | -| `servertype` | symbol | Process type (e.g. `` `rdb ``, `` `hdb ``) | -| `attributes` | dict | Key-value attribute pairs (e.g. `` `date`sym!(2024.01.01;`AAPL) ``) | +| Function | Description | +|---|---| +| `addserverfull[h;pname;st;hp;att]` | Register a server with full details: handle, procname, servertype, hpup, attributes | +| `addserverattr[h;st;att]` | Register a server with servertype and attributes; procname and hpup default to null | +| `addserver[h;st]` | Register a server with no attributes | +| `setserveractive[h;active]` | Mark a server active (`1b`) or inactive (`0b`) | +| `addserversfromtable[proctypes;conntable]` | Bulk-register from a connection table, filtered by proctype | +| `getserverstable[]` | Return the full registered server table | + +`addserversfromtable` skips handles that are already active. Pass `` `ALL `` as `proctypes` to register all process types. The `conntable` must have columns `w` (int handle), `proctype` (symbol), `attributes` (dict per row); `procname` and `hpup` are optional. ```q -srvsel.addserverattr[5i; `hdb; `date`sym!(2024.01.01 2024.01.02; `AAPL`MSFT)] +/ register on connect +srvsel.addserverattr[h; getproctype[h]; getattributes[h]] + +/ mark inactive on disconnect +srvsel.setserveractive[h; 0b] + +/ bulk registration from TorQ connection table +srvsel.addserversfromtable[`rdb`hdb; .servers.SERVERS] ``` --- -### `addserver[handle;servertype]` - -Register a server with no attributes. Convenience wrapper over `addserverattr`. +### Query and Selection -```q -srvsel.addserver[5i; `rdb] -``` +| Function | Description | +|---|---| +| `getservers[nameortype;lookups;req]` | Return active servers matching a servertype or procname filter, with per-attribute match scoring | +| `selector[servertable;selection]` | Pick one row from a server table using a selection strategy | +| `getserverbytype[ptype;col;sel]` | Return one column value for a servertype using the given strategy; updates `lastp` and `hits` | +| `gethandlebytype[ptype;sel]` | Convenience projection of `getserverbytype` returning `handle` | +| `gethpbytype[ptype;sel]` | Convenience projection of `getserverbytype` returning `hpup` | ---- +`getservers` returns a table including an `attribmatch` column — a dictionary of `attrname!(complete_match_bool;matched_values)` per attribute key in `req`. -### `setserveractive[handle;active]` +`selector` supports three strategies: -Mark a registered server active or inactive. Call with `0b` on disconnect, `1b` on reconnect. +| Strategy | Behaviour | +|---|---| +| `` `roundrobin `` | Pick server with the oldest `lastp` (least recently used) | +| `` `any `` | Pick a random server | +| `` `last `` | Pick server with the newest `lastp` (most recently used) | -| Parameter | Type | Description | -|---|---|---| -| `handle` | int | Connection handle of the server to update | -| `active` | boolean | `1b` = active, `0b` = inactive | +All three return `()` if no matching server is found. ```q -srvsel.setserveractive[5i; 0b] / server disconnected -srvsel.setserveractive[5i; 1b] / server reconnected +/ get a handle, round-robin across rdbs +srvsel.gethandlebytype[`rdb; `roundrobin] + +/ get host/port for an hdb +srvsel.gethpbytype[`hdb; `roundrobin] ``` --- -### `getserverids[att]` +### Server ID Lookup + +```q +srvsel.getserverids[att] +``` -Return server IDs matching the given servertype list or attribute requirement dictionary. +Returns server IDs matching a servertype list or attribute requirement dictionary. Used as the primary dispatch input — pass the result to your async or sync query handler to target specific servers. -| Parameter | Type | Description | -|---|---|---| -| `att` | symbol list or dict | Servertype list (11h) — returns IDs grouped by type. Attribute dict (99h) — returns IDs satisfying the requirements. | +#### Symbol list path -**Symbol list path** — validates that all requested types exist and are active: +Pass a symbol or symbol list of servertypes. Validates that all requested types are registered and currently active. ```q srvsel.getserverids[`rdb] srvsel.getserverids[`rdb`hdb] ``` -**Attribute dict path** — matches servers whose attribute dictionaries satisfy the requirements. Supports two matching strategies controlled by the reserved key `` `attributetype ``: +Throws if any requested type is null, unregistered, or all-inactive. -| `attributetype` | Behaviour | -|---|---| -| `` `cross `` (default) | Every combination of attribute values must be coverable — e.g. every date must be available for every sym | -| `` `independent `` | Each attribute value only needs to be matched by at least one server | +#### Attribute dict path -The reserved key `` `besteffort `` (boolean, default `1b`) controls whether a partial match is acceptable. If `0b`, an error is thrown when the requirements cannot be fully satisfied. +Pass a dictionary of attribute requirements. Servers whose attribute dictionaries satisfy the requirements are returned. ```q -/ cross match (default): find servers covering the full date × sym cross product +/ cross match (default): every date must be available for every sym srvsel.getserverids[`date`sym!(2024.01.01 2024.01.02; `AAPL`MSFT)] -/ independent match: each date and each sym just needs one server somewhere +/ independent match: each date and sym just needs one server somewhere srvsel.getserverids[`date`sym`attributetype!(2024.01.01 2024.01.02; `AAPL`MSFT; `independent)] / scope to a specific servertype then apply attribute filter @@ -148,66 +162,50 @@ srvsel.getserverids[`servertype`date!(`hdb; enlist 2024.01.01)] srvsel.getserverids[`date`besteffort!(enlist 2024.01.01; 0b)] ``` ---- - -### `addserversfromtable[proctypes;conntable]` +##### Attribute matching strategies -Register servers from a connection table, skipping handles already active in the pool. - -| Parameter | Type | Description | -|---|---|---| -| `proctypes` | symbol or symbol list | Process types to register; pass `` `ALL `` to register all | -| `conntable` | table | Must have columns: `w` (int handle), `proctype` (symbol), `attributes` (dict per row) | - -```q -/ TorQ stack — pass .servers.SERVERS directly -srvsel.addserversfromtable[`rdb`hdb; .servers.SERVERS] +| `` `attributetype `` | Behaviour | +|---|---| +| `` `cross `` (default) | Every combination of attribute values must be coverable by a single server | +| `` `independent `` | Each individual attribute value only needs to be matched by at least one server | -/ custom connection table -conntab:([] w:1i 2i; proctype:`rdb`hdb; attributes:(()!(); `date!enlist 2024.01.01)) -srvsel.addserversfromtable[`ALL; conntab] -``` +The reserved key `` `besteffort `` (boolean, default `1b`) controls whether a partial match is acceptable. Set to `0b` to throw if requirements cannot be fully satisfied. --- -### `getserverstable[]` - -Return the current registered server table. +## :test_tube: Example ```q -srvsel.getserverstable[] -``` - -Schema: +/ load module +srvsel:use`di.serverselect -| Column | Type | Description | -|---|---|---| -| `serverid` | int (keyed) | Unique auto-assigned server ID | -| `handle` | int | Connection handle | -| `servertype` | symbol | Process type | -| `active` | boolean | Whether the server is currently active | -| `attributes` | any | Attribute dictionary registered with the server | +/ wire log dependency +log:use`di.log +logdep:`info`warn`error!(log.info;log.warn;log.error) +srvsel.init[enlist[`log]!enlist logdep] -## Log dependency contract +/ register servers as they connect (.z.po handler) +.z.po:{[h] + srvsel.addserverattr[h; getproctype[h]; getattributes[h]]; + } -`di.serverselect` requires a log dependency dictionary with keys `` `info`warn`error ``: +/ mark inactive on disconnect (.z.pc handler) +.z.pc:{[h] + srvsel.setserveractive[h; 0b]; + } -```q -`info`warn`error!({[ctx;msg] ...};{[ctx;msg] ...};{[ctx;msg] ...}) -``` +/ bulk register from TorQ stack on startup +srvsel.addserversfromtable[`rdb`hdb; .servers.SERVERS] -`di.log` satisfies this contract: +/ get server IDs for a query by servertype +serverids:srvsel.getserverids[`rdb`hdb] -```q -log:use`di.log -logdep:`info`warn`error!(log.info;log.warn;log.error) -srvsel.init[enlist[`log]!enlist logdep] -``` +/ get server IDs by attribute requirement +serverids:srvsel.getserverids[`date`sym!(enlist 2024.01.01; enlist `AAPL)] -Context symbols used by `di.serverselect` in log calls: +/ get a handle directly (round-robin across rdbs) +h:srvsel.gethandlebytype[`rdb; `roundrobin] -| Context | Level | When | -|---|---|---| -| `` `addserverattr `` | info | Server registered | -| `` `setserveractive `` | info | Server active flag changed | -| `` `addserversfromtable `` | info | Bulk registration — count of servers being added | +/ inspect registered server pool +srvsel.getserverstable[] +``` diff --git a/di/serverselect/test.csv b/di/serverselect/test.csv index 42e88888..febd74c2 100644 --- a/di/serverselect/test.csv +++ b/di/serverselect/test.csv @@ -133,3 +133,120 @@ run,0,0,q,"srvsel.addserversfromtable[`rdb;([]w:enlist 98i;proctype:enlist`rdb;a true,0,0,q,any caplog[`fn]=`info,1,1,addserversfromtable emits info log entries true,0,0,q,"any caplog[`msg] like ""*servers from connection table*""",1,1,addserversfromtable logs count of servers being registered true,0,0,q,`addserversfromtable~first (select from caplog where fn=`info)[`ctx],1,1,addserversfromtable log uses addserversfromtable context symbol + +/ reset to no-op logger for edge case tests +run,0,0,q,srvsel.init[enlist[`log]!enlist mylog],1,1,reset to no-op logger for edge case tests + +/ ---- addserverfull: edge cases ---- +comment,,,,,,,addserverfull - edge cases +run,0,0,q,"srvsel.addserverfull[70i;`first;`rdb;`:first:7000;()!()]",1,1,register first entry at handle 70 +run,0,0,q,"srvsel.addserverfull[70i;`second;`hdb;`:second:7070;()!()]",1,1,register second entry at same handle (keyed by serverid not handle) +true,0,0,q,2=count select from srvsel.getserverstable[] where handle=70i,1,1,addserverfull with duplicate handle creates two distinct rows (new serverid each time) +fail,0,0,q,"srvsel.addserverfull[`badhandle;`;`rdb;`;()!()]",1,1,addserverfull rejects symbol handle (type error on int column) +fail,0,0,q,"srvsel.addserverfull[71i;`;""rdb"";`;()!()]",1,1,addserverfull rejects string servertype (type error on symbol column) + +/ ---- addserver: edge cases ---- +comment,,,,,,,addserver - edge cases +true,0,0,q,(()!())~first exec attributes from srvsel.getserverstable[] where handle=1i,1,1,addserver stores empty attributes dict +true,0,0,q,null first exec procname from srvsel.getserverstable[] where handle=1i,1,1,addserver stores null procname +true,0,0,q,null first exec hpup from srvsel.getserverstable[] where handle=1i,1,1,addserver stores null hpup +fail,0,0,q,"srvsel.addserver[`badhandle;`rdb]",1,1,addserver rejects symbol handle (type error) +fail,0,0,q,"srvsel.addserver[72i;""rdb""]",1,1,addserver rejects string servertype (type error) + +/ ---- addserverattr: edge cases ---- +comment,,,,,,,addserverattr - edge cases +true,0,0,q,null first exec procname from srvsel.getserverstable[] where handle=3i,1,1,addserverattr stores null procname +true,0,0,q,null first exec hpup from srvsel.getserverstable[] where handle=3i,1,1,addserverattr stores null hpup +fail,0,0,q,"srvsel.addserverattr[`badhandle;`hdb;()!()]",1,1,addserverattr rejects symbol handle (type error) +fail,0,0,q,"srvsel.addserverattr[73i;""hdb"";()!()]",1,1,addserverattr rejects string servertype (type error) + +/ ---- setserveractive: edge cases ---- +comment,,,,,,,setserveractive - edge cases +run,0,0,q,srvsel.setserveractive[9999i;0b],1,1,setserveractive on non-existent handle is a no-op +true,0,0,q,0=count select from srvsel.getserverstable[] where handle=9999i,1,1,non-existent handle not inserted by setserveractive +fail,0,0,q,"srvsel.setserveractive[1i;`yes]",1,1,setserveractive rejects symbol active flag (type error) +fail,0,0,q,"srvsel.setserveractive[""badhandle"";0b]",1,1,setserveractive rejects string handle (type error) + +/ ---- getservers: edge cases ---- +comment,,,,,,,getservers - edge cases +run,0,0,q,srvsel.setserveractive[1i;0b],1,1,mark handle 1 inactive for getservers inactive-exclusion test +true,0,0,q,not (1i) in exec handle from srvsel.getservers[`servertype;`rdb;()!()],1,1,getservers excludes inactive servers from results +run,0,0,q,srvsel.setserveractive[1i;1b],1,1,restore handle 1 active after inactive-exclusion test +true,0,0,q,"all (exec servertype from srvsel.getservers[`servertype;`rdb`hdb;()!()]) in `rdb`hdb",1,1,getservers accepts symbol list as lookups +true,0,0,q,4i in exec handle from srvsel.getservers[`procname;`myproc;()!()],1,1,getservers procname filter returns server with matching procname + +/ ---- selector: edge cases ---- +comment,,,,,,,selector - edge cases +run,0,0,q,"singleseltab:([]handle:enlist 77i;lastp:enlist 2024.01.15D00:00)",1,1,create single-row selector table +true,0,0,q,77i~(srvsel.selector[singleseltab;`roundrobin])`handle,1,1,selector roundrobin on single row returns that row +true,0,0,q,77i~(srvsel.selector[singleseltab;`last])`handle,1,1,selector last on single row returns that row +true,0,0,q,77i~(srvsel.selector[singleseltab;`any])`handle,1,1,selector any on single row returns that row +run,0,0,q,"emptyseltab:([]handle:`int$();lastp:`timestamp$())",1,1,create empty selector table +true,0,0,q,0Ni~(srvsel.selector[emptyseltab;`roundrobin])`handle,1,1,selector roundrobin on empty table returns null handle +true,0,0,q,0Ni~(srvsel.selector[emptyseltab;`last])`handle,1,1,selector last on empty table returns null handle + +/ ---- getserverbytype: direct tests ---- +comment,,,,,,,getserverbytype - direct tests +true,0,0,q,`rdb~srvsel.getserverbytype[`rdb;`servertype;`roundrobin],1,1,getserverbytype returns correct column value for servertype +true,0,0,q,()~srvsel.getserverbytype[`nonexistent;`handle;`roundrobin],1,1,getserverbytype returns () for unknown servertype +true,0,0,q,"(srvsel.getserverbytype[`hdb;`hpup;`roundrobin]) in exec hpup from srvsel.getserverstable[] where servertype=`hdb",1,1,getserverbytype returns hpup value for hdb type + +/ ---- gethpbytype: edge cases ---- +comment,,,,,,,gethpbytype - edge cases +true,0,0,q,()~srvsel.gethpbytype[`nonexistent;`roundrobin],1,1,gethpbytype returns () for unknown servertype + +/ ---- getserverids: edge cases ---- +comment,,,,,,,getserverids - edge cases +run,0,0,q,srvsel.getserverids[`symbol$()],1,1,getserverids with empty symbol list runs without error +true,0,0,q,"0 Date: Thu, 25 Jun 2026 11:21:51 +0100 Subject: [PATCH 04/14] Migrated serverselect from dilogging to kx logging, bugfixes, added argument validation, updates to test.csv and added integeration testing --- di/serverselect/init.q | 5 + di/serverselect/integration.q | 200 ++++++++++++++++++++++++++++++++ di/serverselect/serverselect.md | 56 ++++----- di/serverselect/serverselect.q | 143 ++++++++++++++--------- di/serverselect/test.csv | 117 +++++-------------- 5 files changed, 349 insertions(+), 172 deletions(-) create mode 100644 di/serverselect/integration.q diff --git a/di/serverselect/init.q b/di/serverselect/init.q index 5512c3c2..63dd8baa 100644 --- a/di/serverselect/init.q +++ b/di/serverselect/init.q @@ -1,3 +1,8 @@ \l ::serverselect.q +/ logging is an injected dependency: the start-up script that wires the modules together - +/ or the user at run time - must call init with a required `log dependency before using the +/ module. kx.log is intentionally NOT loaded here. +/ note: the injected log dict is stored in .z.m.log and called as .z.m.log[`info]["msg"] + export:([init;addserverfull;addserverattr;addserver;setserveractive;getserverstable;addserversfromtable;getservers;selector;getserverbytype;gethandlebytype;gethpbytype;getserverids]) diff --git a/di/serverselect/integration.q b/di/serverselect/integration.q new file mode 100644 index 00000000..f76b3b4f --- /dev/null +++ b/di/serverselect/integration.q @@ -0,0 +1,200 @@ +/ =========================================================================== +/ di.serverselect integration test +/ End-to-end narrative test driving every exported function through a +/ realistic gateway lifecycle: register -> activate/deactivate -> query -> +/ select -> bulk-register -> error handling. Complements the k4unit suite +/ in test.csv with assertions on real usage rather than per-call checks. +/ Run (QPATH must include this repo and the kx module dir): +/ QPATH=/path/to/kx/mod:/path/to/kdbx-modules q integration.q +/ Exits with a non-zero code equal to the number of failed assertions. +/ =========================================================================== + +srvsel:use`di.serverselect + +/ ---- tiny assertion harness ---- +/ note: avoid `desc`/`exp`/`log` as names - they are reserved words in KDB-X +PASS:0; FAIL:0; +lbl:{[ok] $[ok;" ok | ";" FAIL | "]}; +chk:{[nm;got;expd] + ok:got~expd; + $[ok;PASS+:1;FAIL+:1]; + m:lbl[ok],nm; + if[not ok; m:m," || got=",(.Q.s1 got)," exp=",.Q.s1 expd]; + -1 m; + }; +errtok:`$"__ERRORED__"; +chkerr:{[nm;f] + r:@[f;();{[e](errtok;e)}]; + ok:(0 null procname";1b;all null exec procname from srvsel.getserverstable[]] +chk["addserver -> null hpup";1b;all null exec hpup from srvsel.getserverstable[]] +chk["addserver -> empty attributes dict";(()!());first exec attributes from srvsel.getserverstable[] where handle=4i] +chk["addserver -> active by default";1b;all exec active from srvsel.getserverstable[]] + +/ =========================================================================== +hdr"STEP 3 addserverattr (servertype + attributes)" +srvsel.addserverattr[6i;`hdb;`date`sym!((d1;d2);`A`B)] +srvsel.addserverattr[7i;`hdb;`date`sym!((d2;d3);`B`C)] +chk["two hdb servers added";4=count srvsel.getserverstable[];1b] +chk["addserverattr stores attributes";`date`sym!((d1;d2);`A`B);first exec attributes from srvsel.getserverstable[] where handle=6i] +chk["addserverattr -> null procname";1b;null first exec procname from srvsel.getserverstable[] where handle=6i] + +/ =========================================================================== +hdr"STEP 4 addserverfull (full details)" +srvsel.addserverfull[8i;`gw1;`gateway;`:host:5000;(enlist`region)!enlist`EU] +chk["fifth server registered";5=count srvsel.getserverstable[];1b] +chk["addserverfull -> procname populated";`gw1;first exec procname from srvsel.getserverstable[] where handle=8i] +chk["addserverfull -> hpup populated";`:host:5000;first exec hpup from srvsel.getserverstable[] where handle=8i] +chk["serverid sequence is 1..5";1 2 3 4 5i;exec serverid from srvsel.getserverstable[]] +chk["handles as registered";4 5 6 7 8i;exec handle from srvsel.getserverstable[]] +chk["servertypes as registered";`rdb`rdb`hdb`hdb`gateway;exec servertype from srvsel.getserverstable[]] + +/ =========================================================================== +hdr"STEP 5 setserveractive (deactivate / reactivate)" +srvsel.setserveractive[5i;0b] +chk["handle 5 now inactive";0b;first exec active from srvsel.getserverstable[] where handle=5i] +chk["getservers excludes inactive rdb";1b;not 5i in exec handle from srvsel.getservers[`servertype;`rdb;()!()]] +srvsel.setserveractive[5i;1b] +chk["handle 5 reactivated";1b;first exec active from srvsel.getserverstable[] where handle=5i] +srvsel.setserveractive[9999i;0b] +chk["setserveractive on missing handle is a no-op";0=count select from srvsel.getserverstable[] where handle=9999i;1b] + +/ =========================================================================== +hdr"STEP 6 getservers (lookup + attribute match scoring)" +chk["lookups=` returns all active";5;count srvsel.getservers[`;`;()!()]] +chk["servertype=hdb returns 2";2;count srvsel.getservers[`servertype;`hdb;()!()]] +chk["servertype list rdb,hdb returns 4";4;count srvsel.getservers[`servertype;`rdb`hdb;()!()]] +chk["procname=gw1 returns the gateway";enlist 8i;exec handle from srvsel.getservers[`procname;`gw1;()!()]] +chk["unknown servertype returns empty";0;count srvsel.getservers[`servertype;`nope;()!()]] +chk["attribmatch column present";1b;`attribmatch in cols srvsel.getservers[`;`;()!()]] +chk["complete date match flagged for some hdb";1b;any {x[`date]0} each (srvsel.getservers[`servertype;`hdb;(enlist`date)!enlist enlist d1])`attribmatch] + +/ =========================================================================== +hdr"STEP 7 selector (strategies on a known table)" +seltab:([]serverid:101 102 103i;handle:201 202 203i;lastp:(2024.01.03D0;2024.01.01D0;2024.01.02D0)) +chk["roundrobin picks oldest lastp";202i;(srvsel.selector[seltab;`roundrobin])`handle] +chk["last picks newest lastp";201i;(srvsel.selector[seltab;`last])`handle] +chk["any picks a row from the table";1b;((srvsel.selector[seltab;`any])`handle) in 201 202 203i] +single:([]serverid:enlist 9i;handle:enlist 99i;lastp:enlist 2024.06.01D0) +chk["single-row roundrobin returns that row";99i;(srvsel.selector[single;`roundrobin])`handle] +empt:([]serverid:`int$();handle:`int$();lastp:`timestamp$()) +chk["empty table roundrobin -> null handle";0Ni;(srvsel.selector[empt;`roundrobin])`handle] +chk["empty table any -> null handle";0Ni;(srvsel.selector[empt;`any])`handle] + +/ =========================================================================== +hdr"STEP 8 getserverbytype / gethandlebytype / gethpbytype" +chk["gethandlebytype rdb returns a live rdb handle";1b;(srvsel.gethandlebytype[`rdb;`roundrobin]) in 4 5i] +chk["gethpbytype gateway returns its hpup";`:host:5000;srvsel.gethpbytype[`gateway;`roundrobin]] +chk["getserverbytype returns requested column value";`gw1;srvsel.getserverbytype[`gateway;`procname;`roundrobin]] +chk["gethandlebytype unknown type -> ()";();srvsel.gethandlebytype[`nope;`roundrobin]] +chk["gethpbytype unknown type -> ()";();srvsel.gethpbytype[`nope;`roundrobin]] +/ round-robin rotation: with exactly 2 active rdbs, 4 LRU picks must cover both +rot:srvsel.gethandlebytype[`rdb] each 4#`roundrobin +chk["roundrobin rotates across both rdbs";4 5i;asc distinct rot] +/ selection bumps hits +hitsbefore:exec sum hits from srvsel.getserverstable[] where servertype=`gateway +srvsel.gethandlebytype[`gateway;`roundrobin] +chk["selection increments hits";1b;(hitsbefore+1)=exec sum hits from srvsel.getserverstable[] where servertype=`gateway] + +/ =========================================================================== +hdr"STEP 9 getserverids - symbol (servertype) path" +chk["single servertype rdb -> rdb serverids";1 2i;asc raze srvsel.getserverids[`rdb]] +chk["servertype list rdb,hdb -> all four";1 2 3 4i;asc raze srvsel.getserverids[`rdb`hdb]] +srvsel.setserveractive[4i;0b]; srvsel.setserveractive[5i;0b] +chkerr["all requested servertype inactive -> error";{srvsel.getserverids[`rdb]}] +srvsel.setserveractive[4i;1b]; srvsel.setserveractive[5i;1b] + +/ =========================================================================== +hdr"STEP 10 getserverids - attribute dict path" +chk["single attribute date=d1 matches hdb 6";1b;3i in raze srvsel.getserverids[enlist[`date]!enlist enlist d1]] +chk["cross (d1,d2)x(A,B) satisfied by hdb 6 alone";1b;3i in raze srvsel.getserverids[`date`sym!((d1;d2);`A`B)]] +chk["independent date(d1,d2,d3)";1b;0 hdbs 6,7";3 4i;asc raze srvsel.getserverids[`servertype`date!(`hdb;enlist d2)]] +chk["empty requirement dict -> all active serverids";1 2 3 4 5i;asc raze srvsel.getserverids[()!()]] +chkerr["besteffort=0b impossible -> error";{srvsel.getserverids[`date`besteffort!(enlist 2099.01.01;0b)]}] +chkerr["no server has requested attribute value -> error";{srvsel.getserverids[enlist[`date]!enlist enlist 2099.01.01]}] + +/ =========================================================================== +hdr"STEP 11 addserversfromtable (bulk registration)" +conntab:([]w:10 11i;proctype:`rdb`hdb;attributes:(()!();`date`sym!((d1;d2);`A`B))) +srvsel.addserversfromtable[`rdb`hdb;conntab] +chk["bulk-registered handles 10,11";2=count select from srvsel.getserverstable[] where handle in 10 11i;1b] +/ skip already-active handles (4 active, 12 new) +conntab2:([]w:4 12i;proctype:`rdb`rdb;attributes:(()!();()!())) +n0:count srvsel.getserverstable[] +srvsel.addserversfromtable[`rdb;conntab2] +chk["already-active handle 4 skipped, only 12 added";n0+1;count srvsel.getserverstable[]] +/ proctype filter: only rdb of a mixed table +conntab3:([]w:20 21i;proctype:`tickerplant`rdb;attributes:(()!();()!())) +srvsel.addserversfromtable[`rdb;conntab3] +chk["proctype filter registers only the rdb (handle 21)";1b;(0type configs; - '"di.serverselect: configs must be a dict with `log key"]; - if[not `log in key configs; +init:{[deps] + / wire injectable dependencies - must be called before any other function + / the log dependency is required; there is no fallback + / deps: a dict with a `log key; the log value is `info`warn`error!(...) - monadic + / message loggers (a kx.log instance, or a bespoke logger of the same shape) + / example: di.serverselect.init[enlist[`log]!enlist logdep] + if[99h<>type deps; + '"di.serverselect: deps must be a dict with `log key"]; + if[not `log in key deps; '"di.serverselect: log dependency is required; pass `info`warn`error functions keyed on `log"]; - if[99h<>type configs`log; + if[99h<>type deps`log; '"di.serverselect: log value must be a dict; pass `info`warn`error functions"]; - if[not all (`info`warn`error) in key configs`log; - '"di.serverselect: log dict must have `info`warn`error keys; got: ",(", " sv string key configs`log)]; - .z.m.log:configs`log; + if[not all (`info`warn`error) in key deps`log; + '"di.serverselect: log dict must have `info`warn`error keys; got: ",(", " sv string key deps`log)]; + .z.m.log:deps`log; + }; + +raiseerror:{[msg] + / internal - log an error then signal it, so failures are observable as well as thrown + .z.m.log[`error] msg; + 'msg; }; -/ internal - increment and return the next unique server ID nextserverid:{ + / internal - increment and return the next unique server ID .z.m.serverid:serverid+1i; :.z.m.serverid; }; -/ internal - update last-access timestamp and hit count for a server handle -updatestats:{[h] - .z.m.servers:update lastp:.z.p,hits:hits+1i from servers where handle=h; +updatestats:{[sid] + / internal - update last-access timestamp and hit count for a single server + / keyed on serverid (unique) rather than handle, which may be shared by multiple servers + .z.m.servers:update lastp:.z.p,hits:hits+1i from servers where serverid=sid; }; addserverfull:{[h;pname;st;hp;att] / register a server with full details: handle, procname, servertype, hpup and attribute dictionary - .z.m.log[`info][`addserverfull;"registering server: handle=",(string h),", procname=",string[pname],", type=",string st]; + if[not -6h=type h;raiseerror"di.serverselect: addserverfull: handle must be an int; got type ",string type h]; + if[not -11h=type st;raiseerror"di.serverselect: addserverfull: servertype must be a symbol; got type ",string type st]; + .z.m.log[`info]["addserverfull: registering server: handle=",(string h),", procname=",string[pname],", type=",string st]; .z.m.servers:servers upsert (nextserverid[];h;pname;st;hp;1b;0Np;0i;att); }; @@ -48,7 +69,9 @@ addserver:{[h;st] setserveractive:{[h;a] / mark a registered server active (1b) or inactive (0b); called on connect and disconnect - .z.m.log[`info][`setserveractive;"marking handle=",(string h)," active=",string a]; + if[not -6h=type h;raiseerror"di.serverselect: setserveractive: handle must be an int; got type ",string type h]; + if[not -1h=type a;raiseerror"di.serverselect: setserveractive: active flag must be a boolean; got type ",string type a]; + .z.m.log[`info]["setserveractive: marking handle=",(string h)," active=",string a]; .z.m.servers:update active:a from servers where handle=h; }; @@ -62,11 +85,14 @@ addserversfromtable:{[proctypes;conntable] / conntable must have columns: w (int handle), proctype (symbol), attributes (dict per row) / optional columns: procname (symbol), hpup (symbol) - populated from conntable if present / pass proctypes:`ALL to register all process types + if[not all `w`proctype`attributes in cols conntable; + raiseerror"di.serverselect: addserversfromtable: conntable must have columns w, proctype and attributes; got: ",", " sv string cols conntable; + ]; activehandles:(0i;0Ni),exec handle from servers where active; rows:select from conntable where ((proctype in proctypes) or proctypes~`ALL), not w in activehandles; - .z.m.log[`info][`addserversfromtable;"registering ",(string count rows)," servers from connection table"]; + .z.m.log[`info]["addserversfromtable: registering ",(string count rows)," servers from connection table"]; pnames:$[`procname in cols rows; rows`procname; count[rows]#`]; hpups:$[`hpup in cols rows; rows`hpup; count[rows]#`]; addserverfull'[rows`w;pnames;rows`proctype;hpups;rows`attributes]; @@ -90,7 +116,9 @@ getservers:{[nameortype;lookups;req] select serverid,procname,servertype,hpup,handle,lastp,attributes from servers where active; nameortype~`servertype; select serverid,procname,servertype,hpup,handle,lastp,attributes from servers where active,servertype in lookups; - select serverid,procname,servertype,hpup,handle,lastp,attributes from servers where active,procname in lookups]; + nameortype~`procname; + select serverid,procname,servertype,hpup,handle,lastp,attributes from servers where active,procname in lookups; + raiseerror"di.serverselect: getservers: nameortype must be `servertype or `procname; got: ",string nameortype]; if[0=count r;:update attribmatch:attributes from r]; am:attributematch[req] each r`attributes; :update attribmatch:am from r; @@ -102,7 +130,7 @@ selector:{[servertable;selection] :$[selection=`roundrobin; first `lastp xasc servertable; selection=`any; rand servertable; selection=`last; last `lastp xasc servertable; - '"di.serverselect: unknown selection strategy: ",string selection]; + raiseerror"di.serverselect: unknown selection strategy: ",string selection]; }; getserverbytype:{[ptype;serverval;selection] @@ -111,7 +139,7 @@ getserverbytype:{[ptype;serverval;selection] r:getservers[`servertype;ptype;()!()]; if[not count r;:()]; r:selector[r;selection]; - updatestats[r`handle]; + updatestats[r`serverid]; :r serverval; }; @@ -121,36 +149,43 @@ gethpbytype:getserverbytype[;`hpup;]; getserverids:{[att] / return server IDs matching a servertype list or attribute requirement dictionary / att: symbol list of servertypes, or dict of attribute requirements (optionally keyed on `servertype) - if[99h<>type att; - if[not 11h=abs type att; - '"di.serverselect: servertype must be a symbol list (11h) or attribute dict (99h)"; - ]; - servertype:distinct att,(); - activeservers:exec distinct servertype from servers where active; - allservers:exec distinct servertype from servers; - activeserversmsg:". available servers: ",", " sv string activeservers; - if[any null att; - '"di.serverselect: null servertype passed as argument",activeserversmsg; - ]; - if[count servertype except activeservers; - '"di.serverselect: ", - $[max not servertype in allservers; - "not valid servers: ",", " sv string servertype except allservers; - "requested servers currently inactive: ",", " sv string servertype except activeservers - ],activeserversmsg; - ]; - :(exec serverid by servertype from servers where active)[servertype]; - ]; + / dispatch the symbol-list path to getserveridsbytype and the dict path to getserveridstype + if[99h<>type att; :getserveridsbytype att]; serverids:$[`servertype in key att; raze getserveridstype[delete servertype from att] each (),att`servertype; getserveridstype[att;`all]]; - if[all 0=count each serverids;'"di.serverselect: no servers match requested attributes"]; + if[all 0=count each serverids; + raiseerror"di.serverselect: no servers match requested attributes"; + ]; :serverids; }; -/ internal - filter active servers by servertype and attribute requirements +getserveridsbytype:{[att] + / internal - resolve server IDs for a servertype symbol or symbol list + / validates each requested type is registered and currently active + if[not 11h=abs type att; + raiseerror"di.serverselect: servertype must be a symbol list (11h) or attribute dict (99h)"; + ]; + servertype:distinct att,(); + activeservers:exec distinct servertype from servers where active; + allservers:exec distinct servertype from servers; + activeserversmsg:". available servers: ",", " sv string activeservers; + if[any null att; + raiseerror"di.serverselect: null servertype passed as argument",activeserversmsg; + ]; + if[count servertype except activeservers; + raiseerror"di.serverselect: ", + $[max not servertype in allservers; + "not valid servers: ",", " sv string servertype except allservers; + "requested servers currently inactive: ",", " sv string servertype except activeservers + ],activeserversmsg; + ]; + :(exec serverid by servertype from servers where active)[servertype]; + }; + getserveridstype:{[att;typ] - / resolve besteffort and attributetype control keys then dispatch to cross or independent matching + / internal - filter active servers by servertype and attribute requirements + / resolves besteffort and attributetype control keys then dispatches to cross or independent matching besteffort:1b; attype:`cross; svrs:$[typ=`all; @@ -168,28 +203,30 @@ getserveridstype:{[att;typ] getserversindependent[att;svrs;besteffort]; getserverscross[att;svrs;besteffort]]; serverids:first value flip $[99h=type res; key res; res]; - if[all 0=count each serverids;'"di.serverselect: no servers match ",string[typ]," requested attributes"]; + if[all 0=count each serverids; + raiseerror"di.serverselect: no servers match ",string[typ]," requested attributes"; + ]; :serverids; }; / internal - build a cross product table from a nested dictionary buildcross:{(cross/){flip (enlist y)#x}[x] each key x}; -/ internal - initial filter shared by cross and independent matching -/ drops servers missing any required attribute key, ranks survivors by coverage getserversinitial:{[req;att] + / internal - initial filter shared by cross and independent matching + / drops servers missing any required attribute key, ranks survivors by coverage if[0=count req; :([]serverid:enlist key att)]; att:(where all each (key req) in/: key each att)#att; - if[not count att;'"di.serverselect: no servers report all requested attributes"]; + if[not count att;raiseerror"di.serverselect: no servers report all requested attributes"]; s:update serverid:key att from value req in'/: (key req)#/:att; s:s idesc value min each sum each' `serverid xkey s; s:`serverid xkey 0!(key req) xgroup s; :s; }; -/ internal - find servers satisfying the cross product of all attribute requirements -/ each attribute combination must be coverable by a single server getserverscross:{[req;att;besteffort] + / internal - find servers satisfying the cross product of all attribute requirements + / each attribute combination must be coverable by a single server if[0=count req; :([]serverid:enlist key att)]; s:getserversinitial[req;att]; reqcross:buildcross[req]; @@ -198,22 +235,22 @@ getserverscross:{[req;att;besteffort] {[x;y;z] (y[0] except found; y[0] inter found:$[0=count y[0];y[0];buildcross x@'where each z])}[req]\ )[(reqcross;());value s]; if[(count last util`remaining) and not besteffort; - '"di.serverselect: cannot satisfy query - cross product of all attributes cannot be matched"; + raiseerror"di.serverselect: cannot satisfy query - cross product of all attributes cannot be matched"; ]; s:1!(0!s) w:where not 0=count each util`found; :(key s)!distinct each' flip each util[w]`found; }; -/ internal - find servers satisfying attribute requirements independently -/ each individual requirement only needs to be matched by one server getserversindependent:{[req;att;besteffort] + / internal - find servers satisfying attribute requirements independently + / each individual requirement only needs to be matched by one server if[0=count req; :([]serverid:enlist key att)]; s:getserversinitial[req;att]; / mask out server groups whose contribution is already covered by earlier groups filter:(value s)¬ -1 _ (0b&(value s) enlist 0),maxs value s; alldone:1+first where all each all each' maxs value s; if[(null alldone) and not besteffort; - '"di.serverselect: cannot satisfy query - not all attributes can be matched"; + raiseerror"di.serverselect: cannot satisfy query - not all attributes can be matched"; ]; s:1!(0!s) w:where any each any each' filter; :(key s)!{(key x)!(value x)@'where each y key x}[req] each value s&filter w; diff --git a/di/serverselect/test.csv b/di/serverselect/test.csv index febd74c2..7ef42284 100644 --- a/di/serverselect/test.csv +++ b/di/serverselect/test.csv @@ -1,20 +1,7 @@ action,ms,bytes,lang,code,repeat,minver,comment before,0,0,q,srvsel:use`di.serverselect,1,1,load module -before,0,0,q,mylog:`info`warn`error!({[c;m]};{[c;m]};{[c;m]}),1,1,no-op mock logger -before,0,0,q,"caplog:([] fn:`symbol$();ctx:`symbol$();msg:())",1,1,initialise log capture table -before,0,0,q,logcap:`info`warn`error!({[c;m] `caplog upsert(`info;c;m)};{[c;m] `caplog upsert(`warn;c;m)};{[c;m] `caplog upsert(`error;c;m)}),1,1,define capturing mock logger -before,0,0,q,srvsel.init[enlist[`log]!enlist mylog],1,1,init with no-op logger for setup - -/ ---- init: dependency injection validation ---- -comment,,,,,,,init - dependency injection validation -run,0,0,q,srvsel.init[enlist[`log]!enlist mylog],1,1,init accepts valid log dependency -fail,0,0,q,srvsel.init[(::)],1,1,init rejects :: (not a dict) -fail,0,0,q,srvsel.init[()!()],1,1,init rejects empty config (missing log key) -fail,0,0,q,srvsel.init[enlist[`other]!enlist mylog],1,1,init rejects config without log key -fail,0,0,q,srvsel.init[enlist[`log]!enlist(::)],1,1,init rejects log dep that is not a dict -fail,0,0,q,srvsel.init[enlist[`log]!enlist 42],1,1,init rejects non-dict log value -fail,0,0,q,srvsel.init[enlist[`log]!enlist(`info`warn!(mylog`info;mylog`warn))],1,1,init rejects log dict missing required key -true,0,0,q,"(15#@[srvsel.init;enlist[`log]!enlist(`info`warn!(mylog`info;mylog`warn));{x}])~""di.serverselect""",1,1,error message is prefixed di.serverselect +before,0,0,q,"noop:`info`warn`error!({[m]};{[m]};{[m]})",1,1,define a no-op monadic logger +before,0,0,q,srvsel.init[enlist[`log]!enlist noop],1,1,init the module with the required log dependency before use / ---- getserverstable: initial state ---- comment,,,,,,,getserverstable - initial state before any addserverattr @@ -73,6 +60,14 @@ true,0,0,q,(srvsel.gethpbytype[`hdb;`roundrobin]) in exec hpup from srvsel.getse run,0,0,q,srvsel.gethandlebytype[`rdb;`roundrobin],1,1,gethandlebytype call updates lastp and hits via updatestats true,0,0,q,"0<(exec hits from srvsel.getserverstable[] where handle in exec handle from srvsel.getserverstable[] where servertype=`rdb,active)[0]",1,1,hits incremented after gethandlebytype call +/ ---- updatestats: keyed on serverid not handle ---- +comment,,,,,,,updatestats - a selection updates only the chosen serverid even when a handle is shared +run,0,0,q,"srvsel.addserverfull[60i;`dupa;`zdb;`:h:1;()!()]",1,1,register first server on shared handle 60 (unique servertype zdb) +run,0,0,q,"srvsel.addserverfull[60i;`dupb;`zdb;`:h:2;()!()]",1,1,register second server on same handle 60 +run,0,0,q,srvsel.gethandlebytype[`zdb;`roundrobin],1,1,select one zdb server by handle +true,0,0,q,1=sum exec hits from srvsel.getserverstable[] where servertype=`zdb,1,1,exactly one hit recorded across both shared-handle servers (not two) +true,0,0,q,"1=count select from srvsel.getserverstable[] where servertype=`zdb,not null lastp",1,1,only the selected serverid had its lastp updated + / ---- getserverids: symbol list path ---- comment,,,,,,,getserverids - symbol list (servertype) path true,0,0,q,0 Date: Thu, 25 Jun 2026 16:46:34 +0100 Subject: [PATCH 05/14] updating tests --- di/serverselect/integration.q | 4 ++-- di/serverselect/serverselect.md | 22 +++++++++------------- di/serverselect/test.csv | 6 +++--- 3 files changed, 14 insertions(+), 18 deletions(-) diff --git a/di/serverselect/integration.q b/di/serverselect/integration.q index f76b3b4f..4d00205c 100644 --- a/di/serverselect/integration.q +++ b/di/serverselect/integration.q @@ -185,8 +185,8 @@ chk["injected logger receives the message text";1b;any caplog[`msg] like "*regis caplog:0#caplog @[{srvsel.selector[([]handle:1 2i;lastp:2#.z.p);`bogus]};();{}] chk["injected logger captures an error message";1b;0 Date: Mon, 29 Jun 2026 11:33:54 +0100 Subject: [PATCH 06/14] adding in the standard logging block and ensuring .z.m is used correctly throughout. Fixed a couple errors --- di/serverselect/init.q | 6 ++- di/serverselect/integration.q | 12 +++--- di/serverselect/serverselect.md | 28 +++++++------- di/serverselect/serverselect.q | 67 ++++++++++++++++++++------------- di/serverselect/test.csv | 6 +-- 5 files changed, 69 insertions(+), 50 deletions(-) diff --git a/di/serverselect/init.q b/di/serverselect/init.q index 63dd8baa..135b1e20 100644 --- a/di/serverselect/init.q +++ b/di/serverselect/init.q @@ -3,6 +3,8 @@ / logging is an injected dependency: the start-up script that wires the modules together - / or the user at run time - must call init with a required `log dependency before using the / module. kx.log is intentionally NOT loaded here. -/ note: the injected log dict is stored in .z.m.log and called as .z.m.log[`info]["msg"] +/ note: the injected log dict is stored in .z.m.log and called as .z.m.log[`info][`ctx;"msg"] -export:([init;addserverfull;addserverattr;addserver;setserveractive;getserverstable;addserversfromtable;getservers;selector;getserverbytype;gethandlebytype;gethpbytype;getserverids]) +export:([init; + addserverfull;addserverattr;addserver;setserveractive;getserverstable;addserversfromtable; + getservers;selector;getserverbytype;gethandlebytype;gethpbytype;getserverids]) diff --git a/di/serverselect/integration.q b/di/serverselect/integration.q index 4d00205c..c73a8e32 100644 --- a/di/serverselect/integration.q +++ b/di/serverselect/integration.q @@ -35,7 +35,7 @@ hdr:{[t] -1""; -1"==== ",t," ===="; }; / init must be called before use (log is a required injected dependency, no default). / use a no-op logger for the behavioural steps; STEP 13 exercises init itself. -srvsel.init[enlist[`log]!enlist `info`warn`error!({[m]};{[m]};{[m]})] +srvsel.init[enlist[`log]!enlist `info`warn`error!({[c;m]};{[c;m]};{[c;m]})] d1:2024.01.01; d2:2024.01.02; d3:2024.01.03 @@ -91,7 +91,8 @@ chk["servertype list rdb,hdb returns 4";4;count srvsel.getservers[`servertype;`r chk["procname=gw1 returns the gateway";enlist 8i;exec handle from srvsel.getservers[`procname;`gw1;()!()]] chk["unknown servertype returns empty";0;count srvsel.getservers[`servertype;`nope;()!()]] chk["attribmatch column present";1b;`attribmatch in cols srvsel.getservers[`;`;()!()]] -chk["complete date match flagged for some hdb";1b;any {x[`date]0} each (srvsel.getservers[`servertype;`hdb;(enlist`date)!enlist enlist d1])`attribmatch] +chk["complete date match flagged for some hdb";1b; + any {x[`date]0} each (srvsel.getservers[`servertype;`hdb;(enlist`date)!enlist enlist d1])`attribmatch] / =========================================================================== hdr"STEP 7 selector (strategies on a known table)" @@ -151,7 +152,8 @@ chk["already-active handle 4 skipped, only 12 added";n0+1;count srvsel.getserver / proctype filter: only rdb of a mixed table conntab3:([]w:20 21i;proctype:`tickerplant`rdb;attributes:(()!();()!())) srvsel.addserversfromtable[`rdb;conntab3] -chk["proctype filter registers only the rdb (handle 21)";1b;(0type deps; '"di.serverselect: deps must be a dict with `log key"]; @@ -28,13 +28,26 @@ init:{[deps] '"di.serverselect: log value must be a dict; pass `info`warn`error functions"]; if[not all (`info`warn`error) in key deps`log; '"di.serverselect: log dict must have `info`warn`error keys; got: ",(", " sv string key deps`log)]; - .z.m.log:deps`log; + .z.m.log:normlog deps`log; }; -raiseerror:{[msg] - / internal - log an error then signal it, so failures are observable as well as thrown - .z.m.log[`error] msg; - 'msg; +normlog:{[logdict] + / internal - normalise the injected logger to a binary `info`warn`error!{[c;m]} dict + / detect kx.log instance by presence of kx.log-specific keys (getlvl, sinks, fmts) + / kx.log functions are monadic - wrap each into binary {[c;m]} and embed context in the message + / plain {[c;m]} log dicts (info`warn`error only) pass through unchanged + $[any `getlvl`sinks`fmts in key logdict; + `info`warn`error!( + {[fn;c;m] fn[string[c],": ",m]}[logdict`info;]; + {[fn;c;m] fn[string[c],": ",m]}[logdict`warn;]; + {[fn;c;m] fn[string[c],": ",m]}[logdict`error;]); + logdict] + }; + +raiseerror:{[ctx;msg] + / internal - log an error under ctx then signal it, so failures are observable as well as thrown + .z.m.log[`error][ctx;msg]; + '"di.serverselect: ",string[ctx],": ",msg; }; nextserverid:{ @@ -51,9 +64,9 @@ updatestats:{[sid] addserverfull:{[h;pname;st;hp;att] / register a server with full details: handle, procname, servertype, hpup and attribute dictionary - if[not -6h=type h;raiseerror"di.serverselect: addserverfull: handle must be an int; got type ",string type h]; - if[not -11h=type st;raiseerror"di.serverselect: addserverfull: servertype must be a symbol; got type ",string type st]; - .z.m.log[`info]["addserverfull: registering server: handle=",(string h),", procname=",string[pname],", type=",string st]; + if[not -6h=type h;raiseerror[`addserverfull;"handle must be an int; got type ",string type h]]; + if[not -11h=type st;raiseerror[`addserverfull;"servertype must be a symbol; got type ",string type st]]; + .z.m.log[`info][`addserverfull;"registering server: handle=",(string h),", procname=",string[pname],", type=",string st]; .z.m.servers:servers upsert (nextserverid[];h;pname;st;hp;1b;0Np;0i;att); }; @@ -69,9 +82,9 @@ addserver:{[h;st] setserveractive:{[h;a] / mark a registered server active (1b) or inactive (0b); called on connect and disconnect - if[not -6h=type h;raiseerror"di.serverselect: setserveractive: handle must be an int; got type ",string type h]; - if[not -1h=type a;raiseerror"di.serverselect: setserveractive: active flag must be a boolean; got type ",string type a]; - .z.m.log[`info]["setserveractive: marking handle=",(string h)," active=",string a]; + if[not -6h=type h;raiseerror[`setserveractive;"handle must be an int; got type ",string type h]]; + if[not -1h=type a;raiseerror[`setserveractive;"active flag must be a boolean; got type ",string type a]]; + .z.m.log[`info][`setserveractive;"marking handle=",(string h)," active=",string a]; .z.m.servers:update active:a from servers where handle=h; }; @@ -86,13 +99,13 @@ addserversfromtable:{[proctypes;conntable] / optional columns: procname (symbol), hpup (symbol) - populated from conntable if present / pass proctypes:`ALL to register all process types if[not all `w`proctype`attributes in cols conntable; - raiseerror"di.serverselect: addserversfromtable: conntable must have columns w, proctype and attributes; got: ",", " sv string cols conntable; + raiseerror[`addserversfromtable;"conntable must have columns w, proctype and attributes; got: ",", " sv string cols conntable]; ]; activehandles:(0i;0Ni),exec handle from servers where active; rows:select from conntable where ((proctype in proctypes) or proctypes~`ALL), not w in activehandles; - .z.m.log[`info]["addserversfromtable: registering ",(string count rows)," servers from connection table"]; + .z.m.log[`info][`addserversfromtable;"registering ",(string count rows)," servers from connection table"]; pnames:$[`procname in cols rows; rows`procname; count[rows]#`]; hpups:$[`hpup in cols rows; rows`hpup; count[rows]#`]; addserverfull'[rows`w;pnames;rows`proctype;hpups;rows`attributes]; @@ -118,7 +131,7 @@ getservers:{[nameortype;lookups;req] select serverid,procname,servertype,hpup,handle,lastp,attributes from servers where active,servertype in lookups; nameortype~`procname; select serverid,procname,servertype,hpup,handle,lastp,attributes from servers where active,procname in lookups; - raiseerror"di.serverselect: getservers: nameortype must be `servertype or `procname; got: ",string nameortype]; + raiseerror[`getservers;"nameortype must be `servertype or `procname; got: ",string nameortype]]; if[0=count r;:update attribmatch:attributes from r]; am:attributematch[req] each r`attributes; :update attribmatch:am from r; @@ -130,7 +143,7 @@ selector:{[servertable;selection] :$[selection=`roundrobin; first `lastp xasc servertable; selection=`any; rand servertable; selection=`last; last `lastp xasc servertable; - raiseerror"di.serverselect: unknown selection strategy: ",string selection]; + raiseerror[`selector;"unknown selection strategy: ",string selection]]; }; getserverbytype:{[ptype;serverval;selection] @@ -155,7 +168,7 @@ getserverids:{[att] raze getserveridstype[delete servertype from att] each (),att`servertype; getserveridstype[att;`all]]; if[all 0=count each serverids; - raiseerror"di.serverselect: no servers match requested attributes"; + raiseerror[`getserverids;"no servers match requested attributes"]; ]; :serverids; }; @@ -164,21 +177,21 @@ getserveridsbytype:{[att] / internal - resolve server IDs for a servertype symbol or symbol list / validates each requested type is registered and currently active if[not 11h=abs type att; - raiseerror"di.serverselect: servertype must be a symbol list (11h) or attribute dict (99h)"; + raiseerror[`getserveridsbytype;"servertype must be a symbol list (11h) or attribute dict (99h)"]; ]; servertype:distinct att,(); activeservers:exec distinct servertype from servers where active; allservers:exec distinct servertype from servers; activeserversmsg:". available servers: ",", " sv string activeservers; if[any null att; - raiseerror"di.serverselect: null servertype passed as argument",activeserversmsg; + raiseerror[`getserveridsbytype;"null servertype passed as argument",activeserversmsg]; ]; if[count servertype except activeservers; - raiseerror"di.serverselect: ", + raiseerror[`getserveridsbytype; $[max not servertype in allservers; "not valid servers: ",", " sv string servertype except allservers; "requested servers currently inactive: ",", " sv string servertype except activeservers - ],activeserversmsg; + ],activeserversmsg]; ]; :(exec serverid by servertype from servers where active)[servertype]; }; @@ -204,7 +217,7 @@ getserveridstype:{[att;typ] getserverscross[att;svrs;besteffort]]; serverids:first value flip $[99h=type res; key res; res]; if[all 0=count each serverids; - raiseerror"di.serverselect: no servers match ",string[typ]," requested attributes"; + raiseerror[`getserveridstype;"no servers match ",string[typ]," requested attributes"]; ]; :serverids; }; @@ -217,7 +230,7 @@ getserversinitial:{[req;att] / drops servers missing any required attribute key, ranks survivors by coverage if[0=count req; :([]serverid:enlist key att)]; att:(where all each (key req) in/: key each att)#att; - if[not count att;raiseerror"di.serverselect: no servers report all requested attributes"]; + if[not count att;raiseerror[`getserversinitial;"no servers report all requested attributes"]]; s:update serverid:key att from value req in'/: (key req)#/:att; s:s idesc value min each sum each' `serverid xkey s; s:`serverid xkey 0!(key req) xgroup s; @@ -235,7 +248,7 @@ getserverscross:{[req;att;besteffort] {[x;y;z] (y[0] except found; y[0] inter found:$[0=count y[0];y[0];buildcross x@'where each z])}[req]\ )[(reqcross;());value s]; if[(count last util`remaining) and not besteffort; - raiseerror"di.serverselect: cannot satisfy query - cross product of all attributes cannot be matched"; + raiseerror[`getserverscross;"cannot satisfy query - cross product of all attributes cannot be matched"]; ]; s:1!(0!s) w:where not 0=count each util`found; :(key s)!distinct each' flip each util[w]`found; @@ -250,7 +263,7 @@ getserversindependent:{[req;att;besteffort] filter:(value s)¬ -1 _ (0b&(value s) enlist 0),maxs value s; alldone:1+first where all each all each' maxs value s; if[(null alldone) and not besteffort; - raiseerror"di.serverselect: cannot satisfy query - not all attributes can be matched"; + raiseerror[`getserversindependent;"cannot satisfy query - not all attributes can be matched"]; ]; s:1!(0!s) w:where any each any each' filter; :(key s)!{(key x)!(value x)@'where each y key x}[req] each value s&filter w; diff --git a/di/serverselect/test.csv b/di/serverselect/test.csv index 57116579..59128c93 100644 --- a/di/serverselect/test.csv +++ b/di/serverselect/test.csv @@ -1,6 +1,6 @@ action,ms,bytes,lang,code,repeat,minver,comment before,0,0,q,srvsel:use`di.serverselect,1,1,load module -before,0,0,q,"noop:`info`warn`error!({[m]};{[m]};{[m]})",1,1,define a no-op monadic logger +before,0,0,q,"noop:`info`warn`error!({[c;m]};{[c;m]};{[c;m]})",1,1,define a no-op binary logger {[c;m]} before,0,0,q,srvsel.init[enlist[`log]!enlist noop],1,1,init the module with the required log dependency before use / ---- getserverstable: initial state ---- @@ -175,7 +175,7 @@ fail,0,0,q,"srvsel.getserverids[`servertype`date!(`nonexistent;enlist 2024.01.01 / ---- init: dependency injection (required log) ---- comment,,,,,,,init - inject and validate the required log dependency run,0,0,q,"caplog:([]lvl:`symbol$();msg:())",1,1,initialise log capture table -run,0,0,q,"caplogger:`info`warn`error!({[m]`caplog upsert(`info;m)};{[m]`caplog upsert(`warn;m)};{[m]`caplog upsert(`error;m)})",1,1,define a bespoke capturing logger (info/warn/error single-arg) +run,0,0,q,"caplogger:`info`warn`error!({[c;m]`caplog upsert(`info;m)};{[c;m]`caplog upsert(`warn;m)};{[c;m]`caplog upsert(`error;m)})",1,1,define a bespoke capturing binary logger {[c;m]} run,0,0,q,srvsel.init[enlist[`log]!enlist caplogger],1,1,re-init with the bespoke capturing logger run,0,0,q,caplog:0#caplog,1,1,reset capture run,0,0,q,"srvsel.addserver[40i;`rdb]",1,1,register a server while the capturing logger is active @@ -188,6 +188,6 @@ fail,0,0,q,srvsel.init[(::)],1,1,init rejects a non-dictionary deps fail,0,0,q,srvsel.init[()!()],1,1,init rejects deps missing the log key fail,0,0,q,srvsel.init[enlist[`other]!enlist noop],1,1,init rejects deps without a log key fail,0,0,q,srvsel.init[enlist[`log]!enlist(::)],1,1,init rejects a log value that is not a dict -fail,0,0,q,"srvsel.init[enlist[`log]!enlist(`info`warn!({[m]};{[m]}))]",1,1,init rejects a log dict missing the error key +fail,0,0,q,"srvsel.init[enlist[`log]!enlist(`info`warn!({[c;m]};{[c;m]}))]",1,1,init rejects a log dict missing the error key true,0,0,q,"""di.serverselect""~15#@[srvsel.init;(::);{x}]",1,1,init error message is prefixed di.serverselect run,0,0,q,srvsel.init[enlist[`log]!enlist noop],1,1,restore the no-op logger From 5160a23c8ec147ae62ef1e433998c6a64e1a69bc Mon Sep 17 00:00:00 2001 From: ascottDI Date: Mon, 29 Jun 2026 12:10:58 +0100 Subject: [PATCH 07/14] fixing the serverselect.md --- di/serverselect/serverselect.md | 41 +++++++++++++++++++++++++++++---- 1 file changed, 36 insertions(+), 5 deletions(-) diff --git a/di/serverselect/serverselect.md b/di/serverselect/serverselect.md index 1a571e2f..8e8a8f8e 100644 --- a/di/serverselect/serverselect.md +++ b/di/serverselect/serverselect.md @@ -4,7 +4,7 @@ --- -## :sparkles: Features +## Features - Register backend servers with servertype, procname, host/port, and attribute dictionaries - Track active/inactive state per server, updated on connect and disconnect @@ -16,7 +16,7 @@ --- -## :gear: Initialisation & Dependencies +## Initialisation & Dependencies `init` wires the module's injected dependencies and **must be called before any other function**. The `log` dependency is **required** — there is no fallback, and the module does not load `kx.log` itself. Initialising the logging framework is the job of the start-up script that ties the modules together, or of the user at run time. @@ -46,7 +46,7 @@ srvsel.init[enlist[`log]!enlist mylog] --- -## :label: Server Table Schema +## Server Table Schema Servers are tracked in the `servers` keyed table (keyed on `serverid`): @@ -64,7 +64,7 @@ Servers are tracked in the `servers` keyed table (keyed on `serverid`): --- -## :wrench: Functions +## Functions ### Registration @@ -174,7 +174,7 @@ The reserved key `` `besteffort `` (boolean, default `1b`) controls whether a pa --- -## :test_tube: Example +## Example ```q / load and initialise - init must be called before any other function @@ -201,3 +201,34 @@ h:srvsel.gethandlebytype[`rdb; `roundrobin] / inspect registered server pool srvsel.getserverstable[] ``` + +--- + +## Testing + +Both test suites require KDB-X (the `use` module system). Set `QPATH` so `di.*` and `kx.*` modules resolve — it must include this repository and the `kx` module directory: + +```bash +export QPATH=/path/to/kx/mod:/path/to/kdbx-modules +``` + +### Unit tests (k4unit) + +`test.csv` is a k4unit manifest covering every exported function. Run it from a q session (or script): + +```q +k4unit:use`di.k4unit; +k4unit.moduletest`di.serverselect; / prints the results table; "All tests passed" on success +``` + +The `before` rows load the module and `init` it with a no-op logger, so the tests run standalone with no external logging framework. + +### Integration test + +`integration.q` is an end-to-end narrative test that drives every exported function through a realistic gateway lifecycle (register → activate/deactivate → query → select → bulk-register → error handling) and prints a `PASS`/`FAIL` summary. Run it directly: + +```bash +QPATH=/path/to/kx/mod:/path/to/kdbx-modules q integration.q +``` + +It exits with a non-zero code equal to the number of failed assertions (`0` when all pass). From ee7a90e7193c9f51ca81255c3f7974dc4cea1a9b Mon Sep 17 00:00:00 2001 From: ascottDI Date: Tue, 30 Jun 2026 14:06:16 +0100 Subject: [PATCH 08/14] changing intergration tests to actually open other processes rather than pretend, updated testing to cover this and added more robust logging to main script --- di/serverselect/integration.q | 370 +++++++++++++++++++------------- di/serverselect/serverselect.md | 11 +- di/serverselect/serverselect.q | 5 + di/serverselect/test.csv | 1 + 4 files changed, 229 insertions(+), 158 deletions(-) diff --git a/di/serverselect/integration.q b/di/serverselect/integration.q index c73a8e32..b3c334f4 100644 --- a/di/serverselect/integration.q +++ b/di/serverselect/integration.q @@ -1,15 +1,27 @@ / =========================================================================== -/ di.serverselect integration test -/ End-to-end narrative test driving every exported function through a -/ realistic gateway lifecycle: register -> activate/deactivate -> query -> -/ select -> bulk-register -> error handling. Complements the k4unit suite -/ in test.csv with assertions on real usage rather than per-call checks. +/ di.serverselect integration test (real processes) +/ Spawns child q listeners, opens REAL IPC handles to them, and drives every +/ exported function through a realistic gateway lifecycle - register -> +/ query -> select -> route a live query to the chosen handle -> bulk-register +/ -> disconnect -> error handling -> init. Unlike a simulated test, the +/ selection functions hand back handles that are actually queried, so the +/ assertions prove queries reach the expected backend process. +/ . +/ Self-contained: no helper files. The child backends are bare `q` listeners +/ whose identity (`whoami`) and `ping` api are injected over IPC at setup; the +/ test launches and tears them down itself. +/ . +/ Requirements: +/ - run THIS script with KDB-X q (needs `use`) +/ - a q on PATH (override with $QBIN) to launch the child listeners +/ - some free TCP ports: the test scans for free ports from a pid-derived +/ base (so concurrent runs don't collide); set $SSPORT to pin the base / Run (QPATH must include this repo and the kx module dir): -/ QPATH=/path/to/kx/mod:/path/to/kdbx-modules q integration.q +/ QPATH=/path/to/kx/mod:/path/to/kdbx-modules /q integration.q / Exits with a non-zero code equal to the number of failed assertions. / =========================================================================== -srvsel:use`di.serverselect +srvsel:use`di.serverselect; / ---- tiny assertion harness ---- / note: avoid `desc`/`exp`/`log` as names - they are reserved words in KDB-X @@ -33,169 +45,217 @@ chkerr:{[nm;f] }; hdr:{[t] -1""; -1"==== ",t," ===="; }; -/ init must be called before use (log is a required injected dependency, no default). -/ use a no-op logger for the behavioural steps; STEP 13 exercises init itself. -srvsel.init[enlist[`log]!enlist `info`warn`error!({[c;m]};{[c;m]};{[c;m]})] +d1:2024.01.01; d2:2024.01.02; d3:2024.01.03; -d1:2024.01.01; d2:2024.01.02; d3:2024.01.03 +/ ---- child-process management ---- +qbin:$[0count free; + '"could not find ",string[n]," free ports scanning up from ",string start]; + :n#free; + }; +ports:findfree[baseport;count names]; -/ =========================================================================== -hdr"STEP 2 addserver (no attributes)" -srvsel.addserver[4i;`rdb] -srvsel.addserver[5i;`rdb] -chk["two rdb servers registered";2=count srvsel.getserverstable[];1b] -chk["serverids autoincrement from 1";1 2i;exec serverid from srvsel.getserverstable[]] -chk["addserver -> null procname";1b;all null exec procname from srvsel.getserverstable[]] -chk["addserver -> null hpup";1b;all null exec hpup from srvsel.getserverstable[]] -chk["addserver -> empty attributes dict";(()!());first exec attributes from srvsel.getserverstable[] where handle=4i] -chk["addserver -> active by default";1b;all exec active from srvsel.getserverstable[]] +launch:{[port] + / start a bare q listener detached (no script needed - identity is injected over IPC) + system qbin," -p ",string[port]," -q /dev/null 2>&1 &"; + }; -/ =========================================================================== -hdr"STEP 3 addserverattr (servertype + attributes)" -srvsel.addserverattr[6i;`hdb;`date`sym!((d1;d2);`A`B)] -srvsel.addserverattr[7i;`hdb;`date`sym!((d2;d3);`B`C)] -chk["two hdb servers added";4=count srvsel.getserverstable[];1b] -chk["addserverattr stores attributes";`date`sym!((d1;d2);`A`B);first exec attributes from srvsel.getserverstable[] where handle=6i] -chk["addserverattr -> null procname";1b;null first exec procname from srvsel.getserverstable[] where handle=6i] +connect:{[port] + / open a handle, retrying for up to ~10s while the child binds and accepts + hp:`$":localhost:",string port; + step:{[hp;h] if[not null h; :h]; system"sleep 0.2"; @[hopen;(hp;500);{0Ni}]}[hp]; + :50 step/ 0Ni; + }; -/ =========================================================================== -hdr"STEP 4 addserverfull (full details)" -srvsel.addserverfull[8i;`gw1;`gateway;`:host:5000;(enlist`region)!enlist`EU] -chk["fifth server registered";5=count srvsel.getserverstable[];1b] -chk["addserverfull -> procname populated";`gw1;first exec procname from srvsel.getserverstable[] where handle=8i] -chk["addserverfull -> hpup populated";`:host:5000;first exec hpup from srvsel.getserverstable[] where handle=8i] -chk["serverid sequence is 1..5";1 2 3 4 5i;exec serverid from srvsel.getserverstable[]] -chk["handles as registered";4 5 6 7 8i;exec handle from srvsel.getserverstable[]] -chk["servertypes as registered";`rdb`rdb`hdb`hdb`gateway;exec servertype from srvsel.getserverstable[]] +teardown:{[] + / always-run cleanup. first kill by captured pid (exact); then a port-based + / fallback so children are reaped even when we never connected (no pid). + / the [-]p bracket keeps the pattern from matching this very kill command. + live:pids where not null pids; + if[count live; @[system;"kill ",(" " sv string live)," 2>/dev/null; true";{}]]; + {@[system;"pkill -f \"bin/q [-]p ",string[x],"\" 2>/dev/null; true";{}]} each ports; + @[hclose;;{}] each handles where not null handles; + }; -/ =========================================================================== -hdr"STEP 5 setserveractive (deactivate / reactivate)" -srvsel.setserveractive[5i;0b] -chk["handle 5 now inactive";0b;first exec active from srvsel.getserverstable[] where handle=5i] -chk["getservers excludes inactive rdb";1b;not 5i in exec handle from srvsel.getservers[`servertype;`rdb;()!()]] -srvsel.setserveractive[5i;1b] -chk["handle 5 reactivated";1b;first exec active from srvsel.getserverstable[] where handle=5i] -srvsel.setserveractive[9999i;0b] -chk["setserveractive on missing handle is a no-op";0=count select from srvsel.getserverstable[] where handle=9999i;1b] +/ launch the fleet, connect, capture each child's own pid (.z.i) for exact-pid teardown +-1 "selected free ports: ",.Q.s1 ports; +launch each ports; +handles:connect each ports; +pids:{$[null x; 0Ni; @[x;".z.i";{0Ni}]]} each handles; +/ guarantee the fleet is reaped on ANY exit (normal, error-abort, or exit code) +.z.exit:{teardown[]}; +if[any null handles; + -2 "failed to connect to child processes on ports: ",(" " sv string ports where null handles); + exit 3]; +/ inject each backend's identity + ping api over IPC +{[h;nm] h "whoami:`",string nm; h "ping:{whoami}";}'[handles;names]; +-1 "spawned ",(string count handles)," backend processes; real handles: ",.Q.s1 handles; -/ =========================================================================== -hdr"STEP 6 getservers (lookup + attribute match scoring)" -chk["lookups=` returns all active";5;count srvsel.getservers[`;`;()!()]] -chk["servertype=hdb returns 2";2;count srvsel.getservers[`servertype;`hdb;()!()]] -chk["servertype list rdb,hdb returns 4";4;count srvsel.getservers[`servertype;`rdb`hdb;()!()]] -chk["procname=gw1 returns the gateway";enlist 8i;exec handle from srvsel.getservers[`procname;`gw1;()!()]] -chk["unknown servertype returns empty";0;count srvsel.getservers[`servertype;`nope;()!()]] -chk["attribmatch column present";1b;`attribmatch in cols srvsel.getservers[`;`;()!()]] -chk["complete date match flagged for some hdb";1b; - any {x[`date]0} each (srvsel.getservers[`servertype;`hdb;(enlist`date)!enlist enlist d1])`attribmatch] +/ route: select a server of type t via strategy sel, then SEND ping[] over the chosen handle +route:{[t;sel] h:srvsel.gethandlebytype[t;sel]; $[()~h; `NONE; h "ping[]"]}; +/ map serverids -> their handles (to route attribute-path selections) +sidhandles:{[sids] exec handle from srvsel.getserverstable[] where serverid in sids}; +/ the gateway's host/port for the hpup column - its real (dynamically chosen) address +gwhp:`$":localhost:",string ports 4; -/ =========================================================================== -hdr"STEP 7 selector (strategies on a known table)" -seltab:([]serverid:101 102 103i;handle:201 202 203i;lastp:(2024.01.03D0;2024.01.01D0;2024.01.02D0)) -chk["roundrobin picks oldest lastp";202i;(srvsel.selector[seltab;`roundrobin])`handle] -chk["last picks newest lastp";201i;(srvsel.selector[seltab;`last])`handle] -chk["any picks a row from the table";1b;((srvsel.selector[seltab;`any])`handle) in 201 202 203i] -single:([]serverid:enlist 9i;handle:enlist 99i;lastp:enlist 2024.06.01D0) -chk["single-row roundrobin returns that row";99i;(srvsel.selector[single;`roundrobin])`handle] -empt:([]serverid:`int$();handle:`int$();lastp:`timestamp$()) -chk["empty table roundrobin -> null handle";0Ni;(srvsel.selector[empt;`roundrobin])`handle] -chk["empty table any -> null handle";0Ni;(srvsel.selector[empt;`any])`handle] +/ test steps run at top level (a single enclosing function would exceed q's +/ per-function constant limit). chk/chkerr trap their own assertion failures; +/ .z.exit guarantees teardown regardless of how the script ends. -/ =========================================================================== -hdr"STEP 8 getserverbytype / gethandlebytype / gethpbytype" -chk["gethandlebytype rdb returns a live rdb handle";1b;(srvsel.gethandlebytype[`rdb;`roundrobin]) in 4 5i] -chk["gethpbytype gateway returns its hpup";`:host:5000;srvsel.gethpbytype[`gateway;`roundrobin]] -chk["getserverbytype returns requested column value";`gw1;srvsel.getserverbytype[`gateway;`procname;`roundrobin]] -chk["gethandlebytype unknown type -> ()";();srvsel.gethandlebytype[`nope;`roundrobin]] -chk["gethpbytype unknown type -> ()";();srvsel.gethpbytype[`nope;`roundrobin]] -/ round-robin rotation: with exactly 2 active rdbs, 4 LRU picks must cover both -rot:srvsel.gethandlebytype[`rdb] each 4#`roundrobin -chk["roundrobin rotates across both rdbs";4 5i;asc distinct rot] -/ selection bumps hits -hitsbefore:exec sum hits from srvsel.getserverstable[] where servertype=`gateway -srvsel.gethandlebytype[`gateway;`roundrobin] -chk["selection increments hits";1b;(hitsbefore+1)=exec sum hits from srvsel.getserverstable[] where servertype=`gateway] +/ init must be called before use (log is a required injected dependency, no default) +srvsel.init[enlist[`log]!enlist `info`warn`error!({[c;m]};{[c;m]};{[c;m]})]; -/ =========================================================================== -hdr"STEP 9 getserverids - symbol (servertype) path" -chk["single servertype rdb -> rdb serverids";1 2i;asc raze srvsel.getserverids[`rdb]] -chk["servertype list rdb,hdb -> all four";1 2 3 4i;asc raze srvsel.getserverids[`rdb`hdb]] -srvsel.setserveractive[4i;0b]; srvsel.setserveractive[5i;0b] -chkerr["all requested servertype inactive -> error";{srvsel.getserverids[`rdb]}] -srvsel.setserveractive[4i;1b]; srvsel.setserveractive[5i;1b] +/ ========================================================================= +hdr"STEP 1 initial empty state"; +chk["fresh module: server table empty";0=count srvsel.getserverstable[];1b]; +chk["schema columns correct";`serverid`handle`procname`servertype`hpup`active`lastp`hits`attributes;cols srvsel.getserverstable[]]; -/ =========================================================================== -hdr"STEP 10 getserverids - attribute dict path" -chk["single attribute date=d1 matches hdb 6";1b;3i in raze srvsel.getserverids[enlist[`date]!enlist enlist d1]] -chk["cross (d1,d2)x(A,B) satisfied by hdb 6 alone";1b;3i in raze srvsel.getserverids[`date`sym!((d1;d2);`A`B)]] -chk["independent date(d1,d2,d3)";1b;0 hdbs 6,7";3 4i;asc raze srvsel.getserverids[`servertype`date!(`hdb;enlist d2)]] -chk["empty requirement dict -> all active serverids";1 2 3 4 5i;asc raze srvsel.getserverids[()!()]] -chkerr["besteffort=0b impossible -> error";{srvsel.getserverids[`date`besteffort!(enlist 2099.01.01;0b)]}] -chkerr["no server has requested attribute value -> error";{srvsel.getserverids[enlist[`date]!enlist enlist 2099.01.01]}] +/ ========================================================================= +hdr"STEP 2 registration of live handles (addserver / addserverattr / addserverfull)"; +srvsel.addserver[handles 0;`rdb]; +srvsel.addserver[handles 1;`rdb]; +srvsel.addserverattr[handles 2;`hdb;`date`sym!((d1;d2);`A`B)]; +srvsel.addserverattr[handles 3;`hdb;`date`sym!((d2;d3);`B`C)]; +srvsel.addserverfull[handles 4;`gw1;`gateway;gwhp;(enlist`region)!enlist`EU]; +chk["five servers registered";5=count srvsel.getserverstable[];1b]; +chk["serverids autoincrement 1..5";1 2 3 4 5i;exec serverid from srvsel.getserverstable[]]; +chk["handles stored are the real open handles";handles til 5;exec handle from srvsel.getserverstable[]]; +chk["servertypes as registered";`rdb`rdb`hdb`hdb`gateway;exec servertype from srvsel.getserverstable[]]; +chk["addserverfull populated procname";`gw1;first exec procname from srvsel.getserverstable[] where servertype=`gateway]; +chk["addserverfull populated hpup";gwhp;first exec hpup from srvsel.getserverstable[] where servertype=`gateway]; +chk["addserver -> empty attributes";(()!());first exec attributes from srvsel.getserverstable[] where handle=handles 0]; +chk["all active by default";1b;all exec active from srvsel.getserverstable[]]; -/ =========================================================================== -hdr"STEP 11 addserversfromtable (bulk registration)" -conntab:([]w:10 11i;proctype:`rdb`hdb;attributes:(()!();`date`sym!((d1;d2);`A`B))) -srvsel.addserversfromtable[`rdb`hdb;conntab] -chk["bulk-registered handles 10,11";2=count select from srvsel.getserverstable[] where handle in 10 11i;1b] -/ skip already-active handles (4 active, 12 new) -conntab2:([]w:4 12i;proctype:`rdb`rdb;attributes:(()!();()!())) -n0:count srvsel.getserverstable[] -srvsel.addserversfromtable[`rdb;conntab2] -chk["already-active handle 4 skipped, only 12 added";n0+1;count srvsel.getserverstable[]] -/ proctype filter: only rdb of a mixed table -conntab3:([]w:20 21i;proctype:`tickerplant`rdb;attributes:(()!();()!())) -srvsel.addserversfromtable[`rdb;conntab3] -chk["proctype filter registers only the rdb (handle 21)";1b; - (0 null handle";0Ni;(srvsel.selector[empt;`roundrobin])`handle]; -/ =========================================================================== -hdr"STEP 13 init (required log dependency - kx.log or bespoke)" -caplog:([]lvl:`symbol$();msg:()) -caplogger:`info`warn`error!({[c;m]`caplog upsert(`info;m)};{[c;m]`caplog upsert(`warn;m)};{[c;m]`caplog upsert(`error;m)}) -srvsel.init[enlist[`log]!enlist caplogger] -caplog:0#caplog -srvsel.addserver[40i;`rdb] -chk["injected logger captures an info message";1b;0 ()";();srvsel.gethandlebytype[`nope;`roundrobin]]; +chk["a single hdb query is answered by an hdb";1b;(route[`hdb;`roundrobin]) in `hdb1`hdb2]; +hitsbefore:exec sum hits from srvsel.getserverstable[] where servertype=`gateway; +srvsel.gethandlebytype[`gateway;`roundrobin]; +chk["selection increments hits";1b;(hitsbefore+1)=exec sum hits from srvsel.getserverstable[] where servertype=`gateway]; + +/ ========================================================================= +hdr"STEP 6 getserverids - servertype (symbol) path"; +chk["rdb -> the two rdb serverids";1 2i;asc raze srvsel.getserverids[`rdb]]; +chk["rdb,hdb -> all four serverids";1 2 3 4i;asc raze srvsel.getserverids[`rdb`hdb]]; +srvsel.setserveractive[handles 0;0b]; srvsel.setserveractive[handles 1;0b]; +chkerr["all requested servertype inactive -> error";{srvsel.getserverids[`rdb]}]; +srvsel.setserveractive[handles 0;1b]; srvsel.setserveractive[handles 1;1b]; + +/ ========================================================================= +hdr"STEP 7 getserverids - attribute path + routing to the matched backend"; +/ hdb1 covers (d1,d2)/(A,B); hdb2 covers (d2,d3)/(B,C) +chk["date=d1 -> hdb1 only";enlist`hdb1; + distinct {x"ping[]"} each sidhandles raze srvsel.getserverids[enlist[`date]!enlist enlist d1]]; +chk["cross (d1,d2)x(A,B) -> hdb1 alone covers it";enlist`hdb1; + distinct {x"ping[]"} each sidhandles raze srvsel.getserverids[`date`sym!((d1;d2);`A`B)]]; +chk["independent over d1,d2,d3 + A,B,C -> both hdbs contribute";`hdb1`hdb2; + asc distinct {x"ping[]"} each sidhandles raze srvsel.getserverids[`date`sym`attributetype!((d1;d2;d3);`A`B`C;`independent)]]; +chk["servertype key scopes attribute filter to hdbs";3 4i;asc raze srvsel.getserverids[`servertype`date!(`hdb;enlist d2)]]; +chk["empty requirement dict -> all active serverids";1 2 3 4 5i;asc raze srvsel.getserverids[()!()]]; +chkerr["besteffort=0b impossible -> error";{srvsel.getserverids[`date`besteffort!(enlist 2099.01.01;0b)]}]; +chkerr["no server has requested attribute value -> error";{srvsel.getserverids[enlist[`date]!enlist enlist 2099.01.01]}]; + +/ ========================================================================= +hdr"STEP 8 addserversfromtable (bulk registration of live handles)"; +conntab:([]w:handles 5 6;proctype:`rdb`hdb;attributes:(()!();`date`sym!((d1;d3);`A`C))); +srvsel.addserversfromtable[`rdb`hdb;conntab]; +chk["bulk-registered handles 5,6 now present";2=count select from srvsel.getserverstable[] where handle in handles 5 6;1b]; +chk["bulk-registered rdb3 is reachable via its handle";`rdb3;(handles 5)"ping[]"]; +chk["bulk-registered rdb3 now appears in rdb serverids";1b;6i in raze srvsel.getserverids[`rdb]]; +/ skip-already-active: re-offering a live handle must not duplicate it +n0:count srvsel.getserverstable[]; +srvsel.addserversfromtable[`rdb;([]w:enlist handles 0;proctype:enlist`rdb;attributes:enlist()!())]; +chk["already-active handle skipped (no new row)";n0;count srvsel.getserverstable[]]; +/ (proctype filter, `ALL, and optional procname/hpup columns are covered in test.csv) + +/ ========================================================================= +hdr"STEP 9 disconnect: kill a backend, deactivate it, prove routing adapts"; +/ a real gateway's .z.pc handler would call setserveractive[h;0b] on the dropped handle +@[system;"kill ",string[pids 0]," 2>/dev/null; true";{}]; system"sleep 0.3"; +srvsel.setserveractive[handles 0;0b]; +/ active rdbs are now rdb2 (live) and rdb3 (live); the dead rdb1 must never be picked +postresp:{[i] route[`rdb;`roundrobin]} each til 6; +chk["killed/deactivated rdb1 is never selected";1b;not `rdb1 in postresp]; +chk["no routed query hits a dead handle (all answered)";1b;not `NONE in postresp]; +chk["the two remaining live rdbs both serve";`rdb2`rdb3;asc distinct postresp]; + +/ ========================================================================= +hdr"STEP 10 error handling (every guarded path signals + logs)"; +chkerr["addserverfull rejects non-int handle";{srvsel.addserverfull[`bad;`;`rdb;`;()!()]}]; +chkerr["addserverfull rejects non-symbol servertype";{srvsel.addserverfull[71i;`;"rdb";`;()!()]}]; +chkerr["addserver rejects bad handle";{srvsel.addserver[`bad;`rdb]}]; +chkerr["addserverattr rejects bad servertype";{srvsel.addserverattr[73i;"hdb";()!()]}]; +chkerr["setserveractive rejects non-int handle";{srvsel.setserveractive["bad";0b]}]; +chkerr["setserveractive rejects non-boolean flag";{srvsel.setserveractive[1i;`yes]}]; +chkerr["getservers rejects bad nameortype";{srvsel.getservers[`badname;`x;()!()]}]; +chkerr["selector rejects unknown strategy";{srvsel.selector[seltab;`bogus]}]; +chkerr["getserverids rejects null servertype";{srvsel.getserverids[`]}]; +chkerr["getserverids rejects unregistered servertype";{srvsel.getserverids[`nope]}]; +chkerr["getserverids rejects non-symbol non-dict arg";{srvsel.getserverids[42]}]; +chkerr["getserverids rejects nested/malformed attribute value";{srvsel.getserverids[(enlist`date)!enlist enlist d1,d2]}]; +chkerr["addserversfromtable rejects missing columns";{srvsel.addserversfromtable[`rdb;([]x:enlist 1i)]}]; + +/ ========================================================================= +hdr"STEP 11 init (required log dependency - bespoke + kx.log)"; +caplog:([]lvl:`symbol$();msg:()); +caplogger:`info`warn`error!({[c;m]`caplog upsert(`info;m)};{[c;m]`caplog upsert(`warn;m)};{[c;m]`caplog upsert(`error;m)}); +srvsel.init[enlist[`log]!enlist caplogger]; +caplog:0#caplog; +srvsel.addserver[40i;`rdb]; +chk["injected logger captures an info message";1b;0type att; :getserveridsbytype att]; + / validate attribute-requirement values are atoms or simple vectors (control keys excepted); + / a nested/general-list value (type 0h) would otherwise crash the cross matcher with a raw 'type + reqkeys:key[att] except `servertype`attributetype`besteffort; + if[count badkeys:reqkeys where 0h=type each att reqkeys; + raiseerror[`getserverids;"attribute requirement values must be atoms or simple vectors; nested/mixed for: ",", " sv string badkeys]]; serverids:$[`servertype in key att; raze getserveridstype[delete servertype from att] each (),att`servertype; getserveridstype[att;`all]]; diff --git a/di/serverselect/test.csv b/di/serverselect/test.csv index 59128c93..1db409d5 100644 --- a/di/serverselect/test.csv +++ b/di/serverselect/test.csv @@ -171,6 +171,7 @@ true,0,0,q,"0 Date: Fri, 3 Jul 2026 09:07:46 +0100 Subject: [PATCH 09/14] Added a fix from the automatic reviewer --- di/serverselect/integration.q | 29 +++++++++++++++++++---------- 1 file changed, 19 insertions(+), 10 deletions(-) diff --git a/di/serverselect/integration.q b/di/serverselect/integration.q index b3c334f4..1975277a 100644 --- a/di/serverselect/integration.q +++ b/di/serverselect/integration.q @@ -56,7 +56,10 @@ pids:`int$(); / no hardcoded ports: derive a starting point from this pid (so concurrent runs / on a shared host don't collide) - or take an explicit base from $SSPORT - then / scan upward for ports that are actually free. nothing assumes a fixed port. -inuse:{[p] @[{hclose hopen x; 1b}; (`$":localhost:",string p;200); 0b]}; +/ probe whether a localhost port is in use: a free port refuses instantly, so this short +/ timeout only bounds the wait for a filtered (silently-dropped) port - keep it small so a +/ full scan of 10*count names candidates cannot stall for many seconds +inuse:{[p] @[{hclose hopen x; 1b}; (`$":localhost:",string p;50); 0b]}; baseport:$[0/dev/null 2>&1 &"; + / start a bare q listener detached and capture its shell background pid ($!) - so every + / child can be reaped by exact pid later, whether or not we ever connect to it, and with + / no dependence on the qbin path (identity/api are injected over IPC once connected) + :"I"$first system qbin," -p ",string[port]," -q /dev/null 2>&1 & echo $!"; }; connect:{[port] @@ -81,20 +86,20 @@ connect:{[port] }; teardown:{[] - / always-run cleanup. first kill by captured pid (exact); then a port-based - / fallback so children are reaped even when we never connected (no pid). - / the [-]p bracket keeps the pattern from matching this very kill command. + / always-run cleanup. kill every launched child by its captured pid (exact). the pkill + / fallback is a last resort for the rare case a pid was never captured - keyed on our own + / unique port and -q flag (NOT the qbin path, which is configurable), so a custom qbin + / cannot break it. the [-]p bracket keeps the pattern from matching this very kill command. live:pids where not null pids; if[count live; @[system;"kill ",(" " sv string live)," 2>/dev/null; true";{}]]; - {@[system;"pkill -f \"bin/q [-]p ",string[x],"\" 2>/dev/null; true";{}]} each ports; + {@[system;"pkill -f \"[-]p ",string[x]," -q\" 2>/dev/null; true";{}]} each ports; @[hclose;;{}] each handles where not null handles; }; -/ launch the fleet, connect, capture each child's own pid (.z.i) for exact-pid teardown +/ launch the fleet (capturing each child's shell pid for exact-pid teardown), then connect -1 "selected free ports: ",.Q.s1 ports; -launch each ports; +pids:launch each ports; handles:connect each ports; -pids:{$[null x; 0Ni; @[x;".z.i";{0Ni}]]} each handles; / guarantee the fleet is reaped on ANY exit (normal, error-abort, or exit code) .z.exit:{teardown[]}; if[any null handles; @@ -212,6 +217,10 @@ chk["already-active handle skipped (no new row)";n0;count srvsel.getserverstable / ========================================================================= hdr"STEP 9 disconnect: kill a backend, deactivate it, prove routing adapts"; / a real gateway's .z.pc handler would call setserveractive[h;0b] on the dropped handle +/ guard: the kill is only a genuine test if rdb1's pid was captured. if pids 0 were null, +/ string 0Ni -> "0Ni" and the kill silently no-ops, degrading STEP 9 into a deactivation-only +/ test. fail loudly here rather than pass on a half-exercised path. +chk["rdb1 pid captured (so the kill below is real)";1b;not null pids 0]; @[system;"kill ",string[pids 0]," 2>/dev/null; true";{}]; system"sleep 0.3"; srvsel.setserveractive[handles 0;0b]; / active rdbs are now rdb2 (live) and rdb3 (live); the dead rdb1 must never be picked From d8167c3e0bd4e8603a1e42e91507385256ddb060 Mon Sep 17 00:00:00 2001 From: ascottDI Date: Tue, 14 Jul 2026 14:43:21 +0100 Subject: [PATCH 10/14] updated following automated reviewer comments --- di/serverselect/init.q | 4 +- di/serverselect/integration.q | 8 ++- di/serverselect/serverselect.md | 29 ++++++----- di/serverselect/serverselect.q | 91 ++++++++++++++++----------------- di/serverselect/test.csv | 2 +- 5 files changed, 69 insertions(+), 65 deletions(-) diff --git a/di/serverselect/init.q b/di/serverselect/init.q index 135b1e20..6d78d41f 100644 --- a/di/serverselect/init.q +++ b/di/serverselect/init.q @@ -3,7 +3,9 @@ / logging is an injected dependency: the start-up script that wires the modules together - / or the user at run time - must call init with a required `log dependency before using the / module. kx.log is intentionally NOT loaded here. -/ note: the injected log dict is stored in .z.m.log and called as .z.m.log[`info][`ctx;"msg"] +/ note: the injected log dict must already be binary `info`warn`error!{[c;m]} - no adaptation +/ is done here; init fans it out into .z.m.loginfo/.z.m.logwarn/.z.m.logerr, called as +/ .z.m.loginfo[`ctx;"msg"] export:([init; addserverfull;addserverattr;addserver;setserveractive;getserverstable;addserversfromtable; diff --git a/di/serverselect/integration.q b/di/serverselect/integration.q index 1975277a..6e4ace17 100644 --- a/di/serverselect/integration.q +++ b/di/serverselect/integration.q @@ -261,7 +261,13 @@ chkerr["init rejects non-dictionary deps";{srvsel.init[(::)]}]; chkerr["init rejects deps missing log key";{srvsel.init[()!()]}]; chkerr["init rejects log value that is not a dict";{srvsel.init[enlist[`log]!enlist(::)]}]; chkerr["init rejects log dict missing error key";{srvsel.init[enlist[`log]!enlist(`info`warn!({[c;m]};{[c;m]}))]}]; -srvsel.init[enlist[`log]!enlist (use`kx.log)[`createLog][]]; +/ the module does no adaptation, so wrap the monadic kx.log instance into a binary {[c;m]} dict caller-side +kxinst:(use`kx.log)[`createLog][]; +kxbinary:`info`warn`error!( + {[c;m]kxinst[`info][string[c],": ",m]}; + {[c;m]kxinst[`warn][string[c],": ",m]}; + {[c;m]kxinst[`error][string[c],": ",m]}); +srvsel.init[enlist[`log]!enlist kxbinary]; srvsel.addserver[41i;`rdb]; chk["kx.log logger wired and usable";1b;41i in exec handle from srvsel.getservers[`servertype;`rdb;()!()]]; diff --git a/di/serverselect/serverselect.md b/di/serverselect/serverselect.md index 4e7147e4..de6716a3 100644 --- a/di/serverselect/serverselect.md +++ b/di/serverselect/serverselect.md @@ -12,7 +12,7 @@ - Query servers by servertype list or attribute requirement dictionary - Attribute matching supports cross-product and independent strategies with configurable best-effort mode - Bulk registration from a TorQ-compatible connection table -- Injected logging — supply your own logger (a `kx.log` instance or bespoke) via `init`; required, no default +- Injected logging — supply your own binary `` `info`warn`error `` logger via `init`; required, no default --- @@ -20,19 +20,15 @@ `init` wires the module's injected dependencies and **must be called before any other function**. The `log` dependency is **required** — there is no fallback, and the module does not load `kx.log` itself. Initialising the logging framework is the job of the start-up script that ties the modules together, or of the user at run time. -The injected logger is normalised by `normlog` to a binary `` `info`warn`error!{[c;m]} `` dict — each function takes a context symbol `c` and a message string `m`, called as `.z.m.log[\`info][\`ctx;"msg"]`. Pass **either** form: - -| Form | What to pass | Behaviour | -|---|---|---| -| `kx.log` instance | the full `(use\`kx.log)[\`createLog][]` dict | detected by its `getlvl`/`sinks`/`fmts` keys; its monadic functions are wrapped to `{[c;m]}`, folding the context in as a `"ctx: msg"` prefix | -| bespoke logger | a `` `info`warn`error!({[c;m]};{[c;m]};{[c;m]}) `` dict | already binary — passed through unchanged | +The `log` value must **already** be a binary `` `info`warn`error!{[c;m]} `` dict — each function takes a context symbol `c` and a message string `m`. `init` performs **no** adaptation and fans the dict out into `.z.m.loginfo`/`.z.m.logwarn`/`.z.m.logerr`, called as `.z.m.loginfo[\`ctx;"msg"]`. Build it from `di.log` (the standard logger, which exports binary `info`/`warn`/`error`) or hand-roll one. A raw monadic `kx.log` instance must be wrapped by the caller first — the module will not do it. ```q srvsel:use`di.serverselect -/ option 1: wire in kx.log (the standard logger; typically done once in the start-up script) -/ pass the WHOLE instance so normlog can detect and wrap it - do NOT strip it to a 3-key dict -srvsel.init[enlist[`log]!enlist (use`kx.log)[`createLog][]] +/ option 1: di.log (the standard logger) - build the dict from its exports +logger:use`di.log +logdep:`info`warn`error!(logger.info;logger.warn;logger.error) +srvsel.init[enlist[`log]!enlist logdep] / option 2: a bespoke binary logger {[c;m]} (context symbol, message string) mylog:`info`warn`error!( @@ -40,9 +36,13 @@ mylog:`info`warn`error!( {[c;m] .my.log.warn string[c],": ",m}; {[c;m] .my.log.error string[c],": ",m}); srvsel.init[enlist[`log]!enlist mylog] + +/ a raw kx.log instance is monadic - wrap it to binary {[c;m]} before passing: +/ kxinst:(use`kx.log)[`createLog][] +/ `info`warn`error!({[c;m]kxinst[`info][string[c],": ",m]};…) ``` -`init` throws with prefix `di.serverselect:` if `deps` is not a dictionary, is missing the `` `log `` key, or the `log` value is not a dictionary exposing `` `info`warn`error ``. All other `di.serverselect:` error conditions are logged via `.z.m.log[\`error]` (with the function as context) before being signalled. +`init` throws with prefix `di.serverselect:` if `deps` is not a dictionary, is missing the `` `log `` key, or the `log` value is not a dictionary exposing `` `info`warn`error ``. All other `di.serverselect:` error conditions are logged via `.z.m.logerr` (with the function as context) before being signalled. --- @@ -104,7 +104,7 @@ srvsel.addserversfromtable[`rdb`hdb; .servers.SERVERS] `getservers` returns a table including an `attribmatch` column — a dictionary of `attrname!(complete_match_bool;matched_values)` per attribute key in `req`. When `lookups` is not `` ` ``, `nameortype` must be `` `servertype `` or `` `procname ``; any other value throws (and logs) a `di.serverselect:` error rather than silently falling through to a `procname` lookup. -All `di.serverselect:` error conditions — including the input-type checks on `addserverfull`/`setserveractive` and the connection-table column check on `addserversfromtable` — are logged via `.z.m.log[\`error]` before being signalled. +All `di.serverselect:` error conditions — including the input-type checks on `addserverfull`/`setserveractive` and the connection-table column check on `addserversfromtable` — are logged via `.z.m.logerr` before being signalled. `selector` supports three strategies: @@ -178,9 +178,10 @@ The reserved key `` `besteffort `` (boolean, default `1b`) controls whether a pa ```q / load and initialise - init must be called before any other function -/ pass the whole kx.log instance; normlog wraps it to the binary {[c;m]} contract +/ pass an already-binary `info`warn`error logger (from di.log or hand-rolled) srvsel:use`di.serverselect -srvsel.init[enlist[`log]!enlist (use`kx.log)[`createLog][]] +logger:use`di.log +srvsel.init[enlist[`log]!enlist `info`warn`error!(logger.info;logger.warn;logger.error)] / register servers as they connect / mark inactive on disconnect / on connect: srvsel.addserverattr[h; getproctype[h]; getattributes[h]] diff --git a/di/serverselect/serverselect.q b/di/serverselect/serverselect.q index 0d057f46..da344a5b 100644 --- a/di/serverselect/serverselect.q +++ b/di/serverselect/serverselect.q @@ -15,11 +15,10 @@ servers:([serverid:`u#`int$()] serverid:0i; init:{[deps] - / wire injectable dependencies (and any optional config) - must be called before any other function - / the log dependency is required; there is no fallback - / deps: a dict with a `log key; the log value is either a kx.log instance or a `info`warn`error - / dict of binary {[c;m]} loggers (context symbol, message string) - normalised via normlog - / example: di.serverselect.init[enlist[`log]!enlist logdep] + / wire the injected logger - required, no silent fallback. deps: a dict with a `log key + / holding a binary `info`warn`error dict of {[c;m]} loggers (from di.log or hand-rolled). + / no adaptation here, so a monadic kx.log instance must be wrapped first. + / e.g. di.serverselect.init[enlist[`log]!enlist logdep] if[99h<>type deps; '"di.serverselect: deps must be a dict with `log key"]; if[not `log in key deps; @@ -28,26 +27,9 @@ init:{[deps] '"di.serverselect: log value must be a dict; pass `info`warn`error functions"]; if[not all (`info`warn`error) in key deps`log; '"di.serverselect: log dict must have `info`warn`error keys; got: ",(", " sv string key deps`log)]; - .z.m.log:normlog deps`log; - }; - -normlog:{[logdict] - / internal - normalise the injected logger to a binary `info`warn`error!{[c;m]} dict - / detect kx.log instance by presence of kx.log-specific keys (getlvl, sinks, fmts) - / kx.log functions are monadic - wrap each into binary {[c;m]} and embed context in the message - / plain {[c;m]} log dicts (info`warn`error only) pass through unchanged - $[any `getlvl`sinks`fmts in key logdict; - `info`warn`error!( - {[fn;c;m] fn[string[c],": ",m]}[logdict`info;]; - {[fn;c;m] fn[string[c],": ",m]}[logdict`warn;]; - {[fn;c;m] fn[string[c],": ",m]}[logdict`error;]); - logdict] - }; - -raiseerror:{[ctx;msg] - / internal - log an error under ctx then signal it, so failures are observable as well as thrown - .z.m.log[`error][ctx;msg]; - '"di.serverselect: ",string[ctx],": ",msg; + .z.m.loginfo:deps[`log]`info; + .z.m.logwarn:deps[`log]`warn; + .z.m.logerr:deps[`log]`error; }; nextserverid:{ @@ -64,9 +46,9 @@ updatestats:{[sid] addserverfull:{[h;pname;st;hp;att] / register a server with full details: handle, procname, servertype, hpup and attribute dictionary - if[not -6h=type h;raiseerror[`addserverfull;"handle must be an int; got type ",string type h]]; - if[not -11h=type st;raiseerror[`addserverfull;"servertype must be a symbol; got type ",string type st]]; - .z.m.log[`info][`addserverfull;"registering server: handle=",(string h),", procname=",string[pname],", type=",string st]; + if[not -6h=type h;.z.m.logerr[`addserverfull;err:"di.serverselect: handle must be an int; got type ",string type h];'err]; + if[not -11h=type st;.z.m.logerr[`addserverfull;err:"di.serverselect: servertype must be a symbol; got type ",string type st];'err]; + .z.m.loginfo[`addserverfull;"registering server: handle=",(string h),", procname=",string[pname],", type=",string st]; .z.m.servers:servers upsert (nextserverid[];h;pname;st;hp;1b;0Np;0i;att); }; @@ -82,9 +64,9 @@ addserver:{[h;st] setserveractive:{[h;a] / mark a registered server active (1b) or inactive (0b); called on connect and disconnect - if[not -6h=type h;raiseerror[`setserveractive;"handle must be an int; got type ",string type h]]; - if[not -1h=type a;raiseerror[`setserveractive;"active flag must be a boolean; got type ",string type a]]; - .z.m.log[`info][`setserveractive;"marking handle=",(string h)," active=",string a]; + if[not -6h=type h;.z.m.logerr[`setserveractive;err:"di.serverselect: handle must be an int; got type ",string type h];'err]; + if[not -1h=type a;.z.m.logerr[`setserveractive;err:"di.serverselect: active flag must be a boolean; got type ",string type a];'err]; + .z.m.loginfo[`setserveractive;"marking handle=",(string h)," active=",string a]; .z.m.servers:update active:a from servers where handle=h; }; @@ -99,13 +81,14 @@ addserversfromtable:{[proctypes;conntable] / optional columns: procname (symbol), hpup (symbol) - populated from conntable if present / pass proctypes:`ALL to register all process types if[not all `w`proctype`attributes in cols conntable; - raiseerror[`addserversfromtable;"conntable must have columns w, proctype and attributes; got: ",", " sv string cols conntable]; + .z.m.logerr[`addserversfromtable;err:"di.serverselect: conntable must have columns w, proctype and attributes; got: ",", " sv string cols conntable]; + 'err; ]; activehandles:(0i;0Ni),exec handle from servers where active; rows:select from conntable where ((proctype in proctypes) or proctypes~`ALL), not w in activehandles; - .z.m.log[`info][`addserversfromtable;"registering ",(string count rows)," servers from connection table"]; + .z.m.loginfo[`addserversfromtable;"registering ",(string count rows)," servers from connection table"]; pnames:$[`procname in cols rows; rows`procname; count[rows]#`]; hpups:$[`hpup in cols rows; rows`hpup; count[rows]#`]; addserverfull'[rows`w;pnames;rows`proctype;hpups;rows`attributes]; @@ -125,13 +108,15 @@ getservers:{[nameortype;lookups;req] / nameortype: `servertype or `procname; pass ` as lookups to return all active servers / req: attribute requirements dict - use ()!() for no attribute filtering / returns table with attribmatch column showing (complete_bool;matched_values) per attribute key + if[(not `~lookups) and not nameortype in `servertype`procname; + .z.m.logerr[`getservers;err:"di.serverselect: nameortype must be `servertype or `procname; got: ",string nameortype]; + 'err; + ]; r:$[`~lookups; select serverid,procname,servertype,hpup,handle,lastp,attributes from servers where active; nameortype~`servertype; select serverid,procname,servertype,hpup,handle,lastp,attributes from servers where active,servertype in lookups; - nameortype~`procname; - select serverid,procname,servertype,hpup,handle,lastp,attributes from servers where active,procname in lookups; - raiseerror[`getservers;"nameortype must be `servertype or `procname; got: ",string nameortype]]; + select serverid,procname,servertype,hpup,handle,lastp,attributes from servers where active,procname in lookups]; if[0=count r;:update attribmatch:attributes from r]; am:attributematch[req] each r`attributes; :update attribmatch:am from r; @@ -140,10 +125,13 @@ getservers:{[nameortype;lookups;req] selector:{[servertable;selection] / pick one row from servertable using the given strategy / selection: `roundrobin (least recently used), `any (random), `last (most recently used) + if[not selection in `roundrobin`any`last; + .z.m.logerr[`selector;err:"di.serverselect: unknown selection strategy: ",string selection]; + 'err; + ]; :$[selection=`roundrobin; first `lastp xasc servertable; selection=`any; rand servertable; - selection=`last; last `lastp xasc servertable; - raiseerror[`selector;"unknown selection strategy: ",string selection]]; + last `lastp xasc servertable]; }; getserverbytype:{[ptype;serverval;selection] @@ -168,12 +156,14 @@ getserverids:{[att] / a nested/general-list value (type 0h) would otherwise crash the cross matcher with a raw 'type reqkeys:key[att] except `servertype`attributetype`besteffort; if[count badkeys:reqkeys where 0h=type each att reqkeys; - raiseerror[`getserverids;"attribute requirement values must be atoms or simple vectors; nested/mixed for: ",", " sv string badkeys]]; + .z.m.logerr[`getserverids;err:"di.serverselect: attribute requirement values must be atoms or simple vectors; nested/mixed for: ",", " sv string badkeys]; + 'err]; serverids:$[`servertype in key att; raze getserveridstype[delete servertype from att] each (),att`servertype; getserveridstype[att;`all]]; if[all 0=count each serverids; - raiseerror[`getserverids;"no servers match requested attributes"]; + .z.m.logerr[`getserverids;err:"di.serverselect: no servers match requested attributes"]; + 'err; ]; :serverids; }; @@ -182,21 +172,23 @@ getserveridsbytype:{[att] / internal - resolve server IDs for a servertype symbol or symbol list / validates each requested type is registered and currently active if[not 11h=abs type att; - raiseerror[`getserveridsbytype;"servertype must be a symbol list (11h) or attribute dict (99h)"]; + .z.m.logerr[`getserveridsbytype;err:"di.serverselect: servertype must be a symbol list (11h) or attribute dict (99h)"]; + 'err; ]; servertype:distinct att,(); activeservers:exec distinct servertype from servers where active; allservers:exec distinct servertype from servers; activeserversmsg:". available servers: ",", " sv string activeservers; if[any null att; - raiseerror[`getserveridsbytype;"null servertype passed as argument",activeserversmsg]; + .z.m.logerr[`getserveridsbytype;err:"di.serverselect: null servertype passed as argument",activeserversmsg]; + 'err; ]; if[count servertype except activeservers; - raiseerror[`getserveridsbytype; - $[max not servertype in allservers; + .z.m.logerr[`getserveridsbytype;err:"di.serverselect: ",$[max not servertype in allservers; "not valid servers: ",", " sv string servertype except allservers; "requested servers currently inactive: ",", " sv string servertype except activeservers ],activeserversmsg]; + 'err; ]; :(exec serverid by servertype from servers where active)[servertype]; }; @@ -222,7 +214,8 @@ getserveridstype:{[att;typ] getserverscross[att;svrs;besteffort]]; serverids:first value flip $[99h=type res; key res; res]; if[all 0=count each serverids; - raiseerror[`getserveridstype;"no servers match ",string[typ]," requested attributes"]; + .z.m.logerr[`getserveridstype;err:"di.serverselect: no servers match ",string[typ]," requested attributes"]; + 'err; ]; :serverids; }; @@ -235,7 +228,7 @@ getserversinitial:{[req;att] / drops servers missing any required attribute key, ranks survivors by coverage if[0=count req; :([]serverid:enlist key att)]; att:(where all each (key req) in/: key each att)#att; - if[not count att;raiseerror[`getserversinitial;"no servers report all requested attributes"]]; + if[not count att;.z.m.logerr[`getserversinitial;err:"di.serverselect: no servers report all requested attributes"];'err]; s:update serverid:key att from value req in'/: (key req)#/:att; s:s idesc value min each sum each' `serverid xkey s; s:`serverid xkey 0!(key req) xgroup s; @@ -253,7 +246,8 @@ getserverscross:{[req;att;besteffort] {[x;y;z] (y[0] except found; y[0] inter found:$[0=count y[0];y[0];buildcross x@'where each z])}[req]\ )[(reqcross;());value s]; if[(count last util`remaining) and not besteffort; - raiseerror[`getserverscross;"cannot satisfy query - cross product of all attributes cannot be matched"]; + .z.m.logerr[`getserverscross;err:"di.serverselect: cannot satisfy query - cross product of all attributes cannot be matched"]; + 'err; ]; s:1!(0!s) w:where not 0=count each util`found; :(key s)!distinct each' flip each util[w]`found; @@ -268,7 +262,8 @@ getserversindependent:{[req;att;besteffort] filter:(value s)¬ -1 _ (0b&(value s) enlist 0),maxs value s; alldone:1+first where all each all each' maxs value s; if[(null alldone) and not besteffort; - raiseerror[`getserversindependent;"cannot satisfy query - not all attributes can be matched"]; + .z.m.logerr[`getserversindependent;err:"di.serverselect: cannot satisfy query - not all attributes can be matched"]; + 'err; ]; s:1!(0!s) w:where any each any each' filter; :(key s)!{(key x)!(value x)@'where each y key x}[req] each value s&filter w; diff --git a/di/serverselect/test.csv b/di/serverselect/test.csv index 1db409d5..f92b617c 100644 --- a/di/serverselect/test.csv +++ b/di/serverselect/test.csv @@ -184,7 +184,7 @@ true,0,0,q,0 Date: Fri, 4 Sep 2026 09:51:44 +0100 Subject: [PATCH 11/14] Adding versioning convention --- di/serverselect/VERSION | 1 + 1 file changed, 1 insertion(+) create mode 100644 di/serverselect/VERSION diff --git a/di/serverselect/VERSION b/di/serverselect/VERSION new file mode 100644 index 00000000..6e8bf73a --- /dev/null +++ b/di/serverselect/VERSION @@ -0,0 +1 @@ +0.1.0 From 81df47c87e8411ada376223289f6581c38275c4d Mon Sep 17 00:00:00 2001 From: alowrydi Date: Tue, 8 Sep 2026 15:12:32 +0100 Subject: [PATCH 12/14] di.serverselect: read version from VERSION file and export it di.depcheck resolves a dependency's version from the module's export dict (checkdepversion) and classes a missing one as a FAILURE, not a warning - so any process loading a module that declares di.serverselect as a hard dependency could not start. di/dataaccess/deps.q already pins it at "0.1.0". Follows the convention already used by di.eodtime, di.dataaccess and di.k4unit: read with @[{trim first read0 x};`:::VERSION;...] rather than a bare `first read0`. trim matters because read0 strips the line terminator but not a trailing \r on a CRLF file, nor trailing spaces, and depcheck compares versions as STRINGS - so a padded value would silently fail every dependent's check. A missing, unreadable or empty file now fails loudly and names the module. VERSION stripped to 5 bytes (no trailing newline) to match the sibling modules. --- di/serverselect/VERSION | 2 +- di/serverselect/init.q | 16 +++++++++++++++- 2 files changed, 16 insertions(+), 2 deletions(-) diff --git a/di/serverselect/VERSION b/di/serverselect/VERSION index 6e8bf73a..6c6aa7cb 100644 --- a/di/serverselect/VERSION +++ b/di/serverselect/VERSION @@ -1 +1 @@ -0.1.0 +0.1.0 \ No newline at end of file diff --git a/di/serverselect/init.q b/di/serverselect/init.q index 6d78d41f..edb5d6f9 100644 --- a/di/serverselect/init.q +++ b/di/serverselect/init.q @@ -7,6 +7,20 @@ / is done here; init fans it out into .z.m.loginfo/.z.m.logwarn/.z.m.logerr, called as / .z.m.loginfo[`ctx;"msg"] +/ module version, read from the VERSION file rather than hardcoded, so a release bump touches one +/ plain-text file. read module-relative (`:::` resolves to di/serverselect) and BEFORE the export +/ line, since export:([...]) evaluates each name. +/ NB `version` must STAY in the export: di.depcheck resolves a dependency's version from the export +/ dict (checkdepversion) and classes a missing one as a FAILURE - which makes di.depcheck.init throw +/ for any process loading a module that declares this one as a hard dependency +/ trim, and fail LOUD on a missing/unreadable/empty VERSION, rather than a bare `first read0`: a raw +/ OS error names no module, and read0 strips the line terminator but NOT a trailing \r on a CRLF file +/ or trailing spaces - and di.depcheck compares versions as STRINGS, so a padded value silently fails +/ every dependent module's check. an empty value is worse still: it reads to depcheck as +/ "exports no version", i.e. the exact failure the VERSION file was added to prevent +version:@[{trim first read0 x};`:::VERSION;{'"di.serverselect: VERSION file missing or unreadable"}]; +if[0=count version;'"di.serverselect: VERSION file is empty"]; + export:([init; addserverfull;addserverattr;addserver;setserveractive;getserverstable;addserversfromtable; - getservers;selector;getserverbytype;gethandlebytype;gethpbytype;getserverids]) + getservers;selector;getserverbytype;gethandlebytype;gethpbytype;getserverids;version]) From b584f842faa8e3a1c6be821bc0898ec8f4c249db Mon Sep 17 00:00:00 2001 From: alowrydi Date: Tue, 8 Sep 2026 15:13:39 +0100 Subject: [PATCH 13/14] di.serverselect: fix 10 defects, add selector/purge/api hooks, harden inputs Three adversarial passes over the attribute-matching engine and the code added to it. The engine (getserverscross/getserversindependent/getserversinitial/ buildcross) was previously reached only INDIRECTLY, via getserverids' end-to-end behaviour, so the suite was green without ever unit-testing it. It is now driven directly through the internal .m.di.0serverselect. path, matching the convention di.dataaccess's suite already uses for getrouting/buildshardquery. The four classic cases all turned out CORRECT: empty requirements degrade to any server of the type, partial attribute coverage excludes only servers actually missing a key, besteffort 0b/1b diverge as intended, and best-match ranking really does prefer the widest server rather than the first candidate found. Defects fixed ------------- 1. Multi-servertype requests aborted entirely if ANY one type matched nothing, discarding good matches from the others. getserverids' own all-empty guard was unreachable dead code - evidence the fan-out was meant to tolerate misses. Per-type misses are now logged at warn; only an all-types miss errors. 2. An atom requirement value ((enlist`date)!enlist 2024.01.01) threw a raw, UNLOGGED 'rank. getservers/attributematch always accepted an atom, so the two entry points disagreed. Atom values are promoted at the boundary. 3. besteffort/attributetype were silently ignored when wrong-typed: besteffort:0 (int) kept the 1b default, an unknown attributetype fell back to cross. Both are now validated - at the boundary, deliberately, because the fan-out in (1) runs under a protected apply and would downgrade a deeper error into a "this servertype did not match" warning. 4. attributes was never validated. A non-dict registered happily and surfaced later, elsewhere, as a raw 'type; and if the FIRST registration was malformed the column took that value's type and every later valid registration failed. 5. removeinactive[0Wn] - infinite age, i.e. "never purge" - DELETED every inactive server: disconnecttime+0Wn overflows the timestamp range and wraps to the year 1734. 0Nn failed likewise via 0Np. Both are now handled. 6. A repeated servertype (`hdb`hdb) was resolved twice and returned the same serverids in two groups. The symbol path already deduped; these disagreed. 7. The single-servertype/`all path had been routed through the tolerant fan-out, losing the engine's specific wording and logging a spurious warn. 8. The nested attrs form (below) silently swallowed unknown top-level keys, so a requirement left outside attrs vanished and a mistyped besteffor was ignored - the flat form errors on both, making the new shape LESS safe than the old. 9. selectorarity gave up on a projection of a projection, accepting an arity-1 strategy that would still throw 'rank later. 10. A keyed table is type 99h too, so (4)'s check let one through; so did a dictionary keyed on non-symbols, reaching the matcher as a raw 'length. New capability -------------- - setselector: pluggable selection strategy. getserverbytype dispatches through a live pointer; selector itself is unchanged, still exported, still the default. Arity is checked at the setselector call site rather than left to throw 'rank at the next getserverbytype. - setserveridactive: retire ONE registration by serverid. setserveractive is unchanged (handle-level, bulk) and remains correct for a real disconnect, but updatestats already keyed on serverid "rather than handle, which may be shared" - activation was the one place that ignored that. - removeinactive + disconnecttime column: age-based purge, so a process that connected once and went away does not sit in the registry forever. Caller-invoked; di.torq schedules it with clearinactivetime. - requireinit on every public entry point, so a pre-init call names itself instead of leaking a raw .m.di.0serverselect.loginfo. version and getapimeta are deliberately exempt - di.torq collects api metadata at startup, possibly before init. - getapimeta: api metadata for all 16 callable exports, for di.torq to register with di.api. init/getapimeta omitted as plumbing. - Nested `attrs request shape: attrs holds the requirements explicitly, leaving the rest of the dict to the controls, so an attribute may be named servertype/ besteffort/attributetype without colliding. Purely additive - the flat form is untouched. - Optional config: cp (injected clock, makes the purge testable without sleeping), clearinactivetime, maxcrossproduct (default 1,000,000, 0W to disable) bounding the combinatorial cost of a client-supplied requirement. - raiseerror/getopt/checkopt factored out, matching di.dataaccess, so the two modules read alike before they sit side by side in a gateway. Two existing assertions had to change ------------------------------------- test.csv:9 and integration.q:129 both assert the exact column list of getserverstable[], and disconnecttime is new. Updated rather than loosened. getservers' columns are selected explicitly and are unaffected, so di.dataaccess (the only real consumer, calling getservers[`servertype;`;()!()]) is unaffected. Testing ------- test.csv: 408 rows, all pass (was 194). integration.q: 91 assertions against real child processes and real IPC, all pass (was 75). qlint clean. --- di/serverselect/init.q | 5 +- di/serverselect/integration.q | 78 ++++++- di/serverselect/serverselect.md | 342 +++++++++++++++++++++++++++- di/serverselect/serverselect.q | 380 ++++++++++++++++++++++++++------ di/serverselect/test.csv | 353 ++++++++++++++++++++++++++++- 5 files changed, 1081 insertions(+), 77 deletions(-) diff --git a/di/serverselect/init.q b/di/serverselect/init.q index edb5d6f9..c0af8685 100644 --- a/di/serverselect/init.q +++ b/di/serverselect/init.q @@ -22,5 +22,6 @@ version:@[{trim first read0 x};`:::VERSION;{'"di.serverselect: VERSION file miss if[0=count version;'"di.serverselect: VERSION file is empty"]; export:([init; - addserverfull;addserverattr;addserver;setserveractive;getserverstable;addserversfromtable; - getservers;selector;getserverbytype;gethandlebytype;gethpbytype;getserverids;version]) + addserverfull;addserverattr;addserver;setserveractive;setserveridactive;getserverstable;addserversfromtable; + getservers;selector;setselector;getserverbytype;gethandlebytype;gethpbytype;getserverids; + removeinactive;getapimeta;version]) diff --git a/di/serverselect/integration.q b/di/serverselect/integration.q index 6e4ace17..ab74ae68 100644 --- a/di/serverselect/integration.q +++ b/di/serverselect/integration.q @@ -126,7 +126,7 @@ srvsel.init[enlist[`log]!enlist `info`warn`error!({[c;m]};{[c;m]};{[c;m]})]; / ========================================================================= hdr"STEP 1 initial empty state"; chk["fresh module: server table empty";0=count srvsel.getserverstable[];1b]; -chk["schema columns correct";`serverid`handle`procname`servertype`hpup`active`lastp`hits`attributes;cols srvsel.getserverstable[]]; +chk["schema columns correct";`serverid`handle`procname`servertype`hpup`active`lastp`hits`attributes`disconnecttime;cols srvsel.getserverstable[]]; / ========================================================================= hdr"STEP 2 registration of live handles (addserver / addserverattr / addserverfull)"; @@ -223,6 +223,10 @@ hdr"STEP 9 disconnect: kill a backend, deactivate it, prove routing adapts"; chk["rdb1 pid captured (so the kill below is real)";1b;not null pids 0]; @[system;"kill ",string[pids 0]," 2>/dev/null; true";{}]; system"sleep 0.3"; srvsel.setserveractive[handles 0;0b]; +chk["disconnect stamped disconnecttime on the dropped handle";1b; + not null first exec disconnecttime from srvsel.getserverstable[] where handle=handles 0]; +chk["a still-active server keeps a null disconnecttime";1b; + null first exec disconnecttime from srvsel.getserverstable[] where handle=handles 1]; / active rdbs are now rdb2 (live) and rdb3 (live); the dead rdb1 must never be picked postresp:{[i] route[`rdb;`roundrobin]} each til 6; chk["killed/deactivated rdb1 is never selected";1b;not `rdb1 in postresp]; @@ -271,6 +275,78 @@ srvsel.init[enlist[`log]!enlist kxbinary]; srvsel.addserver[41i;`rdb]; chk["kx.log logger wired and usable";1b;41i in exec handle from srvsel.getservers[`servertype;`rdb;()!()]]; +/ ========================================================================= +hdr"STEP 12 setselector / removeinactive / module metadata"; +/ STEP 11 registered synthetic rdb handles (40i, 41i) purely to prove each logger was wired - they +/ are NOT live processes, so retire everything that is not one of this test's own child handles +/ before routing again, or a round-robin pick lands on a dead handle. done by difference against +/ `handles` rather than by listing 40i/41i, so adding another synthetic handle upstream cannot rot this +srvsel.setserveractive[;0b] each exec distinct handle from srvsel.getserverstable[] where active,not handle in handles; +/ setselector must demonstrably change WHICH live backend a query reaches, not merely run. +/ pin the strategy to rdb3 and prove ping[] comes back from rdb3 every time. +/ NB the pinned handle is captured by PROJECTION, not read from a global: the strategy is invoked +/ from inside the module, where a bare name resolves against the module's own .z.m rather than the +/ root namespace this file defines, so a global `pinned` reads as unset and the select throws 'type +srvsel.setselector[{[h;servertable;selection] first select from servertable where handle=h}[handles 5]]; +chk["setselector overrode routing - every rdb query reaches the pinned backend";enlist`rdb3; + distinct {route[`rdb;`roundrobin]} each til 4]; +chk["the exported selector itself is unchanged by setselector";handles 1; + (srvsel.selector[select from srvsel.getserverstable[] where handle=handles 1;`roundrobin])`handle]; +srvsel.setselector[srvsel.selector]; +chk["restoring the default puts the live strategy back";1b;.m.di.0serverselect.selector~srvsel.selector]; +chk["default routing spreads across both live rdbs again";`rdb2`rdb3; + asc distinct {route[`rdb;`roundrobin]} each til 8]; +chkerr["setselector rejects a non-function";{srvsel.setselector[42]}]; + +/ removeinactive: rdb1 was killed and deactivated in STEP 9, so it is the real purge candidate +chk["the disconnected backend is still on record before any purge";1b; + 0type d;raiseerror[ctx;nm," must be a dictionary; got type ",string type d]]; + if[not $[0=count k:key d;not 98h=type k;11h=type k]; + raiseerror[ctx;nm," must be a symbol-keyed dictionary; got keys of type ",string type k]]; + }; + +requireinit:{[ctx] + / internal - every public entry point needs init to have run first. without this guard a bare read + / of an unwritten .z.m name surfaces as a raw '.m.di.0serverselect. error, leaking the mangled + / internal namespace to the caller instead of naming the actual problem. signals plainly rather than + / via raiseerror - there is no logger to log through yet. + / getapimeta is deliberately NOT guarded: it is pure data and di.torq collects it at startup, which + / may be before this module's init has run + if[not `logerr in key .z.m;'"di.serverselect: ",string[ctx],": init must be called first"]; + }; + +selectorarity:{[f] + / internal - arity of f WHERE Q CAN TELL, else 0N. + / a lambda reports its own parameter list. a projection's REMAINING arity is its underlying + / function's rank minus the arguments already supplied - counting elided (::) placeholders alone is + / wrong, because a trailing partial application such as f[x] carries none at all and would read as + / arity 0, rejecting a perfectly valid strategy. the underlying rank is resolved recursively so a + / projection OF a projection also works. + / primitives, compositions and adverb-derived functions report nothing usable, so they yield 0N and + / the caller accepts them unchecked - admitting the check does not apply beats rejecting valid input + :$[100h=type f; count value[f]1; + 104h=type f; $[null b:selectorarity first value f; + 0N; + b-count where not (::)~/:1_value f]; + 0N]; + }; + +getopt:{[deps;k;dflt] + / internal - read an optional config key from the deps dict, falling back to a default + $[k in key deps;deps k;dflt] + }; + +checkopt:{[deps;k;ok;what] + / internal - validate an optional config value's TYPE at init, so a misconfiguration fails loudly + / at startup instead of silently at first use + if[k in key deps; + if[not ok deps k;'"di.serverselect: ",string[k]," must be ",what]]; + }; + init:{[deps] - / wire the injected logger - required, no silent fallback. deps: a dict with a `log key - / holding a binary `info`warn`error dict of {[c;m]} loggers (from di.log or hand-rolled). - / no adaptation here, so a monadic kx.log instance must be wrapped first. + / wire the injected logger - required, no silent fallback - and the optional config. + / deps keys: + / log (required) `info`warn`error!{[c;m]} dict - binary, already conforming. + / a raw monadic kx.log instance is NOT adapted here and will 'rank at first use + / cp (optional) current-time fn, default {.z.p}. stamps disconnecttime and drives + / removeinactive; override it to fast-forward in tests without sleeping + / clearinactivetime (optional) timespan, default 0D01:00. NOT read by any function here - + / removeinactive is caller-invoked, so this is the age di.torq passes it when + / it schedules the purge, mirroring di.dataaccess's requestkeeptime / e.g. di.serverselect.init[enlist[`log]!enlist logdep] if[99h<>type deps; '"di.serverselect: deps must be a dict with `log key"]; @@ -27,9 +102,17 @@ init:{[deps] '"di.serverselect: log value must be a dict; pass `info`warn`error functions"]; if[not all (`info`warn`error) in key deps`log; '"di.serverselect: log dict must have `info`warn`error keys; got: ",(", " sv string key deps`log)]; + checkopt[deps;`cp;{type[x] within 100 112h};"a function"]; + checkopt[deps;`clearinactivetime;{-16h=type x};"a timespan"]; + checkopt[deps;`maxcrossproduct;{(-7h=type x) and 0inactive transition and clears it on reactivation, so + / removeinactive can age out servers that disconnected and never came back + requireinit`setserveractive; + if[not -6h=type h;raiseerror[`setserveractive;"handle must be an int; got type ",string type h]]; + if[not -1h=type a;raiseerror[`setserveractive;"active flag must be a boolean; got type ",string type a]]; .z.m.loginfo[`setserveractive;"marking handle=",(string h)," active=",string a]; - .z.m.servers:update active:a from servers where handle=h; + / the conditional is hoisted OUT of the update deliberately: inside q-sql, $ resolves as the dyadic + / cast operator rather than the cond special form, so an inline $[a;0Np;.z.m.cp[]] throws 'rank + dtime:$[a;0Np;.z.m.cp[]]; + .z.m.servers:update active:a,disconnecttime:dtime from servers where handle=h; + }; + +setserveridactive:{[sid;a] + / mark ONE registration active/inactive by serverid, independent of any other servertype sharing + / the same physical handle. setserveractive keys on handle and is the right shape for a genuine + / disconnect - a closed handle really does take every registration on it down together - but it + / cannot express "pull just this servertype out of routing". updatestats already keys on serverid + / precisely because a handle may be shared; this closes the same gap for activation. + / stamps and clears disconnecttime exactly as setserveractive does, so removeinactive ages a + / registration retired this way out on the same terms + requireinit`setserveridactive; + if[not -6h=type sid;raiseerror[`setserveridactive;"serverid must be an int; got type ",string type sid]]; + if[not -1h=type a;raiseerror[`setserveridactive;"active flag must be a boolean; got type ",string type a]]; + .z.m.loginfo[`setserveridactive;"marking serverid=",(string sid)," active=",string a]; + / hoisted out of the update for the same reason as setserveractive - $ is the cast operator in q-sql + dtime:$[a;0Np;.z.m.cp[]]; + / no existence check: an unmatched serverid is a natural no-op, matching setserveractive's own + / established behaviour on an unregistered handle. do not add asymmetric strictness here + .z.m.servers:update active:a,disconnecttime:dtime from servers where serverid=sid; + }; + +removeinactive:{[age] + / purge inactive servers that disconnected more than age ago, to bound growth of the server table. + / setserveractive[h;0b] only flips a flag, so without this a process that connected once and went + / away stays in the table forever. caller-invoked - di.torq schedules it with clearinactivetime + requireinit`removeinactive; + if[not -16h=type age;raiseerror[`removeinactive;"age must be a timespan; got type ",string type age]]; + if[null age;raiseerror[`removeinactive;"age must not be null"]]; + if[age<0D;raiseerror[`removeinactive;"age must not be negative; got ",string age]]; + / 0Wn means "retain forever" and MUST short-circuit: it cannot be expressed by the comparison below, + / because disconnecttime+0Wn overflows the timestamp range (wrapping back to the year 1734), so every + / inactive row would compare as aged out - the exact opposite of an infinite retention. 0Nn is + / rejected above for the same reason: disconnecttime+0Nn is 0Np, and cp[]>0Np is true for everything + if[0Wn=age;:()]; + .z.m.servers:delete from servers where not active,not null disconnecttime,.z.m.cp[]>disconnecttime+age; }; getserverstable:{[] / return the current registered server table + requireinit`getserverstable; :servers; }; @@ -80,10 +211,14 @@ addserversfromtable:{[proctypes;conntable] / conntable must have columns: w (int handle), proctype (symbol), attributes (dict per row) / optional columns: procname (symbol), hpup (symbol) - populated from conntable if present / pass proctypes:`ALL to register all process types + requireinit`addserversfromtable; + / unkeyed table required: cols works on a keyed table but the select below does not, so without + / this the caller gets a raw 'type instead of a message naming the argument + if[not 98h=type conntable; + raiseerror[`addserversfromtable;"conntable must be an unkeyed table; got type ",string type conntable]]; if[not all `w`proctype`attributes in cols conntable; - .z.m.logerr[`addserversfromtable;err:"di.serverselect: conntable must have columns w, proctype and attributes; got: ",", " sv string cols conntable]; - 'err; - ]; + raiseerror[`addserversfromtable; + "conntable must have columns w, proctype and attributes; got: ",", " sv string cols conntable]]; activehandles:(0i;0Ni),exec handle from servers where active; rows:select from conntable where ((proctype in proctypes) or proctypes~`ALL), @@ -108,10 +243,10 @@ getservers:{[nameortype;lookups;req] / nameortype: `servertype or `procname; pass ` as lookups to return all active servers / req: attribute requirements dict - use ()!() for no attribute filtering / returns table with attribmatch column showing (complete_bool;matched_values) per attribute key + requireinit`getservers; if[(not `~lookups) and not nameortype in `servertype`procname; - .z.m.logerr[`getservers;err:"di.serverselect: nameortype must be `servertype or `procname; got: ",string nameortype]; - 'err; - ]; + raiseerror[`getservers;"nameortype must be `servertype or `procname; got: ",string nameortype]]; + requiredict[`getservers;"req";req]; r:$[`~lookups; select serverid,procname,servertype,hpup,handle,lastp,attributes from servers where active; nameortype~`servertype; @@ -125,21 +260,37 @@ getservers:{[nameortype;lookups;req] selector:{[servertable;selection] / pick one row from servertable using the given strategy / selection: `roundrobin (least recently used), `any (random), `last (most recently used) + requireinit`selector; if[not selection in `roundrobin`any`last; - .z.m.logerr[`selector;err:"di.serverselect: unknown selection strategy: ",string selection]; - 'err; - ]; + raiseerror[`selector;"unknown selection strategy: ",string selection]]; :$[selection=`roundrobin; first `lastp xasc servertable; selection=`any; rand servertable; last `lastp xasc servertable]; }; +setselector:{[f] + / replace the strategy getserverbytype uses to pick one row from a candidate table. + / the built-in selector stays exported and unchanged - pass it back here to restore the default + requireinit`setselector; + if[not type[f] within 100 112h;raiseerror[`setselector;"selector must be a function; got type ",string type f]]; + / check arity HERE where the mistake is, not at the next getserverbytype. a monadic or niladic + / strategy used to be accepted happily and then throw a bare 'rank from inside the module, far from + / the call site. 0N means q cannot report an arity for this function type, so it is accepted + ar:selectorarity f; + if[not null ar; + if[2<>ar; + raiseerror[`setselector;"selector must take 2 arguments (servertable;selection); got ",string ar]]]; + .z.m.selector:f; + }; + getserverbytype:{[ptype;serverval;selection] / return a single server attribute value for a servertype using the given selection strategy / ptype: servertype symbol; serverval: column to return e.g. `handle or `hpup; selection: `roundrobin`any`last + / dispatches through the live .z.m.selector so setselector can override the strategy + requireinit`getserverbytype; r:getservers[`servertype;ptype;()!()]; if[not count r;:()]; - r:selector[r;selection]; + r:.z.m.selector[r;selection]; updatestats[r`serverid]; :r serverval; }; @@ -147,76 +298,133 @@ getserverbytype:{[ptype;serverval;selection] gethandlebytype:getserverbytype[;`handle;]; gethpbytype:getserverbytype[;`hpup;]; +normreq:{[req] + / internal - promote atom attribute-requirement values to one-element lists, so a caller may write + / (enlist`date)!enlist 2024.01.01 as well as the enlist form. getservers already tolerates an atom + / via attributematch; without this the cross matcher threw a raw, unlogged 'rank on the same input. + / req is PURE requirements by this point - control keys are split off at the boundary - so every + / value is promoted, including one whose key happens to be spelled like a control key + :(key req)!{(),x} each value req; + }; + +raisecaught:{[ctx;msg] + / internal - log an error the engine already prefixed, then re-signal it VERBATIM. the engine's own + / text ("no servers match hdb requested attributes") is more specific than anything composed here, + / and signalnomatch deliberately does not log - the boundary decides the level. used on the + / single-servertype path, where there is nothing to tolerate: one type missing IS the whole answer + .z.m.logerr[ctx;msg]; + 'msg; + }; + +fanoutids:{[req;besteffort;attype;typ] + / internal - resolve one servertype's ids, tolerating a miss. a miss is NOT a query failure: another + / requested type may match, and getserverids errors only when every type comes back empty. this + / revives getserverids' all-empty guard, which was unreachable while this call threw instead + :@[getserveridstype[req;besteffort;attype;];typ; + {[t;e] .z.m.logwarn[`getserverids;"no servers matched servertype ",string[t],": ",e];()}[typ]]; + }; + getserverids:{[att] / return server IDs matching a servertype list or attribute requirement dictionary / att: symbol list of servertypes, or dict of attribute requirements (optionally keyed on `servertype) / dispatch the symbol-list path to getserveridsbytype and the dict path to getserveridstype + requireinit`getserverids; if[99h<>type att; :getserveridsbytype att]; - / validate attribute-requirement values are atoms or simple vectors (control keys excepted); - / a nested/general-list value (type 0h) would otherwise crash the cross matcher with a raw 'type - reqkeys:key[att] except `servertype`attributetype`besteffort; - if[count badkeys:reqkeys where 0h=type each att reqkeys; - .z.m.logerr[`getserverids;err:"di.serverselect: attribute requirement values must be atoms or simple vectors; nested/mixed for: ",", " sv string badkeys]; - 'err]; - serverids:$[`servertype in key att; - raze getserveridstype[delete servertype from att] each (),att`servertype; - getserveridstype[att;`all]]; + / TWO request shapes. flat (the original): control keys sit alongside the attribute requirements, so + / the three control names are reserved and an attribute cannot be called servertype/besteffort/ + / attributetype. nested: an `attrs key holds the requirements EXPLICITLY, leaving the rest of the + / dict to the controls - which lets an attribute carry any name at all, control names included. + / both are supported; nested is the way out of the namespace collision, flat stays for compatibility + nested:`attrs in key att; + if[nested; + requiredict[`getserverids;"attrs";att`attrs]; + / in the nested form EVERY top-level key that is not `attrs is a control, so an unrecognised one + / would be silently swallowed - dropping a requirement left outside attrs by a half-migrated + / caller, or ignoring a mistyped control. the flat form cannot have this problem (an unknown key + / there is simply a requirement), so the nested form has to be strict to stay as safe + if[count unknown:(key att) except `attrs,ctrlkeys; + raiseerror[`getserverids;"unknown keys beside attrs: ",(", " sv string unknown), + "; only attrs and the control keys belong at the top level of an attrs request"]]]; + ctl:$[nested;`attrs _ att;(key[att] inter ctrlkeys)#att]; + req:normreq $[nested;att`attrs;ctrlkeys _ att]; + / controls are validated HERE, at the boundary, and never inside getserveridstype: the fan-out below + / runs under a protected apply, so an error raised deeper down is caught and downgraded to a "this + / servertype did not match" warning. a misconfigured request is categorically not a per-type miss and + / must stay loud. before this, besteffort:0 (int) silently kept the 1b default and an unknown + / attributetype silently fell back to cross matching + if[`servertype in key ctl; + if[not 11h=abs type ctl`servertype; + raiseerror[`getserverids;"servertype must be a symbol or symbol list; got type ",string type ctl`servertype]]]; + if[`besteffort in key ctl; + if[not -1h=type ctl`besteffort; + raiseerror[`getserverids;"besteffort must be a boolean; got type ",string type ctl`besteffort]]]; + if[`attributetype in key ctl; + if[not -11h=type ctl`attributetype; + raiseerror[`getserverids;"attributetype must be a symbol; got type ",string type ctl`attributetype]]; + if[not (ctl`attributetype) in `cross`independent; + raiseerror[`getserverids;"attributetype must be `cross or `independent; got: ",string ctl`attributetype]]]; + / a nested/general-list requirement value (type 0h) would crash the cross matcher with a raw 'type + if[count badkeys:(key req) where 0h=type each value req; + raiseerror[`getserverids; + "attribute requirement values must be atoms or simple vectors; nested/mixed for: ",", " sv string badkeys]]; + besteffort:$[`besteffort in key ctl;ctl`besteffort;1b]; + attype:$[`attributetype in key ctl;ctl`attributetype;`cross]; + / bound the cross product before building it. cost is the PRODUCT of the requirement value counts, so + / a request a caller forwards straight from a client can get very large very cheaply. only cross + / matching builds it - independent matching is not combinatorial, so it is not bounded here. + / maxcrossproduct:0W disables the bound + / count req guard is NOT optional: prd of an empty list is () rather than 1, so an empty requirement + / dict would make the comparison return () and if[()] throw a raw 'type + if[attype=`cross; + if[.z.m.maxcrossproduct Date: Tue, 8 Sep 2026 22:10:25 +0100 Subject: [PATCH 14/14] di.serverselect: fix a dead integration suite, document maxcrossproduct, declare deps integration.q could not run at all. Every invocation died at the first step: 'type [4] integration.q:78: launch: :"I"$first system qbin," -p ",string[port]," -q ... & echo $!"; system returns the generic null instead of the echoed pid whenever the backgrounded child is itself a q process. A backgrounded sleep captures its pid fine, which is what makes this read like a shell-quoting problem rather than a q-child one. first :: then fed "I"$ and threw, before the module was ever touched. The crash also happened before .z.exit teardown was installed, so every run leaked its child process. nohup fixes it. Chosen over setsid because it execs rather than forking, so $! is unambiguously the q child's own pid - verified against pgrep, and load-bearing because teardown reaps by exact pid. It is also POSIX where setsid is util-linux. Both redirects are retained, so no nohup.out is written. This matters beyond the test: serverselect.md tells reviewers to run this file. Also: - init's deps docstring omitted maxcrossproduct, though init validates and sets it and serverselect.md documents it in full. Added, ordered to match the checkopt/assignment sequence below it. - Added an explicit empty deps.q. init.q and serverselect.q contain no `use` at all, so the module is genuinely standalone; the empty manifest records that as checked rather than leaving the file absent, which di.depcheck cannot distinguish from undecided. Verified through di.depcheck's own reader (finddepsq now resolves it, readdeps reports no malformed-manifest failure) - not merely that it parses. Verified: 408 k4unit assertions and 91 integration assertions pass, the latter for the first time. qlint clean. No behavioural change to the module itself. --- di/serverselect/deps.q | 10 ++++++++++ di/serverselect/integration.q | 11 +++++++++-- di/serverselect/serverselect.q | 4 ++++ 3 files changed, 23 insertions(+), 2 deletions(-) create mode 100644 di/serverselect/deps.q diff --git a/di/serverselect/deps.q b/di/serverselect/deps.q new file mode 100644 index 00000000..5b7801b9 --- /dev/null +++ b/di/serverselect/deps.q @@ -0,0 +1,10 @@ +/ hard module dependencies and their minimum versions, validated by di.depcheck. +/ di.serverselect has NONE, and the empty manifest is deliberate rather than the file being absent: +/ di.depcheck's finddepsq returns (::) for a module that ships no deps.q, which is indistinguishable +/ from "nobody has decided yet". an explicit empty dict records that the STANDALONE classification in +/ the modularisation plan was checked against the source and holds. +/ verified: init.q and serverselect.q contain no `use` at all. logging is INJECTED via init, not a +/ hard dep - the plan's tier table excludes logging, timer and handler management from the dependency +/ tree by design. integration.q does load kx.log, but it is a harness run directly rather than loaded +/ by init.q, so it is not an edge of this module. +deps:(`$())!(); diff --git a/di/serverselect/integration.q b/di/serverselect/integration.q index ab74ae68..0eb4f2b5 100644 --- a/di/serverselect/integration.q +++ b/di/serverselect/integration.q @@ -74,8 +74,15 @@ ports:findfree[baseport;count names]; launch:{[port] / start a bare q listener detached and capture its shell background pid ($!) - so every / child can be reaped by exact pid later, whether or not we ever connect to it, and with - / no dependence on the qbin path (identity/api are injected over IPC once connected) - :"I"$first system qbin," -p ",string[port]," -q /dev/null 2>&1 & echo $!"; + / no dependence on the qbin path (identity/api are injected over IPC once connected). + / nohup is REQUIRED, not cosmetic: without it system returns the generic null rather than the + / echoed pid whenever the backgrounded child is itself a q process, so "I"$first threw a bare + / 'type here and the fleet never launched. a backgrounded sleep captures fine, which is what + / makes it look like a shell-quoting problem rather than a q-child one. measured, not assumed. + / nohup over setsid: it execs rather than forking, so $! is unambiguously the q child's own pid + / (verified against pgrep), and it is POSIX where setsid is util-linux. both redirects stay, so + / nohup writes no nohup.out + :"I"$first system "nohup ",qbin," -p ",string[port]," -q /dev/null 2>&1 & echo $!"; }; connect:{[port] diff --git a/di/serverselect/serverselect.q b/di/serverselect/serverselect.q index 05a8e2af..8b7cffd3 100644 --- a/di/serverselect/serverselect.q +++ b/di/serverselect/serverselect.q @@ -93,6 +93,10 @@ init:{[deps] / clearinactivetime (optional) timespan, default 0D01:00. NOT read by any function here - / removeinactive is caller-invoked, so this is the age di.torq passes it when / it schedules the purge, mirroring di.dataaccess's requestkeeptime + / maxcrossproduct (optional) long, default 1000000. upper bound on the requirement cross + / product getserverids builds when cross matching, checked before it is built - + / cost is the PRODUCT of the requirement value counts, so a gateway forwarding + / client-supplied requirements can get very large very cheaply. 0W disables it / e.g. di.serverselect.init[enlist[`log]!enlist logdep] if[99h<>type deps; '"di.serverselect: deps must be a dict with `log key"];