Feature serverselect

[ascottDI]

di.serverselect — TorQ Modularisation PR

Summary

Extracts the server-selection logic from TorQ's .gw namespace (gatewaylib.q and gateway.q) into a standalone kdb-x module: di.serverselect. The module maintains a registered pool of backend servers and selects from them by servertype or attribute requirements. It satisfies the di.* module contract: dependency injection via init, a clean exported API, and no hard module dependencies beyond an injected logger.

---

Background

TorQ's gateway uses a .gw namespace to track connected backend servers and route queries to them. Server registration, active-flag management, and selection logic were entangled with the gateway's query execution and connection-handling code. This PR extracts the server-selection layer — the pool of registered servers and the logic to pick from it — as an independently loadable and testable unit.

This PR is part of the broader TorQ → kdb-x modularisation effort. The extracted module removes the dependency on TorQ's global process framework and can be used in any gateway or proxy that needs to maintain a backend server pool.

---

Changes

New files

File	Description
di/serverselect/serverselect.q	Core implementation — init, addserverfull, addserverattr, addserver, setserveractive, getserverstable, addserversfromtable, getservers, selector, getserverbytype, gethandlebytype, gethpbytype, getserverids and internal helpers
di/serverselect/init.q	Module entry point — loads serverselect.q and declares the export list
di/serverselect/test.csv	k4unit unit-test suite (128 assertions)
di/serverselect/integration.q	End-to-end integration test (76 assertions)
di/serverselect/serverselect.md	Module README

Differences from TorQ original

Aspect	TorQ .gw	di.serverselect
Logging	Hard-coded .lg.o/.lg.e calls	Injected log dependency, required via init (no fallback)
Server state	Global .gw.servers table	.z.m.servers module-local mutable state
Error handling	Mix of silent failures and hard exits	Every error path logged then signalled with a di.serverselect: prefix
Module contract	None	kdb-x use singleton, init[deps] pattern, export: list
Attribute matching	Inlined in query dispatch	Extracted as attributematch (internal); exposed via the getservers attribmatch column
Bulk registration	.servers.SERVERS hard-coded lookup	Generic addserversfromtable[proctypes;conntable] accepting any connection table

---

Logging contract (injected dependency)

Logging is an injected dependency, wired via init — there is no default logger and the module does not load kx.log itself; initialising the logging framework is the job of the start-up script or the user. init must be called before any other function.

• init[deps] takes a single dict carrying the required log dependency (plus any future optional config). It errors immediately (plain signal) if deps is not a dict, is missing `log, or log is not a dict exposing `info`warn`error.
• normlog normalises the injected logger to a binary `info`warn`error!{[c;m]} dict — each function takes a context symbol c and a message string m. It accepts either:
  • a whole kx.log instance ((use\kx.log)[`createLog][]) — detected by its getlvl/sinks/fmtskeys; its monadic functions are wrapped to{[c;m]}, folding context in as a "ctx: msg"` prefix; or
  • a bespoke binary `info`warn`error!({[c;m]};{[c;m]};{[c;m]}) dict — passed through unchanged.
• Every call site uses the binary form .z.m.log[\info][`ctx;"msg"](which maps 1:1 with TorQ's.lg.o[`ctx;"msg"]`).
• Errors are routed through an internal raiseerror[ctx;msg] helper that logs via .z.m.log[\error]**then** signals'"di.serverselect: ",ctx,": ",msg`, so every failure is observable in the log as well as thrown.

---

Exported API

srvsel:use`di.serverselect

/ wire the required log dependency first (pass the whole kx.log instance; normlog wraps it)
srvsel.init[enlist[`log]!enlist (use`kx.log)[`createLog][]]

srvsel.addserverfull[h;pname;st;hp;att]   / register a server with full details
srvsel.addserverattr[h;st;att]            / register a server with servertype and attributes
srvsel.addserver[h;st]                    / register a server with no attributes
srvsel.setserveractive[h;active]          / mark a server active (1b) or inactive (0b)
srvsel.getserverstable[]                  / return the full registered server table
srvsel.addserversfromtable[types;ctab]    / bulk-register from a connection table
srvsel.getservers[nameortype;lookups;req] / look up active servers with attribute scoring
srvsel.selector[servertable;selection]    / pick one row using roundrobin / any / last strategy
srvsel.getserverbytype[ptype;col;sel]     / return one column value for a servertype
srvsel.gethandlebytype[ptype;sel]         / convenience projection: return handle
srvsel.gethpbytype[ptype;sel]             / convenience projection: return hpup
srvsel.getserverids[att]                  / return server IDs by servertype list or attribute dict

---

Hardening and fixes

• updatestats keyed on serverid, not handle. handle is not unique (the table is keyed on serverid, and a freed handle can be reused after reconnect). Keying stats updates on the unique serverid prevents a single selection from skewing hits/lastp across every server that happens to share a handle. Covered by a dedicated regression test.
• Input validation on the public mutators, routed through raiseerror so the failure is logged: addserverfull/setserveractive check handle (int) and servertype/active-flag types; addserversfromtable checks the connection table has the required w/proctype/attributes columns.
• getservers validates nameortype — it must be `servertype or `procname (when lookups is not `); any other value errors rather than silently falling through to a procname lookup.
• selector empty-table behaviour documented — it returns a null-valued row; the *bytype helpers guard against this and return () when no active server matches.

---

Test coverage

Two suites, both green.

Unit tests — test.csv (k4unit), 128 assertions

Area	Coverage
init — dependency injection	accepts a valid log dep; rejects non-dict deps, missing log, non-dict log, log dict missing a key; di.serverselect: error prefix
init — injected logger used	a capturing binary logger receives info messages on registration and error messages on failure paths
registration	addserver/addserverattr/addserverfull — row counts, handles, servertypes, procname/hpup population, serverid autoincrement, type-error rejection
setserveractive	active flag toggled; missing handle is a no-op; type-error rejection
updatestats	a selection updates only the chosen serverid, even when a handle is shared
getservers	column schema incl. attribmatch; servertype/procname filters; null lookups; per-attribute match scoring; invalid nameortype rejected
selector	roundrobin/last/any strategies; single-row and empty-table behaviour; unknown strategy rejected
getserverbytype / gethandlebytype / gethpbytype	column value returned; () for unknown type; round-robin rotation; hits increment
getserverids — symbol path	IDs for registered types; rejects null / unregistered / all-inactive
getserverids — attribute path	date/sym match; cross and independent matching; servertype-scoped dict; besteffort strict mode
addserversfromtable	bulk register; skips already-active handles; proctype filter; `ALL; optional procname/hpup columns; missing-column rejection

Integration test — integration.q, 76 assertions

Drives every exported function through a realistic gateway lifecycle (register → activate/deactivate → query → select → bulk-register → error handling → init re-wiring with a capturing logger), with a PASS/FAIL summary and a non-zero exit code equal to the number of failures.

Running the tests

export QPATH=/path/to/kx/mod:/path/to/kdbx-modules

/ unit tests
k4unit:use`di.k4unit;
k4unit.moduletest`di.serverselect;

/ integration test
QPATH=/path/to/kx/mod:/path/to/kdbx-modules q integration.q

---

Integration notes

• di.serverselect has no hard module dependencies. The injected log is required — init throws if it is not provided, and must be called before any other function.
• The caller wires kx.log by passing the whole createLog[] instance (srvsel.init[enlist[\log]!enlist (use`kx.log)[`createLog][]]); normlogdetects and adapts it. A stripped 3-key dict of kx.log's monadic functions must **not** be passed (it would bypass detection and fail with'rank`).
• addserversfromtable accepts a TorQ .servers.SERVERS-style table directly: columns w (handle), proctype, attributes are required; procname and hpup are optional.
• getserverids supports the full TorQ gateway attribute-matching contract: cross-product matching (default), independent matching, per-servertype scoping, and besteffort mode. Its result is consumed by the gateway via inter/: + first each (and raze for emptiness checks), which is robust to its per-path return shape.
• All error messages are prefixed di.serverselect: to identify the source in stack traces.

Test results screenshot

[DI-Software-Engineering]

DIReview Summary

1 critical | 7 warning(s) | 0 suggestion(s)

⚠️ Spec check skipped — tracker lookup failed (NO_REF_FOUND). Standards axis only.

[DI-Software-Engineering]

DIReview Summary

1 critical | 2 warning(s) | 0 suggestion(s)

⚠️ Spec check skipped — tracker lookup failed (NO_REF_FOUND). Standards axis only.

[DI-Software-Engineering]

DIReview Summary

1 critical | 0 warning(s) | 0 suggestion(s)

⚠️ Spec check skipped — tracker lookup failed (NO_REF_FOUND). Standards axis only.

[DI-Software-Engineering]

checkopt[deps;maxcrossproduct;{(-7h=type x) and 0<x};"a positive long"]rejects0as maxcrossproduct since0<0is false. The docs say0Wdisables the bound;0is not discussed. The error message says "a positive long" and0would be rejected. But the actual semantics ingetserveridsisif[.z.m.maxcrossproduct<sz:...]— if maxcrossproduct were0, every non-empty request would fail (since sz>=1). So 0` being rejected is arguably correct. No defect. Disregarding.

[DI-Software-Engineering]

VERSION file loses its trailing newline (as shown by the \ No newline at end of file in the diff). The read0 call in init.q reads the file and takes first, then trims. If the file has no trailing newline, read0 returns a one-element list with the version string (no trailing \r or space issues). trim first read0 handles this correctly. However, other tooling (e.g. cat, diff, some CI version-check scripts) that expects a newline-terminated file may be broken. This is a minor portability concern but not a code defect in the q module itself.

[DI-Software-Engineering]

removeinactive does not call requireinit before checking not -16h=type age. If called before init, the age type check and subsequent checks may run, and if[0Wn=age;:()] would return without touching .z.m — but the delete at the end references .z.m.servers and .z.m.cp, which are unset, producing a cryptic '.m.di.0serverselect.servers error instead of the expected 'di.serverselect: removeinactive: init must be called first. Looking at the actual code: requireinit\removeinactiveis at line 200 (first line of the function body). Checking:removeinactive:{[age] requireinit`removeinactive; ...}. The test confirms this at the requireinit section. So requireinit` IS called first. No defect.

[DI-Software-Engineering]

DIReview Summary

1 critical | 2 warning(s) | 5 suggestion(s)

⚠️ Spec check skipped — tracker lookup failed (NO_REF_FOUND). Standards axis only.

Suggestions

• di/serverselect/serverselect.q:119 — The checkopt validator for maxcrossproduct uses {(-7h=type x) and 0<x}, which rejects 0W (positive infinity long). But 0W is documented and tested as the "disable the bound" sentinel, and the code in getserverids explicitly compares .z.m.maxcrossproduct<sz meaning 0W must pass init. The type check (-7h=type x) is correct for 0W, but 0<0W is 1b, so 0W actually does pass the check. However the error message says "a positive long" and 0W is technically valid — the real defect is that the check also rejects 0W being passed as maxcrossproduct only if the condition 0<x were the issue. Re-examining: 0W is a long atom with type -7h and 0<0W is 1b, so 0W passes. But separately: the test at line 536 in test.csv asserts fail for srvsel.init[logmaxcrossproduct!(noop;0W)]... wait, that test asserts fail on 0W? No, looking again at test.csv line 536: run,0,0,q,"srvsel.init[logmaxcrossproduct!(noop;0W)]",1,1,re-init with 0W to disable the bound — that is a run (expected to succeed). So 0W must pass checkopt. Since -7h=type 0W and 0<0W are both true, 0W does pass. No defect here after careful re-examination. Disregarding this finding.
• di/serverselect/serverselect.q:59 — The requiredict check for an empty dictionary ()!() relies on not $[0=count k:key d; not 98h=type k; 11h=type k]. When d is ()!(), key d is an empty general list (type 0h), so count k is 0 and the branch taken is not 98h=type k, which evaluates to not 98h=0h → not 0b → 1b, meaning the condition not (1b) = 0b, so the raiseerror is NOT triggered and ()!() is accepted. That is the intended behaviour. However when d is an empty keyed table ([a:int$()]b:int$()), key d is an empty table (type 98h), so count k is 0 (empty table has 0 rows) and the branch is again not 98h=type k → not 98h=98h → not 1b → 0b, so raiseerror is NOT triggered and an empty keyed table is accepted. But the test at the G3 section asserts fail for srvsel.addserverattr[9i;hdb;([a:int$()]b:int$())]. This means requiredictmust reject an empty keyed table, but the logic as written would NOT reject it because an empty keyed table also hascount key = 0. The guard 0=count k:key dis true for both()!()and an empty keyed table, so both take thenot 98h=type kbranch — but98h=type kdiffers: for()!()it is0hgivingnot 0b=1b(accepted), for an empty keyed tablekeyreturns an empty table of type98hgivingnot 1b=0b(rejected). Actually for an empty keyed tablecount key ktwherekt:([a:int$()]b:int$())—key ktreturns the key table which has 0 rows but type98h. So count kis0AND98h=type kis1b, so not 98h=type kis0b, so raiseerror` IS triggered. The logic is correct. Disregarding.
• di/serverselect/serverselect.q:79 — selectorarity handles a projection (type 104h) by calling selectorarity first value f recursively to get the underlying rank, then subtracts count where not (::)~/:1_value f. For a trailing partial application like {[h;t;s]h}[9i], value f is ({[h;t;s]h}; 9i), so 1_value f is enlist 9i, and (::)~/:enlist 9i is enlist 0b, so count where not is 1. The underlying rank is 3 (from selectorarity {[h;t;s]h}), so the result is 3-1=2. This is correct. But for {[h;t;s]h}[9i;t], 1_value fis(9i;t), count where not is 2, result is 3-2=1. That also looks correct per the test. No defect. Disregarding.
• di/serverselect/serverselect.q:366 — In getserverids, the cross-product size check uses prd count each value req. req is the result of normreq, which promotes atoms to one-element lists via {(),x} each value req. After normreq, values of req are lists. count each value req gives the count of each requirement list. prd of that gives the cross-product size. However, normreq is called with ctrlkeys _ att (flat form) or att\attrs(nested form), and this happens before thebadkeyscheck (nested list check). Ifreqhas a value that is a general list (type0h, e.g. enlist enlist 2024.01.01), normreqwraps it in()making it(enlist 2024.01.01)— a one-element general list. Thencountof that is1, contributing 1to the product. Thebadkeyscheck at line ~357 catches this:0h=type each value req would be true for the wrapped general list (type enlist enlist 2024.01.01is0h). Wait — after normreq, the value enlist enlist 2024.01.01gets wrapped as(),(enlist enlist 2024.01.01)=enlist enlist 2024.01.01(since(),on a list is identity). Actually{(),x}[enlist enlist 2024.01.01]=(),(enlist enlist 2024.01.01)which is a general list containing one element. Its type is0h`. So the badkeys check still fires. No defect. Disregarding.
• di/serverselect/serverselect.q:349 — getserverids validates the servertype control key with not 11h=abs type ctl\servertype, which accepts both an atom symbol (-11h) and a symbol list (11h). However, when servertypeis an atom symbol (e.g. ``hdb ``), distinct (),ctl\servertype` correctly makes it a one-element list. The fan-out `fanoutids[req;besteffort;attype] each distinct (),ctl`servertype` then works correctly. No defect.

[DI-Software-Engineering]

The connect function's retry loop 50 step/ 0Ni uses converge (/ without a count), not a counted loop. Because step returns the handle integer on success and 0Ni on failure, the converge will stop as soon as two successive calls return the same value — meaning it stops immediately on the first successful connect (correct), but it also stops if two consecutive retries both return 0Ni (i.e. after just two failed attempts, not fifty). The intent is to retry up to 50 times; use 50 step/ 0Ni as a counted iterate by making the count an integer: this already is an integer 50, so in q n f/ seed with integer n applies f exactly n times. Verify this is the intended semantics — if so it is correct as written and this note can be disregarded.

[DI-Software-Engineering]

DIReview Summary

0 critical | 1 warning(s) | 3 suggestion(s)

⚠️ Spec check skipped — tracker lookup failed (NO_REF_FOUND). Standards axis only.

Suggestions

• di/serverselect/serverselect.q:183 — nextserverid reads serverid (the module-level variable) directly by bare name rather than through .z.m.serverid, then writes the result back as .z.m.serverid. On the first call, .z.m.serverid does not exist yet (it is only written by this function, never initialised in init or at module level under .z.m), so serverid resolves to the top-level 0i initialisation, which is correct on the first call. However, subsequent calls read .z.m.serverid via the bare name serverid only if the module namespace aliases it; if the bare name keeps resolving to the top-level 0i constant every call, every server would get serverid=1i. Confirm that the bare serverid read on this line resolves to .z.m.serverid after the first write, or change it to .z.m.serverid+1i explicitly.
• di/serverselect/serverselect.q:126 — checkopt for maxcrossproduct requires (-7h=type x) and 0<x, which rejects 0W (positive infinity for longs, type -7h, and 0W>0 is 1b). But the doc and test both state 0W disables the bound and is accepted. 0W satisfies -7h=type 0W and 0W>0 is 1b, so the check actually passes for 0W — that is correct. However the check 0<x also rejects 0 as a bound (zero would mean reject every non-empty cross product, which may be intentional). Verify that 0 is intentionally rejected; if a zero bound should be allowed, change to 0<=x.
• di/serverselect/serverselect.q:270 — In getserverids, the @[getserveridstype[req;besteffort;attype]; all; raisecaught[getserverids;]] call passes the atom `all as the second argument to .[] — which means it is used as the argument to getserveridstype[req;besteffort;attype]. getserveridstype is a 4-argument function that has already been projected to 3 arguments, so this single remaining argument is typ. Inside getserveridstype, typ=all`` triggers the branch that returns all active servers regardless of servertype, which is the intended behaviour. The trap raisecaught[getserverids;]` is a projection awaiting one string argument — the error string — which is correct for `@[f;x;h]`. This appears correct. No defect.