Add the virtual-table capture pack - #1
Open
DI-Dexter wants to merge 1 commit into
Open
Conversation
A TorQ application overlay that captures data partitioned by date AND instrument, and serves it through kdb-x virtual tables. No RDB, no HDB, no sort process and no gateway: the writer writes where the readers read, and nothing moves at end of day. The instrument becomes a directory rather than a column, so a selective lookup is a directory lookup instead of a scan. That needs no p# attribute, which means the live day is as fast as history and there is nothing to rebuild overnight. Contents code/wdb/vtwrite.q writer overrides (5 of them) code/processes/vtidb.q the reader code/processes/vtcompress.q the weekend compression job appconfig/ process list, settings, compression tiers testfiles/ 16 assertion tests + 9 evidence scripts docs/ architecture document and status report Verification ./regress.sh 16 assertion tests, 16/16 ./selftest.sh end-to-end through the live stack, 7/7 Results checked against the same bytes in a stock kdb+ database: 16 of 19 queries identical, 3 raise an error, none silently different. Known limitation: a single virtual table cannot span both this format and existing date-partitioned history, because the column list is taken from the first directory only. Raised with KX; see section 10 of the doc. Parts 2 (split capture stacks) and 3 (chained tickerplant) are not included. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
jonnypress
reviewed
Sep 1, 2026
| @@ -0,0 +1,26 @@ | |||
| discovery:pass | |||
| feed:pass | |||
| gateway:pass | |||
Contributor
There was a problem hiding this comment.
Don't think we need all of these in here given we only have a few processes
| // them - on the reference partition (doc §7.3). | ||
| // set 0 to compress everything, as stock TorQ does | ||
|
|
||
| // the hdbstructure override that teaches compression to see this layout is NOT here, even |
Contributor
There was a problem hiding this comment.
What is all this nonesense in the comments?
Can you compress these to something sensible.
| @@ -0,0 +1,2 @@ | |||
| table,minage,column,calgo,cblocksize,clevel | |||
| default,1,default,2,16,9 | |||
Contributor
There was a problem hiding this comment.
if this is a test file we shouldn't have it in the main code base, it should be in a test directory somewhere
| inputcsv:@[value;`inputcsv;.proc.getconfigfile["compressionconfig.csv"]]; | ||
| hdbpath:@[value;`hdbpath;`:hdb]; | ||
| maxage:@[value;`maxage;365]; | ||
| dryrun:@[{[x] `dryrun in key .proc.params};::;0b]; |
Contributor
There was a problem hiding this comment.
This seems an odd way to do it. Should this not just be "`dryrun in key .proc.params"?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A TorQ application overlay that captures data partitioned by date AND instrument, and serves it through kdb-x virtual tables. No RDB, no HDB, no sort process and no gateway: the writer writes where the readers read, and nothing moves at end of day.
The instrument becomes a directory rather than a column, so a selective lookup is a directory lookup instead of a scan. That needs no
p#attribute, which means the live day is as fast as history and there is nothing to rebuild overnight.Contents
Verification
Notes for review
appconfig/passwords/is the stock TorQ demo:passlist, same as the Finance Starter Pack ships. No real credentials.mkPviause`kx.pq.t, anduseis a kdb-x keyword.