feat(memory): completion-log-reconcile + crontab wiring + PGUSER fix (#561, #562, #564) - #567
Merged
Merged
Conversation
added 10 commits
August 8, 2026 04:50
Revises TEST-CASES-ISSUE-561.md per SE #608 Step 4 review. All 8 spec ambiguities flagged at Step 3 are now resolved with authoritative rulings; every affected test case encodes a single unambiguous expected outcome. Key changes: - Two-phase append/commit (grep pre-check -> append -> set watermark); new TC-561-07b (adjacent-day grep) and rewritten TC-561-08 (crash recovery via pre-check, deterministic not probabilistic). - work_queue watermark: COALESCE(completed_at, last_checked_at, created_at); new TC-561-15b for the (structurally unreachable but defensively guarded) all-NULL skip+warn path. - workflow_runs watermark: COALESCE(completed_at, started_at). - Description cap: 120 chars + trailing ellipsis (trigger_context's separate ~80-char rule unaffected). - Sanitization narrowed to whitespace-collapse only; Markdown-escaping assertions removed (structurally moot given fixed line prefix). - stale status added to the work_queue decision table -- fully resolved, no unresolved cells. - Backfill: migration seeds completion_logged_at at deploy time (guarded by IS NULL on re-apply); lookback-window design and its boundary test replaced with a pre/post-deploy boundary test. - Concurrency vs generate-daily-log.py: retargeted from empirical race probing to deterministic flock contention/mutual-exclusion verification (flock addition to generate-daily-log.py is in scope for #561). Case count: 34 -> 36 (TC-561-07b, TC-561-15b added; TC-561-28 repurposed). Ambiguities section replaced with Design-Gate Rulings (authoritative). Ref: nova-mind#561, nova-workspace#123, SE run #608 Step 4. (cherry picked from commit 9bf259074ad56011f7253e44b431254fc2a6a1a4)
…flock (#561) Implements deterministic completion-side daily-log reconcile for work_queue and workflow_runs: - memory/migrations/087_completion_log_watermark.sql: adds completion_logged_at timestamptz to both tables and idempotently seeds the watermark for already-closed rows. - memory/scripts/completion-log-reconcile.py: LLM-free script that scans terminal-status rows, appends one line per row to the correct YYYY-MM-DD.md file, and watermarks. Uses two-phase grep-precheck -> append -> set watermark for crash recovery. - memory/scripts/generate-daily-log.py: surgical companion change to take the same shared advisory flock around its read/write/rename critical section. Ref: nova-mind#561, nova-workspace#123, SE run #608
Adds tests/TEST-CASES-ISSUE-561.md implementation covering: - line format, sanitization, and watermark fallback - idempotency, crash recovery, and adjacent-day grep - midnight-boundary and backfill behaviour - migration idempotency and backfill guard - flock mutual exclusion with generate-daily-log.py Ref: nova-mind#561
…elog updates - Fix WORKFLOW_RUNS_MARKER numeric-prefix collision by inserting a word boundary after the formatted id in the grep pre-check (e.g. #1 no longer matches inside #10). WORK_QUEUE_MARKER is also anchored for consistency. - Add TC-561-35 regression tests covering both tables for the prefix-collision scenario. - Update database/schema.sql and schema-reference.md with the new completion_logged_at columns. - Add CHANGELOG entry for #561. - Update TC-561-26 prose to match accepted amended semantics. - Add one-line risk-disclosure comment to migration 087 about re-applying against a live system. Tested: pytest tests/test_completion_log_reconcile.py tests/test_generate_daily_log.py 93 passed, 0 failed.
Real cron does not set USER, so the previous fallback resolved to a numeric UID string (e.g. "1005"). psycopg2 then tried to authenticate as DB user "1005", .pgpass didn't match, and libpq fell back to no-password auth, causing and exit 1. Replace the fallback with in both completion-log-reconcile.py and generate-daily-log.py. This checks LOGNAME/USER/LNAME/USERNAME and falls back to pwd.getpwuid(os.getuid()), always returning a username string. Also update the pytest fixtures to connect as the current OS user instead of hardcoding PGUSER=nova, so the TC-561-32 flock test can run against staging databases owned by nova-staging. Closes #561
…(SE run #608 step 9) Feature docs: - memory/docs/daily-log-generation.md: new Completion Log Reconcile section (line formats, watermark fallback, two-phase idempotency, shared flock), PGUSER cron-resolution note, updated cron schedule/tests tables to cover completion-log-reconcile.py alongside generate-daily-log.py - CHANGELOG.md, memory/CHANGELOG.md: record #564 PGUSER cron fix and 64-test suite count - memory/README.md, memory/ARCHITECTURE.md, memory/INSTALLATION.md: reference the new companion cron script and shared flock Repo-wide documentation audit (all enumerated doc files read against current source): - README.md: corrected memory-maintenance.py pipeline description (missing lessons-dedup + re-embed-modified-facts phases), added missing --no-cron/--regenerate-agents-json flags - ARCHITECTURE.md: corrected stale ghost-embeddings/connection-pool claims against actual maintenance script and pg-pool.ts sources - database/schema-reference.md: corrected social_interactions column count (16->17, missing kind column) - cognition/docs/shell-environment.md, cognition/focus/protocols/jobs-system.md: added status banners for non-functional/unimplemented designs (#398, dead agent_jobs schema) - cognition/docs/cross-database-replication.md, cognition/metacognition/confidence-check/README.md, cognition/tests/README.md: corrected stale claims (installer behavior, LOC count, orphaned README content) - motivation/README.md: replaced stale unsolved_problems snapshot table with live-query pointer + corrected current count (13) - psyche/README.md, psyche/ARCHITECTURE-entities-users.md: corrected stale project status, documented assertion_intent/mutability_class columns - relationships/README.md, relationships/CONTRIBUTING.md: corrected npm test invocation Docs-only change; no code/tests/installers/schema touched. Refs: nova-mind#561, nova-mind#564. Follow-up issues filed from audit findings: nova-mind#565 (schema.sql CHECK constraint drift), nova-mind#566 (dead agent_jobs/job_messages schema).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the completion-log reconciliation system for
workflow_runs, wires it into the installer's crontab, and fixes a cron-environment bug in PGUSER resolution. Includes a full documentation audit across CHANGELOGs and memory docs.Closes #561
Closes #562
Fixes #564
What's Included
#561 — completion-log-reconcile script + migration + shared flock
memory/scripts/completion-log-reconcile.py— reconcilesworkflow_runscompletion state against the daily log, using a shared flock to coordinate safely withgenerate-daily-log.pymemory/migrations/087_completion_log_watermark.sql— adds the watermark tracking column/table backing incremental reconciliation6cb95f3)#562 — installer crontab wiring
agent-install.sh— idempotent crontab wiring forcompletion-log-reconcile.py(safe to re-run without duplicating cron entries)tests/test_completion_log_reconcile.py::TC-561-07— call-ordering regression test added to lock in correct invocation order between the reconcile script and the daily-log generator#564 — PGUSER cron-env fix
memory/scripts/completion-log-reconcile.py— resolvesPGUSERviagetpass.getuser()when running under cron's minimal environment, instead of relying on an unset env varDocumentation audit (SE Run #608, Step 9)
cb69be7— full documentation pass: CHANGELOGs (root + memory),memory/docs/daily-log-generation.md,memory/README.md,memory/ARCHITECTURE.md,memory/INSTALLATION.md, plus a broader audit touching 20 files total (README/ARCHITECTURE/CONTRIBUTING docs acrosscognition/,motivation/,psyche/,relationships/,database/)Testing
tests/test_completion_log_reconcile.py, including theTC-561-07call-ordering regression test added under completion-log-reconcile.py: no crontab/sweeper invocation wiring — feature is inert in production (nova-mind#561 requirement #3 unimplemented) #562tests/TEST-CASES-ISSUE-561.mddocuments the test case design for completion-log-reconcile: deterministic daily-log completion lines for work_queue + workflow_runs #561Verification
reports/and other historical artifacts untouchedDiffstat
27 files changed, 3077 insertions(+), 56 deletions(-)