pktLog

_{Dashboard · Log explorer · Alerts}

Syslog ingest management and visualization platform. Receives syslog data over UDP/TCP, stores events in ClickHouse, and provides a React UI for search, alerting, and reporting.

Part of the pktSuite platform (SSO with pktHub/pktFlow via a shared suite_token).

Quick Start

Requires a fresh Ubuntu Server 22.04/24.04 LTS host with sudo access, and Node.js 20.x LTS installed for the frontend build (not installed by install.sh — see Requirements).

# 1. Clone the repository
git clone https://github.com/bsnwgit/pktlog.git
cd pktlog

# 2. Install Node.js 20.x LTS first if it isn't already present (see
#    Requirements below) — install.sh builds the frontend automatically
#    when npm is on PATH, and only falls back to a manual build otherwise.

# 3. Run the installer. Interactively, it prompts for the install directory
#    (default /opt/pktlog) and the app port (default 8768); it then handles
#    system packages, ClickHouse, Python deps, ClickHouse schema, config.yaml
#    + secret key, admin user (random password), one seeded collector entry
#    (this host's own IP), the frontend build (if npm is present), and the
#    systemd service (installed + started).
bash install.sh
# Prints the admin username/password at the end — save it, it is not shown again.

# 4. If npm was NOT found during install, install.sh prints the exact
#    fallback commands to build the frontend manually and restart the
#    service — see Installation § 8 below.

# 5. Open the firewall (adjust if you chose a different port, or
#    PKTLOG_SYSLOG_PORT differs from the 5514 default)
sudo ufw allow 8768/tcp
sudo ufw allow 5514/tcp
sudo ufw allow 5514/udp

# 6. Open http://<server-ip>:<port> and log in with the admin credentials
#    from step 3

For a fully manual walkthrough of what install.sh does (e.g. to customize the install path or run steps individually), see Installation.

Environment variables

All settings in config.example.yaml can also be passed as PKTLOG_* environment variables instead of editing config.yaml — environment variables take priority. Commonly used ones:

Variable	Default	Description
`PKTLOG_CONFIG`	(none)	Path to `config.yaml` to load
`PKTLOG_INSTALL_DIR`	(none)	Install directory `install.sh` deployed to; used by `pktlog.service`
`PKTLOG_HOST`	`0.0.0.0`	Bind address
`PKTLOG_PORT`	`8768`	Listen port (HTTP; HTTPS if SSL cert configured)
`PKTLOG_DB_PATH`	`/opt/pktlog/pktlog.db`	SQLite app database path
`PKTLOG_CLICKHOUSE_HOST` / `_PORT` / `_DATABASE` / `_USER` / `_PASSWORD`	`localhost` / `9000` / `pktlog` / `default` / ``	ClickHouse connection
`PKTLOG_SYSLOG_PORT`	`5514`	Syslog ingest port (UDP + TCP)
`PKTLOG_SECRET_KEY`	(required)	JWT signing key — `openssl rand -hex 32`
`PKTLOG_CORS_ORIGINS`	`["*"]`	Restrict to your dashboard origin in production
`PKTLOG_LOG_LEVEL` / `PKTLOG_LOG_FILE`	`info` / `/opt/pktlog/logs/pktlog.log`	Logging

Architecture

Syslog Collectors ──UDP/TCP:5514──► pktLog Ingest Listener
                                          │
                                    Parse + Enrich
                                    (RFC 3164/5424 → org/group/site)
                                          │
                                    ClickHouse (pktlog.syslog_events)
                                          │
                                    FastAPI Backend (:8768)
                                          │
                                    React Frontend (SPA)

Stack

Layer	Technology
Backend	FastAPI (Python 3.11)
Frontend	React + TypeScript + Tailwind
Log storage	ClickHouse
App database	SQLite (users, settings, alerts, device/collector registry)
Auth	Local + SAML/Okta SSO + pktSuite `suite_token`
Ingest	Async UDP + TCP (RFC 3164 / RFC 5424)

Collectors are an allowlist, not just labels

A device can be actively sending syslog data on the wire, but nothing is stored until its IP is approved and marked enabled. Data from an unregistered or disabled collector_ip is dropped at ingest (app/ingest/normalizer.py), not just missing hierarchy metadata — this is intentional, so a stray/misconfigured device on the network can't silently fill up storage.

New senders are admitted on the Approval page (see below); Settings → Collectors edits the ones already approved and also has Export CSV / Import CSV / template-download buttons for provisioning many collectors at once instead of one at a time (columns: collector_ip, collector_name, org, log_group, site, notes, enabled). Duplicate IPs are skipped on import, not overwritten — use the existing Edit action for changes to an entry already in the registry.

Approval — admitting a new collector

Approval is a top-level, admin-only page sitting directly above Settings. Senders that ingest drops because they aren't in the registry are recorded in pending_collectors and listed there with first/last seen, how many messages have been dropped, and a sample raw line to identify the device by.

Approve creates the collector_registry row (that is what actually admits the data) and clears the queue entry.
Ignore hides a sender without admitting it — its messages keep being dropped and its counter keeps rising, so a chatty unwanted device can't bury real ones.
Forget removes the row; it returns if the device sends again. Neither Ignore nor Forget is a block.

It is deliberately not a Settings tab. Settings goes read-only whenever pktHub manages the app (hub_settings_managed, see Hub-managed direct-access lock), which would otherwise force a managed install over to pktHub just to admit a device — an operational act, not a configuration one.

Counters accumulate in memory on the ingest path (app/ingest/pending.py) and are folded into SQLite once per alert-engine tick (60s), so a device that has just started sending takes up to a minute to appear and its total lags slightly behind the wire. A sender already in the registry but disabled also hits the drop path; it is filtered out at flush time rather than being listed as awaiting approval, because it was deliberately turned off.

API (all admin-only): GET /api/approval/pending, GET /api/approval/count, POST /api/approval/approve, POST /api/approval/{ip}/ignore, POST /api/approval/{ip}/unignore, DELETE /api/approval/{ip}.

Timestamps and device timezone

RFC 3164 syslog (MMM DD HH:MM:SS, no timezone marker) is interpreted using the app's configured Timezone setting (Settings → General) as the device's local clock, then converted to UTC for storage — not assumed to already be UTC. Many real devices (UniFi APs/gateways observed in practice) log in local time, and getting this wrong silently shifts every such event by the zone offset. The received_at column is always the server's own UTC receipt time regardless of this setting, and is the field to check first if stored timestamp values ever look wrong.

Alert-event and user last-login timestamps (alert_events.fired_at, users.last_login) are stored as naive UTC and are explicitly normalized before being parsed for display, so they render correctly in the configured display timezone regardless of the browser's own system timezone.

Application Logs time-range filtering

The Application Logs page (search + level filter) also has a time-range dropdown — 1h/6h/24h/7d/30d/All time, plus Custom range… with two date/time pickers (defaulting to today, 12:00 AM–11:59 PM). The custom range validates that the end is after the start (same-day-with-earlier-end-time counts as invalid too) and disallows future times on either side, showing an inline error instead of silently applying an impossible filter.

Application Logs, Syslog Explorer, and Alerts (active and history tabs, sized independently) all paginate with a page-number bar above the table — a sliding window of 5 numbers that follows the current page (Next from page 5 moves to 6-10, Prev the same way in reverse), a 1 .. shortcut back to page 1 once past the first block, and a .. N shortcut to the last page — plus a per-table page-size dropdown (25/50/75/100, defaulting to 25) that resets to page 1 on change. Logs and Syslog Explorer thread the chosen size into their server-side limit/offset fetch; Alerts fetches its full active/history set and re-slices it client-side per page.

Alert rules bulk import/export

Alerts → Rules has Export CSV / Import CSV / template-download buttons alongside "+ New rule", for provisioning many rules at once. Columns: name, description, rule_type, conditions, time_window_min, severity, channels, cooldown_min, enabled — conditions round-trips as a JSON object string (shape depends on rule_type), channels as a comma-separated column drawn from the six supported values: inapp, slack, email, pagerduty, webhook, tracecat (e.g. inapp,slack).

Alert Investigate button

Every active/history alert card has an Investigate ↗ button that jumps straight to Syslog Explorer, pre-filtered to the alert's collector_ip (a new collector_ip search param/filter, distinct from the existing name-based collector_name filter) and a time window around when it fired. The "Unknown collector" link still appears alongside it for new_host alerts, now pointing at the Approval page.

`dest_ip` — destination IP from netfilter-style logs

Alongside source_ip, the syslog schema has a dest_ip column parsed from a DST=<ip> key/value pair embedded in the message body (firewall/netfilter-style log lines). It's a separate concept from collector_ip/source_ip and shows up as its own column and filter in Syslog Explorer. Most syslog lines have no destination-IP concept at all, so dest_ip is blank for them — that's expected, not a parsing failure.

IP intelligence / reputation lookup

Every public IP address shown anywhere in the UI (Syslog Explorer, Dashboard, Alerts) is clickable — it opens a lookup panel combining ipinfo.io (geolocation/ASN/hostname, plus company/privacy/abuse-contact on paid plans), ipapi.is (geolocation, ASN/org, company, abuse contact, VPN/proxy/Tor/datacenter/abuser detection — all in one call, no plan gating), AbuseIPDB (abuse confidence score, report history), and MXToolbox (reverse DNS/PTR, ASN, and a blacklist/RBL check) via GET /api/ip-info/{ip}, all four called concurrently. Private/loopback/link-local/multicast addresses aren't clickable — external providers have nothing useful to say about them.

This is per-user, not a global app setting: each user adds their own API keys under Settings → User Keys, and lookups run under the logged-in user's own keys/quota. Keys are Fernet-encrypted at rest (app/crypto.py, using a dedicated credential_key — separate from secret_key, which only signs JWTs) — decrypted only in memory when a lookup runs or the owning user views their own key. Five providers can have a key stored and tested there (AbuseIPDB, ipinfo.io, ipapi.is, MXToolbox, IPQualityScore), but only four of them are actually used by the lookup panel today — an IPQualityScore key can be saved and tested but isn't consumed anywhere yet.

MXToolbox's other commands — email/DNS record checks (SPF, DMARC, DKIM, MX, DNS, TXT, SOA, BIMI, MTA-STS, TLSRPT, A, AAAA) and active probes (ping, traceroute, TCP/HTTP/HTTPS/SMTP connect, run from MXToolbox's own infrastructure) — are reachable via POST /api/mxtoolbox/lookup ({command, argument, port?}) but aren't surfaced in the lookup panel yet; that's backend-only reach for now.

Settings layout

The Settings page is split into two sections, chosen from a section bar above the tab bar: Common (General · Security · Data · Notifications · User Keys · System — identical across every pkt* app) and pktLog (Collectors · Ingest). Selecting a section swaps the tab bar beneath it, so only one group's tabs is visible at a time; these previously shared a single row separated by a thin divider. Deep links still work unchanged — /settings?tab=devices, including the deep-link from an "Unknown collector" alert event, selects the right section automatically.

Contextual help throughout the UI

Small "?" buttons next to section headers (Dashboard, Alerts, Logs, Syslog Explorer, and most Settings sections — Auth, Suite Integration, Notifications, Data, etc.) open a short explainer of how that feature actually behaves — e.g. what a toggle does, what a bulk-import column means, what "Send Test" actually sends. Worth checking before assuming default behavior when a setting's effect isn't obvious from its label alone.

Notification channels

Alert rules can dispatch to six channels, each configured under Settings → Notifications: in-app (inapp, the alert itself), Slack (incoming webhook), Email (SMTP), PagerDuty (Events API v2), a generic Webhook (Jinja2-templated payload), and TraceCat SOAR. Each channel has a real "Send Test" button that performs an actual dispatch (real Slack post, real SMTP send, etc.) using whatever's currently filled in, even if unsaved — not a dry run.

User roles

Three roles: admin (full access, incl. user management), analyst (read + export, most write actions), viewer (read-only). When a request arrives via a pktHub-issued suite_token/SSO session, pktHub's own roles map onto these: admin→admin, analyst→analyst, and pktHub's viewer→pktlog's analyst (pktlog's own SSO role map treats "viewer" over SSO as read-only-but-still-useful rather than fully locked down — see _SUITE_ROLE_MAP in app/dependencies.py). A locally-created viewer user is genuinely read-only.

Requirements

Component	Version	Notes
OS	Ubuntu Server 22.04 LTS or 24.04 LTS	systemd required
Python	3.10+ (ships with Ubuntu 22.04/24.04)	venv created via `python3-venv`
ClickHouse	24.x+ (installed by `install.sh` from the official apt repo)
Node.js	20.x LTS	Frontend build only, not installed by `install.sh`
npm	10+	Frontend build only
System packages	`python3-venv`, `python3-pip`, `libxmlsec1-dev`, `libxmlsec1-openssl`, `xmlsec1`, `pkg-config`, `gcc`, `openssl`, `curl`, `ca-certificates`, `gnupg`, `apt-transport-https`	Installed by `install.sh`; `libxmlsec1*`/`pkg-config`/`gcc` are required to build `python3-saml`'s xmlsec bindings

Node.js is not installed by install.sh — install it yourself before the frontend build step, e.g. via NodeSource:

curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash -
sudo apt-get install -y nodejs

Python packages

See requirements.txt. Key dependencies:

fastapi, uvicorn[standard] — web framework
clickhouse-driver — ClickHouse backend (default); duckdb is an alternate/experimental backend
aiosqlite — app database
python-jose[cryptography], passlib[bcrypt] — JWT auth
python3-saml, authlib — SAML/OIDC SSO (Okta)

Frontend

React 18, TypeScript, Vite, Tailwind CSS, Recharts.

Installation

install.sh (see Quick Start) automates everything below, including step 8 (build the frontend) as long as npm is already on PATH when it runs — it falls back to printing the manual build commands only if npm isn't found. Step 11 (open the firewall) is always manual. This section is the full manual walkthrough — useful to customize the install, run steps individually, or understand what the script does.

1. Clone the repository

git clone https://github.com/bsnwgit/pktlog.git
cd pktlog

All commands below assume you're in the repo root unless otherwise noted.

2. Create the install directory

INSTALL_DIR=/opt/pktlog
sudo mkdir -p "$INSTALL_DIR" "$INSTALL_DIR/logs"
sudo chown "$(whoami):$(whoami)" "$INSTALL_DIR" "$INSTALL_DIR/logs"

/opt is root-owned by default, so this needs sudo. Steps 5–8 below run as your regular user against this now-owned directory; step 9 re-owns everything to whichever user/group the systemd service runs as.

3. System packages + ClickHouse

sudo apt-get update
sudo apt-get install -y --no-install-recommends \
    python3 python3-venv python3-pip \
    libxmlsec1-dev libxmlsec1-openssl xmlsec1 pkg-config gcc \
    curl ca-certificates gnupg apt-transport-https openssl

# ClickHouse — official apt repo
curl -fsSL https://packages.clickhouse.com/rpm/lts/repodata/repomd.xml.key \
    | sudo gpg --dearmor -o /usr/share/keyrings/clickhouse-keyring.gpg
ARCH="$(dpkg --print-architecture)"
echo "deb [signed-by=/usr/share/keyrings/clickhouse-keyring.gpg arch=${ARCH}] https://packages.clickhouse.com/deb stable main" \
    | sudo tee /etc/apt/sources.list.d/clickhouse.list
sudo apt-get update
sudo apt-get install -y clickhouse-server clickhouse-client
sudo systemctl enable --now clickhouse-server

libxmlsec1-dev, libxmlsec1-openssl, pkg-config, and gcc are required to build python3-saml's xmlsec native bindings.

4. Apply ClickHouse schema

clickhouse-client --multiquery < clickhouse/schema.sql

Creates the pktlog database and the syslog_events table. See the note at the top of clickhouse/schema.sql — this file was reconstructed from the app's insert code rather than an existing checked-in schema; if your running deployment's table differs, treat that as the source of truth.

5. Install Python dependencies

python3 -m venv /opt/pktlog/venv
/opt/pktlog/venv/bin/pip install -r requirements.txt

6. Copy application files

pktlog.service runs uvicorn app.main:app with WorkingDirectory=/opt/pktlog, so the app package must live there:

cp -r app migrations clickhouse /opt/pktlog/

7. Configure

cp config.example.yaml /opt/pktlog/config.yaml
# Edit config.yaml — set secret_key, db_path, cors_origins
openssl rand -hex 32   # use this as secret_key

config.yaml reference:

Key	Default	Description
`host`	`0.0.0.0`	Bind address
`port`	`8768`	Listen port
`db_path`	`/opt/pktlog/pktlog.db`	SQLite database path
`clickhouse_host`	`localhost`	ClickHouse host
`clickhouse_port`	`9000`	ClickHouse native protocol port
`clickhouse_database`	`pktlog`	ClickHouse database name
`syslog_port`	`5514`	Syslog ingest port (UDP + TCP)
`secret_key`	CHANGE THIS	JWT signing key (32+ random bytes)
`credential_key`	CHANGE THIS	Fernet key encrypting stored secrets (user API keys) at rest — generate with `python3 -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"`
`cors_origins`	`["*"]`	Restrict to your dashboard origin in production
`log_file`	`/opt/pktlog/logs/pktlog.log`	Log path

The initial admin user is created directly in SQLite (see step 9's Python snippet, or use seed_admin.py after install) — there is no admin_user/admin_password config.yaml field.

8. Build the frontend

Requires Node.js 20.x LTS. The frontend must be built on Linux — not on Windows (Windows node_modules lacks the Linux rollup native binary).

install.sh does this step for you automatically if npm is already installed when it runs — the commands below are only needed if you're doing a fully manual install, or if npm wasn't present at install time (in which case install.sh prints these same commands as a fallback and leaves the web UI returning {"detail":"Not Found"} until you run them):

cp -r frontend /tmp/pktlog-fe
cd /tmp/pktlog-fe
npm install
npm run build > /dev/null 2>&1 && echo "build ok" || echo "BUILD FAILED"
cp -r dist /opt/pktlog/frontend/dist

9. Initialize the database and create the admin user

/opt/pktlog/venv/bin/python3 - << 'PYEOF'
import asyncio, sys, os
sys.path.insert(0, '/opt/pktlog')
os.environ['PKTLOG_CONFIG'] = '/opt/pktlog/config.yaml'

from app.database import init_db
from app.auth.local import hash_password
import aiosqlite
from app.config import get_settings

async def setup():
    await init_db()
    async with aiosqlite.connect(get_settings().db_path) as db:
        hashed = hash_password('CHANGE_ME')
        await db.execute(
            "INSERT OR IGNORE INTO users (username, email, hashed_password, role) VALUES (?,?,?,?)",
            ('admin', 'admin@pktlog.local', hashed, 'admin')
        )
        await db.commit()

asyncio.run(setup())
PYEOF

Replace CHANGE_ME with a real password before running this. init_db() also applies all migrations/*.sql files (idempotent — safe to call on every startup, which is what app/main.py does).

install.sh itself does more than this minimal snippet: it generates a random admin password (openssl rand -base64 12, printed once at the end of the run — this snippet's hardcoded password is only for a fully manual/scripted install), flags that account as the default admin (is_default_admin — see User roles and the auto-login note under Security below), seeds one collector_registry row for the install host's own detected IP (so there's at least one working collector out of the box instead of an empty registry), and stores a base_url setting derived from that IP and the chosen port (used to build SAML ACS/metadata URLs on the Auth settings tab).

10. Install and start the systemd service

pktlog.service is a template — substitute the placeholders before installing it, or just run install.sh which does this for you:

sed \
    -e "s#__INSTALL_DIR__#/opt/pktlog#g" \
    -e "s#__LOG_DIR__#/opt/pktlog/logs#g" \
    -e "s#__SERVICE_USER__#$(whoami)#g" \
    -e "s#__SERVICE_GROUP__#$(whoami)#g" \
    pktlog.service | sudo tee /etc/systemd/system/pktlog.service
sudo systemctl daemon-reload
sudo systemctl enable --now pktlog
sudo systemctl status pktlog

11. Open the firewall

sudo ufw allow 8768/tcp        # web UI / API
sudo ufw allow 5514/tcp        # syslog ingest (TCP)
sudo ufw allow 5514/udp        # syslog ingest (UDP)

12. Verify

curl -sk https://localhost:8768/api/health

Log in at http://<server-ip>:8768 (or https:// if SSL is configured) with the admin credentials from step 9.

SSL / HTTPS

pktLog auto-detects SSL on startup. If <INSTALL_DIR>/ssl/server.crt and server.key exist, it starts in HTTPS mode; otherwise HTTP. pktlog.service's ExecStart runs start.sh (not uvicorn directly), which implements this detection — so SSL just works under systemd once a cert/key are present, no unit changes needed.

To enable HTTPS: upload cert/key via Settings → Security → SSL / TLS, then restart the service.

To disable HTTPS: remove the cert via the same Settings panel (or delete the files under <INSTALL_DIR>/ssl/), then restart.

Gotcha — a previously-installed unit can bypass start.sh. If /etc/systemd/system/pktlog.service was installed before this start.sh-based ExecStart existed (or was hand-edited to invoke uvicorn directly), SSL/other start.sh behavior silently won't apply — check with systemctl cat pktlog | grep ExecStart and reinstall the unit from the repo's pktlog.service template if it doesn't match, then sudo systemctl daemon-reload && sudo systemctl restart pktlog.

pktSuite Integration

pktLog supports SSO with pktHub/pktFlow via a shared suite_token:

Generated on first call to GET /api/suite/token, or set/regenerated via Settings → Security → Suite Integration (this tab was labeled "pktHub Integration" before it was renamed to "Suite Integration")
Stored in SQLite (settings table), not in config.yaml
Requests carrying a matching X-Suite-Token header are trusted as coming from pktHub (see app/dependencies.py, app/api/suite.py)
GET /api/suite/whoami is what a sibling app's "Test Connection" button actually calls (not the public /api/health), so a wrong/revoked token fails the test instead of silently reporting a healthy connection
Copying the token from the Settings UI works over plain HTTP as well as HTTPS (falls back off the browser clipboard API, which requires a secure context, when needed)

Hub-managed direct-access lock

pktHub can remotely lock pktLog's direct UI so users are redirected to sign in via pktHub instead: POST /api/suite/direct-access {"locked": true} (authenticated with X-Suite-Token). While locked, every non-API/non-asset request is redirected to whichever Hub Redirect URL is configured on the Suite Integration tab (also settable via PATCH /api/suite/hub-redirect-url).

This can't permanently strand admins out of the UI: a heartbeat (refreshed on every suite-token-authenticated request) auto-clears the lock if it goes stale for more than 5 minutes, and the lock is also cleared automatically at application startup if pktHub itself is unreachable. If direct access ever seems unexpectedly blocked, check GET /api/suite/mode (no auth required) for the current direct_ui_locked/hub_redirect_url state.

Outbound integrations (sibling apps)

Separately from the inbound suite_token above, pktLog has a backend API (/api/integrations) for storing named, admin-managed connections from pktlog to other pkt* apps (pktIPAM, pktFlow, pktSNMP, pktPCAP, pktWiFi, pktHub) — same pattern as the equivalent feature in pktIPAM/pktFlow/pktWiFi. As of this writing there is no Settings tab exposing it and no feature actually consumes a configured connection yet; it exists as forward-looking scaffolding, not a working integration. Each connection's suite_token is Fernet-encrypted at rest (app/crypto.py, the same credential_key used for user API keys) — decrypted only in memory when actually used to authenticate an outbound call.

Nav manifest (pktHub's APPS sidebar)

GET /api/nav/manifest (app/api/nav.py) publishes pktLog's own left-nav so pktHub can mirror it under APPS in its sidebar. Entries are {path, label, icon, admin_only, divider_before}. pktHub's health poller reads the endpoint on every cycle and caches the result, so a page added here shows up in the hub within one poll interval with no change on the hub side.

Selecting one of those rows opens pktLog's real page inside pktHub — proxied, and chromeless so it renders without this app's own sidebar or header. It is not a re-implementation and cannot drift from what the page actually does.

NAV_MANIFEST in app/api/nav.py and NAV in frontend/src/components/Layout.tsx are two declarations of one menu, and each carries a comment pointing at the other — a page added to one belongs in both. The endpoint is gated by require_suite_token for the same reason the widget endpoints are: it discloses this app's page structure.

admin_only controls only what the hub draws. The real authorisation is this app's own role check against the X-Suite-Role pktHub asserts.

Chromeless layout needs a definite height

Layout.tsx's chromeless branch uses h-screen overflow-auto, not min-h-screen. A page that fills its container sizes itself with h-full, which resolves against the parent's height — and collapses to zero against an auto-height parent, rendering blank. Maps and canvases hit this first.

Widget endpoints now require the suite token

app/api/widgets.py previously mounted its router with a bare APIRouter(), so the server-rendered widget views — which read internal data — answered anyone who could reach the port. The router now carries dependencies=[Depends(require_suite_token)], matching the NOC Builder's actual access path. Anything calling those URLs without X-Suite-Token now gets a 401.

Resonance (embedded assistant)

Resonance is the suite's shared assistant. It mounts as a launcher in the bottom corner of every authenticated page — the same place the removed in-app AI Assistant sat — but the assistant runs on the resonance server, not inside pktLog.

pktLog is the reference implementation for the whole suite: app/integrations/resonance/ and frontend/src/resonance/ are vendored, meaning they are copied between pkt* apps byte-for-byte except for APP_SLUG. They are deliberately not a published package, because install.sh builds a venv on customer hosts and a private index would put a credentialed network dependency in the middle of every install.

How a session is established

browser                 pktLog                       resonance
───────                 ──────                       ─────────
embed.js  ──GET──▶  /api/resonance/code  ──POST──▶  /embed/session
          ◀─code──                       ◀─code───
frame ──────────────────────────────────────────────▶  /embed?c=<code>

pktLog vouches for whoever is signed in and receives a short-lived, single-use code. The key never reaches the browser, and resonance never sees a pktLog credential. The identity sent is pktlog-<username> — app and login together, so resonance's audit trail shows both. No email address is sent.

GET /api/resonance/code is the one cookie-authenticated route in the app. embed.js fetches data-code-url itself, outside the SPA, and pktLog's access token lives in memory by design, so the refresh cookie is the only credential the browser will attach. It is validated the way /api/auth/refresh validates it and is not rotated. Sec-Fetch-Site and Origin are both checked before the cookie is honoured, so the route does not rest on SameSite alone.

Setting it up

On the resonance side

Create a key for this pktLog install. One key is one placement.
Note the interface server address — the one enrolled under SETTINGS ▸ ENROLL ▸ Enroll Embed Server, which is not the admin portal. The admin portal serves embed.js as well, so pointing at it looks correct until /embed/session answers 404.
Set its session TTL to 480 minutes, matching pktLog's own session_timeout_minutes. The panel warns when they disagree.
Turn on Speakers Name (needs_user). Without it resonance records nothing at all — no audit trail for who asked what.
Add this install's origin to the key's allow-list. The exact string is shown, ready to copy, under Settings → Resonance → Diagnostics.
Authorise the key against a profile scoped to pktLog — see Guardrails below.

On the pktLog side

Settings → Resonance: paste the interface server address and key, choose which roles may use it, press Test Connection, then switch Enabled on. Test works whether or not the feature is enabled, on purpose — an admin must be able to prove a key before putting a widget in front of users, and to diagnose one after turning it off.

Prerequisites

Resonance must be reachable from the browser over HTTPS with a certificate those browsers already trust. A self-signed or internal-CA certificate produces an empty widget with nothing in the console explaining why. There is no verify=False for browsers.
pktLog itself must be on HTTPS for voice. getUserMedia is gated on a secure context, so over plain HTTP the microphone cannot work however the key is configured. pktLog detects this and narrows the microphone away rather than showing a control that does nothing. Text chat is unaffected.

Guardrails — enforced in resonance, not here

pktLog cannot restrict what the assistant will discuss. The input box lives in resonance's iframe on resonance's origin; pktLog cannot see what is typed, cannot intercept it, and has no field in the session contract to assert a topic scope. Anything claiming otherwise would be theatre.

Scope is enforced by the profile the key is authorised against. Give each pkt* app its own profile, scoped by its own instructions, with the app's own documentation as its corpus, and tick that app's key onto that profile alone. Because a key is per-placement, resonance already knows which app is calling without pktLog telling it.

GET /api/resonance/docs serves that corpus: every docs/*.md file from the running install, each with a SHA-256, behind the existing suite token (or an admin session). It sends an ETag and honours If-None-Match, so a resonance that polls costs a 304 rather than a re-ingest. Documentation only — no log data, no user data, nothing from the settings table. Pointing resonance at it means upgrading pktLog updates what the assistant knows, instead of leaving a profile quietly describing last year's UI.

When it does not appear

embed.js logs once to the console and gives up permanently if its script fails to load, which from a user's side is indistinguishable from the feature not existing. pktLog covers the observable half:

The mount retries a missing script with backoff for about two minutes, which covers a service restart during page load.
A script that never arrives is reported to POST /api/resonance/report, and Settings → Resonance → Diagnostics shows "the widget failed to load for N users in the last 7 days." The usual causes are an ad blocker eating third-party script tags, a wrong server address, or resonance being unreachable.
Repeated failures from /code open a circuit breaker (resonance_breaker). Resonance applies a geometric per-IP backoff to bad-key attempts and pktLog is a single IP, so continuing to knock would take the widget down for everyone and keep the backoff growing. The panel shows the paused state; fixing the key and pressing Test Connection clears it.

A failed renewal is not detected. embed.js renews through an endpoint the frontend cannot observe, so there is no signal that distinguishes a dead session from a quiet one, and a timer that assumed the worst would destroy live conversations to fix a rare failure. With a 480-minute session most users never reach a renewal at all.

Developing without a resonance server

scripts/resonance_stub.py is a stand-in for /embed/session implementing the documented contract, with the key selecting the branch (good., disabled., adminport., backoff., anything else) so every error path is reachable offline.

python3 -m uvicorn scripts.resonance_stub:app --port 9911

Then point Server address at http://127.0.0.1:9911 and use the key good.secret.

Deployment

Backend changes

Copy changed files to /opt/pktlog/ on the server (same relative path as the repo), e.g. via deploy_backend.py
sudo systemctl restart pktlog
Verify: curl -sk https://localhost:8768/api/health

Frontend changes

The frontend must be built on Linux — build on the server itself or a Linux CI runner, not on a Windows machine.

cp -r frontend /tmp/pktlog-fe
cd /tmp/pktlog-fe
npm install
npm run build
cp -r dist /opt/pktlog/frontend/dist
sudo systemctl restart pktlog

deploy_fe.py automates this over SSH from a local checkout.

Operational scripts

These live in the repo root and are SSH/SFTP-based tools for managing a remote deployment (all take --host/--user/--key flags or PKTLOG_SSH_* env vars — no hardcoded infrastructure):

Script	Purpose
`deploy_backend.py`	Push backend files and restart the service
`deploy_fe.py`	Sync frontend source, build remotely, deploy `dist/`
`deploy_initial.py`	Fresh install over SSH (mirrors `install.sh` minus system packages/ClickHouse)
`check_server.py`	Quick remote status check (service, disk, logs)
`seed_admin.py`	Create/reset the admin user's password remotely
`backup.py`	Local 2-rotation backup of the project directory

Directory Structure

pktlog/
├── app/
│   ├── api/
│   │   ├── auth.py          Login, SAML, token refresh
│   │   ├── syslog.py        Syslog search/stats/timeseries
│   │   ├── logs.py          App log viewer
│   │   ├── collectors.py    Collector registry CRUD
│   │   ├── settings.py      App settings CRUD
│   │   ├── users.py         User management
│   │   ├── system.py        Health, restart, SSL upload, backup
│   │   ├── suite.py         pktSuite suite_token issuance/registration, hub direct-access lock
│   │   ├── integrations.py  Outbound connections to sibling pkt* apps (backend-only, no UI yet)
│   │   ├── ip_info.py       Per-user IP intelligence/reputation lookup (ipinfo.io + ipapi.is +
│   │   │                     AbuseIPDB + MXToolbox ptr/asn/blacklist)
│   │   ├── mxtoolbox.py     Generic MXToolbox command passthrough (/api/mxtoolbox/lookup) —
│   │   │                     DNS/email records + active probes
│   │   ├── user_api_keys.py Per-user external API key storage (AbuseIPDB/ipinfo.io/ipapi.is/
│   │   │                     MXToolbox/IPQualityScore)
│   │   ├── ws.py             WebSocket for live dashboard/alert updates
│   │   ├── widgets.py       Dashboard widgets
│   │   └── pktlog.py        Misc endpoints
│   ├── auth/                Local (JWT+bcrypt), Okta OIDC, SAML
│   ├── alerts/               Alert evaluation engine (6 notification channels)
│   ├── integrations/          SuiteClient — shared HTTP client for calling sibling pkt* apps
│   ├── ingest/
│   │   ├── listener.py       Async UDP+TCP syslog listener (port 5514)
│   │   ├── parser.py         RFC 3164 / RFC 5424 parsing, dest_ip (DST=) extraction
│   │   ├── normalizer.py     org/group/site enrichment, collector allowlist gate
│   │   └── writer.py         Batch writer → storage backend
│   ├── models/syslog.py       SyslogRecord dataclass
│   ├── storage/
│   │   ├── clickhouse.py     ClickHouse backend (production)
│   │   ├── duckdb.py         DuckDB backend (alternate)
│   │   └── factory.py        Backend selector
│   ├── config.py             Settings loader (YAML + env)
│   ├── database.py           SQLite init + migration runner
│   └── main.py                App factory, lifespan, router registration, direct-access-lock middleware
├── clickhouse/schema.sql       syslog_events table (MergeTree, 18 columns incl. dest_ip)
├── frontend/src/
│   ├── pages/                 Login, Dashboard, Alerts, Settings, Users, Logs, SyslogExplorer
│   └── api/client.ts           Typed API client
├── migrations/                 SQLite migration scripts (auto-applied on startup)
├── install.sh                  Ubuntu install script (ClickHouse, venv, systemd service)
├── config.example.yaml         Config file template
├── pktlog.service               systemd unit template (placeholders filled in by install.sh)
├── start.sh                     SSL-aware startup wrapper (manual/dev use)
└── requirements.txt

Known Issues & Quirks

One-time data migrations in init_db() must be lock-protected — root-caused and fixed 2026-08-04. pktLog is the only pkt* app that runs uvicorn --workers 2 (start.sh); every sibling app is single-process. FastAPI's startup lifespan — which runs init_db(), including any one-time data migration — fires independently in both worker processes on every restart. An unprotected migration (any function following the SELECT unencrypted rows → loop UPDATE-encrypting them pattern, e.g. _encrypt_legacy_api_keys/_encrypt_legacy_suite_tokens) can run in both processes at once the first time it ever executes, racing to write the same rows — this caused real, repeated pktlog.db corruption (PRAGMA integrity_check btree failures) on 2026-08-04. Fixed with a cross-process fcntl.flock around the entire migration body in init_db() (an asyncio.Lock would not help — these are separate OS processes, not threads). Any new one-time migration added to this file is automatically protected as long as it's called from inside _run_migrations()/init_db(), same as the existing ones — don't add a migration that opens its own separate connection outside that lock.

Log Forwarding

pktLog writes its own application log to the in-app Logs page. It can also ship that log to a syslog collector — normally pktLog, which listens on port 5514 — so this app's events sit alongside the rest of the estate.

Settings keys (Settings → Data → Log Forwarding in apps that expose the UI; otherwise via PUT /api/settings):

Key	Default	Meaning
`log_forward_enabled`	`false`	Turn forwarding on
`log_forward_host`	`""`	Collector hostname or IP
`log_forward_port`	`5514`	pktLog's syslog port
`log_forward_protocol`	`udp`	`udp` or `tcp`
`log_forward_level`	`INFO`	Minimum level forwarded
`log_forward_app_name`	`pktlog`	APP-NAME in the syslog message

Admin endpoints:

GET /api/system/log-forward/status — delivery counters (sent, dropped, errors)
POST /api/system/log-forward/test — send one test line without saving settings
POST /api/system/log-forward/reload — apply settings changes without a restart

Format is RFC 5424, deliberately. pktLog parses both 3164 and 5424, but 3164 timestamps carry no timezone and the collector has to guess the offset — which has produced wrong timestamps in this suite before. 5424 carries a full offset, so there is nothing to guess.

Delivery is fire-and-forget on a background thread, with counters. Log forwarding must never block or crash the thing it observes: a dropped line is a nuisance, a stalled collector loop is an outage. If the collector is unreachable, lines are dropped and counted rather than raised.

If forwarded logs never arrive

pktLog drops syslog from sources that are not registered. Its collector_registry gates what is allowed to persist, so the sending host's IP must be approved (Approval page) and enabled. Until then the messages are accepted on the wire and silently discarded — the sender sees a successful send either way, because UDP cannot tell it otherwise. pktLog also caches that registry for five minutes, so a newly enabled source is not live immediately.

Use the Send test message button (or the test endpoint) to confirm the path end to end rather than assuming it works.

Worker Count

pktLog runs single-process by default (PKTLOG_WORKERS, default 1).

It previously defaulted to 2, which made it the only app in the suite driving its SQLite sidecar database from two OS processes. FastAPI's lifespan hook runs once per worker, so the alert engine, alert cleanup, backup scheduler, SQLite log handler and ingest BatchWriter each ran in duplicate against the same file. pktlog.db corrupted under that arrangement twice — 2026-08-04 and 2026-08-09 — and no single-process sibling ever has.

The second worker also bought very little: the syslog listener can only bind the port once, so the second worker logged "port already bound" and served HTTP only.

Raise PKTLOG_WORKERS if you need more HTTP concurrency, but give the background jobs a single-leader guard first, or the same corruption path reopens.

Security Notes

Change secret_key in config.yaml (or PKTLOG_SECRET_KEY env var) before production use — openssl rand -hex 32
Change the default admin password immediately after first login (install.sh generates a random one and prints it once; it is not recoverable afterward except via seed_admin.py or a direct DB reset)
cors_origins should be restricted to your dashboard origin in production
Don't disable both Local auth and SAML SSO (Settings → Security → Auth) unless the UI is only reachable from a genuinely trusted network. With both off, the login page is skipped entirely and anyone who reaches it is auto-logged in as the designated default admin (users.is_default_admin, or the oldest active admin if none is flagged) via POST /api/auth/auto-login — there is intentionally no login prompt in that state
The pktSuite suite_token is a shared secret across pktHub/pktFlow/pktLog — never commit a real value to a tracked file, and rotating it requires updating it in all three services simultaneously
If a config.yaml with a real secret_key or suite_token is ever accidentally committed, treat both as compromised: rotate secret_key immediately (it only affects this service), but coordinate before rotating suite_token since it will break SSO for pktHub/pktFlow until updated everywhere
Outbound integrations.suite_token rows (see Outbound integrations) are Fernet-encrypted at rest using the same credential_key as user API keys — a legacy plaintext row is encrypted automatically, once, the next time the app starts

The pkt suite

pktLog is one of ten apps in the pkt suite — self-hosted tooling for network and security operations. Each installs and runs standalone, so take only the ones you need; they share one architecture (FastAPI + React), one look, one admin/analyst/viewer role model, and a suite token that lets siblings read one another's data. Default ports don't collide (8760–8769), so any combination runs on a single host.

App	Port	What it does
pktFlow	`8766`	NetFlow, sFlow and IPFIX collection — flow search, traffic analytics, geo and topology views
pktSNMP	`8767`	SNMP polling and trap receiving for any OID — device health and metric history without a full NMS
pktLog (you are here)	`8768`	Syslog over UDP, TCP and TLS — parsing, enrichment, full-text search and forwarding
pktPCAP	`8765`	Packet capture analysis in the browser — drop in a `.pcap` for TCP, DNS and threat findings, no Wireshark install
pktWiFi	`8769`	Access point, RF and client visibility from Meraki and UniFi controllers or plain SNMP polling
pktIPAM	`8761`	IP address management reconciling declared subnets against live DHCP, DNS and device data, flagging conflicts
pktNode	`8764`	Endpoint monitoring and management for Mac, Windows and Linux via a lightweight Go agent
pktSecurity	`8762`	Security operations across the estate — CVE exposure, threat intelligence, ATT&CK-mapped detections and case management
pktCert	`8763`	TLS certificate discovery and expiry tracking, plus an internal CA — issue, revoke and serve CRLs
pktHub	`8760`	The front door — one sign-in, one alert stream, NOC wallboards and user management across every registered app

pktHub is optional — it registers the others and puts them behind a single login with shared alerting and NOC wallboards — but every app is fully usable without it.

More at pktsolution.com.

License

This project is distributed under the PolyForm Noncommercial License 1.0.0 — see LICENSE.

Name		Name	Last commit message	Last commit date
app		app
clickhouse		clickhouse
docs		docs
frontend		frontend
migrations		migrations
scripts		scripts
.gitattributes		.gitattributes
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
SECURITY.md		SECURITY.md
VERSION		VERSION
backup.py		backup.py
check_server.py		check_server.py
config.example.yaml		config.example.yaml
deploy_backend.py		deploy_backend.py
deploy_fe.py		deploy_fe.py
deploy_initial.py		deploy_initial.py
install.sh		install.sh
lockup.svg		lockup.svg
pktlog.service		pktlog.service
requirements.txt		requirements.txt
seed_admin.py		seed_admin.py
start.sh		start.sh
uninstall.sh		uninstall.sh

Latest commit

History

Folders and files

Repository files navigation