logsift

Turn a raw nginx/Apache access log into the report you actually want during an incident: error rates, hot endpoints, latency percentiles, noisy clients — in one command, with no dependencies beyond the Python standard library.

$ logsift /var/log/nginx/access.log

requests: 1,284,311   (+37 malformed lines skipped)
status:   2xx: 1,241,002   4xx: 39,921   5xx: 3,388
errors:   3.37% of requests were 4xx/5xx
latency:  p50: 42ms   p90: 180ms   p95: 311ms   p99: 1204ms

top endpoints (by volume)
   requests    4xx    5xx      p95  endpoint
    401,213     12      0     38ms  GET /health
    204,559    118   2101    950ms  POST /api/orders
    118,224     41      0    122ms  GET /api/users/:id
    ...

slowest endpoints (p95)
   2302ms  GET /api/reports/monthly
    950ms  POST /api/orders
    ...

Why

When something is on fire, the questions are always the same: which endpoint, since when, how bad, who's hitting us? Grep pipelines answer them slowly and differently every time. logsift answers them the same way every time, fast, on any machine that has Python — no agent, no ELK stack, no waiting for the dashboards you wish you had set up.

Install

pip install git+https://github.com/kasak-sh/logsift
# or, from a checkout:
pip install .

Usage

logsift access.log                    # human-readable report
logsift access.log --json            # machine-readable, for jq/scripts
logsift a.log b.log --top 20         # multiple files, bigger tables
zcat access.log.3.gz | logsift       # reads stdin — works with rotated logs

Accepted formats: Apache/nginx common and combined, plus nginx's widespread extension of appending $request_time after the user agent — that's what enables the latency columns. Malformed lines are counted and skipped, never fatal.

What it does that grep doesn't

• Route templating. /api/users/42 and /api/users/9000 are the same endpoint. Numeric IDs, UUIDs and long hex tokens collapse to :id, :uuid, :token, so "top endpoints" ranks routes, not URLs.
• Interpolated percentiles (p50/p90/p95/p99), overall and per endpoint — averages hide exactly the requests you're being paged about.
• Error attribution. 4xx and 5xx are counted per endpoint, so a spike in global error rate turns into a specific route in seconds.
• Exit codes for scripting: 0 on success, 2 when nothing parseable was found.

Development

pip install -e ".[dev]"
pytest

License

MIT