Problem
Log aggregation (pgqueuer_log -> pgqueuer_statistics) currently only runs when someone calls log_statistics(). This means:
- If nobody queries stats, log rows pile up with
aggregated = FALSE
- The first stats query after a long gap has to aggregate a large backlog
- There's no predictable cadence for when aggregation happens
Proposal
Use PgQueuer's own cron scheduler to run aggregation on a configurable interval. For example, every 30 seconds or every minute.
Something like:
@sm.schedule("aggregate_pgqueuer_logs", "* * * * *")
async def aggregate_pgqueuer_logs(schedule: Schedule) -> None:
await queries.log_statistics(limit=0)
Or expose a dedicated aggregate() method that runs just the aggregation query without fetching results.
What needs to change
- Add a built-in scheduled task (or opt-in configuration) that runs aggregation periodically
- The interval should be configurable
log_statistics() should still work as-is (aggregate + fetch), but the periodic task keeps the backlog small
- Consider adding a standalone
aggregate_logs() method that only runs build_aggregate_log_data_to_statistics_query without the fetch
Context
Raised in #621
Problem
Log aggregation (
pgqueuer_log->pgqueuer_statistics) currently only runs when someone callslog_statistics(). This means:aggregated = FALSEProposal
Use PgQueuer's own cron scheduler to run aggregation on a configurable interval. For example, every 30 seconds or every minute.
Something like:
Or expose a dedicated
aggregate()method that runs just the aggregation query without fetching results.What needs to change
log_statistics()should still work as-is (aggregate + fetch), but the periodic task keeps the backlog smallaggregate_logs()method that only runsbuild_aggregate_log_data_to_statistics_querywithout the fetchContext
Raised in #621