Skip to content

Move log-to-statistics aggregation from on-demand to periodic #623

Description

@janbjorge

Problem

Log aggregation (pgqueuer_log -> pgqueuer_statistics) currently only runs when someone calls log_statistics(). This means:

  • If nobody queries stats, log rows pile up with aggregated = FALSE
  • The first stats query after a long gap has to aggregate a large backlog
  • There's no predictable cadence for when aggregation happens

Proposal

Use PgQueuer's own cron scheduler to run aggregation on a configurable interval. For example, every 30 seconds or every minute.

Something like:

@sm.schedule("aggregate_pgqueuer_logs", "* * * * *")
async def aggregate_pgqueuer_logs(schedule: Schedule) -> None:
    await queries.log_statistics(limit=0)

Or expose a dedicated aggregate() method that runs just the aggregation query without fetching results.

What needs to change

  • Add a built-in scheduled task (or opt-in configuration) that runs aggregation periodically
  • The interval should be configurable
  • log_statistics() should still work as-is (aggregate + fetch), but the periodic task keeps the backlog small
  • Consider adding a standalone aggregate_logs() method that only runs build_aggregate_log_data_to_statistics_query without the fetch

Context

Raised in #621

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions