Skip to content

Add hashtag and editor analytics APIs#50

Open
NirrWorks wants to merge 9 commits into
osgeonepal:masterfrom
NirrWorks:feature/hashtag-editor-stats
Open

Add hashtag and editor analytics APIs#50
NirrWorks wants to merge 9 commits into
osgeonepal:masterfrom
NirrWorks:feature/hashtag-editor-stats

Conversation

@NirrWorks

Copy link
Copy Markdown
Contributor

Summary

Adds hashtag and editor analytics API support.

Changes

  • Added /api/v1/hashtag-stats

    • returns top hashtag leaderboard
    • includes hashtag trends over time
    • supports date range, hashtag filter, interval, limit, and offset
  • Added /api/v1/editor-stats

    • returns editor/tool usage stats such as iD, JOSM, StreetComplete, etc.
    • includes changesets, users, map changes, and rank
  • Added PostgreSQL indexes for analytics query performance:

    • changesets.hashtags
    • changesets.editor
    • changeset_stats.changeset_id
  • Added performance check script using EXPLAIN ANALYZE

    • supports real database testing
    • supports synthetic temporary data testing with large row counts
  • Added API tests for new response shapes and SQL behavior

Performance Check

Tested with:

  • local DB with ~47k stats rows
  • synthetic temporary PostgreSQL dataset with:
    • 1M changesets
    • 1M changeset_stats rows

Synthetic 1M-row results were around:

  • hashtag stats: ~3.0s
  • hashtag trends: ~2.9s
  • editor stats: ~3.2s

This suggests the current queries are acceptable for bounded windows, but for planet-scale/all-time analytics we may later need cached/materialized aggregate tables.

Testing

uv run --group api pytest tests/test_api.py
uv run --group api ruff check api tests scripts/check_api_query_performance.py
uv run --group api ty check api

Comment thread docs/api-performance.md

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need for this file to be included in PR you can keep it in your working tree but not here ! Explain analyze is right way to do it , you can directly do it in database level as well !

@NirrWorks

Copy link
Copy Markdown
Contributor Author

Updated the PR with the fixes from the last discussion:

  • Fixed hashtag aggregation so hashtag=tt_event aggregates only #tt_event, instead of also returning co-occurring hashtags from the same changesets.
  • Fixed empty tag stats behavior so tags=true returns {} when no tag breakdown exists, while tags=false still returns null intentionally.
  • Added tests for both cases.

All checks are passing now.

@NirrWorks

Copy link
Copy Markdown
Contributor Author

Update

Resolved the merge conflicts with the latest master and added:

  • /api/v1/map with changeset centroid points for map clustering and heatmaps.
  • Bounding-box export without requiring PostGIS geometry.
  • Compact key aggregation by default, with tag_mode=all for full values.
  • Optional editor versions using include_version=true.
  • Standard pagination metadata across collection endpoints.
  • Updated tests, documentation, and performance checks.

Comment thread api/schemas.py
len: float | None = None


class PaginationMeta(BaseModel):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to write basemodel for pagination try to use the starlette paginator directly !

Comment thread osmsg/pg_schema.py
CREATE INDEX IF NOT EXISTS idx_changesets_geom ON changesets USING GIST (geom);
CREATE INDEX IF NOT EXISTS idx_changesets_hashtags ON changesets USING GIN (hashtags);
CREATE INDEX IF NOT EXISTS idx_changesets_editor ON changesets(editor);
CREATE INDEX IF NOT EXISTS idx_changesets_bbox ON changesets(min_lon, min_lat, max_lon, max_lat);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

create index gist ! use sgist index for bbox !
use btree , specify the type of the index

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants