fix(github_graphql): paginate reviews beyond 100 and fix incremental sync sort order (DPROD-1260) by kpiwko · Pull Request #93 · konflux-ci/devlake

kpiwko · 2026-06-02T19:10:05Z

Summary

Fixes two bugs in github_graphql PR collection that caused incomplete review data:

Sort order mismatch: pullRequests was ordered by CREATED_AT but the incremental sync stop condition compares UpdatedAt. Switched to UPDATED_AT DESC so the cursor-based early exit fires correctly and recently-edited old PRs are not skipped.
Review truncation: reviews(first: 100) silently dropped reviews beyond 100 per PR. Added fetchRemainingReviews() which follows PageInfo.hasNextPage cursor pagination, applied in both collection passes (new PRs and OPEN PR refresh).

Tests

11 unit tests (pr_collector_test.go): cover fetchRemainingReviews pagination (single page, multi-page, nil PageInfo, query error), the incremental sync stop condition (nil since, stop-immediately, partial, empty list), and the review-merge guard.
1 e2e test (e2e/pr_extractor_test.go): loads a raw PR fixture with 5 reviews (4 non-PENDING + 1 PENDING), runs ExtractPrsMeta, and verifies exactly 4 rows in _tool_github_pull_request_reviews (PENDING filtered by extractor).

Test plan

All unit tests pass: go test ./plugins/github_graphql/tasks/... -v
Both e2e tests pass: go test ./plugins/github_graphql/e2e/... -v (requires MySQL lake_test)
No linter errors in changed files

Closes DPROD-1260

🤖 Generated with Claude Code

…sync sort order (DPROD-1260) Two bugs prevented complete PR review collection in the github_graphql plugin: 1. Sort order was CREATED_AT but the incremental stop condition used UpdatedAt, causing missed updates for old PRs edited recently. Changed to UPDATED_AT DESC. 2. The reviews(first: 100) query silently truncated PRs with >100 reviews. Added fetchRemainingReviews() to follow PageInfo.hasNextPage cursors, and applied it in both ResponseParsers (new PRs + OPEN PR refresh). Added unit tests covering the pagination logic and stop-condition edge cases, and an e2e test verifying the extractor correctly filters PENDING reviews. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

qodo-app-for-konflux-ci · 2026-06-02T19:12:04Z

Review Summary by Qodo

Fix PR review pagination and incremental sync sort order

🐞 Bug fix 🧪 Tests

Walkthroughs

Description

• Fixed sort order mismatch: changed PR query from CREATED_AT to UPDATED_AT DESC to align
  incremental sync stop condition with cursor-based early exit
• Added fetchRemainingReviews() function to paginate reviews beyond 100 per PR using GraphQL
  cursor pagination
• Applied review pagination in both collection passes: new PRs and OPEN PR refresh
• Added 11 unit tests covering pagination logic, stop conditions, and error handling
• Added e2e test verifying PENDING reviews are correctly filtered by extractor

Diagram

flowchart LR
  A["PR Query<br/>CREATED_AT DESC"] -->|"Changed to"| B["PR Query<br/>UPDATED_AT DESC"]
  B -->|"Aligns with"| C["Incremental Sync<br/>Stop Condition"]
  D["Reviews first: 100"] -->|"Truncates data"| E["Missing Reviews"]
  D -->|"Fixed by"| F["fetchRemainingReviews<br/>Cursor Pagination"]
  F -->|"Applied in"| G["New PRs Pass"]
  F -->|"Applied in"| H["OPEN PR Refresh"]
  G -->|"Produces"| I["Complete Review Data"]
  H -->|"Produces"| I

File Changes

1. backend/plugins/github_graphql/tasks/pr_collector.go 🐞 Bug fix +62/-2

Fix PR sort order and add review pagination

• Changed PR query orderBy from CREATED_AT to UPDATED_AT DESC to fix incremental sync stop
 condition alignment
• Added GraphqlQueryPrReviewsWrapper struct for paginated review queries
• Added PageInfo field to GraphqlQueryPr.Reviews to detect pagination
• Implemented fetchRemainingReviews() function to follow cursor pagination for reviews beyond 100
• Applied review pagination in both ResponseParsers (new PRs and OPEN PR refresh)

backend/plugins/github_graphql/tasks/pr_collector.go

2. backend/plugins/github_graphql/tasks/pr_collector_test.go 🧪 Tests +233/-0

Add comprehensive unit tests for PR collection

• Added 11 unit tests organized in three groups: pagination logic, stop condition, and review
 merging
• Tests cover single-page and multi-page pagination scenarios
• Tests verify nil PageInfo handling and query error propagation
• Tests validate incremental sync stop condition with various since timestamps
• Tests confirm review merging guard prevents unnecessary queries when no pagination needed

backend/plugins/github_graphql/tasks/pr_collector_test.go

3. backend/plugins/github_graphql/e2e/pr_extractor_test.go 🧪 Tests +57/-0

Add e2e test for PR review extraction

• Added e2e test TestGithubPrExtractor_ManyReviews to verify review extraction
• Test loads raw PR fixture with 5 reviews (4 non-PENDING + 1 PENDING)
• Verifies extractor correctly filters PENDING reviews and produces exactly 4 rows
• Uses CSV-based snapshot comparison for validation

backend/plugins/github_graphql/e2e/pr_extractor_test.go

qodo-app-for-konflux-ci · 2026-06-02T19:12:06Z

Code Review by Qodo

Looking for bugs?

Check back in a few minutes. An AI review agent is analyzing this pull request.

codecov-commenter · 2026-06-02T19:12:58Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 59.41%. Comparing base (f53a790) to head (ae14901).

Additional details and impacted files

@@           Coverage Diff           @@
##             main      #93   +/-   ##
=======================================
  Coverage   59.41%   59.41%           
=======================================
  Files          81       81           
  Lines        6105     6105           
=======================================
  Hits         3627     3627           
  Misses       2403     2403           
  Partials       75       75

Flag	Coverage Δ
unit-tests-go	`52.42% <ø> (ø)`
unit-tests-python	`74.36% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

kpiwko · 2026-06-08T08:44:52Z

this PR should be updated according to #92 to track a divergence from upstream.

rsoaresd

Great work 🚀 Just small questions:

rsoaresd · 2026-06-09T09:11:40Z

+}
+
+// fetchRemainingReviews pages through reviews beyond the first 100 for a PR.
+func fetchRemainingReviews(client graphqlQuerier, owner, repo string, prNumber int, startCursor string) ([]GraphqlQueryReview, errors.Error) {


Asking for learning purposes, is there any possibility of rate limit?

rsoaresd · 2026-06-09T09:14:42Z

+	assert.Equal(t, 1, mock.call)
+}
+
+func TestFetchRemainingReviews_TwoPages(t *testing.T) {


Should we check more than two pages? just to check multiple iterations

rsoaresd · 2026-06-09T09:17:17Z

 			Prs        []GraphqlQueryPr `graphql:"nodes"`
 			TotalCount graphql.Int
-		} `graphql:"pullRequests(first: $pageSize, after: $skipCursor, orderBy: {field: CREATED_AT, direction: DESC})"`
+		} `graphql:"pullRequests(first: $pageSize, after: $skipCursor, orderBy: {field: UPDATED_AT, direction: DESC})"`


Do we want to update an exiting test or add a test to ensure this behavior? Or isn't relevant? (asking for learning purposes)

flacatus approved these changes Jun 8, 2026

View reviewed changes

rsoaresd reviewed Jun 9, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(github_graphql): paginate reviews beyond 100 and fix incremental sync sort order (DPROD-1260)#93

fix(github_graphql): paginate reviews beyond 100 and fix incremental sync sort order (DPROD-1260)#93
kpiwko wants to merge 1 commit into
konflux-ci:mainfrom
kpiwko:fix/dprod-1260-pr-review-pagination

kpiwko commented Jun 2, 2026

Uh oh!

qodo-app-for-konflux-ci Bot commented Jun 2, 2026

Uh oh!

qodo-app-for-konflux-ci Bot commented Jun 2, 2026

Uh oh!

codecov-commenter commented Jun 2, 2026

Uh oh!

kpiwko commented Jun 8, 2026

Uh oh!

rsoaresd left a comment

Uh oh!

rsoaresd Jun 9, 2026

Uh oh!

rsoaresd Jun 9, 2026

Uh oh!

rsoaresd Jun 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

kpiwko commented Jun 2, 2026

Summary

Tests

Test plan

Uh oh!

qodo-app-for-konflux-ci Bot commented Jun 2, 2026

Review Summary by Qodo

Walkthroughs

File Changes

Uh oh!

qodo-app-for-konflux-ci Bot commented Jun 2, 2026

Code Review by Qodo

Looking for bugs?

Uh oh!

codecov-commenter commented Jun 2, 2026

Codecov Report

Uh oh!

kpiwko commented Jun 8, 2026

Uh oh!

rsoaresd left a comment

Choose a reason for hiding this comment

Uh oh!

rsoaresd Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

rsoaresd Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

rsoaresd Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants