Skip to content

Fix precision loss on large integers and bigint#833

Open
kvch wants to merge 1 commit into
mainfrom
fix/824-bigint-precision
Open

Fix precision loss on large integers and bigint#833
kvch wants to merge 1 commit into
mainfrom
fix/824-bigint-precision

Conversation

@kvch
Copy link
Copy Markdown
Collaborator

@kvch kvch commented Jun 2, 2026

Description

Preserve int64 precision on the WAL and snapshots.

Related Issue(s)

Type of Change

Please select the relevant option(s):

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 💥 Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • 📚 Documentation update
  • 🔧 Refactoring (no functional changes)
  • ⚡ Performance improvement
  • 🧪 Test coverage improvement
  • 🔨 Build/CI changes
  • 🧹 Code cleanup

Changes Made

A single primitive applied at both decode sites.

  • internal/json.UnmarshalUseInt64: sonic configured with UseInt64: true. Integer JSON numbers decode to int64 (full int64 range); non-integer numbers still decode to float64.
  • WAL path: pkg/wal/listener/postgres.Listener now uses UnmarshalUseInt64 on the wal2json payload.
  • Snapshot path: internal/postgres.registerTypesToConnMap overrides pgx's default JSONCodec and JSONBCodec with versions whose Unmarshal field is internal/json.UnmarshalUseInt64. This is set on every pgx connection (via pgxpool.Config.AfterConnect / Conn setup), so it covers every place pgx scans a json/jsonb value into any
  • Sequence handling: postgres_wal_dml_adapter.go / _bulk.go now accept int64 (and int/int32) in addition to float64 for sequence columns via a small toInt64 helper, since values coming out of the new decoder are int64.

The internal-json.Unmarshal callers elsewhere in the codebase are unchanged.

Testing

  • Unit tests added/updated
  • Integration tests added/updated
  • Manual testing performed
  • All existing tests pass

Checklist

  • Code follows project style guidelines
  • Self-review completed
  • Code is well-commented
  • Documentation updated where necessary

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Jun 2, 2026

Merging this branch changes the coverage (1 decrease, 1 increase)

Impacted Packages Coverage Δ 🤖
github.com/xataio/pgstream/internal/json 33.33% (+33.33%) 🌟
github.com/xataio/pgstream/internal/postgres 29.31% (ø)
github.com/xataio/pgstream/pkg/stream/integration 0.00% (ø)
github.com/xataio/pgstream/pkg/wal/listener/postgres 97.83% (ø)
github.com/xataio/pgstream/pkg/wal/processor/postgres 79.22% (-0.12%) 👎

Coverage by file

Changed files (no unit tests)

Changed File Coverage Δ Total Covered Missed 🤖
github.com/xataio/pgstream/internal/json/json.go 33.33% (+33.33%) 3 (+1) 1 (+1) 2 🌟
github.com/xataio/pgstream/internal/postgres/pg_utils.go 47.37% (ø) 114 54 60
github.com/xataio/pgstream/pkg/wal/listener/postgres/wal_pg_listener.go 97.83% (ø) 46 45 1
github.com/xataio/pgstream/pkg/wal/processor/postgres/postgres_wal_dml_adapter.go 93.68% (-0.96%) 174 (+6) 163 (+4) 11 (+2) 👎
github.com/xataio/pgstream/pkg/wal/processor/postgres/postgres_wal_dml_adapter_bulk.go 90.70% (-0.07%) 129 (-1) 117 (-1) 12 👎

Please note that the "Total", "Covered", and "Missed" counts above refer to code statements instead of lines of code. The value in brackets refers to the test coverage of that file in the old version of the code.

Changed unit test files

  • github.com/xataio/pgstream/internal/json/json_test.go
  • github.com/xataio/pgstream/pkg/stream/integration/pg_pg_integration_test.go
  • github.com/xataio/pgstream/pkg/stream/integration/snapshot_pg_integration_test.go
  • github.com/xataio/pgstream/pkg/wal/processor/postgres/postgres_wal_dml_adapter_test.go

@kvch kvch marked this pull request as ready for review June 2, 2026 13:45
@kvch kvch requested review from exekias and tsg June 3, 2026 14:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Issues with BigInt replication [Snapshots] Precision loss on json/jsonb large integers

1 participant