Improve performance of new transform for large tables - #68

Merged

brycekbargar merged 18 commits into

library-data-platform:release-v4.0.0from

Five-Colleges-Incorporated:performance-testing

Mar 10, 2026

Commits on Mar 9, 2026

Refactor metadata collection to be per property
brycekbargar
committed
1d7d684
Monitor progress of transformation
brycekbargar
committed
1fee373
Optimize the definitions of postgres json operators
brycekbargar
committed
b37dd1f
Simplify checks for specific data types
brycekbargar
committed
0fe5986
Allow postgres to short circuit data type checks
brycekbargar
committed
f6a5978
Sample large tables for column metadata
brycekbargar
committed
4706f98
Split progress bars between scanning and transforming
brycekbargar
committed
b07ec9f
Remove slow jextract function in favor of more native functionality
brycekbargar
committed
c6081bf
Find all possible keys in json
brycekbargar
committed
8d9a61c
Remove sampling
brycekbargar
committed
b63f7e2
Actually drop the tables in the prefix
brycekbargar
committed
c23c6af
Dont carry over jsonb column
brycekbargar
committed
7d04ee3
Simplify the query to get property names
brycekbargar
committed
f619eb6
Ignore empty source tables
brycekbargar
committed
00775d1
Better display of iterators during transform
brycekbargar
committed
523bd2c

Commits on Mar 10, 2026

Actually fix the indexing
brycekbargar
committed
76d92b2
Dont keep tqdm progress bars around
brycekbargar
committed
9936f85
Be more specific about when to include the object column name
brycekbargar
committed
3fa7cc9