Skip to content

Serialize BigDecimal manually to match Arrow's Decimal128 format#2635

Merged
ljeub-pometry merged 6 commits into
db_v4from
fix-props-decimal
Jun 5, 2026
Merged

Serialize BigDecimal manually to match Arrow's Decimal128 format#2635
ljeub-pometry merged 6 commits into
db_v4from
fix-props-decimal

Conversation

@fabubaker
Copy link
Copy Markdown
Contributor

@fabubaker fabubaker commented Jun 4, 2026

What changes were proposed in this pull request?

Serializes Prop::Decimal, which internally holds a BigDecimal, to match the Decimal128 format used in Arrow.

Additionally, modifies column_as_array to return a Result.

Why are the changes needed?

parquet_tests::node_temp_props_decimal currently fails due to Prop::Decimal being serialized incorrectly using scientific notation. serde_arrow expects floats to be in standard notation.

Furthermore, having column_as_array (and its callers) return a Result allows prop serialization errors to bubble up properly. Currently for props that can't be serialized, errors are swallowed using .ok and return empty columns.

Does this PR introduce any user-facing change? If yes is this documented?

No.

How was this patch tested?

This patch fixes a failing test.

Are there any further changes required?

No.

Copy link
Copy Markdown
Contributor

@github-actions github-actions Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark 'Rust Benchmark'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 2.

Benchmark suite Current: 478d4c6 Previous: 9823ef7 Ratio
lotr_graph/num_edges 5 ns/iter (± 0) 0 ns/iter (± 0) +∞
lotr_graph/num_nodes 5 ns/iter (± 0) 1 ns/iter (± 0) 5
lotr_graph/iterate_exploded_edges 603149 ns/iter (± 3628) 285473 ns/iter (± 2920) 2.11
lotr_graph/graph_latest 3 ns/iter (± 0) 0 ns/iter (± 0) +∞
lotr_graph_materialise/materialize 8156928 ns/iter (± 64618) 1564816 ns/iter (± 35303) 5.21
lotr_graph_window_100/num_nodes 14 ns/iter (± 0) 5 ns/iter (± 0) 2.80
lotr_graph_window_100/iterate_exploded_edges 775785 ns/iter (± 1577) 325242 ns/iter (± 847) 2.39
lotr_graph_window_100_materialise/materialize 8549448 ns/iter (± 35869) 1669150 ns/iter (± 10700) 5.12
lotr_graph_window_10/has_edge_existing 211 ns/iter (± 5) 97 ns/iter (± 11) 2.18
lotr_graph_window_10/has_node_existing 153 ns/iter (± 8) 62 ns/iter (± 11) 2.47
lotr_graph_window_10/iterate nodes 33216 ns/iter (± 90) 11339 ns/iter (± 40) 2.93
lotr_graph_window_10/iterate edges 99624 ns/iter (± 400) 48684 ns/iter (± 211) 2.05
lotr_graph_window_10/iterate_exploded_edges 402719 ns/iter (± 1745) 155788 ns/iter (± 1001) 2.59
lotr_graph_window_10_materialise/materialize 3679340 ns/iter (± 8390) 971980 ns/iter (± 4278) 3.79
lotr_graph_subgraph_10pc_materialise/materialize 2068365 ns/iter (± 18520) 334634 ns/iter (± 1287) 6.18
lotr_graph_subgraph_10pc_windowed/has_node_existing 158 ns/iter (± 8) 62 ns/iter (± 14) 2.55
lotr_graph_subgraph_10pc_windowed/iterate nodes 5439 ns/iter (± 97) 1365 ns/iter (± 3) 3.98
lotr_graph_subgraph_10pc_windowed_materialise/materialize 1257737 ns/iter (± 14141) 230399 ns/iter (± 2617) 5.46
lotr_graph_window_50_layered/num_edges_temporal 149993 ns/iter (± 2450) 70121 ns/iter (± 7586) 2.14
lotr_graph_window_50_layered/has_edge_existing 210 ns/iter (± 6) 104 ns/iter (± 7) 2.02
lotr_graph_window_50_layered/has_node_existing 507 ns/iter (± 16) 129 ns/iter (± 12) 3.93
lotr_graph_window_50_layered/iterate nodes 77449 ns/iter (± 239) 19308 ns/iter (± 47) 4.01
lotr_graph_window_50_layered/iterate edges 200924 ns/iter (± 537) 83616 ns/iter (± 1318) 2.40
lotr_graph_window_50_layered/graph_latest 77935 ns/iter (± 1497) 36649 ns/iter (± 916) 2.13
lotr_graph_window_50_layered_materialise/materialize 32223075 ns/iter (± 154269) 3488825 ns/iter (± 24948) 9.24
lotr_graph_persistent_window_50_layered/num_edges_temporal 591852 ns/iter (± 5325) 192686 ns/iter (± 1569) 3.07
lotr_graph_persistent_window_50_layered/has_edge_existing 227 ns/iter (± 106) 109 ns/iter (± 23) 2.08
lotr_graph_persistent_window_50_layered/has_node_existing 525 ns/iter (± 260) 174 ns/iter (± 83) 3.02
lotr_graph_persistent_window_50_layered/iterate nodes 98797 ns/iter (± 346) 35886 ns/iter (± 191) 2.75
lotr_graph_persistent_window_50_layered/iterate edges 184923 ns/iter (± 525) 84161 ns/iter (± 596) 2.20
lotr_graph_persistent_window_50_layered/iterate_exploded_edges 4297713 ns/iter (± 87226) 1659940 ns/iter (± 19402) 2.59
lotr_graph_persistent_window_50_layered_materialise/materialize 58397960 ns/iter (± 581105) 5298035 ns/iter (± 147912) 11.02

This comment was automatically generated by workflow using github-action-benchmark.

@ljeub-pometry ljeub-pometry merged commit 4919ec4 into db_v4 Jun 5, 2026
34 checks passed
@ljeub-pometry ljeub-pometry deleted the fix-props-decimal branch June 5, 2026 06:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants