Serialize BigDecimal manually to match Arrow's Decimal128 format#2635
Merged
Conversation
Contributor
There was a problem hiding this comment.
⚠️ Performance Alert ⚠️
Possible performance regression was detected for benchmark 'Rust Benchmark'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 2.
| Benchmark suite | Current: 478d4c6 | Previous: 9823ef7 | Ratio |
|---|---|---|---|
lotr_graph/num_edges |
5 ns/iter (± 0) |
0 ns/iter (± 0) |
+∞ |
lotr_graph/num_nodes |
5 ns/iter (± 0) |
1 ns/iter (± 0) |
5 |
lotr_graph/iterate_exploded_edges |
603149 ns/iter (± 3628) |
285473 ns/iter (± 2920) |
2.11 |
lotr_graph/graph_latest |
3 ns/iter (± 0) |
0 ns/iter (± 0) |
+∞ |
lotr_graph_materialise/materialize |
8156928 ns/iter (± 64618) |
1564816 ns/iter (± 35303) |
5.21 |
lotr_graph_window_100/num_nodes |
14 ns/iter (± 0) |
5 ns/iter (± 0) |
2.80 |
lotr_graph_window_100/iterate_exploded_edges |
775785 ns/iter (± 1577) |
325242 ns/iter (± 847) |
2.39 |
lotr_graph_window_100_materialise/materialize |
8549448 ns/iter (± 35869) |
1669150 ns/iter (± 10700) |
5.12 |
lotr_graph_window_10/has_edge_existing |
211 ns/iter (± 5) |
97 ns/iter (± 11) |
2.18 |
lotr_graph_window_10/has_node_existing |
153 ns/iter (± 8) |
62 ns/iter (± 11) |
2.47 |
lotr_graph_window_10/iterate nodes |
33216 ns/iter (± 90) |
11339 ns/iter (± 40) |
2.93 |
lotr_graph_window_10/iterate edges |
99624 ns/iter (± 400) |
48684 ns/iter (± 211) |
2.05 |
lotr_graph_window_10/iterate_exploded_edges |
402719 ns/iter (± 1745) |
155788 ns/iter (± 1001) |
2.59 |
lotr_graph_window_10_materialise/materialize |
3679340 ns/iter (± 8390) |
971980 ns/iter (± 4278) |
3.79 |
lotr_graph_subgraph_10pc_materialise/materialize |
2068365 ns/iter (± 18520) |
334634 ns/iter (± 1287) |
6.18 |
lotr_graph_subgraph_10pc_windowed/has_node_existing |
158 ns/iter (± 8) |
62 ns/iter (± 14) |
2.55 |
lotr_graph_subgraph_10pc_windowed/iterate nodes |
5439 ns/iter (± 97) |
1365 ns/iter (± 3) |
3.98 |
lotr_graph_subgraph_10pc_windowed_materialise/materialize |
1257737 ns/iter (± 14141) |
230399 ns/iter (± 2617) |
5.46 |
lotr_graph_window_50_layered/num_edges_temporal |
149993 ns/iter (± 2450) |
70121 ns/iter (± 7586) |
2.14 |
lotr_graph_window_50_layered/has_edge_existing |
210 ns/iter (± 6) |
104 ns/iter (± 7) |
2.02 |
lotr_graph_window_50_layered/has_node_existing |
507 ns/iter (± 16) |
129 ns/iter (± 12) |
3.93 |
lotr_graph_window_50_layered/iterate nodes |
77449 ns/iter (± 239) |
19308 ns/iter (± 47) |
4.01 |
lotr_graph_window_50_layered/iterate edges |
200924 ns/iter (± 537) |
83616 ns/iter (± 1318) |
2.40 |
lotr_graph_window_50_layered/graph_latest |
77935 ns/iter (± 1497) |
36649 ns/iter (± 916) |
2.13 |
lotr_graph_window_50_layered_materialise/materialize |
32223075 ns/iter (± 154269) |
3488825 ns/iter (± 24948) |
9.24 |
lotr_graph_persistent_window_50_layered/num_edges_temporal |
591852 ns/iter (± 5325) |
192686 ns/iter (± 1569) |
3.07 |
lotr_graph_persistent_window_50_layered/has_edge_existing |
227 ns/iter (± 106) |
109 ns/iter (± 23) |
2.08 |
lotr_graph_persistent_window_50_layered/has_node_existing |
525 ns/iter (± 260) |
174 ns/iter (± 83) |
3.02 |
lotr_graph_persistent_window_50_layered/iterate nodes |
98797 ns/iter (± 346) |
35886 ns/iter (± 191) |
2.75 |
lotr_graph_persistent_window_50_layered/iterate edges |
184923 ns/iter (± 525) |
84161 ns/iter (± 596) |
2.20 |
lotr_graph_persistent_window_50_layered/iterate_exploded_edges |
4297713 ns/iter (± 87226) |
1659940 ns/iter (± 19402) |
2.59 |
lotr_graph_persistent_window_50_layered_materialise/materialize |
58397960 ns/iter (± 581105) |
5298035 ns/iter (± 147912) |
11.02 |
This comment was automatically generated by workflow using github-action-benchmark.
ljeub-pometry
approved these changes
Jun 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Serializes
Prop::Decimal, which internally holds aBigDecimal, to match theDecimal128format used in Arrow.Additionally, modifies
column_as_arrayto return aResult.Why are the changes needed?
parquet_tests::node_temp_props_decimalcurrently fails due toProp::Decimalbeing serialized incorrectly using scientific notation.serde_arrowexpects floats to be in standard notation.Furthermore, having
column_as_array(and its callers) return aResultallows prop serialization errors to bubble up properly. Currently for props that can't be serialized, errors are swallowed using.okand return empty columns.Does this PR introduce any user-facing change? If yes is this documented?
No.
How was this patch tested?
This patch fixes a failing test.
Are there any further changes required?
No.