feat: converter#2
Draft
soraxas wants to merge 20 commits into
Draft
Conversation
Signed-off-by: Tin Lai <tin@tinyiu.com>
Signed-off-by: Tin Lai <tin@tinyiu.com>
Signed-off-by: Tin Lai <tin@tinyiu.com>
Signed-off-by: Tin Lai <tin@tinyiu.com>
Signed-off-by: Tin Lai <tin@tinyiu.com>
Signed-off-by: Tin Lai <tin@tinyiu.com>
Signed-off-by: Tin Lai <tin@tinyiu.com>
Signed-off-by: Tin Lai <tin@tinyiu.com>
Signed-off-by: Tin Lai <tin@tinyiu.com>
Resolve conflicts: - take upstream's asset/blocking/parse refactor (resource/ deleted) - keep converter feature: move resource/buffer.rs -> convert/buffer.rs (port to PotreeAsset API follows in next commits) - merge Cargo.toml deps/features from both sides
The upstream asset refactor (2c43229) deleted the resource/ abstraction the converter's in-memory backend plugged into. Port: - convert/buffer.rs: replace BufferClient (url-keyed ResourceClient) with PotreeBufferAsset, a dataset-shaped PotreeAsset over in-memory metadata/hierarchy/octree buffers - PointCloud::from_buffers: reimplement on PotreeBufferAsset (drops the obsolete name/url parameter), mirroring the per-asset constructor style - convert feature now requires fs (the streaming converter writes via std::fs) - tests/example: Hierarchy::from_url(file://..., ResourceLoader) -> Hierarchy::from_path All 26 tests pass (streaming unit tests + hierarchy/roundtrip/smoke suites, incl. BROTLI roundtrip through the new fs asset reader).
Exact pins (=0.2.106 / =0.4.56) cannot co-resolve in workspaces that already lock a newer wasm-bindgen (e.g. bevy-based consumers locking 0.2.117).
Base URLs like '<base>/pcd?v=<cache-key>' previously produced '<base>/pcd?v=<key>/metadata.json'. Parse the URL and insert the asset filename into the path component, preserving the query/fragment: '<base>/pcd/metadata.json?v=<key>'. Applies to both the async and blocking http assets.
- cube the root bounding box to the largest extent (both streaming and in-memory paths); position attribute keeps the tight data range - accept "UNCOMPRESSED" metadata encoding in the reader (written by the C++ converter; treated identically to "DEFAULT") - default --max-points-per-node to 10k to match the reference - add dump_potree example for diffing converter outputs (positions + rgb) Verified against PotreeConverter on 50k synthetic points plus two real scans (1.7M splat, 27.1M rgb): both lossless, cross-decodable, and now byte-compatible on bbox/tree-shape conventions.
- extra attributes now record the observed data min/max from the bounds pass instead of type-range/[0,1] placeholders (reference behaviour; viewers use these to auto-scale intensity gradients) - normalize CloudCompare 'scalar_<Name>' and LAS-style underscore names to canonical Potree attribute names (scalar_Intensity -> intensity, return_number -> 'return number'); unknown fields keep the stripped name, collisions fall back to the original - snap the metadata position attribute min/max to the quantization grid like the reference converter - fold convert_ply_streaming args into ConvertPlyOptions and build_metadata_json args into MetadataParams; drop the never-used seed plumbing and the dead max_points_per_node param on sample_tree - clippy: &mut Vec -> slice params, builder struct literal; converter code is now clippy-clean
- PointBuffer::get used 'len() > (i+1)*stride', so the last point of every buffer returned None; iter() was unaffected. Now '>='. - Metadata offset/scale/boundingBox (and attribute min/max) are parsed as f64 instead of f32: georeferenced clouds (UTM-scale coordinates) lose decimeter-level precision in f32. Position decoding now runs in f64 and narrows to f32 only for the small node-relative result, at the Aabb/PointBuffer render boundary. AttributeFormat::parse returns f64 so int64/double attributes no longer truncate before normalization. Verified against the official potree.org heidentor dataset (25.8M points, BROTLI): hierarchy and node points load correctly.
Real scanner data (dense, duplicate-heavy after quantization) produced
whole tree levels of num_points=0/byte_size=0 nodes: once a level's
Poisson spacing fell below the quantization scale, the acceptance test
became vacuous and every internal node vacuumed its children's entire
distinct payload, stranding only duplicate points in deep leaves. On a
27M-point scan this left all 225 internal nodes at levels 7-8 empty plus
4163 empty leaves. The reference Potree viewer tolerates such nodes but
warns ('loaded node with 0 bytes'), and the reference converter's own
pristine datasets contain none - clean output shouldn't create them.
Three fixes:
- split_tree: resolution guard - never subdivide once child Poisson
spacing would drop below the quantization scale (those levels cannot
represent additional detail)
- split_tree: create children lazily, only for octants that actually
receive points (was: all 8 eagerly, leaving empty-leaf records)
- prune_empty_nodes: drop nodes whose payload was entirely absorbed by
an ancestor's sample before writing the hierarchy; the octree writer
now picks a node's payload by where it lives (temp bucket vs sample)
so internals pruned down to leaves keep their points
Also folds split_tree args into SplitConfig (clippy too_many_arguments).
Verified on the 27M-point scan: 0 empty real nodes (was 4388), all
27,132,011 points present and decodable; new regression test converts
dense duplicated data and asserts no empty records.
Signed-off-by: Tin Lai <tin@tinyiu.com>
…-stop The resolution guard from the previous commit avoided empty nodes by not subdividing below the quantization grid, but that concentrated dense regions into huge leaves (403k pts / 10.5 MB): sub-resolution levels can't add positional detail, yet they matter for progressive density streaming when zoomed in. Replace it with what the C++ converter effectively does: - sample_tree: when a level's Poisson spacing is below the quantization scale (acceptance vacuous), cap the sample at max_points_per_node so parents can't vacuum entire child payloads; levels with real rejection keep the natural, reference-matching yield - sample_tree: never fully drain a child that has children of its own - flip one accepted record back, since an empty internal can't be pruned while its subtree still holds points - split_tree: drop the spacing guard; instead stop when splitting is futile (all points land in one octant, i.e. duplicate-heavy buckets) On the 27M-point scan the output now matches the C++ reference level for level (depth 9, 11,276 vs 12,023 nodes, per-level counts within ~10%, max node 54k vs 67k) with zero empty real nodes (C++ emits 1,137) and all points present and decodable.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
It's a potree converter (based off the javascript & c++ PotreeConverter) written w/ this rust crate. Might need to clean up some files that was touched along the way, and the cargo.toml
Running the PLY → Potree converter
Build
Binary lands at
target/release/ply_to_potree.Run
Writes
metadata.json,hierarchy.bin,octree.bininto<output_dir>(created if missing). Prints a one-line quality summary (node count, depth,
points per node, spacing) when done.
--name <s>--projection <s>""--scale <f>0.001--max-points-per-node <n>10000--max-depth <n>20--encoding <s>BROTLIBROTLI(SoA + brotli) orDEFAULT(raw AoS, Morton-sorted)There is no
--seed: sampling is fully deterministic (grid-based Poisson).Library API:
potree::convert::streaming::convert_ply_streaming(input, output, &ConvertPlyOptions { .. })— same knobs,ConvertPlyOptions::default()matches the table above except
encoding: "DEFAULT".Input support
x/y/z(any scalar type) → quantized int32 positions.red/green/blue(orr/g/b) → uint16rgbattribute.scalar_Xprefix stripped, LAS-style namescanonicalized (
scalar_Intensity→intensity,return_number→return number); unknown names keep the stripped form, collisions fall back tothe original,
min/maxare the observed data range (scanned in pass 1).Output conventions (aligned with C++ PotreeConverter 2.x)
boundingBoxis cubed to the largest extent; the position attributekeeps the tight range snapped to the quantization grid.
stepSize: 4), 22-byte node records.$TMPDIR, somemory stays flat (tested on a 27M-point / 595 MB PLY).
Inspecting output
examples/dump_potree.rsdecodes a Potree directory through this crate's ownreader and prints every node and point (positions + rgb) as text:
Works on this converter's output and on C++ PotreeConverter output
(
DEFAULT/UNCOMPRESSED/BROTLIencodings).Tests
streaming_roundtripconverts corner points and reloads them through thereader — the end-to-end safety net after touching either side.
Reference C++ PotreeConverter (for comparison)
PotreeConverter. Input is LAS/LAZ only (convert PLY→LASfirst, e.g. via numpy: read PLY dtype,
round(xyz/0.001)→ int32 LAS 1.2records):
Comparison methodology that was used to validate parity (rust vs C++ on the
same points): run both,
dump_potreeboth outputs, then match decodedpositions against the source at ±1 quantization-unit tolerance. Expected
result: both lossless (every input point recovered exactly once); rust matches
the source grid exactly for ~50–77 % of points, the C++ output systematically
drifts ~1 unit (its re-quantization bias) — that difference is cosmetic.