Skip to content

feat: converter#2

Draft
soraxas wants to merge 20 commits into
rlamarche:mainfrom
soraxas:feat/converter
Draft

feat: converter#2
soraxas wants to merge 20 commits into
rlamarche:mainfrom
soraxas:feat/converter

Conversation

@soraxas

@soraxas soraxas commented Jul 3, 2026

Copy link
Copy Markdown

It's a potree converter (based off the javascript & c++ PotreeConverter) written w/ this rust crate. Might need to clean up some files that was touched along the way, and the cargo.toml


Running the PLY → Potree converter

Build

cargo build --release --features convert --bin ply_to_potree

Binary lands at target/release/ply_to_potree.

Run

./target/release/ply_to_potree <input.ply> <output_dir> [flags]

Writes metadata.json, hierarchy.bin, octree.bin into <output_dir>
(created if missing). Prints a one-line quality summary (node count, depth,
points per node, spacing) when done.

Flag Default Meaning
--name <s> input file stem pointcloud name in metadata
--projection <s> "" projection string in metadata
--scale <f> 0.001 quantization grid (meters); 1 mm
--max-points-per-node <n> 10000 split threshold, matches C++ PotreeConverter
--max-depth <n> 20 octree depth cap
--encoding <s> BROTLI BROTLI (SoA + brotli) or DEFAULT (raw AoS, Morton-sorted)

There is no --seed: sampling is fully deterministic (grid-based Poisson).

Library API: potree::convert::streaming::convert_ply_streaming(input, output, &ConvertPlyOptions { .. }) — same knobs, ConvertPlyOptions::default()
matches the table above except encoding: "DEFAULT".

Input support

  • PLY formats: ASCII, binary little-endian, binary big-endian.
  • x/y/z (any scalar type) → quantized int32 positions.
  • red/green/blue (or r/g/b) → uint16 rgb attribute.
  • Every other property passes through as a typed extra attribute:
    • native scalar type preserved (uchar→uint8, ushort→uint16, float, …),
    • name normalized: CloudCompare scalar_X prefix stripped, LAS-style names
      canonicalized (scalar_Intensityintensity, return_numberreturn number); unknown names keep the stripped form, collisions fall back to
      the original,
    • metadata min/max are the observed data range (scanned in pass 1).

Output conventions (aligned with C++ PotreeConverter 2.x)

  • boundingBox is cubed to the largest extent; the position attribute
    keeps the tight range snapped to the quantization grid.
  • offset = data min; full f64 precision in metadata.
  • Hierarchy chunked every 4 levels (stepSize: 4), 22-byte node records.
  • Streaming: two passes over the PLY + temp bucket files in $TMPDIR, so
    memory stays flat (tested on a 27M-point / 595 MB PLY).

Inspecting output

examples/dump_potree.rs decodes a Potree directory through this crate's own
reader and prints every node and point (positions + rgb) as text:

cargo build --release --features fs,tokio_dev --example dump_potree
./target/release/examples/dump_potree <potree_dir>   # 'node …' lines, then 'point x y z [r g b]'

Works on this converter's output and on C++ PotreeConverter output
(DEFAULT/UNCOMPRESSED/BROTLI encodings).

Tests

cargo test --features convert,tokio_dev            # whole crate incl. converter suites
cargo test --features convert,tokio_dev --test streaming_roundtrip

streaming_roundtrip converts corner points and reloads them through the
reader — the end-to-end safety net after touching either side.

Reference C++ PotreeConverter (for comparison)

PotreeConverter. Input is LAS/LAZ only (convert PLY→LAS
first, e.g. via numpy: read PLY dtype, round(xyz/0.001) → int32 LAS 1.2
records):

cmake -S PotreeConverter -B build -DCMAKE_POLICY_VERSION_MINIMUM=3.5
cmake --build build -j 8
./build/PotreeConverter input.las -o outdir -m poisson --encoding UNCOMPRESSED

Comparison methodology that was used to validate parity (rust vs C++ on the
same points): run both, dump_potree both outputs, then match decoded
positions against the source at ±1 quantization-unit tolerance. Expected
result: both lossless (every input point recovered exactly once); rust matches
the source grid exactly for ~50–77 % of points, the C++ output systematically
drifts ~1 unit (its re-quantization bias) — that difference is cosmetic.

Signed-off-by: Tin Lai <tin@tinyiu.com>
Signed-off-by: Tin Lai <tin@tinyiu.com>
Signed-off-by: Tin Lai <tin@tinyiu.com>
Signed-off-by: Tin Lai <tin@tinyiu.com>
Signed-off-by: Tin Lai <tin@tinyiu.com>
Signed-off-by: Tin Lai <tin@tinyiu.com>
Signed-off-by: Tin Lai <tin@tinyiu.com>
Signed-off-by: Tin Lai <tin@tinyiu.com>
Signed-off-by: Tin Lai <tin@tinyiu.com>
Resolve conflicts:
- take upstream's asset/blocking/parse refactor (resource/ deleted)
- keep converter feature: move resource/buffer.rs -> convert/buffer.rs
  (port to PotreeAsset API follows in next commits)
- merge Cargo.toml deps/features from both sides
The upstream asset refactor (2c43229) deleted the resource/ abstraction the
converter's in-memory backend plugged into. Port:

- convert/buffer.rs: replace BufferClient (url-keyed ResourceClient) with
  PotreeBufferAsset, a dataset-shaped PotreeAsset over in-memory
  metadata/hierarchy/octree buffers
- PointCloud::from_buffers: reimplement on PotreeBufferAsset (drops the
  obsolete name/url parameter), mirroring the per-asset constructor style
- convert feature now requires fs (the streaming converter writes via std::fs)
- tests/example: Hierarchy::from_url(file://..., ResourceLoader) ->
  Hierarchy::from_path

All 26 tests pass (streaming unit tests + hierarchy/roundtrip/smoke suites,
incl. BROTLI roundtrip through the new fs asset reader).
Exact pins (=0.2.106 / =0.4.56) cannot co-resolve in workspaces that already
lock a newer wasm-bindgen (e.g. bevy-based consumers locking 0.2.117).
Base URLs like '<base>/pcd?v=<cache-key>' previously produced
'<base>/pcd?v=<key>/metadata.json'. Parse the URL and insert the asset
filename into the path component, preserving the query/fragment:
'<base>/pcd/metadata.json?v=<key>'. Applies to both the async and
blocking http assets.
- cube the root bounding box to the largest extent (both streaming and
  in-memory paths); position attribute keeps the tight data range
- accept "UNCOMPRESSED" metadata encoding in the reader (written by the
  C++ converter; treated identically to "DEFAULT")
- default --max-points-per-node to 10k to match the reference
- add dump_potree example for diffing converter outputs (positions + rgb)

Verified against PotreeConverter on 50k synthetic points plus two real
scans (1.7M splat, 27.1M rgb): both lossless, cross-decodable, and now
byte-compatible on bbox/tree-shape conventions.
- extra attributes now record the observed data min/max from the bounds
  pass instead of type-range/[0,1] placeholders (reference behaviour;
  viewers use these to auto-scale intensity gradients)
- normalize CloudCompare 'scalar_<Name>' and LAS-style underscore names
  to canonical Potree attribute names (scalar_Intensity -> intensity,
  return_number -> 'return number'); unknown fields keep the stripped
  name, collisions fall back to the original
- snap the metadata position attribute min/max to the quantization grid
  like the reference converter
- fold convert_ply_streaming args into ConvertPlyOptions and
  build_metadata_json args into MetadataParams; drop the never-used seed
  plumbing and the dead max_points_per_node param on sample_tree
- clippy: &mut Vec -> slice params, builder struct literal; converter
  code is now clippy-clean
- PointBuffer::get used 'len() > (i+1)*stride', so the last point of
  every buffer returned None; iter() was unaffected. Now '>='.
- Metadata offset/scale/boundingBox (and attribute min/max) are parsed
  as f64 instead of f32: georeferenced clouds (UTM-scale coordinates)
  lose decimeter-level precision in f32. Position decoding now runs in
  f64 and narrows to f32 only for the small node-relative result, at
  the Aabb/PointBuffer render boundary. AttributeFormat::parse returns
  f64 so int64/double attributes no longer truncate before
  normalization.

Verified against the official potree.org heidentor dataset (25.8M
points, BROTLI): hierarchy and node points load correctly.
Real scanner data (dense, duplicate-heavy after quantization) produced
whole tree levels of num_points=0/byte_size=0 nodes: once a level's
Poisson spacing fell below the quantization scale, the acceptance test
became vacuous and every internal node vacuumed its children's entire
distinct payload, stranding only duplicate points in deep leaves. On a
27M-point scan this left all 225 internal nodes at levels 7-8 empty plus
4163 empty leaves. The reference Potree viewer tolerates such nodes but
warns ('loaded node with 0 bytes'), and the reference converter's own
pristine datasets contain none - clean output shouldn't create them.

Three fixes:
- split_tree: resolution guard - never subdivide once child Poisson
  spacing would drop below the quantization scale (those levels cannot
  represent additional detail)
- split_tree: create children lazily, only for octants that actually
  receive points (was: all 8 eagerly, leaving empty-leaf records)
- prune_empty_nodes: drop nodes whose payload was entirely absorbed by
  an ancestor's sample before writing the hierarchy; the octree writer
  now picks a node's payload by where it lives (temp bucket vs sample)
  so internals pruned down to leaves keep their points

Also folds split_tree args into SplitConfig (clippy too_many_arguments).

Verified on the 27M-point scan: 0 empty real nodes (was 4388), all
27,132,011 points present and decodable; new regression test converts
dense duplicated data and asserts no empty records.

Signed-off-by: Tin Lai <tin@tinyiu.com>
…-stop

The resolution guard from the previous commit avoided empty nodes by not
subdividing below the quantization grid, but that concentrated dense
regions into huge leaves (403k pts / 10.5 MB): sub-resolution levels
can't add positional detail, yet they matter for progressive density
streaming when zoomed in.

Replace it with what the C++ converter effectively does:

- sample_tree: when a level's Poisson spacing is below the quantization
  scale (acceptance vacuous), cap the sample at max_points_per_node so
  parents can't vacuum entire child payloads; levels with real rejection
  keep the natural, reference-matching yield
- sample_tree: never fully drain a child that has children of its own -
  flip one accepted record back, since an empty internal can't be pruned
  while its subtree still holds points
- split_tree: drop the spacing guard; instead stop when splitting is
  futile (all points land in one octant, i.e. duplicate-heavy buckets)

On the 27M-point scan the output now matches the C++ reference level
for level (depth 9, 11,276 vs 12,023 nodes, per-level counts within
~10%, max node 54k vs 67k) with zero empty real nodes (C++ emits 1,137)
and all points present and decodable.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant