Proposal: Extensible Point Attribute System for GLIM

[TokyoWarfare]

Proposal: Extensible Point Attribute System for GLIM

Problem Statement

Currently, adding new point attributes to GLIM requires extensive modifications across the entire codebase:

• Intensity support required changes to preprocessing, voxelization, CUDA kernels, serialization, and the viewer
• Normals and covariances are computed during odometry but lost downstream - they don't propagate to submapping or global mapping
• Timestamps cannot be preserved through the pipeline, making temporal analysis impossible
• Scanner ID (for multi-LiDAR setups) has no way to be stored or propagated
• RGB color support would require another major refactoring effort

Each new attribute requires weeks of work touching multiple files. This doesn't scale.

Root Cause

The current architecture tightly couples point geometry (x,y,z) with the processing pipeline:

While GTSAM/Eigen only care about geometry, there's no mechanism to carry additional attributes alongside the geometric data.

Proposed Solution:

A two-tier approach balancing performance and extensibility:

class PointCloud {
    // Core geometry (unchanged, GTSAM/Eigen compatible)
    std::vector<Eigen::Vector3f> points;
    
    // Tier 1: Standard attributes (direct access, zero overhead)
    struct StandardAttributes {
        std::optional<std::vector<float>> intensity;
        std::optional<std::vector<double>> timestamp;
        std::optional<std::vector<Eigen::Vector3f>> normals;
        std::optional<std::vector<uint8_t>> scanner_id;
        std::optional<std::vector<Eigen::Vector3f>> colors;  // RGB

may be:
        std::optional<std::vector<Eigen::Matrix3f>> covariances;

    } attributes;
    
    // Tier 2: Custom attributes (extensible, for advanced/experimental use)
    std::unordered_map<std::string, std::shared_ptr<AttributeBase>> custom_attributes;
    
    // Invariant: all attribute vectors must have size() == points.size()
    // or be empty (not present)
};

Design Principles

1. Non-invasive: points remains untouched - GTSAM and Eigen see exactly what they see today
2. Zero overhead when unused: Empty std::optional has minimal memory footprint
3. Fast path for common cases: Direct member access for standard attributes
4. Extensible: Custom attributes via type-erased map for edge cases
5. Propagation by default: All operations (filtering, voxelization, transformation) automatically handle attributes

How Attributes Flow Through the Pipeline

// 1. INPUT - From sensor
PointCloud cloud;
cloud.points = {p1, p2, p3, ...};
cloud.attributes.intensity = {0.5f, 0.8f, 0.3f, ...};
cloud.attributes.timestamp = {t1, t2, t3, ...};

// 2. PREPROCESSING - Downsampling
PointCloud downsampled = voxel_grid_filter(cloud);
// Automatically aggregates attributes per voxel:
//   - intensity: averaged
//   - timestamp: earliest/latest (configurable)
//   - others: averaged

// 3. NORMAL/COVARIANCE ESTIMATION  
estimate_normals(downsampled);
// Adds to attributes:
downsampled.attributes.normals = computed_normals;
downsampled.attributes.covariances = computed_covs;

// 4. ODOMETRY - GTSAM sees only geometry
auto poses = gtsam_optimize(downsampled.points);  
// Attributes unaffected, still there

// 5. SUBMAPPING - Accumulation
SubMap submap;
submap.cloud = merge_clouds({scan1, scan2, scan3});
// merge_clouds() combines attributes intelligently

// 6. VOXELIZATION - Downsampling again
auto voxelized = voxelize(submap.cloud);
// Attributes aggregated again (normals averaged & normalized, etc.)

// 7. GLOBAL MAPPING - Final optimization
// GTSAM optimizes geometry, attributes transform along

// 8. SERIALIZATION - Save everything
// File format v2 includes attribute section

// 9. OFFLINE EDITOR - Full access
auto loaded = load_map("map.glim");
viewer.colorize_by_intensity(loaded.attributes.intensity);
viewer.show_normals(loaded.attributes.normals);
viewer.filter_by_time(loaded.attributes.timestamp);

Attribute Aggregation

When downsampling/voxelization occurs, attributes could be aggregated appropriately:

class VoxelGrid {
    PointCloud downsample(const PointCloud& input, float voxel_size) {
        PointCloud output;
        // ... group points into voxels ...
        
        for (const auto& [voxel_key, point_indices] : voxel_map) {
            // Average geometry (unchanged)
            output.points.push_back(average_points(input.points, point_indices));
            
            // Aggregate attributes
            if (input.attributes.intensity) {
                float avg_intensity = average(input.attributes.intensity, point_indices);
                output.attributes.intensity->push_back(avg_intensity);
            }
            
            if (input.attributes.timestamp) {
                double earliest = min(input.attributes.timestamp, point_indices);
                output.attributes.timestamp->push_back(earliest);
            }
            
            if (input.attributes.normals) {
                Vector3f avg_normal = average_and_normalize(input.attributes.normals, point_indices);
                output.attributes.normals->push_back(avg_normal);
            }
            
            if (input.attributes.covariances) {
                Matrix3f combined_cov = weighted_sum(input.attributes.covariances, point_indices);
                output.attributes.covariances->push_back(combined_cov);
            }
            
            if (input.attributes.scanner_id) {
                uint8_t mode_id = most_common(input.attributes.scanner_id, point_indices);
                output.attributes.scanner_id->push_back(mode_id);
            }
        }
        
        return output;
    }
};

File Format Extension

Backward-compatible binary format:

GLIM Map File v2
├─ Header
│  ├─ magic: "GLIM"
│  ├─ version: 2
│  └─ num_points: N
├─ Geometry Section (v1 compatible)
│  └─ points: Vector3f[N]
│  └─ may be current intensity support should be retained.
└─ Attributes Section (new in v2)
   ├─ num_attributes: M
   └─ For each attribute:
      ├─ name: string (e.g., "intensity")
      ├─ type: enum (float, double, Vector3f, Matrix3f, etc.)
      ├─ size: N (must match num_points)
      └─ data: raw bytes

Old software (v1) reads geometry + intensity .
New (v2+) reads everything.

Benefits

For Users

• Temporal analysis: Filter/visualize point clouds by timestamp
• Multi-scanner support: Track which LiDAR captured each point
• Better visualization: Access to normals, intensity, color in offline editor, mesh point clouds without inverted normals.
• Richer maps: Preserve all sensor data for downstream analysis

For Developers

• Extensibility: New standard attribute = add 1 line to StandardAttributes
• Custom attributes: Use custom_attributes map without touching core code
• Maintainability: Attribute handling is centralized, not scattered across 30 files
• Testing: Easy to verify attributes propagate correctly

For Performance

• Zero overhead if unused: Empty std::optional is free
• Cache-friendly: Standard attributes are contiguous vectors
• CUDA compatible: Can still upload points array directly to GPU
• No virtual calls: Direct member access for standard attributes

Compatibility Guarantee

• Zero breaking changes: Existing code continues to work
• Opt-in: Attributes are std::optional, code that ignores them is unaffected
• File format: v1 maps load correctly (attributes empty)
• Performance: No regression for current users

Future Possibilities

Once this infrastructure exists:

• LiDAR-Visual-Inertial: Store image features as custom attributes
• Semantic mapping: Per-point class labels
• Quality metrics: Per-point confidence scores
• Multi-session mapping: Session ID per point
• Deformation tracking: Temporal consistency analysis

Offer

If GPU offload, final track export with timestamped coordinates, and this
requested feature were considered to be added, I would be willing to:

• Provide financial support to help prioritize these features
• Provide hardware support to avoid export restrictions for dual-use
  products affecting Hesai & Livox, or provide high-end GNSS/INS like NovAtel

This proposal aims to future-proof GLIM's architecture while maintaining its
performance and simplicity. The other features aim to enable GLIM for real-world
use cases like amateur mapping projects.

I believe this feature addresses a fundamental scalability issue in a pragmatic,
non-invasive way.

Looking forward to your feedback and thoughts. If you would like to explore this collaboration details further we would proceed privatelly yet all the code would remain public.

[themightyoarfish]

Looks AI-generated, did the LLM also provide a compiling implementation of the proposed changes?

[TokyoWarfare]

Not sure what more to add other than the explained. Adding extra attribute is painfull. I started trying to keep the normals from A to B and required way too much changes for a feature already computed just to be kept alive.

I understand now why Intensity took so long to be added. Adding expandable attribute list to the point data structure would massively help IMO.

A fork already implemented time, intensity and I believe normals too, sadly they did not PR and probably there is no way to cleanly merge at this point.

I've a couple of pending pulls pending on a camera driver, I polish some stuff on my MMS rig and I get back to this, which is probably beyond my skills tbh, but will try. THat is why I'm open to provide the AIST department with hardware that is now hard to get in Japan. I plan to do a data capture run this year and after that I'll have a top of the line Pandar in excess I wouldn´t mind let go in exchange for the pipeline be upgraded to a more production ready state. That includes, chagnes mentioned in this thread, GPU offload be added to the offline viewer so that the big SLAM can be exported, unlike now that it can´t be loaded nor exported and a trivial upgrade that exports the final slam trajectory mated with a timetable so that off ROS cameras or even ROS camera frames can be positiooned along the SLAM path just by using the time variable. Hence do coloring, 3DGS in a thirdparty package.

Edit: I just recalled return number would aswell nice to have on the standarde features.

[TokyoWarfare]

Helo, today I had a moment to finally look into this.

I did setup Claude Code for the first time and gave it a spin, it seems like my proposal is already there to some extent!

_** Step 1 of your spec is already implemented. Both PointCloud and PointCloudCPU already have the aux_attributes system:

• PointCloud (base): std::unordered_map<std::string, std::pair<size_t, void*>> aux_attributes + typed getter
  aux_attribute(name)
• PointCloudCPU (owning): std::unordered_map<std::string, std::shared_ptr> aux_attributes_storage + typed setter
  add_aux_attribute(name, values)

The design pattern matches the existing hardcoded fields: PointCloud holds raw pointers, PointCloudCPU owns the storage
via shared_ptr (the same trick used for times_storage, points_storage, etc.).

Serialization is also already done

Step 4 of your spec is also already implemented. Both save() and save_compact() in point_cloud.cpp iterate over
aux_attributes and write them as aux_.bin. Loading in PointCloudCPU::load() uses a regex to find any aux_*.bin
files and deserializes them. It's backward-compatible by construction.

What the aux_attributes system can already do

[TokyoWarfare]

so I was able to get some progress on this, I just move the wip and Claude should take the credits, let this be clear

here a sample of normals carried on till the offline viewer. If I get it fully functional I'll share the repo but I won´t do a PR since the risk of breaking something is real but may be it could be checked as for a prototype implementation. time will tell.

[TokyoWarfare]

normals exported to ply , there was a bit of noise so simply reorient normals by neighbors solved that nicely

[TokyoWarfare]

Ok, I'm really enjoying this, I can implement the many ideas I've and test them.

This time "range" was added. On each frame computes the distance of each point to the sensor and saves it as scalar field. This is usefull because the further points are less accurate and if they overlap later on on the slam to more accurate points captured nearby they will cause geometric noise and even intensity values noise. This is only usable if the voxelization is nerfed, as this data must arrive to Cloud Compare asigned to the original point. It can not be voxelized as if this value was interpolated would contaminate valid points. As of now is a problem as the GPU offload is not yet implemented in the offline viewer so I can not apply this feature on large scans.

In cloud compare simply filter out points using the scalar field. This is a double edge tool, as far elements that are not later captured from close could be lost. This is easy to happen in a forward facing scanner and less probable in a 360 scanner.

Data is cleaned very nicely:

Original data:

Cleaned data

Points captured from a far distance:

____________

An example of the risk of doing this in post by simply cleaning the scalar field without salvaging valid data::

data from far:

only data close to scanner at capture time:

some building would be lost, reducing context of the data.

the correct way to implement this discrimination would be at the voxelization time, evaluating the points withing a voxel and discarding candidates with a capture distance delta greater than n discard greater.

I may try to ask Claude to do so but I think the concept is good enough for @koide3 to consider. Cheers!

Edit, another example (rance distance is 33m and beyond)

Original:

Cleaned:

Captured from far:

Overlay

Reflective surfaces are the ones that have the worst performance when captured from far. Signals, lamplight reflectors etc...

edit2: voxelization is not nerfect at code level but with config values, so yet there may be some voxelization going on and some value interpolation, I'll ad option to not perform interpolation in config settings to fully test this concept.

[TokyoWarfare]

a combination of normal export and data filtering using range, ressults in a much cleaner ground mesh

[themightyoarfish]

The use of the distance for filtering is pretty cool. Although it makes me wonder if one should not limit the scanner range in that case, if faraway points are labeled as unreliable later on anyway.

[TokyoWarfare]

In regards tot he distance.
While you can now limit the far points in the odometry on .json. That also means you need to trade off between no data and less accurate data.
My proposal is:
1)better some data than no data
2)better best(closer) data when available

I'll get to this ASAP as I'm a bit busy but I'll try to iplement this logic of point quality on the voxelization, to discard points when other of higher quality are available. May be some lidars do not degrade that much as mine with diatance. I use Livox Horizon, right now. I'm getting a pair of Hesai in a few weeks.

I've first to fix the visualization of colors first, then implement the GPS-Time scalar field, may be scanner-id, for later if sesion merge be able to know from wich scanner each data is coming from, and then will get into the quality discrimination. 2-3 weeks max I would say.

[TokyoWarfare]

This can be tested here
https://github.com/TokyoWarfare/glim-extended

I strongly recommend to run this on a container

as of now features normal export, and different visualization modes for the mapping like intensity, range, gps time.

gps time is broken on the offline viewer.

---

Once I finish with the visualization part I plan to export full res deskewed frames for later in offline viewer be able to export the odometry path at full res and thus obtain a full resolution SLAM. I need to double check if all features are already available at teh time of export.

There is a moving object cleanup plugin that already exports deskewed frames so I may use that as base.

---

I had to aswell modify GTSAM
https://github.com/TokyoWarfare/gtsam_points_extended

---

Intensity

normals

GPSTime, viewer properly handles Unix time.

---

code comes from Claude Code I pretend to get no merit from that.

[TokyoWarfare]

So integrated as a tool the fitlering by range, it has a previw tool to fine tune values and the apply map wide. I also implmeneted a full resolution mode so the data is threaded from the HD frames, and the filltering occurs at high res frames and it is alter translated to SD lods.

I'm happy with the outocme. I've new lidars too may be not so usefull with the new ones, not sure how widespread this issue is.

[TokyoWarfare]

I'm extending it into an MMS tols suite. If i manage to nail colorization my objective will be accomplished. But this part is quite part to nail. I'll try to use @koide3 calibration tool but I think there is aswell the need to implement a weighting on what each camera colors of each frame as I beleive lens distortion is not enoubh. Yet ressults are acceptable. This si by manually tweaking extrinsics But at this poitn if I "fix" one thing other breaks. I'm assuming Metashape computed intrinsics are valid. I'll try to port @koide3 calibration tool, swap Superglue by Lightglue due to the licensing restrictions and instead of using it with live data, use with data in post, in the offline viewer, that would be easyer if camera was genlocked but it is not so I will need to introduce a new variable in the matching that is a time shift and introduce a range where it would perform n number of matchings and choose among the best ressults, obtaining a fine tuned time shift and extrinsics & intrinsics, all potenitially of course :D

some colorization ressults:
so far theese are my best ressults on this dataset
https://youtu.be/1xojDgCutJU

https://youtu.be/1xojDgCutJU

If anyone whant's to try it:
https://github.com/TokyoWarfare/glim-extended

requires
https://github.com/TokyoWarfare/gtsam_points_extended

[tsarrus]

@TokyoWarfare Hi! I've read your work and wanted to give it a try. Could you please tell me the best way to work with your fork? Should I follow the "Install from source" instructions, or is there a Docker image for working with your fork?

[TokyoWarfare]

@TokyoWarfare Hi! I've read your work and wanted to give it a try. Could you please tell me the best way to work with your fork? Should I follow the "Install from source" instructions, or is there a Docker image for working with your fork?

I'm finishing it and doing inteste testingo n the 1.0 after that I'll prepare a dockerfile or an install md. I need a few weeks.

As of now best way and a must would say is to create a docker container and try to build from there, I don´t think there is any extra dependency than livexiv2, and the github version may not even had that yet.

I need a few weeks, then I'll create a container myself to create a proper step by step guide.