Add existing Labels layers as SAM mask inputs

[arthursw]

Summary

• Add compatible napari Labels layers as mask inputs for interactive segmentation.
• Treat every nonzero label ID as an independent SAM mask prompt and preserve IDs in the merged result.
• Assign point, scribble and shape corrections per input object, so different objects can receive different corrections in one Segment Object operation.
• Let users commit satisfactory input masks unchanged, without running SAM or computing embeddings.

Motivation

This enables segmentations created by Cellpose or other napari tools to become first-class inputs to micro-sam.
Users can combine an existing mask with positive or negative corrections when refinement is useful, while masks that are already satisfactory can bypass SAM entirely.

Addresses #768.
Related to #383 and #731.

User-facing behavior

The annotator includes an Existing Segmentation Inputs checklist with Select all and Unselect all controls.
Internal current_object, auto_segmentation and committed_objects layers are excluded so they cannot be selected accidentally; users can duplicate such a layer in napari if they intentionally want to reuse it.

When at least one Labels layer is selected, Segment Object [S] uses every selected object as a separate SAM mask prompt and writes the merged result to current_object.
With no selected mask input, the action retains the existing point and shape prompt behavior.
This replaces an earlier separate refinement button because manual UX testing showed that two overlapping segmentation actions were confusing.

The Target object ID control is populated from the IDs in the selected Labels layers.
Each point or shape stores its own target ID; selecting prompts updates the control, changing the control updates all selected prompts, and mixed selections are shown explicitly.
If the inputs contain only one ID, all current and future corrections are assigned to it automatically.
Every correction must target an ID present in the selected mask inputs.

The point prompt selector is now grouped under Point and scribble prompts and is labeled Prompt type, with Positive (include) and Negative (exclude) choices.
It updates selected or future points and scribbles.
Rectangles, ellipses and polygons remain positive because SAM box and dense shape prompts do not have negative semantics.

Commit selected masks unchanged copies the selected masks directly into committed_objects without computing image embeddings.
This is useful when an external segmentation is already correct or when SAM mask-prompt downsampling would remove fine structures.

Label and geometry semantics

• Every nonzero label value is an object ID.
• Equal IDs from multiple selected layers are unioned.
• Overlapping different IDs are rejected with an actionable error.
• Source Labels layers are never modified.
• Input and image transforms must match; masks are not resampled.
• A trailing one-pixel surplus is cropped per spatial axis to accommodate the size mismatch produced by newly created napari Labels layers.
• Other shape mismatches remain errors.
• Direct and refined commits preserve sparse IDs and reject collisions atomically.

2D and 3D behavior

Each object is sent to SAM independently as its own binary mask prompt, together with only the point, scribble and closed-shape corrections assigned to that object.
This retains object identity and prevents one object from becoming a foreground hint for another.

For 3D inputs, Refine all occupied slices performs 2D mask refinement on each occupied slice and keeps the existing z extent.
Propagate from seed slices selects the largest cross-section for each object, refines that seed plus any assigned correction anchors, and propagates it through the volume.
SAM-v1 uses slice-to-slice projection, while SAM2 conditions its video predictor with the refined seeds.
Tiled prompt paths are supported for both model generations.

The prompt-based segmentation helpers route mask prompts through SAM-v1, tiled SAM-v1, image-style SAM2 and SAM2 video propagation.
Prompt collection can filter every prompt type by object ID and, for 3D data, by slice.

Limitation

SAM converts each 2D mask prompt to its 256 × 256 low-resolution representation.
Fine structures may therefore be lost during refinement.
The UI warns users before refinement and the unchanged-commit action provides a lossless alternative.

Development progression

1. Added Labels-layer collection, validation, per-ID mask extraction and direct unchanged commits.
2. Unified mask refinement with the existing Segment Object action, excluded micro-sam output layers, and added the safe one-pixel crop for new napari Labels layers.
3. Reorganized the controls and replaced the single global correction target with per-prompt object IDs after manual workflow testing and review.
4. Added selection synchronization and regression coverage for mixed targets, cross-layer defaults, prompt polarity preservation and 3D routing.

Documentation and tests

The annotation documentation explains layer selection, ID merging, geometry validation, per-prompt correction routing, unchanged commits, the 256 × 256 warning, and both 3D modes.

Tests cover input collection and validation, deterministic overlap handling, atomic commits, checklist behavior, per-object 2D and 3D prompt routing, mixed prompt selections, napari trailing-pixel cropping, tiled SAM-v1 prompts, SAM2 mask conditioning and 3D seed propagation.

Local validation:

• Consolidated focused suite: 78 tests passed.
• Additional 3D per-object routing regression: passed.
• Python compilation: passed.
• Ruff checks, excluding the repository's existing long-line style: passed.
• git diff --check: passed.
• Manual napari smoke test of the complete mask-input workflow: passed.

The GitHub test matrix requires a maintainer to approve workflows from the fork.

[arthursw]

Note: this was fully generated with GPT 5.6 Sol with high level of reasoning. If this does not comply with your code of conduct ; I can delete the PR.
Also, I will soon make an update of the PR ; to allow better control on points/shape corrections.

[anwai98]

Hi @arthursw,

First of all, thanks for your interest in micro-sam! 🥳

Quick reply on something important: currently, dev is heavily under active development (working towards our next generation tool and models for micro-sam)

I'm happy to take a PR, but I'd like to go with a route of opening an issue first.

That doesn't mean you should close this PR (yet), but let's do it this way:

1. Open a relevant issue stating clearly what is the feature you have a request for and what's your expectation.
2. Define if it is related to micro-sam / sam or micro-sam2 / sam2 (or both) -- this would define where should we base the changes to (although I'd prefer to stick to supporting changes only for micro-sam -- since we are a few weeks aways from an alpha release for micro-sam2)
3. Once the issue describes the issue, the PR body text (in minimal words) should explain what files are changes and what's the intention for any major changes?

If you are up for it, I'd be happy to take a look at the issue you created and continue the discussion there 😉

Lemme know once it's there!

[arthursw]

Hi @anwai98,

Thanks for your quick answer and interest.
I just created a quick issue describing the feature request.

I didn't know about micro-sam2 ; and I think the request is relevant for both SAM and SAM2 ; so I used general terms.

Here is a screenshot of the current version of the PR:

You can see the "Existing Segmentation Inputs" section enabling to select Label layers ; to commit them directly or use them as SAM inputs.

[anwai98]

I just created #1315 describing the feature request.

Amazing, thanks @arthursw 🥳

I didn't know about micro-sam2

Haha it's still under sincere active development, so would take a couple more weeks before an announcement, but we're definitely making a formal release this summer!

I think the request is relevant for both SAM and SAM2 ; so I used general terms.

Nice, good to know!

And re: screenshot: thanks, that helps for sure!

I'll come back to your issue and look at things carefully over the weekend (I need to finish some other things now before I go afk for the evening hehe)!

[arthursw]

I just updated the PR ; here is the new GUI:

I updated the "promp: positive/negative" and the "Target object ID: N" inputs ; to make them more explicit.

It is now possible (and desired) to affect one ID to each point prompt.

[arthursw]

Nice that you are working on a new release ;-)

And it's nice if you can have a look at this PR ; this weekend, or in the coming days.

Thanks ;-)