Skip to content

Implement ChunkableMixin#4472

Open
alejoe91 wants to merge 29 commits intoSpikeInterface:mainfrom
alejoe91:chunkable-mixin
Open

Implement ChunkableMixin#4472
alejoe91 wants to merge 29 commits intoSpikeInterface:mainfrom
alejoe91:chunkable-mixin

Conversation

@alejoe91
Copy link
Copy Markdown
Member

Rebuilding #4216 on top of #4462

This PR introduces a ChunkableMixin class (for extractors) and a ChunkableSegment. These can be used as base classes for any class that can be chunked in the first dimension (e.g., BaseRecording - BaseImaging).

This is done by "abstracting" the get_traces into a more general get_data, and implementing other key methods as:

  • get_sample_size_in_bytes
  • get_memory_size
  • get_shape

The ChunkableSegment is also in charge of all the low level time handling.

This large refactor allows to centralize the following "machinery" (in chunkable_tools and job_tools)

  • write_binary/write_memory/add_to_zarr
  • parallel execution, with ChunkExecutor (previous ChunkRecordingExecutor)
  • pipeline node and run_node_pipeline

NOTE: Created as draft to show diff before #4462 is merged

@alejoe91 alejoe91 added core Changes to core module refactor Refactor of code, with no change to functionality labels Mar 26, 2026
return i0, i1

def compute(self, traces, start_frame, end_frame, segment_index, max_margin, peak_slice):
def compute(self, chunk, start_frame, end_frame, segment_index, max_margin, peak_slice):
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think in the context of spikeinterface we can keep "traces" no ?
the code will be more redable and and this do not change the signature

need_first_call_before_pipeline = False

def get_trace_margin(self):
def get_data_margin(self):
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe get_marin() no ?

start_frame=start_frame,
end_frame=end_frame,
channel_indices=slice(None),
last_dimension_indices=slice(None),
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

??
why this renaming.
We need to handle the backward compatibility.

Comment on lines +62 to +63
def compute(self, chunk, start_frame, end_frame, segment_index, max_margin):
traces = chunk
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For internal si function we can kepp this no ?

Suggested change
def compute(self, chunk, start_frame, end_frame, segment_index, max_margin):
traces = chunk
def compute(self, traces, start_frame, end_frame, segment_index, max_margin):

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Changes to core module refactor Refactor of code, with no change to functionality

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants