[Data] Support reading multiple paths in read_lance - #64560
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds support for reading from multiple Lance datasets in read_lance by allowing a list of URIs. It updates LanceDatasource to handle multiple datasets, distributing the requested parallelism proportionally across them, and adds corresponding unit tests. Feedback is provided to cap the calculated dataset parallelism at the number of fragments to prevent performance issues and potential OOMs when parallelism is extremely high.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
089bc1a to
2e6b3da
Compare
ray-gardener
Bot
added
data
2e6b3da to
9bb0e6d
Compare
|
This pull request has been automatically marked as stale because it has not had You can always ask for help on our discussion forum or Ray's public slack channel. If you'd like to keep this open, just leave any comment, and the stale label will be removed. |
github-actions
Bot
added
the
stale
github-actions
Bot
added
unstale
owenowenisme
left a comment
owenowenisme
left a comment
There was a problem hiding this comment.
Left minor nits but LGTM now, will approve after conflict solved.
FYI, conflict is from #64881
8f07e79 to
ec437e8
Compare
|
tests failing? |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
Reviewed by Cursor Bugbot for commit 14644d5. Configure here.
14644d5 to
0a67eac
Compare
Extend `read_lance` to accept a list of URIs so that data from multiple Lance datasets is combined into a single Dataset. Fragments across all datasets are flattened and split globally by parallelism, so read tasks are balanced by fragment count rather than requiring at least one task per dataset. A single read task can span multiple datasets by creating one scanner per dataset. Also: - Use the dataset-level schema (unified with type promotion across datasets) as the ReadTask output schema, which is correct under Lance schema evolution. - Reject `scanner_options['fragments']` when reading multiple datasets, since fragments are bound to a specific dataset. - Reject an empty URI list. Signed-off-by: dragongu <andrewgu@vip.qq.com>
0a67eac to
20905b9
Compare
@richardliaw Fixed now, thanks! |
_fill_missing_columns null-filled columns for the unified ReadTask schema but left each block in its per-dataset scanner order. When a dataset's column order differed from the unified schema, the block violated the advertised schema contract and broke positional consumers like Table.cast and RecordBatchReader.from_batches. Reorder each block via select(schema.names) after null-fill; projection under `columns=` is still returned untouched. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: Drakery <dragongu@example.com> Signed-off-by: dragongu <andrewgu@vip.qq.com>
|
@owenowenisme did you review? |
3 participants
Why are these changes needed?
Our upstream produces data as many small Lance directories, but
ray.data.read_lanceonly accepts a singleuri, so reading them all requirescalling
read_lanceper directory and unioning the results by hand.This PR lets
read_lancealso accept a list of URIs:Passing a single string works exactly as before.
Note: a read task can only read fragments from one Lance dataset, so the number
of read tasks is at least the number of directories, even when
parallelismissmaller.
Related issue number
None.
Checks
-sflag, i.e.,git commit -s) in this PR.scripts/format.shto lint the changes in this PR.python/ray/data/tests/test_lance.py.