[Data] Add support for reading ORC files by WFY123wfy · Pull Request #64600 · ray-project/ray

WFY123wfy · 2026-07-08T14:39:38Z

Description

Ray Data can read Parquet, CSV, JSON, and Avro, but not Apache ORC — a columnar format widely used in the Hive/Spark ecosystem (supported by both Dask and Spark). This adds ray.data.read_orc(), backed by a new ORCDatasource that reads each
file with pyarrow.orc.read_table. ORC keeps its metadata footer at the end of the file, so the datasource opens files for random access rather than as a sequential stream. Read-only for now, mirroring how Avro shipped; write support can
follow.

Related issues

Closes #37890

Additional information

Mirrors the read_avro / read_parquet API. file_extensions defaults to ["orc"]. No new dependency (pyarrow already ships pyarrow.orc). Adds tests in test_orc.py and docs in loading_data.rst.

import ray
ds = ray.data.read_orc("s3://bucket/path/")

gemini-code-assist · 2026-07-08T14:49:30Z

Warning

Gemini encountered an error creating the review. You can try again by commenting /gemini review.

Add `ray.data.read_orc()` to read Apache ORC files into a Dataset, backed by a new `ORCDatasource` that reads each file with `pyarrow.orc.read_table`. ORC stores its metadata footer at the end of the file, so the datasource opens files for random access rather than as a sequential stream. Closes ray-project#37890 Signed-off-by: wufeiye1 <wufeiye1@jd.com>

WFY123wfy · 2026-07-09T02:31:40Z

Adds ORC read support (#37890), mirroring the existing read_avro / read_parquet datasource pattern. Read-only for now; write support can follow separately.

Tests in test_orc.py pass locally against a nightly build, and the readthedocs and Bugbot checks are green. The microcheck failure (#49686) appears to be a CI infra issue — the docker image pull failed after retries, not a lint/test failure (I reproduced all pre-commit hooks locally with no failures).

@scottjlee @richardliaw — would you be able to review, or help re-trigger CI?Happy to iterate on any feedback.

WFY123wfy · 2026-07-13T02:45:35Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces support for reading ORC files in Ray Data by implementing ORCDatasource, exposing the read_orc API, and adding corresponding unit tests and documentation. Feedback on the changes highlights a bug where **open_args are silently ignored in _open_input_source, preventing configuration arguments from being passed to the filesystem. Additionally, it is recommended to remove the redundant __init__ method in ORCDatasource to simplify the class.

gemini-code-assist · 2026-07-13T02:47:01Z

+    def __init__(
+        self,
+        paths: Union[str, List[str]],
+        **file_based_datasource_kwargs,
+    ):
+        super().__init__(paths, **file_based_datasource_kwargs)


The __init__ method is redundant as it only calls super().__init__ with the exact same arguments. Removing this boilerplate simplifies the class and improves maintainability.

Good catch — removed in the latest commit. Thanks!

The __init__ only forwarded its arguments to the parent FileBasedDatasource without adding any behavior, so it can be dropped and the class inherits the parent constructor directly. Also drop the now-unused List/Union imports. Signed-off-by: wufeiye1 <wufeiye1@jd.com>

WFY123wfy · 2026-07-13T14:07:03Z

CI is green and I've addressed the review comments. This adds ORC read support (closes #37890), read-only for now, following the existing read_avro datasource pattern. Tests pass locally against a nightly build.

@bveeramani @justinvyu @owenowenisme — would any of you have a chance to review
when you get a moment? Happy to iterate on any feedback. Thanks!

bveeramani · 2026-07-13T16:23:43Z

@WFY123wfy I don't have bandwidth to review, but I'll ask a colleague

dstrodtman

Doc changes are straightforward. Stamping to make sure the Vale changes don't remain blocking (just adding the API entry).

ayushk7102

Thanks a lot for contributing! This PR will be a good addition to Ray Data for ORC as a V1 datasource.

We are eventually planning to move all readers (that can support chunking, indexing) to the Datasource V2 API.

A good reference for this is parquet_reader.py. Would it make sense to migrate ORC to the V2 API as a follow-up?

Read one ORC stripe at a time through a BlockOutputBuffer instead of loading the whole file at once. This bounds per-task memory usage on large files while keeping output blocks at the target block size (small stripes are coalesced, large ones are split), mirroring AvroDatasource's buffering. Add tests for reading Hive-partitioned files (with and without a partition filter) and for reading files with multiple stripes. Signed-off-by: wufeiye1 <wufeiye1@jd.com>

WFY123wfy · 2026-07-22T15:12:27Z

Thanks a lot for contributing! This PR will be a good addition to Ray Data for ORC as a V1 datasource.

We are eventually planning to move all readers (that can support chunking, indexing) to the Datasource V2 API.

A good reference for this is parquet_reader.py. Would it make sense to migrate ORC to the V2 API as a follow-up?

Thanks! Migrating to the Datasource V2 API makes sense, and parquet_reader.py is a helpful reference. I'd prefer to do it as a separate follow-up PR so this one stays focused on landing V1 ORC support. Happy to take that on afterward.

Drop the manual BlockOutputBuffer in ORCDatasource._read_stream. Since read_stripe yields Arrow tables (which are already blocks), the read operator's BlockMapTransformFn handles output block shaping against the target block size. _read_stream now yields one stripe at a time, which still bounds per-task memory on large files without duplicating the shaping logic. Signed-off-by: wufeiye1 <wufeiye1@jd.com>

ORC uses random access (open_input_file) and doesn't consume stream open args, so exposing arrow_open_stream_args in read_orc was a no-op that could mislead users. Remove it from the public signature, matching read_images which also omits it. Signed-off-by: wufeiye1 <wufeiye1@jd.com>

ayushk7102

LGTM except 1 nit. Thanks for working on this!

Co-authored-by: Ayush Kumar <ayushk7102@gmail.com> Signed-off-by: feiye_wu <46945405+WFY123wfy@users.noreply.github.com>

Signed-off-by: Richard Liaw <rliaw@berkeley.edu>

- Explicitly import pyarrow.fs for the type annotation and use the PartitionStyle enum so the datasource and tests pass pyrefly type checking. - Add a regression test that reads an empty ORC file as an empty dataset. Signed-off-by: wufeiye1 <wufeiye1@jd.com>

WFY123wfy · 2026-07-30T13:43:02Z

Heads up: The earlier premerge failure has been fixed, and I’ve force-pushed the commit f7e30a6. All checks are green, and the two existing approvals remain valid. The branch is now out-of-date with master. Auto-merge was disabled due to the force push.
@richardliaw Could you please re-enable auto-merge (or update the branch) when you have time? Thanks!

WFY123wfy · 2026-08-05T02:19:30Z

The branch has been updated with master, and all checks are green. @richardliaw could you please re‑enable auto‑merge at your convenience? Thanks!

ayushk7102 · 2026-08-06T22:12:48Z

The branch has been updated with master, and all checks are green. @richardliaw could you please re‑enable auto‑merge at your convenience? Thanks!

Hey @WFY123wfy, thanks for contributing to a V1 of Ray Data's read_orc API!

With Ray 2.56, we have set the default path to be datasource V2 for all formats that can support it. With newer Ray verisons, we have also made the decision that we want to support V2 from the start with any new format (again, if compatible).

Would you be interested in moving this PR towards a V2 path for read_orc? I'd be happy to spend time on review, and we can move this along in a expedited way given that a majority of the legwork is done for V1.

Also, we could support an enriched ORC reader with projection pushdown and (potentially predicate pushdown),

Also, this draft PR that we are working on implementing bin-packing for load balanced Parquet reads should also apply to ORC. We can wait for this to merge, as we're going to genericise this algorithm so that it is usable by any datasource.

ayushk7102 · 2026-08-07T00:34:46Z

The branch has been updated with master, and all checks are green. @richardliaw could you please re‑enable auto‑merge at your convenience? Thanks!

Hey @WFY123wfy, thanks for contributing to a V1 of Ray Data's read_orc API!

With Ray 2.56, we have set the default path to be datasource V2 for all formats that can support it. With newer Ray verisons, we have also made the decision that we want to support V2 from the start with any new format (again, if compatible).

Would you be interested in moving this PR towards a V2 path for read_orc? I'd be happy to spend time on review, and we can move this along in a expedited way given that a majority of the legwork is done for V1.

Also, we could support an enriched ORC reader with projection pushdown and (potentially predicate pushdown),

Also, this draft PR that we are working on implementing bin-packing for load balanced Parquet reads should also apply to ORC. We can wait for this to merge, as we're going to genericise this algorithm so that it is usable by any datasource.

Hey @WFY123wfy, sorry for the back-and-forth. After further discussion with the team, we can land V1 ORC as a first step. Can we support projection pushdown as part of V1?

We can then work on V2 as a follow up.

richardliaw · 2026-08-07T04:28:44Z

actually, https://github.com/ray-project/ray/pull/64540/changes seems to be a V2 implementation

WFY123wfy · 2026-08-07T06:53:08Z

@ayushk7102 previously confirmed V1 + projection pushdown as the path forward. Given the ongoing V2 work in #64540, I’d like to double‑check this is still the agreed plan. Should I keep implementing projection pushdown within this V1‑based PR?

richardliaw · 2026-08-07T07:02:07Z

Yes, that seems fine. We can get your current PR merged by tomorrow, with projection pushdown.

…

On Thu, Aug 6, 2026 at 11:53 PM feiye_wu ***@***.***> wrote: *WFY123wfy* left a comment (ray-project/ray#64600) <#64600 (comment)> @ayushk7102 <https://github.com/ayushk7102> previously confirmed V1 + projection pushdown as the path forward. Given the ongoing V2 work in #64540 <#64540>, I’d like to double‑check this is still the agreed plan. Should I keep implementing projection pushdown within this V1‑based PR? — Reply to this email directly, view it on GitHub <#64600?email_source=notifications&email_token=ABCRZZMUF6TYECVMKSNM2A35IV4GVA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMRRGM2TONJUGI42M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-5213575429>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCRZZPHZJ2CZAWP6BSSVET5IV4GVAVCNFSNUABEKJSXA33TNF2G64TZHM3TCOJTGIZTIOJ3JFZXG5LFHM2DQMZYGQ3DKMZWGKQXMAQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

cursor · 2026-08-07T11:36:10Z

+        # First pass: walk in declared order and pick the first "cheap" type.
+        for field in orc_schema:
+            if str(field.type).split("[", 1)[0].split("(", 1)[0] in cheap_kinds:
+                return field.name


Carrier type matching never hits ints

Medium Severity

_pick_carrier_column treats cheap kinds as exact base names like int, date, float, and time, but str(field.type) yields forms such as int64, date32, and double. The intended preference for narrow fixed-width columns almost never matches, so empty or partition-only projections often fall back to the first schema field and can pull large string or binary columns on big ORC files.

^{Reviewed by Cursor Bugbot for commit 5ecc1dd. Configure here.}

richardliaw · 2026-08-08T03:43:33Z

seems like test failing?

ORC now prunes physical columns at stripe-read time via the supports_projection_pushdown protocol, so pure select_columns() is removed from the optimized plan. Adds 6 tests covering physical-only, multi-stripe, empty, partition-only, mixed, and include_paths projections. Signed-off-by: wufeiye1 <wufeiye1@jd.com>

cursor

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{Reviewed by Cursor Bugbot for commit c125382. Configure here.}

cursor · 2026-08-08T15:39:20Z

+
+    ds = ray.data.read_orc(path).select_columns([])
+    assert ds.count() == 5000
+    assert ds.take_all() == []


Empty projection take assertion wrong

Medium Severity

test_read_orc_projection_pushdown_empty asserts both count() == 5000 and take_all() == []. An empty projection still yields one row per input row (backed by the internal stub column), so take_all() cannot be an empty list while count() is nonzero. This matches the reported failing test after projection pushdown landed.

^{Reviewed by Cursor Bugbot for commit c125382. Configure here.}

richardliaw · 2026-08-08T16:53:26Z

test still failing

WFY123wfy force-pushed the feature/data-read-orc branch from 8e92ffb to bacf7df Compare July 8, 2026 15:07

WFY123wfy marked this pull request as ready for review July 8, 2026 15:11

WFY123wfy requested review from a team as code owners July 8, 2026 15:11

Merge branch 'master' into feature/data-read-orc

a2d1a39

ray-gardener Bot added docs An issue or change related to documentation data Ray Data-related issues community-contribution Contributed by the community labels Jul 8, 2026

Merge branch 'master' into feature/data-read-orc

8be25da

gemini-code-assist Bot reviewed Jul 13, 2026

View reviewed changes

dstrodtman reviewed Jul 13, 2026

View reviewed changes

dstrodtman approved these changes Jul 13, 2026

View reviewed changes

ayushk7102 requested changes Jul 21, 2026

View reviewed changes

Comment thread python/ray/data/_internal/datasource/orc_datasource.py Outdated

Comment thread python/ray/data/_internal/datasource/orc_datasource.py

Comment thread python/ray/data/tests/datasource/test_orc.py

WFY123wfy requested a review from ayushk7102 July 22, 2026 15:13

WFY123wfy added 4 commits July 23, 2026 10:13

Merge branch 'master' into feature/data-read-orc

ba24a1d

Merge branch 'master' into feature/data-read-orc

f96a3ff

Merge branch 'master' into feature/data-read-orc

ae848f5

Merge branch 'master' into feature/data-read-orc

78a231e

ayushk7102 requested changes Jul 28, 2026

View reviewed changes

Comment thread python/ray/data/_internal/datasource/orc_datasource.py Outdated

WFY123wfy requested a review from ayushk7102 July 28, 2026 13:02

Merge branch 'master' into feature/data-read-orc

b8afdaa

ayushk7102 reviewed Jul 28, 2026

View reviewed changes

Comment thread python/ray/data/read_api.py Outdated

WFY123wfy requested a review from ayushk7102 July 29, 2026 10:22

ayushk7102 reviewed Jul 29, 2026

View reviewed changes

Comment thread python/ray/data/tests/datasource/test_orc.py Outdated

ayushk7102 approved these changes Jul 29, 2026

View reviewed changes

Update python/ray/data/tests/datasource/test_orc.py

c1e86a1

Co-authored-by: Ayush Kumar <ayushk7102@gmail.com> Signed-off-by: feiye_wu <46945405+WFY123wfy@users.noreply.github.com>

richardliaw reviewed Jul 30, 2026

View reviewed changes

Comment thread python/ray/data/read_api.py Outdated

Apply suggestion from @richardliaw

825845c

Signed-off-by: Richard Liaw <rliaw@berkeley.edu>

richardliaw enabled auto-merge (squash) July 30, 2026 06:09

github-actions Bot added the go add ONLY when ready to merge, run all tests label Jul 30, 2026

cursor Bot reviewed Jul 30, 2026

View reviewed changes

Comment thread python/ray/data/_internal/datasource/orc_datasource.py Outdated

auto-merge was automatically disabled July 30, 2026 08:39
Head branch was pushed to by a user without write access

WFY123wfy force-pushed the feature/data-read-orc branch from 1e32a71 to f7e30a6 Compare July 30, 2026 10:23

richardliaw assigned ayushk7102 Aug 7, 2026

cursor Bot reviewed Aug 7, 2026

View reviewed changes

WFY123wfy force-pushed the feature/data-read-orc branch from 5ecc1dd to c125382 Compare August 8, 2026 15:35

cursor Bot reviewed Aug 8, 2026

View reviewed changes

Uh oh!

Conversation

WFY123wfy commented Jul 8, 2026

Description

Related issues

Additional information

Uh oh!

gemini-code-assist Bot commented Jul 8, 2026

Uh oh!

WFY123wfy commented Jul 9, 2026

Uh oh!

WFY123wfy commented Jul 13, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

gemini-code-assist Bot Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

WFY123wfy Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

WFY123wfy commented Jul 13, 2026

Uh oh!

bveeramani commented Jul 13, 2026

Uh oh!

dstrodtman left a comment

Choose a reason for hiding this comment

Uh oh!

ayushk7102 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

WFY123wfy commented Jul 22, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ayushk7102 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

WFY123wfy commented Jul 30, 2026

Uh oh!

WFY123wfy commented Aug 5, 2026

Uh oh!

ayushk7102 commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ayushk7102 commented Aug 7, 2026

Uh oh!

richardliaw commented Aug 7, 2026

Uh oh!

WFY123wfy commented Aug 7, 2026

Uh oh!

richardliaw commented Aug 7, 2026 via email

Uh oh!

cursor Bot Aug 7, 2026

Choose a reason for hiding this comment

Carrier type matching never hits ints

Uh oh!

richardliaw commented Aug 8, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor Bot Aug 8, 2026

Choose a reason for hiding this comment

Empty projection take assertion wrong

Uh oh!

richardliaw commented Aug 8, 2026

Uh oh!

Reviewers

Assignees

Labels

ayushk7102 commented Aug 6, 2026 •

edited

Loading