Skip to content

cat_file violates fsspec range semantics: end treated as inclusive, start=None returns trailing bytes #166

Description

@victor-zou

Summary

ossfs's cat_file does not implement the range semantics defined by
fsspec.AbstractFileSystem.cat_file, where start/end behave "like usual
python slices", i.e. [start, end) with end exclusive and negative values
counting from EOF. Instead, both OSSFileSystem.cat_file and
AioOSSFileSystem._cat_file forward (start, end) unmodified to
oss2's get_object(byte_range=...), which serializes them into an HTTP
Range: bytes=start-end header — where end is inclusive (RFC 9110).

Actual vs expected behavior

For a file with contents data:

call fsspec contract ossfs actual
cat_file(p, 10, 20) data[10:20] (10 bytes) Range: bytes=10-20data[10:21] (11 bytes)
cat_file(p, None, 500) data[:500] (first 500 bytes) Range: bytes=-500data[-500:] (last 500 bytes)
cat_file(p, -100, None) data[-100:] correct by accident (HTTP suffix range)
cat_file(p, 0, -5) data[:-5] Range: bytes=0--5 → invalid header

The second row is the most damaging: silently wrong bytes, not just a wrong length.
This breaks consumers that stitch byte ranges together, e.g. kerchunk /
ReferenceFileSystem-style workloads, which rely on
cat_file(path, s, e) == data[s:e].

The same translation gap exists in OSSFileSystem.get_object() and
OSSFile._fetch_range (https://github.com/fsspec/ossfs/blob/main/src/ossfs/file.py),
which forward the exclusive end expected by AbstractBufferedFile's caches to the
inclusive OSS API (fetches one extra byte per block; mostly masked by the
readahead cache's final slice, but still incorrect).

Reproducer

import fsspec

content = bytes(range(256)) * 4  # 1024 bytes
fs = fsspec.filesystem("oss", ...)  # any bucket
fs.pipe_file("my-bucket/range-test.bin", content)

for s, e in [(0, 100), (10, 20), (None, 500), (50, None), (-100, None), (0, -5), (7, 7)]:
    got = fs.cat_file("my-bucket/range-test.bin", s, e)
    expected = content[s:e]
    assert got == expected, f"start={s}, end={e}: got {len(got)} bytes, expected {len(expected)}"

All positive-range cases return one extra byte; (None, 500) returns the last 500
bytes; (0, -5) produces an invalid header.

For comparison, s3fs, gcsfs, ocifs and fsspec's own http implementation all
translate to inclusive-end correctly (end - 1 after normalizing None/negatives).

Suggested fix

Normalize in cat_file / _cat_file before calling oss2, mirroring
_process_limits:

  1. resolve negative start/end against the object size (self.info(path)["size"]),
  2. default start to 0 when end is given,
  3. return b"" when end <= start,
  4. call get_object(byte_range=(start, end - 1)).

And in OSSFile._fetch_range, pass end - 1 as the inclusive end.

Happy to open a PR along these lines, including a parametrized conformance test
like the snippet above.

Environment

  • ossfs: current main (core.py cat_file as of bda100d)
  • fsspec: 2026.x

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions