deps(deps): bump the data group across 1 directory with 2 updates

[dependabot]

Bumps the data group with 2 updates in the / directory: datasets and pyarrow.

Updates datasets from 4.4.2 to 5.0.0

Release notes

Sourced from datasets's releases.

5.0.0

Datasets Features

Agent traces

• Parse Agent traces messages for SFT using teich by @​lhoestq in huggingface/datasets#8232

  • Agent traces from claude_code/pi/codex and others can now be loaded with load_dataset
  • Using the teich library (new optional dependency), traces are parsed to messages to enable training on traces using e.g. trl
  • Load the data:

  >>> from datasets import load_dataset
  >>> ds = load_dataset("lhoestq/agent-traces-example", split="train")
  >>> ds[0]["messages"]
  [{'role': 'user', 'content': 'Download a random dataset from Hugging Face, use DuckDB to inspect it, and come back with a short report about it. Be concise and include: dataset name, what files/format you found, row count or rough size if you can determine it,...'
   ...]

  • Train on agent traces:

  trl sft --dataset-name lhoestq/agent-traces-example ...

  • find all the Agent traces datasets on HF here: https://huggingface.co/datasets?format=format:agent-traces&sort=trending

Next-level shuffling in streaming mode

• Use multiple input shards for shuffle buffer by @​lhoestq in huggingface/datasets#8194

  ds = load_dataset(..., streaming=True)
  ds = ds.shuffle(seed=42)
  # or configure local buffer shuffling manually, default is:
  ds = ds.shuffle(seed=42, buffer_size=1000, max_buffer_input_shards=10)

  before👎:

  after✨:

  toy example comparison

  from datasets import IterableDataset
  ds = IterableDataset.from_dict({"i": range(123_456_789)}, num_shards=1024)
  ds = ds.shuffle(seed=42)
  print("Cold start ids:")

... (truncated)

Commits

• 68ac1a9 Release: 5.0.0 (#8239)
• cfe4492 Support composed splits in streaming datasets (#8220)
• fd67320 Keep None as a real null in Json() columns instead of the string "null" (#8231)
• 10cdc81 Fix iterable skip over full Arrow blocks (#8236)
• b7c064d Parse agent traces messages for SFT using teich (#8232)
• 31e92f1 fix: embed_external_files=True for mesh support (#8224)
• d168d5f feat: add TsFile (Apache IoTDB) packaged builder with per-device wide format ...
• 992f3cf fix(map): fix progress bar exceeding total when load_from_cache_file=False (#...
• 8474a91 Fix single lance file form pylance 7.0 (#8225)
• d4284e9 feat: add 3D mesh support and MeshFolder builder (#8055)
• Additional commits viewable in compare view

Updates pyarrow from 21.0.0 to 24.0.0

Release notes

Sourced from pyarrow's releases.

Apache Arrow 24.0.0

Release Notes URL: https://arrow.apache.org/release/24.0.0.html

Apache Arrow 24.0.0 RC0

Release Notes: Release Candidate: 24.0.0 RC0

Apache Arrow 23.0.1

Release Notes URL: https://arrow.apache.org/release/23.0.1.html

Apache Arrow 23.0.1 RC0

Release Notes: Release Candidate: 23.0.1 RC0

Apache Arrow 23.0.0

Release Notes URL: https://arrow.apache.org/release/23.0.0.html

Apache Arrow 23.0.0 RC2

Release Notes: Release Candidate: 23.0.0 RC2

Apache Arrow 22.0.0

Release Notes URL: https://arrow.apache.org/release/22.0.0.html

Apache Arrow 22.0.0 RC1

Release Notes: Release Candidate: 22.0.0 RC1

Apache Arrow 22.0.0 RC0

Release Notes: Release Candidate: 22.0.0 RC0

Commits

• 31b4b6c MINOR: [Release] Update versions for 24.0.0
• 06dbc17 MINOR: [Release] Update .deb/.rpm changelogs for 24.0.0
• a021d80 MINOR: [Release] Update CHANGELOG.md for 24.0.0
• 2d6b12c GH-49716: [C++] FixedShapeTensorType::Deserialize should strictly validate se...
• a74cb6a GH-49697: [C++][CI] Check IPC file body bounds are in sync with decoder outco...
• 871a0c6 GH-49676: [Python][Packaging] Fix gRPC docker image layer being too big for h...
• f9203b3 GH-49586: [C++][CI] StructToStructSubset test failure with libc++ 22.1.1 (#49...
• fe298b4 GH-49628: [Python][Interchange protocol] Suppress warnings for pandas 4.0.0 a...
• 1f94910 GH-49252: [GLib] Deprecate Feather features (#49673)
• 5ba5c3c GH-49671: [CI][Docs] Don't run jobs for push by Dependabot (#49672)
• Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

---

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency
• @dependabot ignore <dependency name> major version will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself)
• @dependabot ignore <dependency name> minor version will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself)
• @dependabot ignore <dependency name> will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself)
• @dependabot unignore <dependency name> will remove all of the ignore conditions of the specified dependency
• @dependabot unignore <dependency name> <ignore condition> will remove the ignore condition of the specified dependency and ignore conditions

[nuntius-review]

Nuntius

Reviewed commits

Commit	Summary
a1b74ea	deps(deps): bump the data group across 1 directory with 2 updates

Here is an analysis of the proposed changes in requirements-lock.txt.

Analysis

The changes represent major version upgrades for two core data-science libraries:

1. datasets (from 4.4.2 to 5.0.0)
2. pyarrow (from 21.0.0 to 24.0.0)

1. Correctness & Compatibility

• Major Version Upgrades: Upgrading across major versions (e.g., v4 to v5 for datasets and v21 to v24 for pyarrow) typically introduces breaking API changes, deprecations, or changes in default behaviors.
• Interdependency: datasets relies heavily on pyarrow for its underlying memory format and serialization. While a package manager usually ensures compatibility during lockfile generation, you must verify that your application code does not rely on APIs or behaviors that were deprecated or removed in these newer major versions.
• Pandas Compatibility: The lockfile contains pandas==2.3.3. Ensure that this version of pandas is fully compatible with pyarrow==24.0.0, especially regarding PyArrow-backed backend types or PyArrow engines for reading/writing Parquet/Feather files.

2. Performance & Security

• Performance: Major updates to pyarrow often bring significant performance improvements in serialization, memory usage, and multi-threaded compute kernels.
• Security: Upgrading dependencies is generally good practice to pull in security patches and bug fixes.

Recommendations

Since lockfiles should be managed via package management tools (such as pip-compile, poetry, or pipenv) rather than manual edits, no direct manual changes to this file are recommended.

Instead, before merging this PR, ensure the following verification steps are completed:

1. Run Integration Tests: Execute your full test suite, paying close attention to tests involving dataset loading, saving, and PyArrow/Pandas conversions.
2. Check Deprecations: Review the release notes for datasets v5.0.0 and pyarrow v24.0.0 to identify any removed APIs that your codebase might be using.