Conversation
Force CPU accelerator device selection, define a shared accelerator protocol, and make Ray worker CPU/GPU resources explicit. Fix automatic Ray worker sizing to honor both CPU and GPU capacity, and add checkpoint, scheduler, and resource regression tests.
Protect externally managed Redis processes and clean up only owned resources. Persist sampler, scaler, and RNG state for reliable training resume.
Partition distributed samples without padding, derive Ray local ranks from node topology, and make accelerator cleanup explicit and idempotent.
…validation, and rank-zero dataset download Ray placement groups now honor a configurable ray_placement_timeout_s (default 120s) and are removed on timeout or error so workers do not scatter into a leaking group. Resource validation now checks GPU capacity in addition to CPU before placement, raising a clearer InsufficientCPUError that names the shortfall (e.g. needed GPU=2.0, available GPU=1.0). The MNIST example is downloaded by rank 0 only, with a barrier before other ranks load from local disk, eliminating duplicated and racing downloads across workers.
Reassign Ray worker ranks from the actual node topology so ranks remain contiguous within each node. Print topology using the final distributed ranks and add regression coverage and documentation.
Replace echo-built demo script with direct TinyExp CLI commands and regenerate README GIF assets.
Move mode and launcher defaults before ray_cfg so the dataclass configuration is constructed correctly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary