Conversation
Load trusted TinyExp checkpoints with weights_only=False and add RNG-state restoration coverage.
Start a fresh local Ray runtime, bound remote waits, and clean up worker actors after the CPU accelerator test.
Apply epoch positioning only before iterator creation, document the epoch-level resume contract, and add regression coverage for iterator reuse.
Remove the Ray head-node resource pin so CPU-only heads can schedule distributed workers. Start the TCPStore on placement-group bundle 0 and keep its dynamic port reserved through worker startup and execution. Add logical multi-node Ray integration coverage and update CPU accelerator tests.
Use available Ray resources for automatic worker-count sizing while retaining total-capacity validation for explicit worker counts. Include requested, total, and currently available CPU/GPU values in placement timeout diagnostics, and cover the behavior with regression tests and documentation.
Move PyTorch dependencies to an optional extra, keep uv resolution universal across platforms, preserve machine-selected accelerator builds, and fix Python 3.14 Hydra compatibility.
Constrain NumPy, rpds-py, and Pillow to releases with CPython 3.14 wheels. Regenerate uv.lock so make install-pytorch avoids source builds on Python 3.14.
Run CI and tox tests with the pytorch extra explicitly selected. Disable Ray UV runtime environment propagation so workers reuse the installed test environment instead of a bare environment without torch.
Use a small run wrapper around the experiment workload so accelerator.destroy() runs from finally without indenting the training body. Add regression coverage for cleanup when MNIST, ResNet, or Pi workloads raise.
Use the HTTP rendezvous only for multi-node Redis Cluster startup. Remove the runtime finish barrier so each wrapper returns the child exit code and immediately cleans up locally owned Redis. Document that whole-job failure and restart belong to the external launcher, and add success/failure lifecycle coverage.
Retry RedisCluster construction and cluster exceptions within a finite readiness deadline. Add hard timeouts for redis-cli cluster creation, propagate startup timeout configuration, and run external startup work outside the rendezvous lock while preventing duplicate creation. Add regression coverage and remote CPU multi-node validation.
Document the infinite stream model, nominal epoch length, sampler and DataLoader drop_last distinction, batch-rounding effects, distributed seed requirements, and the boundary between exact index continuation and epoch-level statistical resume.
Reuse externally owned Ray runtimes and only shut down runtimes initialized by the launcher.
Disable persistent workers when num_workers is zero for training and validation loaders.
Apply Hydra mapping values to plain mapping fields while preserving recursive nested config updates.
Validate static Ray cluster CLI topology and keep readiness checks within the configured deadline.
Treat Redis Cluster-specific operation errors as cache misses or failed writes.
Reject invalid or conflicting Redis Cluster bus ports before starting server processes.
Comment out mypy execution and configuration while retaining the dependency and restore path for future use.
Remove the completed review checklist; future follow-up work can be tracked with local TODO comments when needed.
zengarden
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary