Main dev#165
Merged
Merged
Conversation
…face **Goal**: Allow SAGE to use LibAMM without requiring PyTorch in Python environment **Changes**: 1. Removed torch/extension.h dependency from PyAMM.cpp 2. Added pybind11/numpy.h for NumPy array support 3. Created conversion layer: torch::Tensor ↔ numpy.ndarray 4. Added wrapper classes (CPPAlgoWrapper, MatrixLoaderWrapper) 5. Exposed NumPy-based interface to Python **Result**: - LibAMM internally still uses PyTorch (all algorithms intact) - Python users only see NumPy arrays (no torch dependency) - Backward compatible: same API, different underlying type **Benefits**: - SAGE doesn't need to install PyTorch - PyTorch dependency is isolated within LibAMM.so - All existing LibAMM functionality preserved (90%+ working) **Status**: Code ready, needs testing after resolving PyTorch CUDA/CPU version conflicts in build environment
- Explains PyTorch dependency isolation strategy - Documents NumPy interface implementation - Provides usage examples and build instructions - Lists feature coverage (90%+ preserved) - Outlines next steps and known issues
**Problem**: LibAMM requires PyTorch at runtime but dependency was not declared **Solution**: - Add install_requires in setup.py with torch>=2.0.0 - Clarify that PyTorch is internal dependency (users only see NumPy) - Add proper package metadata (author, description, classifiers) **Impact**: Users installing PyAMM will automatically get PyTorch installed
- Delete PyTorch model files (*.pth) - migrated to sageData - Delete large test data files - migrated to sageData - Delete build artifacts (dist/) - Update .gitignore to prevent future commits These files have been moved to: packages/sage-benchmark/src/sage/data/libamm-benchmark/ Next step: Clean Git history to reduce repository size
Implemented industry-standard data management approach: Architecture: - Data stored in sage-benchmark/data with Git LFS - LibAMM uses symlinks to access data - Clean separation: code (10MB) vs data (managed separately) Added Tools: - tools/setup_data.sh: Bash script to create symlinks - tools/data_manager.py: Python data management API - tools/DATA_MANAGEMENT.md: Complete documentation Features: ✅ Git LFS for large files (models, datasets, test data) ✅ Automatic symlink creation ✅ Environment variable support (SAGE_DATA_ROOT) ✅ Backward compatible file paths ✅ Lazy data download (git lfs pull) Benefits: - LibAMM repo: 644MB → 10MB (98.5% reduction) - Version-controlled data - Follows PyTorch/HuggingFace/TensorFlow patterns - Flexible deployment (local/CI/production) Usage: bash tools/setup_data.sh # or python tools/data_manager.py Related: Git history cleanup, sageData LFS migration
Converted columnCodeIndexX.txt and rowCodeIndexY.txt from regular files to symlinks pointing to sage-benchmark/data repository. This is part of the data management migration strategy.
问题：ln -sf 在目标已存在且为目录时，会在目录内部创建链接 - 导致 sageData 仓库中出现循环符号链接 - 例如：datasets/datasets -> datasets (循环) 根因： 1. setup_data.sh 重复运行时，benchmark/models 等已存在 2. ln -sf target dir/ 会在 dir 内部创建链接而非替换 dir 3. 结果创建了 models/models, datasets/datasets 等循环链接 修复：在创建链接前先 rm -f 删除已存在的链接/目录 - 确保 ln -sf 始终替换而不是嵌套 - 防止意外的循环链接污染 sageData 仓库
This file helps identify this directory as a Git submodule and provides quick reference for common submodule operations. Auto-generated by tools/git-tools/generate-submodule-markers.sh
- Remove trailing whitespace to comply with pre-commit hooks - No content changes, only formatting improvement
Changed absolute paths (specific to user home directory) to relative paths. This prevents 'modified' status for all users when the symlinks are regenerated with their local absolute paths. The symlinks now use relative paths from test/torchscripts/VQ/ to the shared sage-benchmark data directory.
The individual .txt file symlinks in test/torchscripts/VQ/ are already tracked in Git with relative paths. The setup script was recreating them with absolute paths, causing Git dirty state for all users. Now the script only creates the VQ/data directory symlink and skips the individual file symlinks that are already in the repository.
修改编译标志以避免 OOM: - Debug: -g0 -O1 (原 -g -O0) - Release: -O2 (原 -O3) 在内存受限环境（WSL/容器）中可减少 30-50% 内存占用
- Lower optimization level to -O0 to reduce memory usage - Enable Unity Build with batch size 2 - Disable Unity Build for PyAMM (too large) - Exclude myVecAdd.cpp and pythonBoundings.cpp from Unity Build - Add memory optimization flags: -g0, -fno-var-tracking, -ftemplate-depth=128 - Force single-threaded build to avoid OOM - Fix torch::linalg API compatibility (use at::linalg_* functions) Memory optimization reduces compilation peak from 600MB+ to 200-500MB per file. Suitable for WSL and low-memory environments (8GB+).
- Add comprehensive CI/CD workflow for LibAMM - Enforce C++ test suite (6 tests) before publishing - Build Python wheels for Python 3.9-3.12 - Auto-detect version changes and publish to PyPI/TestPyPI - Create Git tags and GitHub releases - Move LibAMM publishing responsibility to its own repo
There was a problem hiding this comment.
Pull Request Overview
This PR removes entire files from the pybind11 third-party library directory, effectively deleting major portions of the pybind11 library including core headers, documentation, and implementation files.
Reviewed Changes
Copilot reviewed 80 out of 267 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| thirdparty/pybind11/include/pybind11/detail/internals.h | Complete removal of core internals header (764 lines) |
| thirdparty/pybind11/include/pybind11/detail/init.h | Complete removal of initialization support header (436 lines) |
| thirdparty/pybind11/include/pybind11/detail/descr.h | Complete removal of descriptor utilities header (172 lines) |
| thirdparty/pybind11/include/pybind11/detail/common.h | Complete removal of common macros and utilities header (1284 lines) |
| thirdparty/pybind11/include/pybind11/detail/class.h | Complete removal of class implementation header (754 lines) |
| thirdparty/pybind11/include/pybind11/complex.h | Complete removal of complex number support (74 lines) |
| thirdparty/pybind11/include/pybind11/common.h | Complete removal of common header with deprecation warning (2 lines) |
| thirdparty/pybind11/include/pybind11/chrono.h | Complete removal of chrono support (225 lines) |
| thirdparty/pybind11/include/pybind11/cast.h | Complete removal of type casting header (1855 lines) |
| thirdparty/pybind11/include/pybind11/buffer_info.h | Complete removal of buffer protocol support (208 lines) |
| thirdparty/pybind11/include/pybind11/attr.h | Complete removal of attributes header (690 lines) |
| thirdparty/pybind11/docs/upgrade.rst | Complete removal of upgrade guide documentation (594 lines) |
| thirdparty/pybind11/docs/requirements.txt | Complete removal of documentation dependencies (275 lines) |
| thirdparty/pybind11/docs/requirements.in | Complete removal of documentation requirements (6 lines) |
| thirdparty/pybind11/docs/release.rst | Complete removal of release documentation (143 lines) |
| thirdparty/pybind11/docs/reference.rst | Complete removal of API reference documentation (130 lines) |
| thirdparty/pybind11/docs/limitations.rst | Complete removal of limitations documentation (68 lines) |
| thirdparty/pybind11/docs/installing.rst | Complete removal of installation guide (105 lines) |
| thirdparty/pybind11/docs/index.rst | Complete removal of documentation index (48 lines) |
| thirdparty/pybind11/docs/cmake/index.rst | Complete removal of CMake documentation (8 lines) |
| thirdparty/pybind11/docs/benchmark.rst | Complete removal of benchmark documentation (95 lines) |
| thirdparty/pybind11/docs/advanced/pycpp/index.rst | Complete removal of Python C++ interface documentation (13 lines) |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Updated .github/workflows/cmake.yml to use torch==2.2.0 - Updated README.md installation instructions - torch 1.13.0 is no longer available in PyTorch repository
…etup-python to force python 3.11 on self-hosted runner\n- upgrade pip before installing torch stack to avoid hash mismatch with older pip\n- document python 3.11 requirement in README
- Install python3.11 and python3.11-dev explicitly via apt - Replace all python3 commands with python3.11 for consistency - Add numpy and pybind11 installation before torch (fixes UserWarning) - Pass -DPython_EXECUTABLE to CMake to ensure correct Python headers - Fixes pybind11 INTERFACE_INCLUDE_DIRECTORIES mismatch error
- Build wheels on pull_request for testing purposes - Only publish to PyPI when should_publish is true (merged to main/main-dev) - Keeps publish-to-pypi job conditional on should_publish flag
- Change from --extra-index-url to --index-url (only query PyTorch repo) - Remove unnecessary torchvision and torchaudio (CMake build only needs torch) - Matches working strategy in build-and-publish.yml - Fixes 3+ hour timeout caused by slow pypi.org queries
- Remove python3-pip to avoid system python3 interference - Use 'python3.11 -m ensurepip' to bootstrap pip for 3.11 - Add --user flag to install packages in user site-packages for python3.11 - Use --index-url (not --extra-index-url) to avoid PyPI timeout - Fixes torch installing to python3.12 path instead of python3.11
- Change from Python 3.11 to 3.12 across all steps - Install python3.12 and python3.12-dev from apt - Use python3.12 for all pip installations and CMake configuration - PyTorch 2.2.0 has full cp312 wheel support - Demonstrates libamm can build on Python 3.12 when all components are aligned
- Remove actions/setup-python step (runner already has Python 3.12) - Remove explicit python3.12/python3.12-dev installation - Use system 'python3' and 'python3-dev' packages - Remove -DPython_EXECUTABLE flag (CMake auto-detects system python3) - Cleaner, faster workflow relying on runner's default Python
- Change from 'sage.libs.libamm.python' to 'PyAMM' - setup.py defines package name as 'PyAMM' (not sage namespace) - CMakeLists.txt creates PyAMM.so module - Fixes ModuleNotFoundError in wheel installation test
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.