Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
358a5a9
feat: Isolate PyTorch dependency in Python bindings using NumPy inter…
ShuhaoZhangTony Nov 12, 2025
61f326b
docs: Add comprehensive dependency isolation documentation
ShuhaoZhangTony Nov 12, 2025
eb20f18
feat: Add PyTorch dependency declaration in setup.py
ShuhaoZhangTony Nov 12, 2025
a5bc63e
chore: Remove large binary files from working tree
ShuhaoZhangTony Nov 12, 2025
81d6e80
feat: Implement SOTA data management with Git LFS + symlinks
ShuhaoZhangTony Nov 12, 2025
b2f1335
chore: Replace test data files with symlinks to sageData
ShuhaoZhangTony Nov 12, 2025
e684436
build: use system pybind11
ShuhaoZhangTony Nov 12, 2025
bea754f
fix: 防止 setup_data.sh 创建嵌套符号链接
ShuhaoZhangTony Nov 12, 2025
69bba8e
docs: add submodule marker file
ShuhaoZhangTony Nov 12, 2025
859c637
docs: fix trailing whitespace in SUBMODULE.md
ShuhaoZhangTony Nov 12, 2025
fee30de
fix: use relative paths for test data symlinks
KimmoZAG Nov 13, 2025
92439b7
fix: don't overwrite Git-tracked symlinks in setup_data.sh
KimmoZAG Nov 14, 2025
e07bc6f
perf: 降低编译优化级别以减少内存消耗
ShuhaoZhangTony Nov 12, 2025
211aa77
feat: optimize LibAMM compilation for low-memory environments
ShuhaoZhangTony Nov 14, 2025
8fcbc1f
feat: add automated build, test and publish workflow
ShuhaoZhangTony Nov 14, 2025
45968a9
Fix LibAMM build deps and CI
ShuhaoZhangTony Nov 15, 2025
60bde68
Make LibAMM CI apt installs non-interactive
ShuhaoZhangTony Nov 15, 2025
b631589
Fix LibAMM CI: remove sudo from pip, use dynamic torch path
ShuhaoZhangTony Nov 15, 2025
5754e90
Add pip timeout and retry for torch downloads
ShuhaoZhangTony Nov 15, 2025
60f6e3e
Add retry and timeout for PyTorch pip installs
ShuhaoZhangTony Nov 16, 2025
4492ed2
Fix pip hash mismatch: add --no-cache-dir flag
ShuhaoZhangTony Nov 16, 2025
0494542
Enable unit tests in CI workflows
ShuhaoZhangTony Nov 16, 2025
55b5ba6
fix: update torch version from 1.13.0 to 2.2.0 to resolve CI/CD error
ShuhaoZhangTony Nov 16, 2025
b878f58
chore: pin python 3.11 for libamm CI torch install\n\n- add actions/s…
ShuhaoZhangTony Nov 17, 2025
96cb406
fix: use python3.11 consistently in libamm CI build
ShuhaoZhangTony Nov 17, 2025
5461e22
ci: enable wheel build testing on PR without publishing
ShuhaoZhangTony Nov 17, 2025
aa90725
fix: use --index-url for torch install to avoid PyPI timeout
ShuhaoZhangTony Nov 17, 2025
6c274ee
fix: ensure Python 3.11 is used consistently in cmake workflow
ShuhaoZhangTony Nov 17, 2025
daa4526
feat: upgrade cmake workflow to use Python 3.12
ShuhaoZhangTony Nov 17, 2025
a3ef242
refactor: simplify cmake workflow to use system Python
ShuhaoZhangTony Nov 17, 2025
2d4d799
fix: correct wheel import test to use PyAMM module name
ShuhaoZhangTony Nov 17, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
170 changes: 170 additions & 0 deletions .github/workflows/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,170 @@
# LibAMM Build and Publish Workflow

## 架构说明

LibAMM 是一个独立的 C++ 库,作为 SAGE 的 submodule。它有自己的构建、测试和发布流程。

### 仓库关系

```
intellistream/SAGE (主仓库)
└── packages/sage-libs/
└── src/sage/libs/
└── libamm/ (submodule -> intellistream/LibAMM)
```

### 发布流程

1. **LibAMM 仓库** (`intellistream/LibAMM`)
- 维护源代码和 C++ 测试
- 构建预编译的 Python wheels
- 发布到 PyPI 作为 `isage-libamm`

2. **SAGE 主仓库** (`intellistream/SAGE`)
- 将 `isage-libamm` 声明为可选依赖
- 用户安装 `isage-libs[amm]` 时自动从 PyPI 获取

## Workflow 说明

### 触发条件

- **自动触发**: Push 到 `main` 或 `main-dev` 分支(源代码变更时)
- **PR 检查**: 每个 PR 都会运行构建和测试,但不发布
- **手动触发**: 可选择目标仓库(PyPI/TestPyPI)

### 工作流程

```
1. check-version
├─ 读取 setup.py 中的版本号
├─ 对比最新的 Git tag
└─ 决定是否需要发布

2. cpp-build-test
├─ 配置 CMake
├─ 编译 C++ 库
└─ 运行 6 个 C++ 测试套件
✅ 所有测试必须通过才继续

3. build-wheels (parallel)
├─ Python 3.9
├─ Python 3.10
├─ Python 3.11
└─ Python 3.12
每个版本:
├─ 构建 wheel
└─ 测试安装和 import

4. publish-to-pypi
├─ 收集所有 wheels
├─ 发布到 PyPI/TestPyPI
├─ 创建 Git tag (vX.Y.Z)
└─ 创建 GitHub Release

5. summary
└─ 生成构建摘要
```

### C++ 测试套件

必须全部通过的测试:
- `cpp_test` - 基础功能测试
- `sketch_test` - Sketch 算法测试
- `crs_test` - 稀疏矩阵测试
- `weighted_cr_test` - 加权测试
- `block_partition_test` - 分块测试
- `streaming_test` - 流式处理测试

## 版本管理

### 手动更新版本

在 `setup.py` 中修改版本号:

```python
setup(
name='PyAMM', # 会被 workflow 自动改为 isage-libamm
version='0.1.5', # 👈 修改这里
...
)
```

### 发布规则

- **版本变化 + Push** → 自动发布
- **版本未变化** → 跳过发布(仅运行测试)
- **main 分支** → PyPI(生产环境)
- **main-dev 分支** → TestPyPI(测试环境)

### 示例:发布新版本

1. 修改代码
2. 更新 `setup.py` 中的版本号(如 `0.1.5` → `0.1.6`)
3. Commit 并 push 到 `main-dev`
4. Workflow 自动:
- ✅ 运行 C++ 测试
- ✅ 构建 4 个 Python wheels
- ✅ 发布到 TestPyPI
5. 测试通过后,merge 到 `main`
6. 自动发布到生产 PyPI

## 在 SAGE 中使用

### 安装 LibAMM

```bash
# 安装 sage-libs with LibAMM
pip install isage-libs[amm]

# 或单独安装 LibAMM
pip install isage-libamm
```

### 导入使用

```python
from sage.libs.libamm.python import PyAMM

# LibAMM 功能...
```

## 配置要求

### GitHub Secrets (LibAMM 仓库)

需要在 **LibAMM 仓库**中配置:

1. **PYPI_API_TOKEN** - PyPI API token
2. **TEST_PYPI_API_TOKEN** - TestPyPI API token
3. **PAT_TOKEN** - Personal Access Token (用于创建 tags)

### Self-hosted Runner

- **标签**: `self-hosted`, `Linux`, `X64`
- **内存**: ≥16GB(推荐 32GB)
- **CPU**: 多核(推荐 8+ 核)

## 故障排除

### C++ 测试失败

检查测试日志,定位失败的测试用例。所有测试必须通过才能发布。

### Wheel 构建失败(OOM)

增加 runner 内存或降低并行度:
```yaml
CMAKE_BUILD_PARALLEL_LEVEL: '1' # 改为 1
```

### PyPI 发布失败

1. 检查版本号是否已经存在(PyPI 不允许重复上传)
2. 验证 API token 权限
3. 首次发布需要先在 PyPI 创建项目

## 参考链接

- **LibAMM 源码**: https://github.com/intellistream/LibAMM
- **PyPI 包**: https://pypi.org/project/isage-libamm/
- **SAGE 文档**: https://intellistream.github.io/SAGE-Pub/
Loading