feat: Add OpenAI-compatible embedding API support

[CZH-THU]

Add OpenAI-Compatible Embedding API Support

Overview

This PR introduces support for OpenAI-compatible embedding APIs to SkillClaw, enabling users to leverage external embedding services (Jina, Azure OpenAI, Ollama, LocalAI, etc.) alongside the existing local embedding model support.

Motivation

The current SkillClaw implementation only supports local embedding models (SentenceTransformer). This PR adds flexibility by:

• Supporting any service that implements the OpenAI embedding API specification
• Enabling users to choose between local and remote embedding services
• Reducing on-device resource requirements when using cloud-based APIs
• Supporting multi-language embeddings through services like Jina

Changes

New Files

• skillclaw/embedding_api_client.py - Complete OpenAI-compatible embedding API client

  • Supports any OpenAI-compatible API (Jina, Azure OpenAI, Ollama, LocalAI, etc.)
  • L2 normalization support
  • Automatic request retry and error handling
  • Lazy-loaded requests session for efficiency

• tests/test_embedding_api.py - Comprehensive test suite (9 tests + 1 integration test)

  • Basic encoding tests
  • Authorization and authentication tests
  • Response ordering and normalization tests
  • Error handling tests
  • Large batch processing tests
  • 100% pass rate

• scripts/demo_embedding_api.py - Usage demo and examples

Modified Files

• skillclaw/config.py - Added embedding API configuration fields

  • embedding_type: "local" or "api"
  • embedding_api_url: API endpoint URL
  • embedding_api_model: Model name
  • embedding_api_key: Optional API key

• skillclaw/skill_manager.py - Extended to support both embedding modes

  • Auto-detects embedding type from config
  • Maintains backward compatibility with local embeddings
  • Seamless switching between local and API modes

• skillclaw/launcher.py - Passes embedding API configuration to SkillManager

• pyproject.toml - Updated dependencies

  • Added requests for API calls
  • Added responses for testing

• requirements.txt - Updated with new dependencies

Features

1. Flexible Configuration

# Use local embeddings
config = SkillClawConfig(embedding_type="local")

# Use Jina API
config = SkillClawConfig(
    embedding_type="api",
    embedding_api_url="https://api.jina.ai/v1",
    embedding_api_model="jina-embeddings-v5-text-small",
    embedding_api_key="your-jina-key"
)

# Use Azure OpenAI
config = SkillClawConfig(
    embedding_type="api",
    embedding_api_url="https://your-resource.openai.azure.com/openai/deployments/your-deployment/",
    embedding_api_model="text-embedding-ada-002",
    embedding_api_key="your-azure-key"
)

### 2. Backward Compatibility
- Existing code using local embeddings continues to work without changes
- Default behavior remains unchanged
- Opt-in to API support

### 3. API Compatibility
Supports any service implementing OpenAI embedding API format:
- OpenAI (https://api.openai.com/v1/embeddings)
- Jina (https://api.jina.ai/v1/embeddings)
- Azure OpenAI
- LocalAI
- Ollama (with OpenAI-compatible server)
- Hugging Face Inference API
- Any custom OpenAI-compatible service

## Testing

### Test Coverage

tests/test_embedding_api.py::TestEmbeddingAPIClient::test_encode_basic PASSED
tests/test_embedding_api.py::TestEmbeddingAPIClient::test_encode_with_normalization PASSED
tests/test_embedding_api.py::TestEmbeddingAPIClient::test_encode_empty_input PASSED
tests/test_embedding_api.py::TestEmbeddingAPIClient::test_encode_with_authorization PASSED
tests/test_embedding_api.py::TestEmbeddingAPIClient::test_encode_without_api_key PASSED
tests/test_embedding_api.py::TestEmbeddingAPIClient::test_encode_out_of_order_responses PASSED
tests/test_embedding_api.py::TestEmbeddingAPIClient::test_api_error_handling PASSED
tests/test_embedding_api.py::TestEmbeddingAPIClient::test_invalid_response_format PASSED
tests/test_embedding_api.py::TestEmbeddingAPIClient::test_large_batch PASSED

**Result: 9 passed, 1 skipped in 0.55s**

Run tests:
```bash
pytest tests/test_embedding_api.py -v

Usage Example

Basic Usage with Jina API

from skillclaw.skill_manager import SkillManager

# Initialize with Jina API
manager = SkillManager(
    skills_dir="./skills",
    retrieval_mode="embedding",
    embedding_type="api",
    embedding_api_url="https://api.jina.ai/v1",
    embedding_api_model="jina-embeddings-v5-text-small",
    embedding_api_key="your-jina-key"
)

# Retrieve relevant skills
results = manager.retrieve("how to handle user input", top_k=5)

Configuration File

embedding:
  type: api
  api_url: https://api.jina.ai/v1
  api_model: jina-embeddings-v5-text-small
  api_key: ${JINA_API_KEY}

Benefits

1. Reduced Local Resources: No need to load large embedding models locally
2. Better Accuracy: Use specialized embedding models (e.g., domain-specific)
3. Multi-language Support: Leverage models like Jina that support 100+ languages
4. Cost Flexibility: Choose between local (zero cost) and cloud-based (per-API-call) solutions
5. Seamless Migration: Switch between local and cloud embedding without code changes

API Compatibility Notes

The implementation follows the OpenAI embedding API specification:

• Request format: {"model": "...", "input": [...]}
• Response format: {"data": [{"embedding": [...], "index": 0}, ...]}
• All responses are automatically normalized to L2 norm when requested

Breaking Changes

None - fully backward compatible with existing code.

Related Issues

Addresses the need for flexible embedding model support.

Checklist

• Code follows project style guidelines
• Comprehensive test coverage (9 tests)
• Documentation updated
• Backward compatible
• No breaking changes
• All tests passing