-
+
- APIPod is the way for building and deploying AI services.
- Combining the developer experience of FastAPI with the power of Serverless GPU computing.
+ APIPod combines the developer experience of FastAPI with the power of Serverless GPU computing.
+ Write your service like FastAPI. Run it anywhere with a single command.
+ Think Vercel — but for AI services.
Why APIPod • Installation • Quick Start • - Deployment + Develop & Test • + Build & Deploy
--- @@ -25,14 +27,16 @@ Building AI services is complex: file handling, long-running inference, job queues, deployment, scaling, and hosting provider choices all create friction at every step. -**APIPod** solves this by standardizing the entire stack. +**APIPod** eliminates that friction. It abstracts away the AI infrastructure stack so you can focus on your model. You write the service; [Socaity](https://www.socaity.ai) handles the deployment and scaling across any cloud. -### 🚀 Highlights -1. **Write Powerful APIs Instantly**: Built on top of FastAPI, it feels familiar but comes with batteries included for AI services. -2. **Standardized I/O**: Painless handling of Images, Audio, and Video via [MediaToolkit](https://github.com/SocAIty/media-toolkit). -3. **Automatic packaging**: The package can configure docker and deployment for you. No Cuda hell; the package knows compatible options. -4. **Streamlined Deployment**: Deploy as a standard container or to serverless providers (**Socaity.ai** or **RunPod**) with zero configuration changes. Auth included. -5. **Native SDK**: Built-in support for **Asynchronous Job Queues**, polling & and progress tracking via [fastSDK](https://github.com/SocAIty/fastSDK) +### Highlights + +1. **Write once, run anywhere** — the same code runs in development, in serverless emulation, or on a real GPU cloud. Zero changes between environments. +2. **Drop-in FastAPI** — if you know FastAPI, you already know APIPod. Built on top of it, with batteries included for AI. +3. **Standardized I/O** — painless Images, Audio and Video via [media-toolkit](https://github.com/SocAIty/media-toolkit). +4. **OpenAI-compatible schemas** — built-in request/response schemas for chat, completions, embeddings, TTS, transcription, image/video generation. OpenAI clients work out of the box. +5. **Built-in job queue** — async jobs, polling and progress tracking, with no Celery/Redis/Kubernetes to wire up. +6. **One-command packaging** — `apipod build` generates your Dockerfile. No CUDA hell; APIPod picks compatible images. ## Installation @@ -42,46 +46,41 @@ pip install apipod ## Quick Start -### 1. Create your Service - -Zero-Hassle Migration: Replacing `FastAPI` with `APIPod` gives you instant access to all APIPod capablities.. +`APIPod` is a drop-in replacement for `FastAPI`. You get all of APIPod's capabilities with no migration cost. ```python from apipod import APIPod, ImageFile -# 1. Initialize APIPod (Drop-in replacement for FastAPI) +# Drop-in replacement for FastAPI app = APIPod() -# 2. Define a standard endpoint (Synchronous) +# A standard endpoint @app.endpoint("/hello") def hello(name: str): return f"Hello {name}!" -# 2. Use built-in media processing -@app.endpoint("/process_image", queue_size=10) +# Built-in media processing — uploads/URLs/base64 are parsed for you +@app.endpoint("/process-image") def process_image(image: ImageFile): - # APIPod handles the file upload/parsing automatically img_array = image.to_np_array() - # ... run your AI model here ... - return ImageFile().from_np_array(img_array) -# 4. Run the server if __name__ == "__main__": app.start() ``` -### 2. Run Locally +Run it and open `http://localhost:8000/docs` for the auto-generated Swagger UI. + ```bash python main.py +# or +apipod start ``` -Visit `http://localhost:8000/docs` to see your auto-generated Swagger UI. -## Features in Depth +## Smart File Handling -### 📁 Smart File Handling -Forget about parsing `multipart/form-data`, `base64`, or `bytes`. APIPod integrates with **MediaToolkit** to handle files as objects. +Forget about parsing `multipart/form-data`, `base64`, or `bytes`. APIPod integrates with **MediaToolkit** to handle files as objects. Whether the client sends a file upload, a URL, or a base64 string, your endpoint receives a ready-to-use object. ```python from apipod import AudioFile @@ -93,216 +92,146 @@ def transcribe(audio: AudioFile): return {"transcription": "..."} ``` -### ☁️ Serverless Routing -When deploying to serverless platforms like **RunPod**, standard web frameworks often fail because they lack the necessary routing logic for the platform's specific entry points. **APIPod** detects the environment and handles the routing automatically—no separate "handler" function required. - - -### 🔄 Scaling services, Asynchronous Jobs, Polling and Job Progress - -Let's say you want to serve your service to many users or you have a long-running task. -Usually you need to set-up a load-balancer, kubernetes, brokers and a lot of other complicated stuff. -If you deploy to socaity / runpod this is taken care of for you. No Dev-Ops for you. - -We allow you to emulate this behaviour for testing. - -For long-running tasks (e.g., inference of a large model), you don't want to block the HTTP request. -Often you want to be able to give a progress bar or updates about the current task to the user. This is what job progress is for. -It allows you to communicate a progress percentage and a status message to your user. +## AI Services Streamlined (OpenAI-compatible) +APIPod provides built-in request/response schemas for common AI tasks (chat, TTS, image gen, etc.) that are fully OpenAI-compatible. This allows you to focus on the model logic while APIPod handles the boilerplate of validation, media parsing, and streaming. +```python +from apipod.common.schemas import ChatCompletionRequest + +@app.endpoint("/chat") +def chat(request: ChatCompletionRequest): + if request.stream: + # Yield plain tokens — APIPod wraps them into ChatCompletionChunk SSE events. + return my_llm.stream(request.messages) + return my_llm.generate(request.messages) # auto-wrapped into ChatCompletionResponse +``` -1. **Setup test environment for serverless (Job Queue)**: - ```python - # Initialize with serverless compute on localhost to enable the local job queue - app = APIPod(compute="serverless", provider="localhost") - ``` +## Asynchronous Jobs & Scaling -2. **Define Endpoint**: - Use `@app.endpoint` (or `@app.post`). It automatically becomes a background task when a queue is configured. - ```python - @app.post("/generate", queue_size=50) - def generate(job_progress: JobProgress, prompt: str): - job_progress.set_status(0.1, "Initializing model...") - # ... heavy computation ... - job_progress.set_status(1.0, "Done!") - return "Generation Complete" - ``` +For long-running tasks, APIPod provides a built-in job queue and progress reporting. When configured for serverless or with a queue, endpoints automatically return a `job_id` and run in the background. - * **Client:** Receives a `job_id` immediately. - * **Server:** Processes the task in the background. - * **[SDK](https://github.com/SocAIty/fastSDK):** Automatically polls for status and result. +```python +from apipod import JobProgress + +@app.post("/generate", queue_size=50) +def generate(job_progress: JobProgress, prompt: str): + job_progress.set_status(0.1, "Initializing model...") + # ... heavy computation ... + job_progress.set_status(1.0, "Done!") + return "Generation Complete" +``` -3. **Opt-out**: - If you want a standard synchronous endpoint even when queue is enabled: - ```python - @app.endpoint("/ping", use_queue=False) - def ping(): - return "pong" - ``` +* **Client:** Receives a `job_id` immediately. +* **Server:** Processes the task in the background. +* **SDK:** Automatically polls for status and result. +## Develop, Test, and Simulate -# Deployment -APIPod is designed to run anywhere by leveraging docker. -- Build & configure • - Deploy -
+Just say *how you want to run the service right now*. -## Create & configure container +### Development (default) -All you need to do is run: +Plain FastAPI. The fastest iteration loop. ```bash -apipod build +apipod start +# or simply +python main.py ``` -This command creates the dockerfile for you, and select the correct docker template and cuda/cudnn versions and comes with ffmpeg installed. - -- For most users this already creates a sufficient solution. -- However you are always free to create or customize the Dockerfile for your needs. -Requirements: -1. docker installed on your system. -2. Depending on your setup a cuda/cudnn installation +### Simulate a deployment +Before you ship, run your service exactly how it will behave in production — locally, with **no code changes**. `apipod simulate` takes an optional target string `{compute}-{provider}` (compute defaults to `serverless`). -### APIPod Configuration -APIPod provides a flexible deployment configuration that allows developers to: -- Run services locally for development -- Deploy via the Socaity orchestration platform -- Deploy directly to cloud providers -- Choose between serverless or dedicated compute - -The configuration is controlled through a combination of: -- orchestrator -- compute -- provider -- region -- CPU/GPU - - -| Orchestrator | Compute | Provider | Resulting Backend | -| ---------------- | ------------ | ----------- | --------------------------------- | -| `socaity` | `dedicated` | `auto` | FastAPI | -| `socaity` | `dedicated` | `localhost` | FastAPI + job queue *(test mode)* | -| `socaity` | `dedicated` | `runpod` | Celery backend *(planned)* | -| `socaity` | `dedicated` | `scaleway` | Celery backend *(planned)* | -| `socaity` | `dedicated` | `azure` | Celery backend *(planned)* | -| `socaity` | `serverless` | `auto` | RunPod router backend | -| `socaity` | `serverless` | `localhost` | FastAPI + job queue *(test mode)* | -| `socaity` | `serverless` | `runpod` | RunPod router backend | -| `socaity` | `serverless` | `scaleway` | ❌ Not supported | -| `socaity` | `serverless` | `azure` | ❌ Not supported | -| `local` / `None` | `dedicated` | `localhost` | FastAPI | -| `local` / `None` | `dedicated` | `runpod` | FastAPI | -| `local` / `None` | `dedicated` | `scaleway` | FastAPI | -| `local` / `None` | `dedicated` | `azure` | FastAPI | -| `local` / `None` | `serverless` | `localhost` | FastAPI + job queue | -| `local` / `None` | `serverless` | `runpod` | RunPod router backend | -| `local` / `None` | `serverless` | `scaleway` | ❌ Not supported | -| `local` / `None` | `serverless` | `azure` | ❌ Not supported | -| `local` / `None` | `localhost` | `localhost` | FastAPI | +```bash +apipod simulate # serverless emulation: FastAPI + local job queue +apipod simulate serverless # same as above +apipod simulate dedicated # plain FastAPI (dedicated compute) +apipod simulate serverless-runpod # emulate Socaity routing requests to RunPod +apipod simulate dedicated-azure # emulate a dedicated Azure deployment +``` +If a provider has no serverless offering, APIPod warns and falls back to the job-queue emulation: +```bash +apipod simulate serverless-azure +# Warning: azure does not support serverless. Defaulting to FastAPI + Local Job Queue. +``` +### Emulate a provider's native worker (`--native`) -### 🔄 Queue Backend Support +By default Socaity is the orchestrator. `--native` skips Socaity and runs the provider's **own** serverless backend locally — e.g. RunPod's serverless worker (requires the `runpod` package): -APIPod supports multiple job queue backends to handle different deployment scenarios and scaling needs. +```bash +apipod simulate serverless-runpod --native +``` -#### Available Backends +### Configure from Python -- **None** (default): Standard FastAPI behavior. No background jobs. - -- **Local Queue** (`local`): In-memory job queue using threading. - - Perfect for local development and single-instance deployments - - No external dependencies required +The same intent can be set in code. Socaity **overrides** it with env vars once the service is actually managed by the platform, so what you test is what you ship. -#### Configuration +```python +app = APIPod() # development (plain FastAPI) +app = APIPod(simulate="serverless") # FastAPI + local job queue +app = APIPod(simulate="serverless-runpod", direct=True) # RunPod native worker (local) +``` - ```python - # Job queues are automatically enabled based on your configuration. - # For example, serverless + localhost enables a local job queue for testing: - app = APIPod(compute="serverless", provider="localhost") +## Build & Deploy - # Or via environment variables - import os - os.environ["APIPOD_COMPUTE"] = "serverless" - os.environ["APIPOD_PROVIDER"] = "localhost" +### Build a container - app = APIPod() # Uses environment config - ``` +```bash +apipod build +``` -### Troubleshooting +This scans your project, picks a compatible base image (CUDA/cuDNN, ffmpeg included) and generates a `Dockerfile`. For most users this is all you need; advanced users can edit or write their own Dockerfile. -You are always free to create or edit the Dockerfile for your needs. -Depending on your OS, your machine or your project setup you might occur one of those issues: -- Build scripts fails -- You can't build the docker container. +Requirements: Docker installed, plus a CUDA/cuDNN setup if your model needs the GPU. -In this cases don't -Advanced users can also configure or write the docker file for themselves +### Deploy -## Deploy to socaity -Right after build you can deploy the service via the [socaity.ai](https://www.socaity.ai) dashboard. -This is the simplest option. +```bash +apipod deploy # coming soon +apipod deploy serverless +apipod deploy dedicated-azure +``` -## Deploy to runpod. -1. You will need to build the your docker image. -2. Push your image to your dockerhub repository. -3. Deploy on RunPod Serverless by using the runpod dashboard. - * *APIPod acts as the handler, managing job inputs/outputs compatible with RunPod's API.* +The managed `deploy` command is on the roadmap. Today, build your container and deploy it through the [Socaity dashboard](https://www.socaity.ai) — the simplest path, with auth, scaling and routing handled for you. -Make sure that the environment variables are set to the following: ```APIPOD_COMPUTE="serverless"``` and ```APIPOD_PROVIDER="runpod"``` +## Client SDK +Generate a typed client for your service using the [fastSDK](https://github.com/SocAIty/fastSDK). It handles authentication, file uploads, and automatic polling for background jobs. -## Debugging APIPod serverless -You can configure your environment variables so that APIPod acts as if it were deployed on socaity.ai or on runpod. ```bash -# Orchestrator -ENV APIPOD_ORCHESTRATOR="local" # Options: "local" (default), "socaity" - -# Compute type -ENV APIPOD_COMPUTE="serverless" # Options: "dedicated" (default), "serverless" - -# Infrastructure provider -ENV APIPOD_PROVIDER="runpod" # Options: "localhost" (default), "auto", "runpod", "scaleway", "azure" +fastsdk generate http://localhost:8009 -o myClient.py ``` - -# Client SDK - -While you can use `curl` or `requests`, our [FastSDK](https://github.com/SocAIty/fastSDK) makes interacting with APIPod services feel like calling native Python functions. - ```python -# The SDK handles authentication, file uploads, and result polling -# create a full working client stub -create_sdk("https://localhost:8000", save_path="my_service.py") - -# Import the client. It will have a method for each of your service endpoints including all parameters and its default values. -from my_service import awesome_client -mySDK = awesome_client() -mySDK.my_method(...) +from myClient import myService -# Blocks until the remote job is finished -result = task.get_result() +client = myService() +client.text_to_speech("what a time to be alive") ``` -# Comparison +## Comparison | Feature | APIPod | FastAPI | Celery | Replicate/Cog | | :--- | :---: | :---: | :---: | :---: | -| **Setup Difficulty** | ⭐ Easy | ⭐ Easy | ⭐⭐⭐ Hard | ⭐⭐ Medium | +| **Setup Difficulty** | Easy | Easy | Hard | Medium | | **Async/Job Queue** | ✅ Built-in | ❌ Manual | ✅ Native | ✅ Native | | **Serverless Ready** | ✅ Native | ❌ Manual | ❌ No | ✅ Native | | **File Handling** | ✅ Standardized | ⚠️ Manual | ❌ Manual | ❌ Manual | | **Router Support** | ✅ | ✅ | ❌ | ❌ | +| **Multi-cloud** | ✅ | ❌ | ❌ | ❌ | ## Roadmap + +- `apipod deploy` managed deployment command. - MCP protocol support. -- OpenAI-compatible default endpoints for LLMs -- Improve async support. --- +- Made with ❤️ by SocAIty + Made with ❤️ by SocAIty
diff --git a/apipod/__init__.py b/apipod/__init__.py index 402e0d9..00f8bc9 100644 --- a/apipod/__init__.py +++ b/apipod/__init__.py @@ -1,7 +1,9 @@ from apipod.api import APIPod from apipod.engine.jobs.base_job import BaseJob, LocalJob from apipod.engine.jobs.job_progress import JobProgress -from apipod.engine.jobs.job_result import FileModel, JobLinks, JobMetrics, JobResult +from socaity_schemas import FileModel +from apipod.engine.jobs.job_result import JobLinks, JobMetrics, JobResult +from apipod.engine.streaming import StreamStore, LocalStreamStore, StreamProducer from media_toolkit import MediaFile, ImageFile, AudioFile, VideoFile, MediaList, MediaDict from apipod.common import constants @@ -25,6 +27,9 @@ "JobLinks", "JobMetrics", "JobResult", + "StreamStore", + "LocalStreamStore", + "StreamProducer", "MediaFile", "ImageFile", "AudioFile", diff --git a/apipod/api.py b/apipod/api.py index 2bd0b81..287f7d2 100644 --- a/apipod/api.py +++ b/apipod/api.py @@ -1,148 +1,132 @@ -from apipod.common import constants -from apipod.common.settings import APIPOD_ORCHESTRATOR, APIPOD_COMPUTE, APIPOD_PROVIDER -from apipod.engine.base_backend import _BaseBackend -from apipod.engine.backend.runpod.router import SocaityRunpodRouter +from typing import Optional, Tuple, Union + +from apipod.common.constants import COMPUTE, PROVIDER +from apipod.common.settings import ( + APIPOD_COMPUTE, + APIPOD_NATIVE, + APIPOD_PROVIDER, + APIPOD_SIMULATE, + IS_MANAGED_DEPLOYMENT, +) from apipod.engine.backend.fastapi.router import SocaityFastAPIRouter -from apipod.engine.queue.job_queue_interface import JobQueueInterface +from apipod.engine.backend.runpod.router import SocaityRunpodRouter +from apipod.engine.streaming.local_stream_store import LocalStreamStore + -from typing import Union +# (backend_class, use_job_queue, runpod_simulate) +_Resolution = Tuple[type, bool, bool] + +_NO_SERVERLESS = (PROVIDER.AZURE, PROVIDER.SCALEWAY) def APIPod( - orchestrator: Union[constants.ORCHESTRATOR, str, None] = None, - compute: Union[constants.COMPUTE, str, None] = None, - provider: Union[constants.PROVIDER, str, None] = None, + simulate: Union[str, None] = None, + direct: Union[bool, None] = None, *args, **kwargs -) -> Union[_BaseBackend, SocaityRunpodRouter, SocaityFastAPIRouter]: +) -> Union[SocaityFastAPIRouter, SocaityRunpodRouter]: """ - Initialize an APIPod router with the appropriate backend based on the deployment configuration. - - The resulting backend is determined by the combination of orchestrator, compute, and provider: - - | Orchestrator | Compute | Provider | Backend | - |------------- |----------- |---------- |--------------------------- | - | socaity | dedicated | auto | FastAPI | - | socaity | dedicated | localhost | FastAPI + job queue (test) | - | socaity | dedicated | socaity | FastAPI + redis (prod) | - | socaity | dedicated | runpod | Celery (planned) | - | socaity | dedicated | scaleway | Celery (planned) | - | socaity | dedicated | azure | Celery (planned) | - | socaity | serverless | auto | RunPod router | - | socaity | serverless | localhost | FastAPI + job queue (test) | - | socaity | serverless | runpod | RunPod router | - | socaity | serverless | scaleway | Not supported | - | socaity | serverless | azure | Not supported | - | local/None | dedicated | * | FastAPI | - | local/None | serverless | localhost | FastAPI + job queue | - | local/None | serverless | runpod | RunPod router | - | local/None | serverless | scaleway | Not supported | - | local/None | serverless | azure | Not supported | + Build the right backend for an APIPod service from a single *intent*. + + Socaity is the implicit orchestrator, so you never wire infrastructure by hand. + You only pick how the service should run *locally*: + + - **Development** (default, ``APIPod()``): plain FastAPI — the fastest loop. + - **Simulation** (``simulate="{compute}-{provider}"``): emulate a deployment + locally. The target collapses compute + provider, e.g. ``"serverless"``, + ``"serverless-runpod"``, ``"dedicated-azure"``. Compute defaults to + ``serverless``. ``direct=True`` bypasses Socaity to emulate the provider's + own serverless worker (currently RunPod). + + In a **managed deployment** (``SOCAITY_DEPLOYMENT_CERT`` verified) ``simulate`` + and ``direct`` are ignored: Socaity injects ``APIPOD_COMPUTE`` / + ``APIPOD_PROVIDER`` and the real backend is selected from them. Args: - orchestrator: "socaity" or "local" (default from env / local). - compute: "dedicated" or "serverless" (default from env / dedicated). - provider: "auto", "localhost", "socaity", "runpod", "scaleway", "azure" (default from env / localhost). + simulate: deployment target to emulate, ``"{compute}-{provider}"``. + ``None`` runs plain FastAPI for development. + direct: emulate the provider's native serverless worker instead of the + Socaity job-queue emulation. Only affects ``serverless-runpod``. """ - orchestrator = _resolve_enum(orchestrator, constants.ORCHESTRATOR, APIPOD_ORCHESTRATOR, constants.ORCHESTRATOR.LOCAL) - compute = _resolve_enum(compute, constants.COMPUTE, APIPOD_COMPUTE, constants.COMPUTE.DEDICATED) - provider = _resolve_enum(provider, constants.PROVIDER, APIPOD_PROVIDER, constants.PROVIDER.LOCALHOST) + if IS_MANAGED_DEPLOYMENT: + backend_class, use_job_queue, runpod_simulate = _resolve_managed() + else: + backend_class, use_job_queue, runpod_simulate = _resolve_intent(simulate, direct) - backend_class, use_job_queue = _resolve_backend(orchestrator, compute, provider) + if backend_class is SocaityRunpodRouter: + return SocaityRunpodRouter(simulate=runpod_simulate, *args, **kwargs) - custom_job_queue = kwargs.pop("job_queue", None) - if custom_job_queue: + # FastAPI backend: a job queue (+ stream store) turns it into the serverless + # emulation; without one it is plain FastAPI. A deployment may inject its own. + job_queue = kwargs.pop("job_queue", None) + if job_queue is not None: use_job_queue = True - job_queue = custom_job_queue - else: - job_queue = _create_job_queue() if use_job_queue else None + elif use_job_queue: + from apipod.engine.queue.job_queue import JobQueue + job_queue = JobQueue() - if backend_class == SocaityFastAPIRouter: - return backend_class(job_queue=job_queue, *args, **kwargs) - else: - return backend_class(*args, **kwargs) - - -def _resolve_enum(value, enum_cls, env_default, fallback): - """Coerce a value into an enum member, falling back through env default and hard default.""" - if value is None: - value = env_default - if isinstance(value, str): - try: - return enum_cls(value) - except ValueError: - raise ValueError(f"Invalid {enum_cls.__name__} value: '{value}'. Choose from: {[e.value for e in enum_cls]}") - if isinstance(value, enum_cls): - return value - return fallback - - -def _resolve_backend( - orchestrator: constants.ORCHESTRATOR, - compute: constants.COMPUTE, - provider: constants.PROVIDER, -) -> tuple: - """ - Apply the configuration matrix and return (backend_class, use_job_queue). - Raises for unsupported or not-yet-implemented combinations. - """ - _raise_if_unsupported(compute, provider) - - if orchestrator == constants.ORCHESTRATOR.SOCAITY: - return _resolve_socaity(compute, provider) + if use_job_queue and "stream_store" not in kwargs: + + kwargs["stream_store"] = LocalStreamStore() - return _resolve_local(compute, provider) + return SocaityFastAPIRouter(job_queue=job_queue, *args, **kwargs) -def _raise_if_unsupported(compute: constants.COMPUTE, provider: constants.PROVIDER): - unsupported = { - (constants.COMPUTE.SERVERLESS, constants.PROVIDER.SCALEWAY), - (constants.COMPUTE.SERVERLESS, constants.PROVIDER.AZURE), - } - if (compute, provider) in unsupported: - raise NotImplementedError( - f"Serverless compute on {provider.value} is not supported. " - f"Use provider='runpod' for serverless or switch to dedicated compute." - ) +def _resolve_intent(simulate: Optional[str], direct: Optional[bool]) -> _Resolution: + """Resolve a local run (development or simulation) into a backend selection.""" + target = APIPOD_SIMULATE if simulate is None else simulate + # Development: no simulation requested -> plain FastAPI. + if simulate is None and not APIPOD_SIMULATE: + return SocaityFastAPIRouter, False, False -def _resolve_socaity(compute: constants.COMPUTE, provider: constants.PROVIDER) -> tuple: - if compute == constants.COMPUTE.DEDICATED: - if provider == constants.PROVIDER.SOCAITY: - return SocaityFastAPIRouter, True + direct = APIPOD_NATIVE if direct is None else bool(direct) + compute, provider = _parse_target(target) - if provider in (constants.PROVIDER.RUNPOD, constants.PROVIDER.SCALEWAY, constants.PROVIDER.AZURE): - raise NotImplementedError( - f"Celery backend for socaity + dedicated + {provider.value} is planned but not yet available." - ) - if provider == constants.PROVIDER.LOCALHOST: - return SocaityFastAPIRouter, True - # auto or any other -> FastAPI without queue - return SocaityFastAPIRouter, False + if compute is COMPUTE.DEDICATED: + # "Standard FastAPI"; with a named provider it emulates a direct client. + return SocaityFastAPIRouter, False, False # serverless - if provider == constants.PROVIDER.LOCALHOST: - return SocaityFastAPIRouter, True - # auto or runpod -> RunPod router - return SocaityRunpodRouter, False + if provider in _NO_SERVERLESS: + print(f"Warning: {provider.value} does not support serverless. " + f"Defaulting to FastAPI + Local Job Queue.") + return SocaityFastAPIRouter, True, False + if provider is PROVIDER.RUNPOD and direct: + # Emulate RunPod's native serverless worker locally. + return SocaityRunpodRouter, False, True -def _resolve_local(compute: constants.COMPUTE, provider: constants.PROVIDER) -> tuple: - if compute == constants.COMPUTE.DEDICATED: - return SocaityFastAPIRouter, False + # Default serverless: Socaity emulation = FastAPI + Local Job Queue. + return SocaityFastAPIRouter, True, False + + +def _resolve_managed() -> _Resolution: + """Pick the real production backend from the env vars Socaity injects.""" + compute = COMPUTE(APIPOD_COMPUTE) + provider = PROVIDER(APIPOD_PROVIDER) + + if compute is COMPUTE.SERVERLESS and provider is PROVIDER.RUNPOD: + return SocaityRunpodRouter, False, False # real serverless worker + if compute is COMPUTE.SERVERLESS: + return SocaityFastAPIRouter, True, False + return SocaityFastAPIRouter, False, False # dedicated (queue injected if needed) - # serverless - if provider == constants.PROVIDER.LOCALHOST: - return SocaityFastAPIRouter, True - if provider == constants.PROVIDER.RUNPOD: - return SocaityRunpodRouter, False - # auto -> RunPod router (same default as socaity serverless auto) - if provider == constants.PROVIDER.AUTO: - return SocaityRunpodRouter, False - raise NotImplementedError(f"Unsupported configuration: local + serverless + {provider.value}") +def _parse_target(target: str) -> Tuple[COMPUTE, Optional[PROVIDER]]: + """Parse a ``"{compute}-{provider}"`` target. Provider is optional; compute defaults to serverless.""" + if not target: + return COMPUTE.SERVERLESS, None + compute_str, _, provider_str = target.partition("-") + try: + compute = COMPUTE(compute_str) + except ValueError: + raise ValueError(f"Invalid compute '{compute_str}'. Choose from: {[c.value for c in COMPUTE]}") -def _create_job_queue() -> JobQueueInterface: - from apipod.engine.queue.job_queue import JobQueue + if not provider_str: + return compute, None + try: + return compute, PROVIDER(provider_str) + except ValueError: + raise ValueError(f"Invalid provider '{provider_str}'. Choose from: {[p.value for p in PROVIDER]}") - return JobQueue() diff --git a/apipod/cli.py b/apipod/cli.py index 936d1ca..34ccd88 100644 --- a/apipod/cli.py +++ b/apipod/cli.py @@ -1,10 +1,11 @@ import argparse +import importlib.util +import os import sys from pathlib import Path from typing import Optional from apipod.deploy.deployment_manager import DeploymentManager -from apipod.common.constants import ORCHESTRATOR, COMPUTE, PROVIDER def input_yes_no(question: str, default: bool = True) -> bool: @@ -67,7 +68,7 @@ def perform_scan(): else: print("Scanning project...") return manager.scan() - + if manager.config_exists: if not input_yes_no(f"Found {manager.config_path.name} in {manager.config_path.parent}/. Overwrite?"): return manager.load_config() @@ -83,7 +84,7 @@ def perform_scan(): return config_data -def run_scan(): +def run_scan(_args=None): """Scan the project and generate apipod.json configuration file.""" manager = DeploymentManager() @@ -99,43 +100,33 @@ def run_build(args): """Run the build process for creating a deployment-ready container.""" manager = DeploymentManager() - target_file = None - if args.build is not None and args.build is not True: - target_file = args.build - + target_file = args.file + if target_file: target_path = Path(target_file) if not target_path.exists(): print(f"Error: Target file '{target_file}' does not exist.") return - + if not target_path.is_file(): print(f"Error: '{target_file}' is not a file.") return - + if not target_path.suffix == '.py': print(f"Warning: '{target_file}' is not a Python file (.py)") if not input_yes_no("Continue anyway?", default=False): return - + print(f"Using target file: {target_file}") if manager.dockerfile_exists and not input_yes_no("Deployment config DOCKERFILE exists. Overwrite your deployment config?"): print("Aborting build configuration.") return - should_create_dockerfile = True - config_data = get_or_create_config(manager, target_file) if not config_data: print("Error: Failed to obtain configuration.") return - config_data["orchestrator"] = args.orchestrator - config_data["compute"] = args.compute - config_data["provider"] = args.provider - if args.region: - config_data["region"] = args.region - service_title = config_data.get("title", "apipod-service") final_image = select_base_image(manager, config_data) @@ -149,117 +140,279 @@ def run_build(args): print("Please configure dependencies and try again.") return - if should_create_dockerfile: - print("Generating Dockerfile...") - dockerfile_content = manager.render_dockerfile(final_image, config_data) - manager.write_dockerfile(dockerfile_content) + print("Generating Dockerfile...") + dockerfile_content = manager.render_dockerfile(final_image, config_data) + manager.write_dockerfile(dockerfile_content) if input_yes_no(f"Build the application now using docker? (Tag: {service_title})"): manager.build_docker_image(service_title) -def run_start(args): - """Start an APIPod service with the given configuration.""" - from apipod import APIPod +def _looks_like_entrypoint(value: str) -> bool: + """Return True when a simulate positional arg is likely a Python entrypoint file.""" + return value.endswith(".py") or Path(value).is_file() - app = APIPod( - orchestrator=args.orchestrator, - compute=args.compute, - provider=args.provider, - ) + +def _resolve_simulate_args(args) -> tuple[str, Optional[str]]: + """Split simulate positionals into deployment target and optional entrypoint.""" + if args.entrypoint is not None: + return args.target, args.entrypoint + + if args.target is None: + return "serverless", None + + if _looks_like_entrypoint(args.target): + return "serverless", args.target + + return args.target, None + + +def _resolve_entrypoint(manager: DeploymentManager, entrypoint: Optional[str]) -> str: + """Return the service entrypoint file, scanning the project when not provided.""" + if entrypoint: + return entrypoint + + config = manager.load_config() if manager.config_exists else None + if not config: + print("No apipod.json found. Scanning project to locate the entrypoint...") + config = manager.scan() + manager.save_config(config) + return config.get("entrypoint", "main.py") + + +def _load_app(entrypoint: str): + """Import the entrypoint module and return its APIPod application instance.""" + from apipod.engine.base_backend import _BaseBackend + + path = Path(entrypoint).resolve() + if not path.exists(): + raise FileNotFoundError(f"Entrypoint '{entrypoint}' not found.") + + spec = importlib.util.spec_from_file_location("apipod_entrypoint", path) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + + app = getattr(module, "app", None) + if isinstance(app, _BaseBackend): + return app + for value in vars(module).values(): # fall back to the first app in the module + if isinstance(value, _BaseBackend): + return value + raise RuntimeError(f"No APIPod app found in '{entrypoint}'. Expected an APIPod() instance.") + + +def _run_service(args, simulate: Optional[str], native: bool = False): + """Resolve the entrypoint, apply the run intent via env vars, and start the app.""" + # The user's service calls APIPod() with no args; it reads the intent from env. + # Set the env BEFORE importing the entrypoint (which imports apipod settings). + if simulate is not None: + os.environ["APIPOD_SIMULATE"] = simulate + if native: + os.environ["APIPOD_NATIVE"] = "true" + + manager = DeploymentManager() + entrypoint = _resolve_entrypoint(manager, args.entrypoint) + app = _load_app(entrypoint) port = args.port or 8000 host = args.host or "0.0.0.0" - print(f"Starting APIPod (orchestrator={args.orchestrator}, compute={args.compute}, provider={args.provider})") + if simulate is None: + mode = "development" + else: + mode = f"simulation '{simulate}'{' --native' if native else ''}" + print(f"Starting APIPod ({mode}) from {entrypoint}") app.start(port=port, host=host) -def main(): - """Main entry point for the APIPod CLI.""" - orchestrator_choices = [e.value for e in ORCHESTRATOR] - compute_choices = [e.value for e in COMPUTE] - provider_choices = [e.value for e in PROVIDER] +def run_start(args): + """Run the service locally for development (plain FastAPI).""" + _run_service(args, simulate=None) + + +def run_simulate(args): + """Run the service locally while emulating a deployment target.""" + target, entrypoint = _resolve_simulate_args(args) + args.entrypoint = entrypoint + _run_service(args, simulate=target, native=args.native) + + +def run_deploy(args): + """Placeholder for the upcoming managed deployment command.""" + target = args.target or "serverless" + print( + f"`apipod deploy {target}` is not available yet.\n" + "Deploy through the Socaity dashboard for now: https://www.socaity.ai\n" + f"Tip: validate the target locally first with `apipod simulate {target}`." + ) + + +def run_help(args, parsers: dict): + """Print top-level or command-specific help.""" + if args.command: + parser = parsers.get(args.command) + if parser is None: + print(f"Unknown command: {args.command}\n") + parsers["__root__"].print_help() + sys.exit(1) + parser.print_help() + else: + parsers["__root__"].print_help() + + +def _add_run_options(parser: argparse.ArgumentParser) -> None: + """Shared host/port and entrypoint options for start and simulate.""" + parser.add_argument( + "entrypoint", + nargs="?", + default=None, + metavar="ENTRYPOINT", + help="Service entrypoint file (auto-detected from apipod.json when omitted).", + ) + parser.add_argument("--host", default=None, help="Host to bind to (default: 0.0.0.0).") + parser.add_argument("--port", type=int, default=None, help="Port to bind to (default: 8000).") + +def _build_parser() -> tuple[argparse.ArgumentParser, dict]: + """Construct the CLI parser tree and return the root parser plus a lookup map.""" parser = argparse.ArgumentParser( - description="APIPod CLI - Build and deploy AI service containers", + description="APIPod CLI - build, simulate and deploy AI services", formatter_class=argparse.RawDescriptionHelpFormatter, epilog=""" Examples: - apipod --scan Scan project and generate apipod.json - apipod --build Build container using current directory - apipod --build ./main.py Build container with specific entry file - apipod --build --provider runpod Build for RunPod deployment - apipod --start Start service locally - apipod --start --compute serverless Start in serverless emulation mode - apipod --start --orchestrator socaity Start with Socaity orchestrator - """ + apipod help Show this overview + apipod start Run locally for development (FastAPI) + apipod start main.py Run a specific entrypoint + apipod simulate Emulate a serverless deployment (FastAPI + job queue) + apipod simulate serverless-runpod Emulate Socaity managed deploy to RunPod + apipod simulate serverless-runpod --native Emulate RunPod's native worker directly + apipod simulate dedicated-azure Emulate a dedicated Azure deployment + apipod scan Scan project and generate apipod.json + apipod build Build the deployment container + apipod build path/to/service.py Build from a specific Python file + apipod deploy Deploy via Socaity (coming soon) + """, ) + subparsers = parser.add_subparsers(dest="command", metavar="command") + parsers: dict = {"__root__": parser} - parser.add_argument( - "--build", - nargs="?", - const=True, - metavar="FILE", - help="Build the service container. Optionally specify a target Python file (e.g., --build main.py)" + start_parser = subparsers.add_parser( + "start", + help="Run the service locally for development (plain FastAPI).", + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog="Examples:\n apipod start\n apipod start main.py --port 8123", ) - parser.add_argument( - "--scan", - action="store_true", - help="Scan project and generate apipod.json configuration file" + _add_run_options(start_parser) + parsers["start"] = start_parser + + simulate_parser = subparsers.add_parser( + "simulate", + help="Emulate a deployment target locally.", + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog=( + "Examples:\n" + " apipod simulate\n" + " apipod simulate serverless-runpod\n" + " apipod simulate serverless-runpod main.py --native" + ), ) - parser.add_argument( - "--start", + simulate_parser.add_argument( + "target", + nargs="?", + default=None, + metavar="TARGET", + help="Deployment target '{compute}-{provider}' (e.g. serverless-runpod, dedicated-azure). " + "Defaults to 'serverless'. A .py value is treated as the entrypoint.", + ) + simulate_parser.add_argument( + "entrypoint", + nargs="?", + default=None, + metavar="ENTRYPOINT", + help="Service entrypoint file (auto-detected when omitted).", + ) + simulate_parser.add_argument( + "--native", action="store_true", - help="Start the APIPod service locally" + help="Emulate the provider's native serverless worker instead of Socaity's job queue.", ) + simulate_parser.add_argument("--host", default=None, help="Host to bind to (default: 0.0.0.0).") + simulate_parser.add_argument("--port", type=int, default=None, help="Port to bind to (default: 8000).") + parsers["simulate"] = simulate_parser - config_group = parser.add_argument_group("deployment configuration") - config_group.add_argument( - "--orchestrator", - choices=orchestrator_choices, - default="local", - help=f"Orchestration platform (default: local). Options: {', '.join(orchestrator_choices)}" - ) - config_group.add_argument( - "--compute", - choices=compute_choices, - default="dedicated", - help=f"Compute type (default: dedicated). Options: {', '.join(compute_choices)}" + scan_parser = subparsers.add_parser( + "scan", + help="Scan project and generate apipod.json configuration file.", ) - config_group.add_argument( - "--provider", - choices=provider_choices, - default="localhost", - help=f"Infrastructure provider (default: localhost). Options: {', '.join(provider_choices)}" + parsers["scan"] = scan_parser + + build_parser = subparsers.add_parser( + "build", + help="Build the service container.", + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog="Examples:\n apipod build\n apipod build path/to/service.py", ) - config_group.add_argument( - "--region", + build_parser.add_argument( + "file", + nargs="?", default=None, - help="Deployment region (provider-specific, e.g. 'us-east-1')" + metavar="FILE", + help="Target Python file to scan (project-wide scan when omitted).", ) - config_group.add_argument( - "--host", - default=None, - help="Host to bind to (default: 0.0.0.0)" + parsers["build"] = build_parser + + deploy_parser = subparsers.add_parser( + "deploy", + help="Deploy the service via Socaity (coming soon).", + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog="Examples:\n apipod deploy\n apipod deploy serverless-runpod", + ) + deploy_parser.add_argument( + "target", + nargs="?", + default="serverless", + metavar="TARGET", + help="Deployment target '{compute}-{provider}' (default: serverless).", + ) + parsers["deploy"] = deploy_parser + + help_parser = subparsers.add_parser( + "help", + help="Show help for apipod or a specific command.", ) - config_group.add_argument( - "--port", - type=int, + help_parser.add_argument( + "command", + nargs="?", default=None, - help="Port to bind to (default: 8000)" + metavar="COMMAND", + help="Command to show help for (omit for the overview).", ) + parsers["help"] = help_parser + return parser, parsers + + +def main(): + """Main entry point for the APIPod CLI.""" + parser, parsers = _build_parser() args = parser.parse_args() - if args.scan: - run_scan() - elif args.build is not None: + if args.command is None: + parser.print_help() + return + + if args.command == "scan": + run_scan(args) + elif args.command == "build": run_build(args) - elif args.start: + elif args.command == "deploy": + run_deploy(args) + elif args.command == "simulate": + run_simulate(args) + elif args.command == "start": run_start(args) - else: - parser.print_help() + elif args.command == "help": + run_help(args, parsers) if __name__ == "__main__": diff --git a/apipod/common/constants.py b/apipod/common/constants.py index 59f9bf8..1b8e0cb 100644 --- a/apipod/common/constants.py +++ b/apipod/common/constants.py @@ -1,11 +1,6 @@ from enum import Enum -class ORCHESTRATOR(Enum): - SOCAITY = "socaity" - LOCAL = "local" - - class COMPUTE(Enum): DEDICATED = "dedicated" SERVERLESS = "serverless" @@ -25,4 +20,4 @@ class SERVER_HEALTH(Enum): BOOTING = "booting" RUNNING = "running" BUSY = "busy" - ERROR = "error" + ERROR = "error" \ No newline at end of file diff --git a/apipod/common/exceptions.py b/apipod/common/exceptions.py index f614b3f..6d96a6e 100644 --- a/apipod/common/exceptions.py +++ b/apipod/common/exceptions.py @@ -24,4 +24,4 @@ def __init__(self, file_name: str = "", message: str = "File upload failed"): class InsufficientBalanceException(JobException): """ Gets raised when a job is rejected because the user has insufficient balance """ def __init__(self, message: str = "Insufficient balance"): - super().__init__(message) + super().__init__(message) \ No newline at end of file diff --git a/apipod/common/schemas.py b/apipod/common/schemas.py deleted file mode 100644 index 488918e..0000000 --- a/apipod/common/schemas.py +++ /dev/null @@ -1,161 +0,0 @@ -from typing import List, Optional, Union, Literal -from pydantic import BaseModel - -# ===================================================== -# Base schema -# ===================================================== - -class OpenAIBaseModel(BaseModel): - model_config = { - "extra": "forbid", - "validate_assignment": True, - "populate_by_name": True, - } - - -# ===================================================== -# Chat Completions - Input schemas -# ===================================================== - -class ChatMessage(OpenAIBaseModel): - role: Literal["system", "user", "assistant"] - content: str - - -class ChatCompletionRequest(OpenAIBaseModel): - model: str - messages: List[ChatMessage] - - temperature: float = 0.7 - max_tokens: Optional[int] = None - top_p: float = 1.0 - n: int = 1 - stream: bool = False - stop: Optional[Union[str, List[str]]] = None - presence_penalty: float = 0.0 - frequency_penalty: float = 0.0 - user: Optional[str] = None - - -# ===================================================== -# Text Completion - Input schemas -# ===================================================== - -class CompletionRequest(OpenAIBaseModel): - model: str - prompt: Union[str, List[str]] - - temperature: float = 0.7 - max_tokens: int = 16 - top_p: float = 1.0 - n: int = 1 - stream: bool = False - stop: Optional[Union[str, List[str]]] = None - - -# ===================================================== -# Embedding - Input schemas -# ===================================================== - -class EmbeddingRequest(OpenAIBaseModel): - model: str - input: Union[str, List[str]] - user: Optional[str] = None - - -# Supported request schemas that should be interpreted as JSON bodies -# by router decorators, even when endpoint authors do not specify Body(...). -SUPPORTED_LLM_REQUEST_SCHEMAS = ( - ChatCompletionRequest, - CompletionRequest, - EmbeddingRequest, -) - - -# ===================================================== -# Shared output schemas -# ===================================================== - -class Usage(OpenAIBaseModel): - prompt_tokens: int - completion_tokens: Optional[int] = None - total_tokens: int - - -# ===================================================== -# Chat Completions - Output schemas -# ===================================================== - -class ChatCompletionMessage(OpenAIBaseModel): - role: Literal["assistant"] - content: str - - -class ChatCompletionChoice(OpenAIBaseModel): - index: int - message: ChatCompletionMessage - finish_reason: Literal["stop", "length", "content_filter"] - - -class ChatCompletionResponse(OpenAIBaseModel): - id: str - object: Literal["chat.completion"] - created: int - model: str - choices: List[ChatCompletionChoice] - usage: Usage - - -# ===================================================== -# Text Completion - Output schemas -# ===================================================== - -class CompletionChoice(OpenAIBaseModel): - text: str - index: int - logprobs: None = None - finish_reason: Literal["stop", "length", "content_filter"] - - -class CompletionResponse(OpenAIBaseModel): - id: str - object: Literal["text_completion"] - created: int - model: str - choices: List[CompletionChoice] - usage: Usage - - -# ===================================================== -# Embedding - Output schemas -# ===================================================== - -class EmbeddingData(OpenAIBaseModel): - object: Literal["embedding"] - embedding: List[float] - index: int - - -class EmbeddingResponse(OpenAIBaseModel): - object: Literal["list"] - data: List[EmbeddingData] - model: str - usage: Usage - -# ====================================================== -# Streaming Models -# ====================================================== -class ChatDelta(BaseModel): - content: Optional[str] = None - -class ChatStreamChoice(BaseModel): - index: int - delta: ChatDelta - finish_reason: Optional[str] = None - -class ChatCompletionChunk(BaseModel): - id: str - object: Literal["chat.completion.chunk"] = "chat.completion.chunk" - created: int - model: str - choices: List[ChatStreamChoice] \ No newline at end of file diff --git a/apipod/common/schemas/__init__.py b/apipod/common/schemas/__init__.py new file mode 100644 index 0000000..5ff2ba9 --- /dev/null +++ b/apipod/common/schemas/__init__.py @@ -0,0 +1,95 @@ +"""Standard request/response schemas, re-exported from socaity-schemas.""" + +from socaity_schemas import ( + APIPodSchemaBase, + AudioFileModel, + ChatCompletionChunk, + ChatCompletionChoice, + ChatCompletionMessage, + ChatCompletionRequest, + ChatCompletionResponse, + ChatDelta, + ChatMessage, + ChatStreamChoice, + CompletionChoice, + CompletionRequest, + CompletionResponse, + CreateVoiceRequest, + EmbeddingData, + EmbeddingRequest, + EmbeddingResponse, + FileModel, + Generation3DRequest, + Generation3DResponse, + ImageFileModel, + ImageGenerationRequest, + ImageGenerationResponse, + MultimodalEmbeddingData, + MultimodalEmbeddingRequest, + MultimodalEmbeddingResponse, + SpeechRequest, + SpeechResponse, + ThreeDFileModel, + TranscriptionRequest, + TranscriptionResponse, + TranscriptionSegment, + TranscriptionWord, + Usage, + VideoFileModel, + VideoGenerationRequest, + VideoGenerationResponse, + VisionData, + VisionLabel, + VisionRequest, + VisionResponse, + VoiceConversionRequest, + VoiceConversionResponse, + VoiceResponse, +) + +__all__ = [ + "APIPodSchemaBase", + "Usage", + "ChatMessage", + "ChatCompletionRequest", + "ChatCompletionResponse", + "ChatCompletionChoice", + "ChatCompletionMessage", + "CompletionRequest", + "CompletionResponse", + "CompletionChoice", + "EmbeddingRequest", + "EmbeddingResponse", + "EmbeddingData", + "ImageGenerationRequest", + "ImageGenerationResponse", + "VideoGenerationRequest", + "VideoGenerationResponse", + "TranscriptionRequest", + "TranscriptionResponse", + "TranscriptionSegment", + "TranscriptionWord", + "SpeechRequest", + "SpeechResponse", + "CreateVoiceRequest", + "VoiceResponse", + "VoiceConversionRequest", + "VoiceConversionResponse", + "Generation3DRequest", + "Generation3DResponse", + "VisionRequest", + "VisionResponse", + "VisionData", + "VisionLabel", + "MultimodalEmbeddingRequest", + "MultimodalEmbeddingResponse", + "MultimodalEmbeddingData", + "ChatDelta", + "ChatStreamChoice", + "ChatCompletionChunk", + "FileModel", + "ImageFileModel", + "AudioFileModel", + "VideoFileModel", + "ThreeDFileModel", +] diff --git a/apipod/common/settings.py b/apipod/common/settings.py index 4dcf34d..dfa68c0 100644 --- a/apipod/common/settings.py +++ b/apipod/common/settings.py @@ -1,12 +1,17 @@ -import sys from os import environ -from apipod.common.constants import ORCHESTRATOR, COMPUTE, PROVIDER +from apipod.common.constants import COMPUTE, PROVIDER -APIPOD_ORCHESTRATOR = environ.get("APIPOD_ORCHESTRATOR", ORCHESTRATOR.LOCAL) -APIPOD_COMPUTE = environ.get("APIPOD_COMPUTE", COMPUTE.DEDICATED) -APIPOD_PROVIDER = environ.get("APIPOD_PROVIDER", PROVIDER.LOCALHOST) +# Deployment target. Socaity overwrites these env vars in a managed deployment so +# the right backend is selected in production (e.g. serverless + runpod). +APIPOD_COMPUTE = environ.get("APIPOD_COMPUTE", COMPUTE.DEDICATED.value) +APIPOD_PROVIDER = environ.get("APIPOD_PROVIDER", PROVIDER.LOCALHOST.value) APIPOD_REGION = environ.get("APIPOD_REGION", "") +# Local simulation intent (ignored in managed deployments). Empty = development. +# Target string is "{compute}-{provider}", e.g. "serverless-runpod". +APIPOD_SIMULATE = environ.get("APIPOD_SIMULATE", "") +APIPOD_NATIVE = environ.get("APIPOD_NATIVE", "").strip().lower() in ("1", "true", "yes") + APIPOD_HOST = environ.get("APIPOD_HOST", "0.0.0.0") APIPOD_PORT = int(environ.get("APIPOD_PORT", 8000)) @@ -14,6 +19,9 @@ DEFAULT_DATE_TIME_FORMAT = environ.get("FTAPI_DATETIME_FORMAT", '%Y-%m-%dT%H:%M:%S.%f%z') -if (APIPOD_ORCHESTRATOR == ORCHESTRATOR.SOCAITY or APIPOD_ORCHESTRATOR == ORCHESTRATOR.LOCAL) and APIPOD_PROVIDER == PROVIDER.RUNPOD and APIPOD_COMPUTE == COMPUTE.SERVERLESS: - sys.argv.extend(['rp_serve_api', '1']) - sys.argv.extend(['--rp_serve_api', '1']) +# Socaity deployment certificate (SHA1 of a shared secret) for detecting a managed +# deployment. When verified, simulate/direct are ignored and the backend is chosen +# from APIPOD_COMPUTE / APIPOD_PROVIDER. +SOCAITY_DEPLOYMENT_CERT = environ.get("SOCAITY_DEPLOYMENT_CERT", "") +_EXPECTED_CERT_HASH = "7b35ca9da2f0c280d48f66c780a0a0d5d3f8ad8a" +IS_MANAGED_DEPLOYMENT = SOCAITY_DEPLOYMENT_CERT == _EXPECTED_CERT_HASH \ No newline at end of file diff --git a/apipod/deploy/docker_template.j2 b/apipod/deploy/docker_template.j2 index a1689d2..420eb7d 100644 --- a/apipod/deploy/docker_template.j2 +++ b/apipod/deploy/docker_template.j2 @@ -58,7 +58,6 @@ COPY . . RUN pip install --no-cache-dir . # Configuration for APIPod deployment -ENV APIPOD_ORCHESTRATOR="{{ orchestrator | default('local') }}" ENV APIPOD_COMPUTE="{{ compute | default('serverless') }}" ENV APIPOD_PROVIDER="{{ provider | default('runpod') }}" ENV APIPOD_HOST="0.0.0.0" diff --git a/apipod/engine/backend/fastapi/file_handling_mixin.py b/apipod/engine/backend/fastapi/file_handling_mixin.py index 0c05d62..094a8b8 100644 --- a/apipod/engine/backend/fastapi/file_handling_mixin.py +++ b/apipod/engine/backend/fastapi/file_handling_mixin.py @@ -3,10 +3,14 @@ from typing import Any, Union, get_args, get_origin, Callable, List, Dict from apipod.engine.backend.fastapi.LimitedUploadFile import LimitedUploadFile from apipod.engine.signatures.upload import is_param_media_toolkit_file -from apipod.engine.jobs.job_result import FileModel, ImageFileModel, AudioFileModel, VideoFileModel +from socaity_schemas import AudioFileModel, FileModel, ImageFileModel, VideoFileModel from apipod.engine.signatures.policies import FastAPISignaturePolicies from apipod.engine.files.base_file_mixin import _BaseFileHandlingMixin from apipod.engine.utils import replace_func_signature +from apipod.engine.signatures.analysis import job_progress_param_names +from apipod.engine.endpoint_config import EndpointExecutionPlan +from apipod.engine.backend.schema_resolve import openapi_schema_annotation, resolve_request_model +from apipod.engine.jobs.job_progress import JobProgress from media_toolkit import MediaList, MediaDict, ImageFile, AudioFile, VideoFile, MediaFile import functools @@ -222,13 +226,42 @@ def _update_signature(self, func: Callable, max_upload_file_size_mb: float = Non func = replace_func_signature(func, inspect.Signature(parameters=body_params)) return func - def _prepare_func_for_media_file_upload_with_fastapi(self, func: callable, max_upload_file_size_mb: float = None) -> callable: + def _apply_openapi_request_model(self, func: Callable, plan: EndpointExecutionPlan | None) -> Callable: + """Swap schema param annotations when ``stream`` must be hidden from OpenAPI.""" + if plan is None or plan.schema_binding is None or plan.is_streaming: + return func + + sig = inspect.signature(func) + new_params = [] + changed = False + for param in sig.parameters.values(): + if resolve_request_model(param.annotation) is None: + new_params.append(param) + continue + new_annotation = openapi_schema_annotation(param.annotation, is_streaming=plan.is_streaming) + if new_annotation is param.annotation: + new_params.append(param) + else: + new_params.append(param.replace(annotation=new_annotation)) + changed = True + + if not changed: + return func + return replace_func_signature(func, sig.replace(parameters=new_params)) + + def _prepare_func_for_media_file_upload_with_fastapi( + self, + func: callable, + max_upload_file_size_mb: float = None, + plan: EndpointExecutionPlan | None = None, + ) -> callable: """ Prepare a function for FastAPI 1. Adds file upload logic to convert parameters 2. Injects a dummy JobProgress if the function expects it but we're not using a queue 3. Removes job progress parameter from the function signature (for OpenAPI docs) - 4. Replaces upload file parameters with FastAPI File type + 4. Hides ``stream`` in OpenAPI (not validation) when the endpoint is not streaming + 5. Replaces upload file parameters with FastAPI File type """ # 1. Add file upload conversion logic (must be first to handle positional args) file_upload_modified = self._handle_file_uploads(func) @@ -239,8 +272,11 @@ def _prepare_func_for_media_file_upload_with_fastapi(self, func: callable, max_u # 3. Remove job progress parameter from signature (so it doesn't show in OpenAPI) no_job_progress = self._remove_job_progress_from_signature(with_dummy_progress) - # 4. Update signature with file upload parameters (converts to Form/File) - with_file_upload_signature = self._update_signature(no_job_progress, max_upload_file_size_mb) + # 4. Hide ``stream`` in OpenAPI for non-streaming schema endpoints (validation keeps the field) + openapi_adjusted = self._apply_openapi_request_model(no_job_progress, plan) + + # 5. Update signature with file upload parameters (converts to Form/File) + with_file_upload_signature = self._update_signature(openapi_adjusted, max_upload_file_size_mb) return with_file_upload_signature @@ -249,14 +285,7 @@ def _inject_dummy_job_progress(self, func: Callable) -> Callable: Wrap the function to inject a dummy JobProgress instance if the original function expects one. This is used for non-queued FastAPI endpoints. """ - from apipod.engine.jobs.job_progress import JobProgress - - sig = inspect.signature(func) - job_progress_params = [ - p.name for p in sig.parameters.values() - if p.name == "job_progress" or "FastJobProgress" in str(p.annotation) - ] - + job_progress_params = job_progress_param_names(func) if not job_progress_params: return func @@ -290,7 +319,7 @@ def _remove_job_progress_from_signature(self, func: Callable) -> Callable: sig = inspect.signature(func) new_sig = sig.replace(parameters=[ p for p in sig.parameters.values() - if p.name != "job_progress" and "FastJobProgress" not in str(p.annotation) + if p.name != "job_progress" and "JobProgress" not in str(p.annotation) ]) if len(new_sig.parameters) != len(sig.parameters): return replace_func_signature(func, new_sig) diff --git a/apipod/engine/backend/fastapi/llm_mixin.py b/apipod/engine/backend/fastapi/llm_mixin.py deleted file mode 100644 index 41bb5e3..0000000 --- a/apipod/engine/backend/fastapi/llm_mixin.py +++ /dev/null @@ -1,29 +0,0 @@ -from fastapi.responses import StreamingResponse - -from apipod.engine.llm.base_llm_mixin import _BaseLLMMixin - - -class _FastApiLlmMixin(_BaseLLMMixin): - """ - LLM execution logic for the FastAPI provider backend. - """ - - async def handle_llm_request(self, func, openai_req, should_use_queue, res_model, endpoint_type, **kwargs): - should_stream = getattr(openai_req, "stream", False) and endpoint_type != "embedding" - - if should_stream: - result = await self._execute_func(func, payload=openai_req, **kwargs) - return StreamingResponse(self._stream_generator(result), media_type="text/event-stream") - - if should_use_queue: - from apipod.engine.jobs.job_result import JobResultFactory - - job = self.job_queue.add_job( - job_function=func, - job_params={"payload": openai_req.dict()} - ) - ret_job = JobResultFactory.from_base_job(job) - return ret_job - - raw_res = await self._execute_func(func, payload=openai_req, **kwargs) - return self._wrap_llm_response(raw_res, res_model, endpoint_type, openai_req) diff --git a/apipod/engine/backend/fastapi/router.py b/apipod/engine/backend/fastapi/router.py index 0c43256..e84b7b8 100644 --- a/apipod/engine/backend/fastapi/router.py +++ b/apipod/engine/backend/fastapi/router.py @@ -3,24 +3,27 @@ import threading import logging from contextlib import asynccontextmanager -from typing import Union, Callable, get_type_hints, Generator, AsyncGenerator, Iterator, AsyncIterator -from fastapi import APIRouter, FastAPI, Request, Response, status +from typing import Union, Callable +from fastapi import APIRouter, FastAPI, Response, status from fastapi.exceptions import HTTPException -from fastapi.responses import JSONResponse, StreamingResponse +from fastapi.responses import JSONResponse from apipod.common.settings import APIPOD_PORT, APIPOD_HOST from apipod.common.constants import SERVER_HEALTH from apipod.engine.jobs.job_result import JobResultFactory, JobResult -from apipod.engine.endpoint_config import FastApiEndpointConfigurator, EndpointExecutionPlan +from apipod.engine.endpoint_config import build_plan, EndpointExecutionPlan +from apipod.engine.streaming.stream_serializer import build_stream_producer from apipod.engine.base_backend import _BaseBackend from apipod.engine.queue.queue_mixin import _QueueMixin from apipod.engine.backend.fastapi.file_handling_mixin import _fast_api_file_handling_mixin -from apipod.engine.utils import normalize_name +from apipod.engine.backend.fastapi.streaming_mixin import _FastAPIStreamingMixin +from apipod.engine.utils import normalize_name, normalize_mount_prefix from apipod.engine.backend.fastapi.exception_handling import _FastAPIExceptionHandler -from apipod.engine.backend.fastapi.llm_mixin import _FastApiLlmMixin +from apipod.engine.backend.schema_resolve import wrap_schema_response +from apipod.engine.jobs.enqueue_payload import is_enqueue_payload -class SocaityFastAPIRouter(APIRouter, _BaseBackend, _QueueMixin, _fast_api_file_handling_mixin, _FastAPIExceptionHandler, _FastApiLlmMixin): +class SocaityFastAPIRouter(APIRouter, _BaseBackend, _QueueMixin, _fast_api_file_handling_mixin, _FastAPIStreamingMixin, _FastAPIExceptionHandler): """ FastAPI router extension that adds support for task endpoints. @@ -36,7 +39,6 @@ def __init__( prefix: str = "", # "/api", max_upload_file_size_mb: float = None, job_queue=None, - lifespan=None, *args, **kwargs): """ @@ -49,59 +51,50 @@ def __init__( prefix: The API route prefix max_upload_file_size_mb: Maximum file size in MB for uploads job_queue: Optional custom JobQueue implementation - lifespan: Optional async context manager for custom startup/shutdown logic args: Additional arguments - kwargs: May include ``stream_store`` (SSE backend for GET /stream/{job_id}) and - ``gateway_stream_url_prefix`` for absolute stream URLs in JobResult, plus - additional keyword arguments for parent classes. + kwargs: May include ``stream_store`` (SSE backend for GET /stream/{job_id}), + plus additional keyword arguments for parent classes. """ - # Extract user-provided lifespan (explicit param or kwarg) before parent init - user_lifespan = lifespan or kwargs.pop('lifespan', None) stream_store = kwargs.pop("stream_store", None) - gateway_stream_url_prefix = kwargs.pop("gateway_stream_url_prefix", "") # Initialize parent classes api_router_params = inspect.signature(APIRouter.__init__).parameters api_router_kwargs = {k: kwargs.get(k) for k in api_router_params if k in kwargs} - api_router_kwargs.pop('lifespan', None) # handled via composed lifespan below - + APIRouter.__init__(self, **api_router_kwargs) _BaseBackend.__init__(self, title=title, summary=summary, *args, **kwargs) _QueueMixin.__init__(self, job_queue=job_queue, *args, **kwargs) _fast_api_file_handling_mixin.__init__(self, max_upload_file_size_mb=max_upload_file_size_mb, *args, **kwargs) - _FastApiLlmMixin.__init__(self) self.status = SERVER_HEALTH.INITIALIZING - # Registry for functions that workers can execute. Keys are function names. - self._job_func_registry: dict = {} + # Queued endpoint plans keyed by worker ``__name__`` (see ``_create_task_endpoint_decorator``). + self._endpoint_plans: dict[str, EndpointExecutionPlan] = {} # Stop event and thread handle for in-process worker (dev mode) self._worker_stop_event = threading.Event() self._worker_thread: threading.Thread | None = None self._logger = logging.getLogger(__name__) - # Build a composed lifespan that merges internal worker hooks with the user-provided lifespan - combined_lifespan = self._build_lifespan(user_lifespan) - # Create or use provided FastAPI app if app is None: app = FastAPI( title=self.title, summary=self.summary, contact={"name": "SocAIty", "url": "https://www.socaity.ai"}, - lifespan=combined_lifespan, ) - else: - # Existing app: replace its lifespan with our composed version - app.router.lifespan_context = combined_lifespan self.app: FastAPI = app - self.prefix = prefix + self.prefix = normalize_mount_prefix(prefix) self.stream_store = stream_store - self.gateway_stream_url_prefix = gateway_stream_url_prefix - self.add_standard_routes() - self._endpoint_configurator = FastApiEndpointConfigurator(self) + # Let the queue produce streaming job output into the stream store so a + # client can consume it via GET /stream/{job_id} (serverless emulation). + if self.job_queue is not None and self.stream_store is not None: + set_store = getattr(self.job_queue, "set_stream_store", None) + if callable(set_store): + set_store(self.stream_store) + + self.add_standard_routes() # Exception handling _FastAPIExceptionHandler.__init__(self) @@ -112,67 +105,6 @@ def __init__( # Save original OpenAPI function and replace it self._orig_openapi_func = self.app.openapi self.app.openapi = self.custom_openapi - - # ------------------------------------------------------------------ - # Lifespan & worker lifecycle - # ------------------------------------------------------------------ - - def _build_lifespan(self, user_lifespan=None): - """ - Build a composed lifespan context manager that runs: - 1. Internal worker startup - 2. User-provided lifespan (if any) - 3. Internal worker shutdown on exit - """ - router_self = self # capture for closure - - @asynccontextmanager - async def _combined_lifespan(app): - router_self._start_background_worker() - try: - if user_lifespan: - async with user_lifespan(app): - yield - else: - yield - finally: - router_self._stop_background_worker() - - return _combined_lifespan - - def _start_background_worker(self): - """Start the in-process job queue worker in a daemon thread (dev convenience).""" - try: - if self.job_queue and hasattr(self.job_queue, "start_worker"): - def _run(): - try: - self.job_queue.start_worker( - func_registry=self._job_func_registry, - worker_name="api-worker", - stop_event=self._worker_stop_event, - ) - except Exception: - self._logger.exception("Worker thread exited with exception") - - thread = threading.Thread(target=_run, daemon=True) - thread.start() - self._worker_thread = thread - except Exception: - self._logger.exception("Failed to start in-process worker on startup") - - def _stop_background_worker(self): - """Signal the background worker to stop and shut down the job queue.""" - try: - self._worker_stop_event.set() - except Exception: - pass - - if self.job_queue and hasattr(self.job_queue, "shutdown"): - try: - self.job_queue.shutdown() - except Exception: - self._logger.exception("Error shutting down job queue") - # ------------------------------------------------------------------ # Standard routes # ------------------------------------------------------------------ @@ -187,6 +119,28 @@ def add_standard_routes(self): self.api_route(path="/stream/{job_id}", methods=["GET"])(self.stream_job_sse) self.api_route(path="/health", methods=["GET"])(self.get_health) + def _apply_mount_prefix(self, mount_prefix: str) -> None: + """Record the mount prefix used for job hypermedia links.""" + mount = normalize_mount_prefix(mount_prefix) + if mount: + self.prefix = mount + + def include_router( + self, + router: "SocaityFastAPIRouter", + prefix: str = "", + tags: list | None = None, + **kwargs, + ) -> None: + """Mount another APIPod FastAPI router (mirrors FastAPI ``include_router``).""" + if not isinstance(router, SocaityFastAPIRouter): + raise TypeError( + f"APIPod include_router expects a SocaityFastAPIRouter instance, got {type(router)!r}" + ) + mount = normalize_mount_prefix(prefix) + router._apply_mount_prefix(mount) + super().include_router(router, prefix=mount, tags=tags, **kwargs) + def get_health(self) -> Response: """ Get server health status. @@ -207,7 +161,16 @@ def custom_openapi(self): if not self.app.openapi_schema: self._orig_openapi_func() - self.app.openapi_schema["info"]["apipod"] = self.version + compute = "dedicated" + if self.job_queue is not None: + compute = "serverless" + + manifest = { + "compute": compute, + "version": self.version, + } + + self.app.openapi_schema["info"]["apipod"] = manifest return self.app.openapi_schema def get_job(self, job_id: str, return_format: str = 'json') -> JobResult: @@ -227,7 +190,7 @@ def get_job(self, job_id: str, return_format: str = 'json') -> JobResult: if self.job_queue is None: return JobResultFactory.job_not_found(job_id) - ret_job = self.job_queue.get_job_result(job_id) + ret_job = self.job_queue.get_job_result(job_id, link_prefix=self.prefix) if ret_job is None: return JobResultFactory.job_not_found(job_id) @@ -237,71 +200,41 @@ def get_job(self, job_id: str, return_format: str = 'json') -> JobResult: return ret_job def post_cancel_job(self, job_id: str) -> dict: - """Cancel a background job (gateway / orchestrator integration).""" + """Cancel a background job via the queue port.""" job_id = job_id.strip().strip('"').strip("'").strip("?").strip("#") if self.job_queue is None: raise HTTPException(status_code=status.HTTP_503_SERVICE_UNAVAILABLE, detail="Job queue not configured.") - cancel_fn = getattr(self.job_queue, "cancel_gateway_job", None) - if callable(cancel_fn): - return cancel_fn(job_id) - try: - self.job_queue.cancel_job(job_id) + summary = self.job_queue.cancel_job(job_id) except NotImplementedError: raise HTTPException( status_code=status.HTTP_501_NOT_IMPLEMENTED, detail="Cancellation is not supported for this job queue.", ) from None - return {"id": job_id, "status": "cancelled", "message": "Job cancelled."} + return summary or {"id": job_id, "status": "cancelled", "message": "Job cancelled."} - async def stream_job_sse(self, job_id: str, request: Request): - """Server-Sent Events for streaming job output (requires stream_store).""" - job_id = job_id.strip().strip('"').strip("'").strip("?").strip("#") - if self.stream_store is None: - raise HTTPException(status_code=status.HTTP_503_SERVICE_UNAVAILABLE, detail="Streaming not configured.") + def add_job(self, func: Callable, job_params: dict) -> JobResult: + """Enqueue a task and return a :class:`JobResult` with endpoint-aware links.""" if self.job_queue is None: - raise HTTPException(status_code=status.HTTP_503_SERVICE_UNAVAILABLE, detail="Job queue not configured.") + raise ValueError("Job Queue is not initialized. Cannot add job.") - jq = self.job_queue - job_data = jq.get_job_status(job_id) if hasattr(jq, "get_job_status") else None - - if job_data is None: - raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail=f"Job '{job_id}' not found.") - - st = (job_data.get("status") or "").lower() - if st != "streaming" and not self.stream_store.stream_exists(job_id): - raise HTTPException( - status_code=status.HTTP_409_CONFLICT, - detail=f"Job '{job_id}' is not streaming (status: {job_data.get('status')}).", - ) - - async def _event_generator(): - try: - async for chunk in self.stream_store.read_chunks(job_id): - if await request.is_disconnected(): - break - yield chunk - except Exception: - self._logger.exception("Error during stream delivery | job_id=%s", job_id) - yield 'data: {"error": "Internal stream error"}\n\n' - - return StreamingResponse( - _event_generator(), - media_type="text/event-stream", - headers={ - "Cache-Control": "no-cache", - "Connection": "keep-alive", - "X-Accel-Buffering": "no", - }, + plan = self._endpoint_plans.get(f"{func.__module__}.{func.__qualname__}") + supports_streaming = ( + self.stream_store is not None and plan is not None and plan.is_streaming + ) + return self.job_queue.add_job( + job_function=func, + job_params=job_params, + supports_streaming=supports_streaming, + link_prefix=self.prefix, ) - def endpoint(self, path: str, methods: list[str] | None = None, max_upload_file_size_mb: int = None, queue_size: int = 500, use_queue: bool = None, *args, **kwargs): """ Unified endpoint decorator. - If LLM configuration is detected on the function, it creates an LLM endpoint. + If a parameter is annotated with a standardized request schema, it creates a schema endpoint. If job_queue is configured (and use_queue is not False), it creates a task endpoint. Otherwise, it creates a standard FastAPI endpoint. @@ -317,181 +250,31 @@ def endpoint(self, path: str, methods: list[str] | None = None, max_upload_file_ should_use_queue = self._determine_queue_usage(use_queue, normalized_path) def decorator(func: Callable) -> Callable: - plan = self._create_endpoint_plan( - func=func, - normalized_path=normalized_path, + plan = build_plan( + func, + path=normalized_path, methods=methods, max_upload_file_size_mb=max_upload_file_size_mb, queue_size=queue_size, should_use_queue=should_use_queue, route_args=args, - route_kwargs=kwargs + route_kwargs=kwargs, ) - - if plan.is_llm: - return self._register_llm_endpoint(func, plan) - if plan.is_streaming: - return self._create_streaming_endpoint_decorator( - path=plan.path, - methods=plan.methods, - max_upload_file_size_mb=plan.max_upload_file_size_mb, - args=plan.route_args, - kwargs=plan.route_kwargs - )(func) + # save the plan for later use + self._endpoint_plans[f"{func.__module__}.{func.__qualname__}"] = plan + + # Streaming with a queue + stream store mimics a real deployment: + # the job is queued, the worker produces chunks into the stream store + # and the client consumes them from GET /stream/{job_id}. Without a + # queue (plain FastAPI) streaming goes straight to the client. + if plan.is_streaming and plan.schema_binding is None and not plan.should_use_queue: + return self._create_streaming_endpoint_decorator(plan)(func) if plan.should_use_queue: - return self._create_task_endpoint_decorator( - path=plan.path, - methods=plan.methods, - max_upload_file_size_mb=plan.max_upload_file_size_mb, - queue_size=plan.queue_size, - args=plan.route_args, - kwargs=plan.route_kwargs - )(func) - - return self._create_standard_endpoint_decorator( - path=plan.path, - methods=plan.methods, - max_upload_file_size_mb=plan.max_upload_file_size_mb, - args=plan.route_args, - kwargs=plan.route_kwargs - )(func) - - return decorator - - def _create_endpoint_plan( - self, - *, - func: Callable, - normalized_path: str, - methods: list[str] | None, - max_upload_file_size_mb: int | None, - queue_size: int, - should_use_queue: bool, - route_args: tuple, - route_kwargs: dict, - ) -> EndpointExecutionPlan: - """Build an orchestration plan for endpoint registration and execution.""" - return self._endpoint_configurator.build_plan( - func=func, - path=normalized_path, - methods=methods, - max_upload_file_size_mb=max_upload_file_size_mb, - queue_size=queue_size, - should_use_queue=should_use_queue, - route_args=route_args, - route_kwargs=route_kwargs, - ) - - def _register_llm_endpoint(self, func: Callable, plan: EndpointExecutionPlan) -> Callable: - """Register a FastAPI LLM endpoint from an endpoint execution plan.""" - req_model = plan.request_model - res_model = plan.response_model - endpoint_type = plan.endpoint_type - - if req_model is None or res_model is None or endpoint_type is None: - raise ValueError("Invalid LLM endpoint plan: missing request/response metadata") - - if plan.should_use_queue: - try: - self._job_func_registry[func.__name__] = func - except Exception: - pass - - @functools.wraps(func) - async def _unified_worker(*w_args, **w_kwargs): - payload = next( - (v for v in w_kwargs.values() if isinstance(v, (dict, req_model))), - next((a for a in w_args if isinstance(a, (dict, req_model))), None) - ) - - if payload is None: - raise ValueError(f"Request does not match expected model {req_model}") - - clean_kwargs = {k: v for k, v in w_kwargs.items() if not isinstance(v, req_model)} - openai_req = self._prepare_llm_payload(req_model=req_model, payload=payload) - - return await self.handle_llm_request( - func=func, - openai_req=openai_req, - should_use_queue=plan.should_use_queue, - res_model=res_model, - endpoint_type=endpoint_type, - **clean_kwargs - ) - - final_handler = self._prepare_func_for_media_file_upload_with_fastapi( - _unified_worker, - plan.max_upload_file_size_mb - ) + return self._create_task_endpoint_decorator(plan)(func) - route_kwargs = dict(plan.route_kwargs) - if plan.should_use_queue: - route_kwargs["response_model_exclude_none"] = True - - self.api_route( - path=plan.path, - methods=plan.active_methods, - response_model=JobResult if plan.should_use_queue else res_model, - *plan.route_args, - **route_kwargs - )(final_handler) + return self._create_standard_endpoint_decorator(plan)(func) - return final_handler - - async def _execute_func(self, func, **kwargs): - """ - Execute a function, handling both sync and async functions. - - Args: - func: Function to execute - **kwargs: Arguments to pass to function - - Returns: - Result of function execution - """ - if inspect.iscoroutinefunction(func): - return await func(**kwargs) - else: - # Sync function - run in thread pool to avoid blocking - import asyncio - loop = asyncio.get_event_loop() - return await loop.run_in_executor(None, functools.partial(func, **kwargs)) - - async def _stream_generator(self, result): - """ - Convert a sync or async generator into an async generator for StreamingResponse. - - Args: - result: Generator (sync or async) that yields streaming chunks - - Yields: - Streaming chunks as strings or bytes - """ - import asyncio - - if inspect.isasyncgen(result): - # Async generator - yield directly - async for chunk in result: - if isinstance(chunk, (str, bytes)): - yield chunk - else: - yield str(chunk) - elif inspect.isgenerator(result): - # Sync generator - run in executor - loop = asyncio.get_event_loop() - try: - while True: - chunk = await loop.run_in_executor(None, next, result, StopIteration) - if chunk is StopIteration: - break - if isinstance(chunk, (str, bytes)): - yield chunk - else: - yield str(chunk) - except StopIteration: - pass - else: - raise TypeError(f"Expected generator, got {type(result)}") + return decorator def _normalize_endpoint_path(self, path: str) -> str: """Normalize the endpoint path to ensure it starts with '/'.""" @@ -507,134 +290,103 @@ def _determine_queue_usage(self, use_queue: bool = None, path: str = None) -> bo return use_queue return self.job_queue is not None - - def _determine_generator_fun(self, func: Callable) -> bool: + + # ------------------------------------------------------------------ + # Endpoint decorators (task / standard / streaming) + # ------------------------------------------------------------------ + def _modify_result_decorator(self, func: Callable, plan: EndpointExecutionPlan, *, queued: bool) -> Callable: """ - Determine whether the function is streaming data. - Uses quick, robust checks for common streaming patterns. + Wraps endpoint responses into their final transport form. + + Streamable results (a token/chunk generator) become a stream: + - ``queued`` (job queue): a :class:`StreamProducer` is returned so the + worker relays chunks into the stream store (consumed via /stream) while + aggregating the full result for /status; + - direct (no queue): a :class:`StreamingResponse` is returned to the client. + + Non-streaming schema results are wrapped into the response model; all + results are then serialized (media files -> JSON). Handlers that return + :class:`~apipod.engine.jobs.enqueue_payload.EnqueuePayload` (job + submission metadata for remote queues) are passed through unchanged. """ - # 1. Direct generator functions - if inspect.isgeneratorfunction(func) or inspect.isasyncgenfunction(func): - return True + @functools.wraps(func) + def sync_wrapper(*args, **kwargs): + result = self.run_callable(func, *args, **kwargs) - # 2. Check return type hints - try: - hints = get_type_hints(func) - return_type = hints.get('return') + producer = build_stream_producer(result, plan.schema_binding) + if producer is not None: + return producer if queued else self._streaming_response_from_producer(producer) - if return_type is not None: - origin = getattr(return_type, '__origin__', return_type) - streaming_types = (Generator, AsyncGenerator, Iterator, AsyncIterator) + if is_enqueue_payload(result): + return result - if origin in streaming_types or return_type in streaming_types: - return True - except Exception: - pass + binding = plan.schema_binding + if binding is not None: + result = wrap_schema_response(result, binding) + return JobResultFactory._serialize_result(result) - # 3. Check source code for yield keyword - try: - source = inspect.getsource(func) - if 'yield' in source: - return True - except Exception: - pass + # Python 3.12+ follows __wrapped__ when evaluating inspect.isgeneratorfunction(). + # sync_wrapper is never a generator (it returns a JobResult or StreamProducer). + # Without this, FastAPI would mistake a task endpoint wrapping a generator for a + # streaming route and serve the JobResult as iterated key-value NDJSON chunks. + if hasattr(sync_wrapper, "__wrapped__"): + # Pin the signature explicitly so FastAPI can parse parameters + sync_wrapper.__signature__ = inspect.signature(func) + del sync_wrapper.__wrapped__ - return False + return sync_wrapper - def _create_task_endpoint_decorator(self, path: str, methods: list[str] | None, max_upload_file_size_mb: int, queue_size: int, args, kwargs): + def _create_task_endpoint_decorator(self, plan: EndpointExecutionPlan): """Create a decorator for task endpoints (background job execution).""" # FastAPI route decorator (returning JobResult) - task_kwargs = dict(kwargs) + task_kwargs = dict(plan.route_kwargs) task_kwargs["response_model_exclude_none"] = True fastapi_route_decorator = self.api_route( - path=path, - methods=["POST"] if methods is None else methods, + path=plan.path, + methods=plan.active_methods, response_model=JobResult, - *args, + *plan.route_args, **task_kwargs ) # Queue decorator queue_decorator = super().job_queue_func( - path=path, - queue_size=queue_size, - *args, - **kwargs + path=plan.path, + queue_size=plan.queue_size, + *plan.route_args, + **plan.route_kwargs ) def decorator(func: Callable) -> Callable: + func = self._modify_result_decorator(func, plan, queued=True) # Add job queue functionality and prepare for FastAPI file handling queue_decorated = queue_decorator(func) - # Register the function so workers can execute it (dev mode). - try: - self._job_func_registry[func.__name__] = func - except Exception: - pass - upload_enabled = self._prepare_func_for_media_file_upload_with_fastapi(queue_decorated, max_upload_file_size_mb) - return fastapi_route_decorator(upload_enabled) - return decorator - - def _create_standard_endpoint_decorator(self, path: str, methods: list[str] | None, max_upload_file_size_mb: int, args, kwargs): - """Create a decorator for standard endpoints (direct execution).""" - # FastAPI route decorator - fastapi_route_decorator = self.api_route( - path=path, - methods=["POST"] if methods is None else methods, - *args, - **kwargs - ) - - def file_result_modification_decorator(func: Callable) -> Callable: - """Wrap endpoint result and serialize it.""" - @functools.wraps(func) - def sync_wrapper(*args, **kwargs): - result = func(*args, **kwargs) - return JobResultFactory._serialize_result(result) - return sync_wrapper - - def decorator(func: Callable) -> Callable: - result_modified = file_result_modification_decorator(func) - with_file_upload_signature = self._prepare_func_for_media_file_upload_with_fastapi(result_modified, max_upload_file_size_mb) - return fastapi_route_decorator(with_file_upload_signature) + upload_enabled = self._prepare_func_for_media_file_upload_with_fastapi( + queue_decorated, plan.max_upload_file_size_mb, plan=plan, + ) + return fastapi_route_decorator(upload_enabled) return decorator - def _create_streaming_endpoint_decorator(self, path: str, methods: list[str] | None, max_upload_file_size_mb: int, args, kwargs): - """Create a decorator for streaming endpoints.""" - from fastapi.responses import StreamingResponse - - # Extract custom headers if provided - custom_headers = kwargs.pop('response_headers', None) - + def _create_standard_endpoint_decorator(self, plan: EndpointExecutionPlan): + """Create a decorator for standard and schema endpoints (direct execution).""" + route_kwargs = dict(plan.route_kwargs) + if plan.is_schema_endpoint: + route_kwargs["response_model_exclude_none"] = True fastapi_route_decorator = self.api_route( - path=path, - methods=["POST"] if methods is None else methods, - *args, - **kwargs + path=plan.path, + methods=plan.active_methods, + response_model=plan.schema_binding.response_model if plan.is_schema_endpoint else None, + *plan.route_args, + **route_kwargs ) def decorator(func: Callable) -> Callable: - @functools.wraps(func) - async def streaming_wrapper(*w_args, **w_kwargs): - result = await self._execute_func(func, *w_args, **w_kwargs) - generator = self._stream_generator(result) - - # Merge default and custom headers - headers = { - "Cache-Control": "no-cache", - "X-Accel-Buffering": "no" - } - if custom_headers: - headers.update(custom_headers) - - return StreamingResponse( - generator, - media_type="text/event-stream", - headers=headers - ) - - with_file_upload_signature = self._prepare_func_for_media_file_upload_with_fastapi(streaming_wrapper, max_upload_file_size_mb) + result_modified = self._modify_result_decorator(func, plan, queued=False) + with_file_upload_signature = self._prepare_func_for_media_file_upload_with_fastapi( + result_modified, plan.max_upload_file_size_mb, plan=plan, + ) return fastapi_route_decorator(with_file_upload_signature) return decorator diff --git a/apipod/engine/backend/fastapi/streaming_mixin.py b/apipod/engine/backend/fastapi/streaming_mixin.py new file mode 100644 index 0000000..2b41c3e --- /dev/null +++ b/apipod/engine/backend/fastapi/streaming_mixin.py @@ -0,0 +1,169 @@ +""" +FastAPI-specific streaming transport, factored out of the router. + +This mixin owns the FastAPI-side mechanics for turning a streaming endpoint +result into an HTTP response (execution itself goes through the shared +``_BaseBackend.run_callable`` / ``run_callable_async`` runtime): + +- ``_stream_generator`` — adapt a sync/async generator into an async generator + for :class:`StreamingResponse` (direct non-queued path); +- ``_streaming_response_from_producer`` — build a :class:`StreamingResponse` from + a :class:`StreamProducer` (direct non-queued path); +- ``_create_streaming_endpoint_decorator`` — register a plain generator endpoint + directly on the FastAPI router (direct non-queued path); +- ``stream_job_sse`` — SSE consumer route ``GET /stream/{job_id}`` that replays + a queued job's chunks from the stream store. + +Serialization details (encoding, aggregation, producer construction) live in +:mod:`apipod.engine.streaming.stream_serializer`; endpoint introspection and +plan building live in :mod:`apipod.engine.endpoint_config`. + +The mixin carries no state and needs no ``__init__``. +""" + +import asyncio +import functools +import inspect +from typing import Callable + +from fastapi import Request, status +from fastapi.exceptions import HTTPException +from fastapi.responses import StreamingResponse + +from apipod.engine.endpoint_config import EndpointExecutionPlan +from apipod.engine.jobs.base_job import JOB_STATUS, STREAM_WAIT_STATUSES +from apipod.engine.streaming.stream_producer import StreamProducer + + +# Sentinel marking the end of a synchronous iterator drained via run_in_executor. +_STREAM_END = object() + + +class _FastAPIStreamingMixin: + """FastAPI streaming transport helpers mixed into the FastAPI router.""" + + # ------------------------------------------------------------------ + # Execution helpers + # ------------------------------------------------------------------ + + async def _stream_generator(self, result): + """Adapt a sync or async generator into an async generator for StreamingResponse.""" + if inspect.isasyncgen(result): + async for chunk in result: + yield chunk if isinstance(chunk, (str, bytes)) else str(chunk) + elif inspect.isgenerator(result): + loop = asyncio.get_event_loop() + while True: + chunk = await loop.run_in_executor(None, next, result, _STREAM_END) + if chunk is _STREAM_END: + break + yield chunk if isinstance(chunk, (str, bytes)) else str(chunk) + else: + raise TypeError(f"Expected generator, got {type(result)}") + + # ------------------------------------------------------------------ + # Direct streaming (non-queued) + # ------------------------------------------------------------------ + + def _streaming_response_from_producer(self, producer: StreamProducer) -> StreamingResponse: + """Build a direct :class:`StreamingResponse` from a producer (non-queued path).""" + async def _event_generator(): + loop = asyncio.get_event_loop() + iterator = iter(producer.raw_chunks) + while True: + item = await loop.run_in_executor(None, next, iterator, _STREAM_END) + if item is _STREAM_END: + break + yield producer.to_chunk(item) + for closing_chunk in producer.closing: + yield closing_chunk + + return StreamingResponse( + _event_generator(), + media_type=producer.media_type, + headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"}, + ) + + def _create_streaming_endpoint_decorator(self, plan: EndpointExecutionPlan): + """Register a plain generator endpoint served directly to the client (no queue).""" + kwargs = dict(plan.route_kwargs) + custom_headers = kwargs.pop("response_headers", None) + + fastapi_route_decorator = self.api_route( + path=plan.path, + methods=plan.active_methods, + *plan.route_args, + **kwargs, + ) + + def decorator(func: Callable) -> Callable: + @functools.wraps(func) + async def streaming_wrapper(*w_args, **w_kwargs): + result = await self.run_callable_async(func, *w_args, **w_kwargs) + generator = self._stream_generator(result) + + headers = {"Cache-Control": "no-cache", "X-Accel-Buffering": "no"} + if custom_headers: + headers.update(custom_headers) + + return StreamingResponse(generator, media_type="text/event-stream", headers=headers) + + with_upload = self._prepare_func_for_media_file_upload_with_fastapi( + streaming_wrapper, plan.max_upload_file_size_mb, plan=plan, + ) + return fastapi_route_decorator(with_upload) + + return decorator + + # ------------------------------------------------------------------ + # SSE consumer route (queued jobs) + # ------------------------------------------------------------------ + + async def stream_job_sse(self, job_id: str, request: Request): + """Server-Sent Events consumer for a queued streaming job (requires stream_store).""" + job_id = job_id.strip().strip('"').strip("'").strip("?").strip("#") + if self.stream_store is None: + raise HTTPException(status_code=status.HTTP_503_SERVICE_UNAVAILABLE, detail="Streaming not configured.") + if self.job_queue is None: + raise HTTPException(status_code=status.HTTP_503_SERVICE_UNAVAILABLE, detail="Job queue not configured.") + + jq = self.job_queue + job_data = jq.get_job_status(job_id) if hasattr(jq, "get_job_status") else None + stream_open = self.stream_store.stream_exists(job_id) + + # Serve the stream even when the job record is already gone (completed + + # cleaned up) as long as the stream is still draining. + if job_data is None and not stream_open: + raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail=f"Job '{job_id}' not found.") + + # Allow pre-stream lifecycle states; read_chunks waits for the producer. + raw_status = (job_data.get("status") if job_data else "") or "" + if isinstance(raw_status, JOB_STATUS): + st = raw_status.value + else: + st = str(raw_status).lower() + if not stream_open and st not in STREAM_WAIT_STATUSES: + raise HTTPException( + status_code=status.HTTP_409_CONFLICT, + detail=f"Job '{job_id}' is not streaming (status: {st or 'unknown'}).", + ) + + async def _event_generator(): + try: + async for chunk in self.stream_store.read_chunks(job_id): + if await request.is_disconnected(): + break + yield chunk + except Exception: + self._logger.exception("Error during stream delivery | job_id=%s", job_id) + yield 'data: {"error": "Internal stream error"}\n\n' + + return StreamingResponse( + _event_generator(), + media_type="text/event-stream", + headers={ + "Cache-Control": "no-cache", + "Connection": "keep-alive", + "X-Accel-Buffering": "no", + }, + ) diff --git a/apipod/engine/backend/runpod/llm_mixin.py b/apipod/engine/backend/runpod/llm_mixin.py deleted file mode 100644 index 225b8b9..0000000 --- a/apipod/engine/backend/runpod/llm_mixin.py +++ /dev/null @@ -1,20 +0,0 @@ -from apipod.engine.llm.base_llm_mixin import _BaseLLMMixin - - -class _RunPodLLMMixin(_BaseLLMMixin): - """ - LLM logic specific to the RunPod Router - """ - - def handle_llm_request(self, func, openai_req, req_model, res_model, endpoint_type, w_args, w_kwargs): - if isinstance(openai_req, dict): - openai_req = req_model.model_validate(openai_req) - w_kwargs["payload"] = openai_req - - should_stream = getattr(openai_req, "stream", False) and endpoint_type != "embedding" - - if should_stream: - return self._yield_native_stream(func, w_args, w_kwargs) - - raw_res = self._execute_sync_or_async(func, w_args, w_kwargs) - return self._wrap_llm_response(raw_res, res_model, endpoint_type, openai_req) diff --git a/apipod/engine/backend/runpod/router.py b/apipod/engine/backend/runpod/router.py index e78d55b..7bea495 100644 --- a/apipod/engine/backend/runpod/router.py +++ b/apipod/engine/backend/runpod/router.py @@ -1,41 +1,80 @@ -import asyncio import functools import inspect import traceback from datetime import datetime, timezone -from typing import Union, Callable +from typing import Union, Callable, Iterator from apipod.common import constants -from apipod.engine.jobs.base_job import JOB_STATUS -from apipod.engine.jobs.job_progress import JobProgressRunpod, JobProgress -from apipod.engine.jobs.job_result import ( - JobResultFactory, - JobResult, - JobMetrics, - _compute_duration_s, - _job_status_to_public, -) +from apipod.engine.jobs.base_job import BaseJob, JOB_STATUS +from apipod.engine.jobs.job_progress import JobProgressRunpod +from apipod.engine.jobs.job_result import JobResultFactory from apipod.engine.base_backend import _BaseBackend +from apipod.engine.endpoint_config import build_plan, EndpointExecutionPlan +from apipod.engine.signatures.analysis import job_progress_param_names from apipod.engine.files.base_file_mixin import _BaseFileHandlingMixin -from apipod.engine.backend.runpod.llm_mixin import _RunPodLLMMixin +from apipod.engine.backend.schema_resolve import ( + SchemaBinding, + SchemaStreamSerializer, + STREAM_CHUNK_SPECS, + iter_media_chunks, + prepare_schema_call, + wrap_schema_response, +) +from apipod.engine.streaming.stream_serializer import as_sync_iter, encode_chunk, is_streaming_result -from apipod.engine.utils import normalize_name -from apipod.common.settings import APIPOD_PROVIDER, APIPOD_PORT +from apipod.engine.utils import normalize_name, normalize_mount_prefix +from apipod.common.settings import APIPOD_PORT +from media_toolkit import AudioFile, VideoFile -class SocaityRunpodRouter(_BaseBackend, _BaseFileHandlingMixin, _RunPodLLMMixin): +class SocaityRunpodRouter(_BaseBackend, _BaseFileHandlingMixin): """ Adds routing functionality for the runpod serverless framework. Provides enhanced file handling and conversion capabilities. """ - def __init__(self, title: str = "APIPod for ", summary: str = None, *args, **kwargs): + def __init__(self, title: str = "APIPod for ", summary: str = None, simulate: bool = False, prefix: str = "", *args, **kwargs): super().__init__(title=title, summary=summary, *args, **kwargs) - _RunPodLLMMixin.__init__(self) + # When True, start() runs RunPod's local API emulator instead of the real + # serverless worker (set by APIPod(simulate="serverless-runpod", direct=True)). + self.simulate = simulate + self.prefix = normalize_mount_prefix(prefix) self.routes = {} # routes are organized like {"ROUTE_NAME": "ROUTE_FUNCTION"} + self._endpoint_plans: dict[str, EndpointExecutionPlan] = {} + self._endpoint_source_funcs: dict[str, Callable] = {} self.add_standard_routes() + def _apply_mount_prefix(self, mount_prefix: str) -> None: + mount = normalize_mount_prefix(mount_prefix) + if mount: + self.prefix = mount + + def include_router( + self, + router: "SocaityRunpodRouter", + prefix: str = "", + **kwargs, + ) -> None: + """Mount another APIPod RunPod router under *prefix* (path prefix only).""" + del kwargs + if not isinstance(router, SocaityRunpodRouter): + raise TypeError( + f"APIPod include_router expects a SocaityRunpodRouter instance, got {type(router)!r}" + ) + mount = normalize_mount_prefix(prefix) + router._apply_mount_prefix(mount) + head = mount.strip("/") + for path, route in router.routes.items(): + if path == "openapi.json": + continue + prefixed = f"{head}/{path.strip('/')}" if head else path.strip("/") + self.routes[prefixed] = route + if path in router._endpoint_plans: + self._endpoint_plans[prefixed] = router._endpoint_plans[path] + if path in router._endpoint_source_funcs: + self._endpoint_source_funcs[prefixed] = router._endpoint_source_funcs[path] + def add_standard_routes(self): self.endpoint(path="openapi.json")(self.get_openapi_schema) @@ -43,97 +82,77 @@ def endpoint(self, path: str = None, use_queue: bool = None, *args, **kwargs): path = normalize_name(path, preserve_paths=True).strip("/") def decorator(func: Callable) -> Callable: - # 1. Auto-Detection - req_model, res_model, endpoint_type = self._get_llm_config(func) - - @functools.wraps(func) - def wrapper(*w_args, **w_kwargs): - self.status = constants.SERVER_HEALTH.BUSY - - try: - if req_model: - payload = w_kwargs.get("payload", None) - - openai_req = self._prepare_llm_payload( - req_model=req_model, - payload=payload - ) - - w_kwargs["payload"] = openai_req - - return self.handle_llm_request( - func=func, - openai_req=openai_req, - req_model=req_model, - res_model=res_model, - endpoint_type=endpoint_type, - w_args=w_args, - w_kwargs=w_kwargs - ) - - # Default execution for standard endpoints - return self._execute_sync_or_async(func, w_args, w_kwargs) - finally: - self.status = constants.SERVER_HEALTH.RUNNING - - self.routes[path] = wrapper - return wrapper + plan = build_plan(func, path=path) + self._endpoint_plans[path] = plan + self._endpoint_source_funcs[path] = func + route = self._build_route(func, plan) + self.routes[path] = route + return route return decorator - def _yield_native_stream(self, func, args, kwargs): - """Bridge for RunPod native generator streaming.""" - from starlette.responses import StreamingResponse - - # Execute the function - result = self._execute_sync_or_async(func, args, kwargs) - - # If it's a StreamingResponse (shouldn't happen in RunPod, but handle it) - if isinstance(result, StreamingResponse): - body_iterator = result.body_iterator - - if inspect.isasyncgen(body_iterator): - while True: - try: - chunk = self._run_in_loop(body_iterator.__anext__()) - yield chunk if isinstance(chunk, (str, bytes)) else str(chunk) - except StopAsyncIteration: - break - else: - for chunk in body_iterator: - yield chunk if isinstance(chunk, (str, bytes)) else str(chunk) - return - - # Handle async generators - if inspect.isasyncgen(result): - while True: - try: - chunk = self._run_in_loop(result.__anext__()) - yield chunk if isinstance(chunk, (str, bytes)) else str(chunk) - except StopAsyncIteration: - break - return - - # Handle sync generators - if inspect.isgenerator(result): - for chunk in result: - yield chunk if isinstance(chunk, (str, bytes)) else str(chunk) - return - - # Not a generator - shouldn't happen for streaming - raise TypeError(f"Expected generator for streaming, got {type(result)}") + def _build_route(self, func: Callable, plan: EndpointExecutionPlan) -> Callable: + """ + Compose the callable registered for a route from its execution plan. - def _execute_sync_or_async(self, func, args, kwargs): - if inspect.iscoroutinefunction(func): - return self._run_in_loop(func(*args, **kwargs)) - return func(*args, **kwargs) + Schema endpoints validate + media-parse their request themselves; plain + endpoints get the generic media file-upload conversion wrapped in. + """ + @functools.wraps(func) + def wrapper(*w_args, **w_kwargs): + self.status = constants.SERVER_HEALTH.BUSY + try: + if plan.is_schema_endpoint: + return self._handle_schema_request(func, plan.schema_binding, w_args, w_kwargs) + result = self.run_callable(func, *w_args, **w_kwargs) + # Plain generator endpoints stream too: turn the generator into a + # RunPod-native stream of JSON-safe chunks (RunPod aggregates it). + native_stream = self._as_native_stream(result) + return native_stream if native_stream is not None else result + finally: + self.status = constants.SERVER_HEALTH.RUNNING - def _run_in_loop(self, coro): - try: - loop = asyncio.get_event_loop() - except RuntimeError: - loop = asyncio.new_event_loop() - asyncio.set_event_loop(loop) - return loop.run_until_complete(coro) + return wrapper if plan.is_schema_endpoint else self._handle_file_uploads(wrapper) + + # ------------------------------------------------------------------ + # Standardized schema endpoints + # ------------------------------------------------------------------ + def _handle_schema_request(self, func: Callable, binding: SchemaBinding, args: tuple, kwargs: dict): + """ + Run a standardized schema endpoint: validate + media-parse the request + (``prepare_schema_call``), then stream or wrap the response into the + registered response model (``wrap_schema_response``). + """ + request = prepare_schema_call(binding, kwargs) + result = self.run_callable(func, *args, **kwargs) + + if getattr(request, "stream", False): + native_stream = self._as_native_stream(result, binding) + if native_stream is not None: + return native_stream + + return wrap_schema_response(result, binding) + + def _as_native_stream(self, result, binding: Union[SchemaBinding, None] = None) -> Union[Iterator, None]: + """ + Wrap a streamable result into a RunPod-native generator of JSON-safe chunks. + + - ``AudioFile`` / ``VideoFile`` → base64-encoded byte chunks; + - schema generator with a registered chunk model (e.g. chat) → standardized + ``ChatCompletionChunk`` stream via :class:`SchemaStreamSerializer`; + - any other generator → ``encode_chunk`` (base64 for binary, str for text). + + Returns ``None`` when the result is not streamable. + """ + if isinstance(result, (AudioFile, VideoFile)): + return (encode_chunk(chunk) for chunk in iter_media_chunks(result)) + + if is_streaming_result(result): + tokens = as_sync_iter(result) + if binding is not None and binding.tag in STREAM_CHUNK_SPECS: + return SchemaStreamSerializer(binding).stream(tokens) + return (encode_chunk(chunk) for chunk in tokens) + + return None def get(self, path: str = None, *args, **kwargs): return self.endpoint(path=path, *args, **kwargs) @@ -153,11 +172,7 @@ def _add_job_progress_to_kwargs(self, func, job, kwargs): Returns: Updated kwargs with job_progress added """ - job_progress_params = [] - for param in inspect.signature(func).parameters.values(): - if param.annotation in (JobProgress, JobProgressRunpod) or param.name == "job_progress": - job_progress_params.append(param.name) - + job_progress_params = job_progress_param_names(func) if job_progress_params: jp = JobProgressRunpod(job) for job_progress_param in job_progress_params: @@ -196,85 +211,37 @@ def _router(self, path, job, **kwargs): if missing_args: raise Exception(f"Arguments {missing_args} are missing") - # Handle file uploads and conversions - route_function = self._handle_file_uploads(route_function) - - start_time = datetime.now(timezone.utc) - result = JobResult(job_id=job["id"], endpoint=path) + # Create a BaseJob record so the result can be assembled by the same + # from_base_job path used by the local queue worker. + job_record = BaseJob(id=job["id"]) + job_record.metrics.started_at = datetime.now(timezone.utc) try: - # Execute the function (Sync or Async Handling) - res = self._execute_route_function(route_function, kwargs) + res = self.run_callable(route_function, **kwargs) - # Check if result is a generator (streaming response) + # Streaming response: hand the generator straight to RunPod, which + # aggregates it (no JobResult envelope is produced for streams). if inspect.isgenerator(res) or inspect.isasyncgen(res): - # For streaming, return the generator directly - # RunPod will handle the streaming return res - # Convert result to JSON if it's a MediaFile / MediaList / Pydantic Model - res = JobResultFactory._serialize_result(res) - - result.result = res - result.status = _job_status_to_public(JOB_STATUS.FINISHED) + job_record.result = res + job_record.status = JOB_STATUS.FINISHED except Exception as e: - result.error = str(e) - result.status = _job_status_to_public(JOB_STATUS.FAILED) + job_record.error = str(e) + job_record.status = JOB_STATUS.FAILED print(f"Job {job['id']} failed: {str(e)}") traceback.print_exc() + # Adopt the progress handle the function reported through, so its final + # progress/message flow into the JobResult. for arg in kwargs.values(): if isinstance(arg, JobProgressRunpod): - result.progress = float(arg._progress) - result.message = arg._message + job_record.job_progress = arg break - inference_time_s = _compute_duration_s(start_time, datetime.now(timezone.utc)) - if inference_time_s is not None: - result.metrics = JobMetrics(inference_time_s=inference_time_s) + job_record.metrics.finished_at = datetime.now(timezone.utc) - return result.model_dump_json(exclude_none=True) - - def _execute_route_function(self, route_function, kwargs): - """ - Execute a route function, handling both sync and async functions. - - Args: - route_function: The function to execute - kwargs: Keyword arguments to pass to the function - - Returns: - The result of the function execution - """ - if not inspect.iscoroutinefunction(route_function): - # Synchronous function - simple execution - return route_function(**kwargs) - - # Async function - need event loop handling - try: - loop = asyncio.get_event_loop() - if loop.is_running(): - # Already in async context - shouldn't happen in RunPod handler - # But if it does, we need to handle it differently - import concurrent.futures - with concurrent.futures.ThreadPoolExecutor() as executor: - future = executor.submit( - asyncio.run, - route_function(**kwargs) - ) - return future.result() - else: - # Loop exists but not running - use it - return loop.run_until_complete(route_function(**kwargs)) - except RuntimeError: - # No event loop exists - create one - loop = asyncio.new_event_loop() - asyncio.set_event_loop(loop) - try: - return loop.run_until_complete(route_function(**kwargs)) - finally: - loop.close() - asyncio.set_event_loop(None) + return JobResultFactory.from_base_job(job_record).model_dump_json(exclude_none=True) def handler(self, job): """ @@ -288,11 +255,11 @@ def handler(self, job): del inputs["path"] result = self._router(route, job, **inputs) - + # If it's a generator, return it directly for RunPod to stream if inspect.isgenerator(result): return result - + # Otherwise return the JSON result return result @@ -316,8 +283,13 @@ def start_runpod_serverless_localhost(self, port): rp_fastapi.RUN_DESCRIPTION = desc + "\n" + rp_fastapi.RUN_DESCRIPTION # hack to print version also in runpod - version = self.version - + # Add APIPod manifest like in the FastAPI router + manifest = { + "compute": "serverless", + "version": self.version, + "simulate": self.simulate, + } + class WorkerAPIWithModifiedInfo(rp_fastapi.WorkerAPI): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) @@ -327,7 +299,7 @@ def __init__(self, *args, **kwargs): def custom_openapi(self): if not self.rp_app.openapi_schema: self._orig_openapi_func() - self.rp_app.openapi_schema["info"]["apipod"] = version + self.rp_app.openapi_schema["info"]["apipod"] = manifest self.rp_app.openapi_schema["info"]["runpod"] = rp_fastapi.runpod_version return self.rp_app.openapi_schema @@ -335,16 +307,20 @@ def custom_openapi(self): runpod.serverless.start({"handler": self.handler, "return_aggregate_stream": True}) - def _create_openapi_compatible_function(self, func: Callable) -> Callable: + def _create_openapi_compatible_function( + self, + func: Callable, + plan: EndpointExecutionPlan | None = None, + ) -> Callable: """ - Create a function compatible with FastAPI OpenAPI generation by applying + Create a function compatible with FastAPI OpenAPI generation by applying the same conversion logic as the FastAPI mixin, but without runtime dependencies. This generates the rich schema with proper file upload handling. Args: func: Original function to convert - max_upload_file_size_mb: Maximum file size in MB + plan: Endpoint execution plan (controls ``stream`` in request schema) Returns: Function with FastAPI-compatible signature for OpenAPI generation @@ -357,7 +333,9 @@ def _create_openapi_compatible_function(self, func: Callable) -> Callable: # Create a temporary instance of the FastAPI mixin to use its conversion methods temp_mixin = _fast_api_file_handling_mixin(max_upload_file_size_mb=5) # Apply the same preparation logic as FastAPI router - with_file_upload_signature = temp_mixin._prepare_func_for_media_file_upload_with_fastapi(func, 5) + with_file_upload_signature = temp_mixin._prepare_func_for_media_file_upload_with_fastapi( + func, 5, plan=plan, + ) # 4. Set proper return type for job-based endpoints sig = inspect.signature(with_file_upload_signature) @@ -429,9 +407,11 @@ def get_openapi_schema(self): fastapi_routes = [] for path, func in self.routes.items(): + source_func = self._endpoint_source_funcs.get(path, func) + plan = self._endpoint_plans.get(path) # Create FastAPI-compatible function for rich OpenAPI generation try: - compatible_func = self._create_openapi_compatible_function(func) + compatible_func = self._create_openapi_compatible_function(source_func, plan) fastapi_routes.append(APIRoute( path=f"/{path.strip('/')}", endpoint=compatible_func, @@ -471,19 +451,24 @@ def minimal_func(): description=self.summary, ) - # Add APIPod version information like the FastAPI router - schema["info"]["apipod"] = self.version + # Add APIPod manifest like in the FastAPI router + manifest = { + "compute": "serverless", + "version": self.version, + "simulate": self.simulate, + } + schema["info"]["apipod"] = manifest return schema - def start(self, port: int = APIPOD_PORT, provider: Union[constants.PROVIDER, str, None] = None, *args, **kwargs): - if provider is None: - provider = APIPOD_PROVIDER - if isinstance(provider, str): - provider = constants.PROVIDER(provider) + def start(self, port: int = APIPOD_PORT, **kwargs): + """Start the RunPod worker. - if provider == constants.PROVIDER.LOCALHOST: + In simulation (``APIPod(simulate="serverless-runpod", direct=True)``) RunPod's + local API emulator is used. In a managed deployment the real serverless worker runs. + """ + if self.simulate: self.start_runpod_serverless_localhost(port=port) else: import runpod.serverless - runpod.serverless.start({"handler": self.handler, "return_aggregate_stream": True}) + runpod.serverless.start({"handler": self.handler, "return_aggregate_stream": True}) \ No newline at end of file diff --git a/apipod/engine/backend/schema_resolve.py b/apipod/engine/backend/schema_resolve.py new file mode 100644 index 0000000..bb3a1e0 --- /dev/null +++ b/apipod/engine/backend/schema_resolve.py @@ -0,0 +1,431 @@ +""" +Registry and backend-neutral helpers for standardized (OpenAI-compatible) schemas. + +``SCHEMA_REGISTRY`` is the single source of truth: it maps every supported +request schema to how it is served (response model + semantic tag). Everything +else derives from it: + +- :func:`resolve_request_model` / :func:`get_schema_binding` detect schema + endpoints by annotation (used by routers and signature policies), +- module-level helpers prepare the incoming request (validation + nested-media + parsing) and wrap raw endpoint results into response models. + +Schemas are a stable part of APIPod, not an optional extension. Routers keep the +normal decorator pipeline; schema helpers are called from the existing standard, +task and streaming decorators where needed. + +Note: response ``id`` fields are intentionally NOT populated here — identifiers +belong to the ``JobResult`` envelope produced by the platform, not to the model +schemas themselves. +""" + +import inspect +import uuid +from dataclasses import dataclass +from datetime import datetime, timezone +from types import UnionType +from typing import Any, Callable, Iterable, Iterator, Optional, Type, Union, get_args, get_origin + +from pydantic import BaseModel, Field, create_model +from pydantic.json_schema import SkipJsonSchema +from media_toolkit import MediaFile + +from socaity_schemas import ( + ChatCompletionChunk, + ChatCompletionChoice, + ChatCompletionRequest, + ChatCompletionResponse, + ChatDelta, + ChatStreamChoice, + CompletionRequest, + CompletionResponse, + CreateVoiceRequest, + EmbeddingRequest, + EmbeddingResponse, + Generation3DRequest, + Generation3DResponse, + ImageGenerationRequest, + ImageGenerationResponse, + MultimodalEmbeddingRequest, + MultimodalEmbeddingResponse, + SpeechRequest, + SpeechResponse, + TranscriptionRequest, + TranscriptionResponse, + VideoGenerationRequest, + VideoGenerationResponse, + VisionRequest, + VisionResponse, + VoiceConversionRequest, + VoiceConversionResponse, + VoiceResponse, +) +from apipod.engine.files.base_file_mixin import parse_schema_media_fields +from apipod.engine.signatures.analysis import is_injected_progress_param + + +@dataclass(frozen=True) +class SchemaEndpointSpec: + """How a registered request schema is served.""" + response_model: Type + tag: str + + +# Single source of truth for all standardized schemas. Adding a schema here +# automatically enables endpoint detection, body-parameter policies and +# response wrapping everywhere. +SCHEMA_REGISTRY: dict[Type, SchemaEndpointSpec] = { + ChatCompletionRequest: SchemaEndpointSpec(ChatCompletionResponse, "chat"), + CompletionRequest: SchemaEndpointSpec(CompletionResponse, "completion"), + EmbeddingRequest: SchemaEndpointSpec(EmbeddingResponse, "embedding"), + ImageGenerationRequest: SchemaEndpointSpec(ImageGenerationResponse, "image_generation"), + VideoGenerationRequest: SchemaEndpointSpec(VideoGenerationResponse, "video_generation"), + TranscriptionRequest: SchemaEndpointSpec(TranscriptionResponse, "transcription"), + SpeechRequest: SchemaEndpointSpec(SpeechResponse, "speech"), + CreateVoiceRequest: SchemaEndpointSpec(VoiceResponse, "voice"), + VoiceConversionRequest: SchemaEndpointSpec(VoiceConversionResponse, "voice_conversion"), + Generation3DRequest: SchemaEndpointSpec(Generation3DResponse, "generation_3d"), + VisionRequest: SchemaEndpointSpec(VisionResponse, "vision"), + MultimodalEmbeddingRequest: SchemaEndpointSpec(MultimodalEmbeddingResponse, "embedding_multimodal"), +} + +# Tags whose streams are token deltas (served as server-sent events). Media +# tags stream raw encoded bytes instead, matching OpenAI's stream_format="audio". +SSE_STREAM_TAGS = {"chat", "completion", "transcription"} + + +@dataclass(frozen=True) +class SchemaBinding: + """Resolved binding between an endpoint function parameter and a registered schema.""" + param_name: str + request_model: Type + response_model: Type + tag: str + + +def _registry_spec(request_model: Type) -> Optional[SchemaEndpointSpec]: + """Find the registry spec for a request model, honoring subclasses of registered schemas.""" + if not inspect.isclass(request_model): + return None + for cls in request_model.__mro__: + if cls in SCHEMA_REGISTRY: + return SCHEMA_REGISTRY[cls] + return None + + +_OPENAPI_WITHOUT_STREAM_CACHE: dict[type, type] = {} +_OPENAPI_DOC_TO_SOURCE: dict[type, type] = {} + + +def source_request_model(model: type[BaseModel]) -> type[BaseModel]: + """Return the author-facing schema for an OpenAPI doc model, or *model* itself.""" + return _OPENAPI_DOC_TO_SOURCE.get(model, model) + + +def openapi_request_model(model: type[BaseModel], *, is_streaming: bool) -> type[BaseModel]: + """Return *model*, or a cached OpenAPI variant that hides ``stream`` when not streaming. + + When ``is_streaming`` is false and the model declares a ``stream`` field, a sibling + model is built via :func:`pydantic.create_model` with ``SkipJsonSchema`` on ``stream``. + The field stays in validation (OpenAI clients may send ``stream: false``) but is + omitted from the generated OpenAPI/JSON schema. + """ + if is_streaming or "stream" not in model.model_fields: + return model + if model in _OPENAPI_WITHOUT_STREAM_CACHE: + return _OPENAPI_WITHOUT_STREAM_CACHE[model] + + stream_field = model.model_fields["stream"] + fields = {} + for name, field in model.model_fields.items(): + if name == "stream": + fields[name] = ( + SkipJsonSchema[bool], + Field(default=False, description=stream_field.description), + ) + else: + fields[name] = (field.annotation, field) + + doc_model = create_model( + f"{model.__name__}OpenAPI", + __config__=model.model_config, + **fields, + ) + _OPENAPI_WITHOUT_STREAM_CACHE[model] = doc_model + _OPENAPI_DOC_TO_SOURCE[doc_model] = model + return doc_model + + +def openapi_schema_annotation(annotation: Any, *, is_streaming: bool) -> Any: + """Apply :func:`openapi_request_model` to a parameter annotation (incl. Optional).""" + resolved = resolve_request_model(annotation) + if resolved is None: + return annotation + + doc_model = openapi_request_model(resolved, is_streaming=is_streaming) + if doc_model is resolved: + return annotation + + origin = get_origin(annotation) + if origin in (Union, UnionType): + args = get_args(annotation) + new_args = tuple( + doc_model if resolve_request_model(arg) is not None else arg + for arg in args + ) + return origin[new_args] + return doc_model + + +def resolve_request_model(annotation: Any) -> Optional[Type]: + """ + Resolve an annotation to a registered request schema (or a subclass of one). + Supports direct annotations and Optional/Union annotations. + """ + candidates = [annotation] + if get_origin(annotation) in (Union, UnionType): + candidates = [arg for arg in get_args(annotation) if arg is not type(None)] + + for candidate in candidates: + if _registry_spec(candidate) is not None: + return candidate + return None + + +def get_schema_binding(func: Callable) -> Optional[SchemaBinding]: + """ + Detect the schema-typed parameter of an endpoint function by annotation — the + parameter may have any name, mirroring how ``JobProgress`` is detected. + + Returns ``None`` for plain (non-schema) functions. When a schema parameter is + found, the rest of the signature is validated: a schema endpoint must take the + request schema as its only user input (``job_progress`` and framework + dependencies aside), so that the whole request lives in the schema body. + """ + for param in inspect.signature(func).parameters.values(): + request_model = resolve_request_model(param.annotation) + if request_model is None: + continue + + spec = _registry_spec(request_model) + binding = SchemaBinding( + param_name=param.name, + request_model=request_model, + response_model=spec.response_model, + tag=spec.tag, + ) + _validate_schema_endpoint_signature(func, binding) + return binding + return None + + +def _is_injected_param(param: inspect.Parameter) -> bool: + """True for parameters the framework supplies (job progress, self/cls).""" + return param.name in ("self", "cls") or is_injected_progress_param(param) + + +def _validate_schema_endpoint_signature(func: Callable, binding: SchemaBinding) -> None: + """Reject schema endpoints that declare user parameters besides the request schema.""" + from apipod.engine.signatures.policies import FastAPISignaturePolicies + + extra = [ + param.name + for param in inspect.signature(func).parameters.values() + if param.name != binding.param_name + and not _is_injected_param(param) + and not FastAPISignaturePolicies.is_fastapi_dependency(param) + ] + if extra: + raise TypeError( + f"Schema endpoint '{func.__name__}' must take the request schema " + f"'{binding.request_model.__name__}' as its only parameter, but also declares " + f"{extra}. Move these inputs into the schema (or a subclass of it)." + ) + + +def prepare_schema_call(binding: SchemaBinding, call_kwargs: dict): + """ + Replace the raw request body in call kwargs with a validated schema object + and return that object. + """ + payload = call_kwargs.get(binding.param_name) + if payload is None: + raise ValueError( + f"Request body for parameter '{binding.param_name}' " + f"({binding.request_model.__name__}) is missing" + ) + + # Prepare the request object + if isinstance(payload, binding.request_model): + request = payload + elif isinstance(payload, dict): + request = binding.request_model.model_validate(payload) + else: + raise ValueError(f"Invalid payload for {binding.request_model.__name__}: {type(payload).__name__}") + request = parse_schema_media_fields(request) + + call_kwargs[binding.param_name] = request + return request + + +def wrap_schema_response(result: Any, binding: SchemaBinding) -> Any: + """ + Wrap a raw endpoint result into the response model of the binding. + + Response models share a uniform envelope (``created``, ``model``) where + applicable. Raw results (strings, vectors, media files, dicts) are normalized + into the target response model. + """ + response_model = binding.response_model + if isinstance(result, response_model): + return result + + payload = _normalize_response_model(result, response_model) + payload = _apply_envelope(payload, response_model) + return response_model.model_validate(payload) + + +# Empty values for required JSON-schema property types (`created` is filled by the envelope). +_SCHEMA_EMPTY = {"string": "", "array": [], "integer": 0, "object": {}} + + +def _schema_defaults(model: Type[BaseModel]) -> dict: + """Required-field empties from pydantic's JSON schema.""" + schema = model.model_json_schema() + properties = schema.get("properties") or {} + # Only top-level required fields; nested item shapes are validated by pydantic later. + return { + name: _SCHEMA_EMPTY[prop["type"]] + for name in schema.get("required") or [] + if name != "created" and (prop := properties.get(name)) and prop.get("type") in _SCHEMA_EMPTY + } + + +def _normalize_response_model(result: Any, response_model: Type) -> dict: + """ + Normalize raw endpoint results into a dictionary matching the response model. + Handles convenient raw shapes for chat, completion, embedding and transcription. + """ + # None → empty text or dict; dict gaps are filled from schema defaults below. + if result is None: + result = "" if response_model in (ChatCompletionResponse, CompletionResponse, TranscriptionResponse) else {} + + # Shorthand raw returns authors may use instead of a full response dict. + if response_model is ChatCompletionResponse and isinstance(result, str): + result = {"choices": [{"index": 0, "message": {"content": result}, "finish_reason": "stop"}]} + elif response_model is CompletionResponse and isinstance(result, str): + result = {"choices": [{"text": result, "index": 0, "finish_reason": "stop"}]} + elif response_model is EmbeddingResponse and isinstance(result, list): + if not result: + result = {"data": []} + elif isinstance(result[0], (list, tuple)): + result = {"data": [{"embedding": list(vector), "index": i} for i, vector in enumerate(result)]} + else: + result = {"data": [{"embedding": list(result), "index": 0}]} + elif response_model is TranscriptionResponse and isinstance(result, str): + result = {"text": result} + + payload = _normalize_result(result, response_model) + if not isinstance(payload, dict): + return payload + # Defaults first, then author payload; drop None so defaults apply. + return {**_schema_defaults(response_model), **{k: v for k, v in payload.items() if v is not None}} + + +def _normalize_result(result: Any, response_model: Type) -> dict: + """Bring raw results into dict form; media files are lifted into the `data` list.""" + # Single file or file list → standard media envelope shape. + if isinstance(result, MediaFile): + return {"data": [result]} + if isinstance(result, (list, tuple)) and result and all(isinstance(item, MediaFile) for item in result): + return {"data": list(result)} + if isinstance(result, dict): + return dict(result) + raise ValueError( + f"Cannot wrap a result of type {type(result).__name__} into {response_model.__name__}. " + f"Return a {response_model.__name__}, a dict matching its fields, or a media-toolkit file." + ) + + +def _apply_envelope(payload: dict, response_model: Type) -> dict: + """Fill the shared response envelope (``created`` timestamp, request ``model``) when applicable.""" + fields = response_model.model_fields + if "created" in fields and payload.get("created") is None: + payload["created"] = int(datetime.now(timezone.utc).timestamp()) + + return payload + + +def iter_media_chunks(media_file: MediaFile, chunk_size: int = 64 * 1024) -> Iterator[bytes]: + """Yield the encoded bytes of a media file in chunks (for raw media streaming).""" + buffer = media_file.to_bytes_io() + while chunk := buffer.read(chunk_size): + yield chunk + + +# ---------------------------------------------------------------------------- +# Streaming: wrap raw token deltas into standardized chunk SSE events +# ---------------------------------------------------------------------------- + +SSE_DONE = "data: [DONE]\n\n" + + +def _to_sse(chunk: BaseModel) -> str: + """Serialize a chunk model into a single server-sent-event data line.""" + return f"data: {chunk.model_dump_json()}\n\n" + + +def _build_chat_chunk(chunk_id: str, created: int, content: Optional[str], finish_reason: Optional[str]) -> ChatCompletionChunk: + return ChatCompletionChunk( + id=chunk_id, + created=created, + choices=[ChatStreamChoice(index=0, delta=ChatDelta(content=content), finish_reason=finish_reason)], + ) + + +@dataclass(frozen=True) +class StreamChunkSpec: + """How raw token deltas of a streaming schema are wrapped into chunk events.""" + id_prefix: str + build: Callable[[str, int, Optional[str], Optional[str]], BaseModel] + + +# Schema tags whose token stream is wrapped into a standardized chunk SSE stream. +# Endpoints for these tags may simply yield text tokens; APIPod owns the envelope. +STREAM_CHUNK_SPECS: dict[str, StreamChunkSpec] = { + "chat": StreamChunkSpec(id_prefix="chatcmpl", build=_build_chat_chunk), +} + + +class SchemaStreamSerializer: + """ + Turns the raw token deltas yielded by a streaming schema endpoint into its + standardized chunk SSE stream (e.g. ``ChatCompletionChunk``). + + The endpoint only yields text tokens; APIPod owns the envelope: a stable + chunk id, the ``created`` timestamp and the ``object`` discriminator are + generated out of the box, closed by a final delta and the ``[DONE]`` sentinel. + """ + + def __init__(self, binding: SchemaBinding): + spec = STREAM_CHUNK_SPECS.get(binding.tag) + if spec is None: + raise ValueError(f"Schema '{binding.tag}' does not support streaming chunks.") + self._build = spec.build + self._chunk_id = f"{spec.id_prefix}-{uuid.uuid4().hex[:8]}" + self._created = int(datetime.now(timezone.utc).timestamp()) + + def delta(self, content: str) -> str: + """Serialize a content token into a chunk SSE event.""" + return _to_sse(self._build(self._chunk_id, self._created, content, None)) + + def finish(self) -> str: + """Serialize the closing chunk (``finish_reason='stop'``).""" + return _to_sse(self._build(self._chunk_id, self._created, None, "stop")) + + def stream(self, tokens: Iterable[str]) -> Iterator[str]: + """Wrap a synchronous token iterable into the full chunk SSE stream (deltas, closing chunk, [DONE]).""" + for token in tokens: + yield self.delta(token) + yield self.finish() + yield SSE_DONE diff --git a/apipod/engine/base_backend.py b/apipod/engine/base_backend.py index 64cd5d2..3830516 100644 --- a/apipod/engine/base_backend.py +++ b/apipod/engine/base_backend.py @@ -1,5 +1,9 @@ +import asyncio +import concurrent.futures +import functools +import inspect from abc import abstractmethod -from typing import Union +from typing import Any, Callable, Union import importlib.metadata from apipod.common import constants from apipod.engine.compatibility.HealthCheck import HealthCheck @@ -23,6 +27,61 @@ def __init__( self._health_check = HealthCheck() self.version = importlib.metadata.version("apipod") + # ------------------------------------------------------------------ + # Execution runtime + # + # Executing a user-registered endpoint while transparently resolving + # sync/async is the one operation every backend shares. There are exactly + # two authoritative entry points so behaviour can never drift between the + # worker thread, the RunPod handler and the direct FastAPI path. + # ------------------------------------------------------------------ + def run_callable(self, func: Callable, *args: Any, **kwargs: Any) -> Any: + """Execute *func* from a synchronous context, resolving coroutines. + + Used by the two real sync execution sites: the local queue worker thread + and the RunPod handler. Sync functions run directly; coroutine functions + are driven to completion on an event loop (a fresh one is created and torn + down when no usable loop exists, or when the current loop is already + running the call is offloaded to a worker thread). + """ + if not inspect.iscoroutinefunction(func): + return func(*args, **kwargs) + + coro = func(*args, **kwargs) + try: + loop = asyncio.get_event_loop() + except RuntimeError: + loop = None + + if loop is not None and not loop.is_running(): + return loop.run_until_complete(coro) + + if loop is not None and loop.is_running(): + # Already inside a running loop (rare in sync handlers): run the + # coroutine to completion on a separate thread to avoid re-entrancy. + with concurrent.futures.ThreadPoolExecutor() as executor: + return executor.submit(asyncio.run, coro).result() + + new_loop = asyncio.new_event_loop() + asyncio.set_event_loop(new_loop) + try: + return new_loop.run_until_complete(coro) + finally: + new_loop.close() + asyncio.set_event_loop(None) + + async def run_callable_async(self, func: Callable, *args: Any, **kwargs: Any) -> Any: + """Execute *func* from an async context without blocking the event loop. + + Used by the direct (non-queued) FastAPI response path. Coroutine + functions are awaited; sync functions are offloaded to the default + thread-pool executor. + """ + if inspect.iscoroutinefunction(func): + return await func(*args, **kwargs) + loop = asyncio.get_event_loop() + return await loop.run_in_executor(None, functools.partial(func, *args, **kwargs)) + @property def status(self) -> constants.SERVER_HEALTH: return self._health_check.status @@ -56,11 +115,15 @@ def cancel_job(self, job_id: str): def start(self, port: int = APIPOD_PORT, *args, **kwargs): raise NotImplementedError("Implement in subclass") + def include_router(self, router, prefix: str = "", **kwargs): + """Mount a nested APIPod router under *prefix* (FastAPI ``include_router`` semantics).""" + raise NotImplementedError("Implement in subclass") + def endpoint(self, path: str = None, *args, **kwargs): """ Add a route to the app. Can be a task route (async job) or standard route depending on configuration. - + :param path: In case of fastapi will be resolved as url in form http://{host:port}/{prefix}/{path} In case of runpod will be resolved as url in form http://{host:port}?route={path} diff --git a/apipod/engine/endpoint_config.py b/apipod/engine/endpoint_config.py index 4f198ae..71fd96b 100644 --- a/apipod/engine/endpoint_config.py +++ b/apipod/engine/endpoint_config.py @@ -1,6 +1,13 @@ -from dataclasses import dataclass +""" +Endpoint planning: backend-neutral analysis of an endpoint function. +""" + +from dataclasses import dataclass, field from typing import Any, Callable +from apipod.engine.backend.schema_resolve import SchemaBinding, get_schema_binding +from apipod.engine.signatures.analysis import is_streaming_endpoint + @dataclass(frozen=True) class EndpointExecutionPlan: @@ -8,66 +15,55 @@ class EndpointExecutionPlan: Immutable plan describing how an endpoint should be registered and executed. The plan separates endpoint configuration decisions from router registration - mechanics. This keeps endpoint decorators compact and easier to reason about. + mechanics. Fields that only apply to a specific backend (e.g. queue options) + default to sensible no-ops so every backend can build a plan with what it needs. """ path: str - methods: list[str] | None - should_use_queue: bool - max_upload_file_size_mb: int | None - queue_size: int - route_args: tuple[Any, ...] - route_kwargs: dict[str, Any] - request_model: type | None = None - response_model: type | None = None - endpoint_type: str | None = None + methods: list[str] | None = None + should_use_queue: bool = False + max_upload_file_size_mb: int | None = None + queue_size: int = 500 + route_args: tuple[Any, ...] = () + route_kwargs: dict[str, Any] = field(default_factory=dict) + schema_binding: SchemaBinding | None = None is_streaming: bool = False @property - def is_llm(self) -> bool: - return self.request_model is not None + def is_schema_endpoint(self) -> bool: + return self.schema_binding is not None @property def active_methods(self) -> list[str]: return ["POST"] if self.methods is None else self.methods -class FastApiEndpointConfigurator: - """ - Builds endpoint execution plans for the FastAPI backend. +def build_plan( + func: Callable, + path: str, + *, + methods: list[str] | None = None, + should_use_queue: bool = False, + max_upload_file_size_mb: int | None = None, + queue_size: int = 500, + route_args: tuple[Any, ...] = (), + route_kwargs: dict[str, Any] | None = None, +) -> EndpointExecutionPlan: + """Build a backend-neutral :class:`EndpointExecutionPlan` by inspecting *func*. - This component configures endpoint behavior (LLM, streaming, queue) - independent from provider mechanics (FastAPI routing). + Both the FastAPI and RunPod backends call this. Backend-specific parameters + (``methods``, ``queue_*``, ``route_*``) are optional; they default to no-ops + so RunPod can simply call ``build_plan(func, path=path)``. """ - - def __init__(self, router): - self._router = router - - def build_plan( - self, - *, - func: Callable, - path: str, - methods: list[str] | None, - max_upload_file_size_mb: int | None, - queue_size: int, - should_use_queue: bool, - route_args: tuple[Any, ...], - route_kwargs: dict[str, Any], - ) -> EndpointExecutionPlan: - request_model, response_model, endpoint_type = self._router._get_llm_config(func) - is_streaming = bool(request_model is None and self._router._determine_generator_fun(func)) - - return EndpointExecutionPlan( - path=path, - methods=methods, - should_use_queue=should_use_queue, - max_upload_file_size_mb=max_upload_file_size_mb, - queue_size=queue_size, - route_args=route_args, - route_kwargs=route_kwargs, - request_model=request_model, - response_model=response_model, - endpoint_type=endpoint_type, - is_streaming=is_streaming, - ) + schema_binding = get_schema_binding(func) + return EndpointExecutionPlan( + path=path, + methods=methods, + should_use_queue=should_use_queue, + max_upload_file_size_mb=max_upload_file_size_mb, + queue_size=queue_size, + route_args=route_args, + route_kwargs=route_kwargs if route_kwargs is not None else {}, + schema_binding=schema_binding, + is_streaming=is_streaming_endpoint(func, schema_binding), + ) diff --git a/apipod/engine/files/base_file_mixin.py b/apipod/engine/files/base_file_mixin.py index eeaf24f..c6f78d9 100644 --- a/apipod/engine/files/base_file_mixin.py +++ b/apipod/engine/files/base_file_mixin.py @@ -1,14 +1,125 @@ import functools import inspect +import json from types import UnionType -from typing import Any, Union, get_args, get_origin, Callable, List, Type +from typing import Any, Optional, Union, get_args, get_origin, Callable, List, Type -from media_toolkit import media_from_any, MediaFile, MediaList, MediaDict +from pydantic import BaseModel + +from media_toolkit import media_from_any, MediaFile, MediaList, MediaDict, ImageFile, AudioFile, VideoFile from apipod.engine.signatures.upload import is_param_media_toolkit_file -from apipod.engine.jobs.job_result import FileModel +from socaity_schemas import AudioFileModel, FileModel, ImageFileModel, VideoFileModel from apipod.common.exceptions import FileUploadException +# Wire-format pydantic file models mapped to the media-toolkit type they parse into. +_FILE_MODEL_MEDIA_TYPES: dict = { + ImageFileModel: ImageFile, + AudioFileModel: AudioFile, + VideoFileModel: VideoFile, + FileModel: MediaFile, +} + + +def _media_type_for_file_model(file_model_cls: Type) -> Type: + """Resolve the media-toolkit type a FileModel subclass should be parsed into.""" + for cls in file_model_cls.__mro__: + if cls in _FILE_MODEL_MEDIA_TYPES: + return _FILE_MODEL_MEDIA_TYPES[cls] + return MediaFile + + +def _file_model_class_from_annotation(annotation: Any) -> Optional[Type[FileModel]]: + """Resolve a FileModel subclass from a parameter annotation (also inside unions).""" + if inspect.isclass(annotation) and issubclass(annotation, FileModel): + return annotation + if get_origin(annotation) in (Union, UnionType): + for arg in get_args(annotation): + if arg is type(None): + continue + resolved = _file_model_class_from_annotation(arg) + if resolved is not None: + return resolved + return None + + +def _coerce_to_file_model(value: Any, file_model_cls: Type[FileModel]) -> Any: + """Parse wire-format FileModel payloads (dict, JSON string) into pydantic models.""" + if isinstance(value, FileModel): + content = value.content + if isinstance(content, str) and content.strip().startswith("{"): + try: + parsed = json.loads(content.strip()) + if isinstance(parsed, dict) and {"file_name", "content_type", "content"}.issubset(parsed.keys()): + return file_model_cls.model_validate(parsed) + except (json.JSONDecodeError, ValueError): + pass + return value + if isinstance(value, dict): + return file_model_cls.model_validate(value) + if isinstance(value, str): + stripped = value.strip() + if stripped.startswith("{"): + try: + return file_model_cls.model_validate(json.loads(stripped)) + except (json.JSONDecodeError, ValueError): + pass + return value + + +def _is_media_list_type(target_type: Any) -> bool: + """True for ``MediaList`` and parameterized forms like ``MediaList[ImageFile]``.""" + if target_type is MediaList: + return True + return get_origin(target_type) is MediaList + + +def _coerce_wire_list(value: Any) -> Any: + """Parse JSON list strings from multipart form fields.""" + if isinstance(value, str): + stripped = value.strip() + if stripped.startswith("["): + try: + return json.loads(stripped) + except json.JSONDecodeError: + pass + return value + + +def _parse_file_model_value(value: Any) -> Any: + """Convert FileModel instances (also inside lists) into parsed media-toolkit objects.""" + if isinstance(value, FileModel): + return media_from_any( + data=value.model_dump(include={"file_name", "content_type", "content"}), + type_hint=_media_type_for_file_model(type(value)), + use_temp_file=True, + temp_dir=None, + allow_reads_from_disk=False, + ) + if isinstance(value, list): + return [_parse_file_model_value(item) for item in value] + return value + + +def parse_schema_media_fields(schema_obj): + """ + Replace FileModel-typed fields of a validated request schema with parsed + media-toolkit objects, so endpoint functions receive ready-to-use media — + exactly as method-level ``def endpoint(image: ImageFile)`` parameters do. + + Lives in the file-handling layer because it is the schema-side counterpart + of the parameter-level upload conversion (``_convert_param_to_media_file``). + """ + for field_name in type(schema_obj).model_fields: + value = getattr(schema_obj, field_name, None) + parsed = _parse_file_model_value(value) + if parsed is not value: + # The runtime type (e.g. ImageFile) intentionally differs from the wire + # annotation (ImageFileModel); bypass pydantic's assignment validation. + object.__setattr__(schema_obj, field_name, parsed) + return schema_obj + + class _BaseFileHandlingMixin: """ Base mixin for handling file uploads and parameter conversions across different deployment environments. @@ -156,16 +267,22 @@ def _convert_param_to_media_file(self, param_value: Any, annotation: Any) -> Any return param_value try: + file_model_cls = _file_model_class_from_annotation(annotation) + if file_model_cls is not None: + coerced = _coerce_to_file_model(param_value, file_model_cls) + if isinstance(coerced, FileModel): + return _parse_file_model_value(coerced) + # Determine target type for conversion target_type = self._get_media_target_type(annotation) - if target_type == MediaList: + if _is_media_list_type(target_type): return MediaList( read_system_files=False, download_files=True, use_temp_file=True, - temp_dir=None - ).from_any(param_value) + temp_dir=None, + ).from_any(_coerce_wire_list(param_value)) # Attempt conversion m = media_from_any( @@ -230,6 +347,49 @@ def _read_upload_files(self, files: dict, media_types: dict, *args, **kwargs) -> raise ValueError(f"Could not parse file {key}. Check if the file is correct. Error: {str(e)}") return converted_files + def _resolve_schema_annotation(self, annotation: Any) -> Optional[Type]: + """ + Resolve an annotation (also ``Optional``/``Union``) to a pydantic request + schema whose nested FileModel fields should be parsed. + + FileModel variants are themselves media handled by the parameter-level upload + path (``_convert_param_to_media_file``), so they are excluded here. + """ + candidates = [annotation] + if get_origin(annotation) in (Union, UnionType): + candidates = [arg for arg in get_args(annotation) if arg is not type(None)] + + for candidate in candidates: + if inspect.isclass(candidate) and issubclass(candidate, BaseModel) and not issubclass(candidate, FileModel): + return candidate + return None + + def _parse_schema_params(self, schema_params: set, named_args: dict) -> dict: + """ + Parse nested FileModel fields inside request-schema arguments. + + Called by ``_handle_file_uploads`` before the media-param conversion so that + endpoint functions always receive ready-to-use media-toolkit objects regardless + of whether files arrive as top-level parameters or nested inside a schema. + + Args: + schema_params: Parameter names whose annotation resolved to a pydantic schema. + named_args: Current mapping of parameter name → value for this call. + + Returns: + Mapping of parameter name → schema instance with media fields replaced. + """ + parsed = {} + for name in schema_params: + value = named_args.get(name) + if value is None: + continue + try: + parsed[name] = parse_schema_media_fields(value) + except Exception as e: + raise FileUploadException(message=f"Could not parse media fields of '{name}': {e}") + return parsed + def _handle_file_uploads(self, func: Callable) -> Callable: """ Wrap a function to handle file uploads and conversions. @@ -243,7 +403,14 @@ def _handle_file_uploads(self, func: Callable) -> Callable: """ sig = inspect.signature(func) media_params = self._get_media_params(sig) - + + # Parameters typed as a pydantic request schema whose nested FileModel fields + # must be parsed into media-toolkit objects before the endpoint executes. + schema_params = { + name for name, annot in self._sig_to_annotations(sig).items() + if name not in media_params and self._resolve_schema_annotation(annot) is not None + } + param_names = list(sig.parameters.keys()) # Check if the original function is async @@ -254,6 +421,9 @@ async def file_upload_wrapper(*args, **kwargs): named_args = {param_names[i]: arg for i, arg in enumerate(args) if i < len(param_names)} named_args.update(kwargs) + # Parse nested media inside schema parameters + named_args.update(self._parse_schema_params(schema_params, named_args)) + # Convert media-related parameters files_to_process = { param_name: param_value @@ -268,7 +438,7 @@ async def file_upload_wrapper(*args, **kwargs): # Update arguments with converted files named_args.update(processed_files) - + # AWAIT the async function return await func(**named_args) else: @@ -278,6 +448,9 @@ def file_upload_wrapper(*args, **kwargs): named_args = {param_names[i]: arg for i, arg in enumerate(args) if i < len(param_names)} named_args.update(kwargs) + # Parse nested media inside schema parameters + named_args.update(self._parse_schema_params(schema_params, named_args)) + # Convert media-related parameters files_to_process = { param_name: param_value @@ -292,7 +465,7 @@ def file_upload_wrapper(*args, **kwargs): # Update arguments with converted files named_args.update(processed_files) - + # Call the sync function directly return func(**named_args) diff --git a/apipod/engine/jobs/base_job.py b/apipod/engine/jobs/base_job.py index c4e20e1..85fd4a8 100644 --- a/apipod/engine/jobs/base_job.py +++ b/apipod/engine/jobs/base_job.py @@ -1,47 +1,78 @@ from datetime import datetime, timedelta, timezone -from typing import Any, Optional -from uuid import uuid4 from enum import Enum +from typing import Any, FrozenSet, Optional +from uuid import uuid4 from apipod.engine.jobs.job_progress import JobProgress class JOB_STATUS(Enum): - QUEUED = "Queued" - PROCESSING = "Processing" - FINISHED = "Finished" - FAILED = "Failed" - TIMEOUT = "Timeout" + """Internal lifecycle states for APIPod's in-process job queue. + + Values are lowercase strings suitable for :class:`JobResult.status` when + APIPod runs standalone. + """ + QUEUED = "queued" + PROCESSING = "processing" + STREAMING = "streaming" + FINISHED = "finished" + FAILED = "failed" + TIMEOUT = "timeout" -class PROVIDERS(Enum): - RUNPOD = "runpod" - OPENAI = "openai" - REPLICATE = "replicate" + @property + def is_terminal(self) -> bool: + return self in { + JOB_STATUS.FINISHED, + JOB_STATUS.FAILED, + JOB_STATUS.TIMEOUT, + } -class BaseJob: - """Essential job record shared by all job types (local thread, remote service, etc.). +# Statuses where GET /stream may attach before the local stream store opens. +STREAM_WAIT_STATUSES: FrozenSet[str] = frozenset( + { + JOB_STATUS.QUEUED.value, + JOB_STATUS.PROCESSING.value, + JOB_STATUS.STREAMING.value, + } +) - Subclasses add domain-specific fields: - * :class:`LocalJob` — thread-pool execution (``job_function``, ``job_progress``). - * Platform-specific subclasses (e.g. gateway ``ServiceJob``) — remote/Redis - (``service_id``, ``api_url``, ``endpoint``, ``input_data``). +class JobMetrics: + """Storage for all job timing and performance data.""" + + def __init__(self): + self.created_at: datetime = datetime.now(timezone.utc) + self.queued_at: Optional[datetime] = None + self.started_at: Optional[datetime] = None + self.finished_at: Optional[datetime] = None + self.time_out_at: Optional[datetime] = None + + @property + def execution_time_s(self) -> Optional[float]: + if self.started_at and self.finished_at: + delta = (self.finished_at - self.started_at).total_seconds() + return round(delta, 2) if delta >= 0 else 0.0 + return None + - :class:`~apipod.engine.jobs.job_result.JobResultFactory.from_base_job` maps any - ``BaseJob`` subclass to the public :class:`~apipod.engine.jobs.job_result.JobResult`. +class BaseJob: + """ + Essential job record shared by all job types (local thread, remote service, etc.). + Subclass to add domain-specific fields like service_id, endpoint, etc. """ - def __init__(self, id=None, status="pending", created_at=None, updated_at=None): + def __init__(self, id: Optional[str] = None): self.id: str = id or str(uuid4()) - self.status = status + self.status = JOB_STATUS.QUEUED self.result: Any = None self.error: Optional[str] = None - self.created_at = created_at or datetime.now(timezone.utc) - self.updated_at = updated_at or datetime.now(timezone.utc) - self.progress: Optional[float] = None - self.message: Optional[str] = None + self.job_progress = JobProgress() + self.metrics = JobMetrics() + # True when the job's output may be consumed via GET /stream/{job_id}. + # Set at enqueue time by the queue; drives the ``links.stream`` hint. + self.supports_streaming: bool = False class LocalJob(BaseJob): @@ -50,31 +81,13 @@ class LocalJob(BaseJob): """ def __init__(self, job_function: callable, job_params: Optional[dict] = None, timeout_seconds: int = 3600): - super().__init__(status=JOB_STATUS.QUEUED) + super().__init__() self.job_function = job_function self.job_params: dict = job_params or {} - self.job_progress = JobProgress() - - self.queued_at: Optional[datetime] = None - self.execution_started_at: Optional[datetime] = None - self.execution_finished_at: Optional[datetime] = None - self.time_out_at = self.created_at + timedelta(seconds=timeout_seconds) + self.metrics.time_out_at = self.metrics.created_at + timedelta(seconds=timeout_seconds) @property def is_timed_out(self) -> bool: - return datetime.now(timezone.utc) > self.time_out_at - - @property - def execution_duration_ms(self) -> int: - if not self.execution_started_at: - return 0 - end_time = self.execution_finished_at or datetime.now(timezone.utc) - return int((end_time - self.execution_started_at).total_seconds() * 1000) - - @property - def delay_time_ms(self) -> int: - if not self.queued_at: - return int((datetime.now(timezone.utc) - self.created_at).total_seconds() * 1000) - if not self.execution_started_at: - return int((datetime.now(timezone.utc) - self.queued_at).total_seconds() * 1000) - return int((self.execution_started_at - self.created_at).total_seconds() * 1000) + if not self.metrics.time_out_at: + return False + return datetime.now(timezone.utc) > self.metrics.time_out_at diff --git a/apipod/engine/jobs/enqueue_payload.py b/apipod/engine/jobs/enqueue_payload.py new file mode 100644 index 0000000..40c1d20 --- /dev/null +++ b/apipod/engine/jobs/enqueue_payload.py @@ -0,0 +1,18 @@ +"""Marker for queue handlers that return submission metadata, not inference output. + +Some :class:`~apipod.engine.queue.job_queue_interface.JobQueueInterface` +implementations call the wrapped handler when a job is *submitted* (not when a +worker runs it) to collect metadata for the queue. Those return values must not +be coerced through schema response wrapping. +""" + + +class EnqueuePayload: + """Base class for handler results that represent job submission metadata.""" + + __apipod_enqueue_payload__ = True + + +def is_enqueue_payload(result: object) -> bool: + """Return True when *result* is job submission metadata, not model output.""" + return isinstance(result, EnqueuePayload) diff --git a/apipod/engine/jobs/job_progress.py b/apipod/engine/jobs/job_progress.py index 099e3c9..9817909 100644 --- a/apipod/engine/jobs/job_progress.py +++ b/apipod/engine/jobs/job_progress.py @@ -1,22 +1,34 @@ +import logging + +logger = logging.getLogger(__name__) + + class JobProgress: - def __init__(self, progress: float = 0, message: str = None): - """ - Used to display _progress of a job while executing. - :param progress: value between 0 and 1.0 - :param message: message to deliver to client. - """ - self._progress = progress - self._message = message + """Live progress handle for a running job. + + Holds the single source of truth for a job's ``progress`` and ``message``: + the endpoint function receives this object (via a ``job_progress`` parameter) + and reports updates through :meth:`set_status`, while the job record exposes + the same object so the public ``JobResult`` can read the current values. + """ + + def __init__(self, progress: float = 0.0, message: str = None): + """:param progress: value between 0 and 1.0. :param message: message to deliver to the client.""" + self.progress = progress + self.message = message + logger.setLevel(logging.INFO) def set_status(self, progress: float = None, message: str = None): if progress is not None: - self._progress = progress + self.progress = progress if message is not None: - self._message = message + self.message = message + logger.info(f"Progress: {self.progress} Message: {self.message}") + return self class JobProgressRunpod(JobProgress): - def __init__(self, runpod_job, progress: float = 0, message: str = None): + def __init__(self, runpod_job, progress: float = 0.0, message: str = None): super().__init__(progress=progress, message=message) self.runpod_job = runpod_job @@ -27,7 +39,7 @@ def set_status(self, progress: float = None, message: str = None): import runpod runpod.serverless.progress_update( self.runpod_job, - f"Progress: {int(self._progress)} Message: {self._message}" + f"Progress: {int(self.progress)} Message: {self.message}" ) except Exception as e: print(f"Problem in progress update: {e}") diff --git a/apipod/engine/jobs/job_result.py b/apipod/engine/jobs/job_result.py index 11c0288..7ded00a 100644 --- a/apipod/engine/jobs/job_result.py +++ b/apipod/engine/jobs/job_result.py @@ -1,176 +1,51 @@ import gzip +import logging +from datetime import datetime from io import BytesIO from typing import Any, List, Optional, Union -from pydantic import BaseModel, AnyUrl +from pydantic import BaseModel -from apipod.common.settings import DEFAULT_DATE_TIME_FORMAT from apipod.engine.jobs.base_job import JOB_STATUS, BaseJob +from apipod.engine.jobs.job_progress import JobProgress, JobProgressRunpod from apipod.engine.signatures.upload import is_param_media_toolkit_file from media_toolkit import IMediaContainer from media_toolkit.utils.data_type_utils import is_file_model_dict +from socaity_schemas import FileModel -class FileModel(BaseModel): - file_name: str - content_type: str - content: Union[str, AnyUrl] # base64 encoded or url - max_size_mb: Optional[float] = 4000 +logger = logging.getLogger(__name__) - class Config: - json_schema_extra = { - "x-media-type": "MediaFile", - "example": { - "file_name": "example.csv", - "content_type": "text/csv", - "content": "https://example.com/example.csv", - } - } - - -class ImageFileModel(FileModel): - class Config: - json_schema_extra = { - "x-media-type": "ImageFile", - "example": { - "file_name": "example.png", - "content_type": "image/png", - "content": "base64 encoded image data", - } - } - - -class AudioFileModel(FileModel): - class Config: - json_schema_extra = { - "x-media-type": "AudioFile", - "example": { - "file_name": "example.mp3", - "content_type": "audio/mpeg", - "content": "base64 encoded audio data", - } - } - -class VideoFileModel(FileModel): - class Config: - json_schema_extra = { - "x-media-type": "VideoFile", - "example": { - "file_name": "example.mp4", - "content_type": "video/mp4", - "content": "base64 encoded video data", - } - } - - -def _job_status_to_public(status: Any) -> Optional[str]: - """Map internal JOB_STATUS (or legacy string) to public API strings (gateway-aligned).""" +def _public_status(status: Any) -> Optional[str]: + """Public status string: a JOB_STATUS carries its own value; pass strings through.""" if status is None: return None - if isinstance(status, JOB_STATUS): - return { - JOB_STATUS.QUEUED: "pending", - JOB_STATUS.PROCESSING: "processing", - JOB_STATUS.FINISHED: "completed", - JOB_STATUS.FAILED: "failed", - JOB_STATUS.TIMEOUT: "failed", - }.get(status, status.value.lower()) - if isinstance(status, str): - lowered = status.lower() - legacy = { - "queued": "pending", - "pending": "pending", - "processing": "processing", - "finished": "completed", - "completed": "completed", - "failed": "failed", - "timeout": "failed", - "rejected": "failed", - } - return legacy.get(lowered, lowered) - return str(status) - - -def _format_date(date: Any) -> Optional[str]: - """Format a datetime or ISO string for the public API.""" - if date is None: - return None - if isinstance(date, str): - return date - try: - return date.strftime(DEFAULT_DATE_TIME_FORMAT) - except Exception: - return str(date) - - -def _parse_iso(value: Any): - """Best-effort parse an ISO 8601 string or datetime into a datetime.""" - if value is None: - return None - if hasattr(value, "timestamp"): - return value - try: - from datetime import datetime, timezone - dt = datetime.fromisoformat(str(value)) - if dt.tzinfo is None: - dt = dt.replace(tzinfo=timezone.utc) - return dt - except (ValueError, TypeError): - return None - - -def _compute_duration_s(start: Any, end: Any) -> Optional[float]: - """Compute seconds between two timestamps, returning None if either is missing.""" - s, e = _parse_iso(start), _parse_iso(end) - if s is None or e is None: - return None - delta = (e - s).total_seconds() - return round(delta, 2) if delta >= 0 else None - - -def _opt_float(value: Any) -> Optional[float]: - """Coerce to positive float, else None.""" - if value is None: - return None - try: - f = float(value) - return round(f, 3) if f > 0 else None - except (ValueError, TypeError): - return None + return status.value if isinstance(status, JOB_STATUS) else str(status) class JobLinks(BaseModel): """Hypermedia links for job status polling, cancellation, and streaming.""" - status: Optional[str] = None cancel: Optional[str] = None stream: Optional[str] = None class JobMetrics(BaseModel): - """Performance metrics populated as a job progresses through the platform. - - Segments (chronological): - upload_time_s – file upload duration (gateway) - platform_queue_time_s – our validation + dispatch + Celery routing - provider_queue_time_s – provider-side GPU / resource wait - inference_time_s – actual model execution - execution_time_s – orchestrator end-to-end (queue + inference, excludes upload) - """ - + """Execution metrics APIPod measures for a job.""" + created_at: Optional[datetime] = None + queued_at: Optional[datetime] = None + started_at: Optional[datetime] = None + finished_at: Optional[datetime] = None execution_time_s: Optional[float] = None - inference_time_s: Optional[float] = None - platform_queue_time_s: Optional[float] = None - provider_queue_time_s: Optional[float] = None - upload_time_s: Optional[float] = None class JobResult(BaseModel): """Public job snapshot returned by GET /status and job submissions. Unified response: same shape whether the client just submitted a job - (``status="pending"``) or is polling for completion. + or is polling for completion. ``status`` is a string, APIPod's local + queue uses :class:`~apipod.engine.jobs.base_job.JOB_STATUS` values. Null fields are excluded from the serialized response so the client only receives relevant information for the current job state. @@ -183,18 +58,30 @@ class JobResult(BaseModel): progress: Optional[float] = None message: Optional[str] = None - service: Optional[str] = None - endpoint: Optional[str] = None - metrics: Optional[JobMetrics] = None links: Optional[JobLinks] = None class JobResultFactory: @staticmethod - def _serialize_result(data: Any) -> Union[FileModel, List[FileModel], list, str, None]: + def _is_job_progress(value: Any) -> bool: + return isinstance(value, (JobProgress, JobProgressRunpod)) + + @staticmethod + def _warn_job_progress_ignored(context: str = "response") -> None: + logger.warning( + "Endpoint returned JobProgress in its %s. JobProgress is input-only and is omitted from the JSON result.", + context, + ) + + @classmethod + def _serialize_result(cls, data: Any) -> Union[FileModel, List[FileModel], list, str, None]: + if cls._is_job_progress(data): + cls._warn_job_progress_ignored() + return None + if isinstance(data, IMediaContainer): - return data.to_json() + return FileModel(**data.to_json()) if is_param_media_toolkit_file(data): return FileModel(**data.to_json()) @@ -208,88 +95,74 @@ def _serialize_result(data: Any) -> Union[FileModel, List[FileModel], list, str, except Exception: pass + if isinstance(data, BaseModel): + return cls._serialize_result(data.model_dump()) + + if isinstance(data, tuple): + items = [] + for index, item in enumerate(data): + if cls._is_job_progress(item): + cls._warn_job_progress_ignored(context=f"response[{index}]") + continue + items.append(cls._serialize_result(item)) + return items + if isinstance(data, list): - return [JobResultFactory._serialize_result(item) for item in data] + items = [] + for index, item in enumerate(data): + if cls._is_job_progress(item): + cls._warn_job_progress_ignored(context=f"response[{index}]") + continue + items.append(cls._serialize_result(item)) + return items if isinstance(data, dict): return { - key: JobResultFactory._serialize_result(value) + key: cls._serialize_result(value) for key, value in data.items() } return data @staticmethod - def from_base_job(job: BaseJob) -> JobResult: - """Map any :class:`BaseJob` subclass to the public :class:`JobResult`. - - Works for :class:`~apipod.engine.jobs.base_job.LocalJob` (thread queue) - and any platform ``BaseJob`` subclass (e.g. gateway ``ServiceJob``). - """ - status = _job_status_to_public(job.status) - result = JobResultFactory._serialize_result(job.result) - - progress = job.progress - message = job.message - - job_progress = getattr(job, "job_progress", None) - if job_progress is not None: - try: - progress = float(job_progress._progress) - message = job_progress._message - except Exception: - pass - - service = getattr(job, "service_id", None) - endpoint = getattr(job, "endpoint", None) - - metrics = JobResultFactory._build_metrics(job) + def _job_link(link_prefix: str, suffix: str) -> str: + prefix = (link_prefix or "").rstrip("/") + return f"{prefix}{suffix}" if prefix else suffix - links = JobLinks( - status=f"/status/{job.id}", - cancel=f"/cancel/{job.id}", - stream=f"/stream/{job.id}", + @staticmethod + def from_base_job( + job: BaseJob, + *, + include_stream_link: bool = False, + link_prefix: str = "", + ) -> JobResult: + """Map a :class:`BaseJob` to the public :class:`JobResult`.""" + m = job.metrics + metrics = JobMetrics( + created_at=m.created_at, + queued_at=m.queued_at, + started_at=m.started_at, + finished_at=m.finished_at, + execution_time_s=m.execution_time_s, ) return JobResult( job_id=job.id, - status=status, + status=_public_status(job.status), + result=JobResultFactory._serialize_result(job.result), error=job.error, - result=result, - progress=progress, - message=message, - service=service, - endpoint=endpoint, + progress=job.job_progress.progress, + message=job.job_progress.message, metrics=metrics, - links=links, - ) - - @staticmethod - def _build_metrics(job: BaseJob) -> Optional[JobMetrics]: - """Derive timing metrics from orchestrator-provided values or timestamps.""" - execution_time_s = _opt_float(getattr(job, "execution_time_s", None)) or _compute_duration_s( - getattr(job, "created_at", None), - getattr(job, "completed_at", None) or getattr(job, "failed_at", None), - ) - upload_time_s = _compute_duration_s( - getattr(job, "upload_started_at", None), - getattr(job, "upload_finished_at", None), - ) - inference_time_s = _opt_float(getattr(job, "inference_time_s", None)) - platform_queue_time_s = _opt_float(getattr(job, "platform_queue_time_s", None)) - provider_queue_time_s = _opt_float(getattr(job, "provider_queue_time_s", None)) - - values = (execution_time_s, upload_time_s, inference_time_s, - platform_queue_time_s, provider_queue_time_s) - if all(v is None for v in values): - return None - - return JobMetrics( - execution_time_s=execution_time_s, - inference_time_s=inference_time_s, - platform_queue_time_s=platform_queue_time_s, - provider_queue_time_s=provider_queue_time_s, - upload_time_s=upload_time_s, + links=JobLinks( + status=JobResultFactory._job_link(link_prefix, f"/status/{job.id}"), + cancel=JobResultFactory._job_link(link_prefix, f"/cancel/{job.id}"), + stream=( + JobResultFactory._job_link(link_prefix, f"/stream/{job.id}") + if include_stream_link + else None + ), + ), ) @staticmethod diff --git a/apipod/engine/llm/__init__.py b/apipod/engine/llm/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/apipod/engine/llm/base_llm_mixin.py b/apipod/engine/llm/base_llm_mixin.py deleted file mode 100644 index 674efff..0000000 --- a/apipod/engine/llm/base_llm_mixin.py +++ /dev/null @@ -1,93 +0,0 @@ -from types import UnionType -from typing import Callable, Type, Any, get_args, get_origin, Union -import uuid -from datetime import datetime, timezone -import inspect -from apipod.common.schemas import ChatCompletionRequest, ChatCompletionResponse, CompletionRequest, CompletionResponse, EmbeddingRequest, EmbeddingResponse, ChatCompletionChoice, CompletionChoice, EmbeddingData - - -class _BaseLLMMixin: - """ - Mixin class for Base LLM functionality - """ - - def __init__(self, *args, **kwargs): - self._llm_configs = { - ChatCompletionRequest: (ChatCompletionResponse, "chat"), - CompletionRequest: (CompletionResponse, "completion"), - EmbeddingRequest: (EmbeddingResponse, "embedding"), - } - self._supported_llm_request_models = tuple(self._llm_configs.keys()) - - def _resolve_supported_llm_request_model(self, annotation: Any): - """Resolve direct/optional request model annotations to a supported LLM request model.""" - if annotation in self._llm_configs: - return annotation - - origin = get_origin(annotation) - if origin in (Union, UnionType): - for arg in get_args(annotation): - if arg is type(None): - continue - if arg in self._llm_configs: - return arg - - return None - - def _get_llm_config(self, func: Callable): - sig = inspect.signature(func) - for param in sig.parameters.values(): - req_model = self._resolve_supported_llm_request_model(param.annotation) - if req_model is not None: - res_model, endpoint_type = self._llm_configs[req_model] - return req_model, res_model, endpoint_type - return None, None, None - - def _prepare_llm_payload(self, req_model: Type, payload: Any) -> Any: - """ - Prepare the LLM request payload. - """ - if isinstance(payload, req_model): - return payload - elif isinstance(payload, dict): - return req_model.model_validate(payload) - else: - raise ValueError(f"Invalid payload type for {req_model}: {type(payload)}") - - def _wrap_llm_response(self, result: Any, response_model: Type, endpoint_type: str, openai_req: Any) -> Any: - """ - Wrap the raw result into the appropriate LLM response model. - """ - if isinstance(result, response_model): - return result - - model_name = getattr(openai_req, "model", "unknown-model") - ts, uid = int(datetime.now(timezone.utc)), uuid.uuid4().hex[:8] - - if endpoint_type == "chat": - return response_model( - id=f"chatcmpl-{ts}-{uid}", - object="chat.completion", - created=ts, - model=model_name, - choices=[ChatCompletionChoice(**choice) for choice in result["choices"]], - usage=result.get("usage") - ) - elif endpoint_type == "completion": - return response_model( - id=f"cmpl-{ts}-{uid}", - object="text.completion", - created=ts, - model=model_name, - choices=[CompletionChoice(**choice) for choice in result["choices"]], - usage=result.get("usage") - ) - elif endpoint_type == "embedding": - return response_model( - object="embedding", - data=[EmbeddingData(**data) for data in result["data"]], - model=model_name, - usage=result.get("usage") - ) - else: - raise ValueError(f"Unknown endpoint type: {endpoint_type}") \ No newline at end of file diff --git a/apipod/engine/queue/job_queue.py b/apipod/engine/queue/job_queue.py index 985d239..292e423 100644 --- a/apipod/engine/queue/job_queue.py +++ b/apipod/engine/queue/job_queue.py @@ -6,8 +6,9 @@ from apipod.engine.queue.job_store import JobStore from apipod.engine.jobs.base_job import BaseJob, LocalJob, JOB_STATUS +from apipod.engine.signatures.analysis import job_progress_param_names from apipod.engine.queue.job_queue_interface import JobQueueInterface -import inspect +from apipod.engine.streaming.stream_producer import StreamProducer T = TypeVar('T', bound=BaseJob) @@ -35,6 +36,21 @@ def __init__(self, delete_orphan_jobs_after_s: int = 60 * 30): self._shutdown = threading.Event() self._job_threads: Dict[str, threading.Thread] = {} self._delete_orphan_jobs_after_seconds = delete_orphan_jobs_after_s + # Optional StreamStore. When set, generator/streaming jobs produce their + # chunks into the store (so a client can consume GET /stream/{job_id}) + # while still aggregating the full result for GET /status/{job_id}. + self._stream_store = None + + def set_stream_store(self, stream_store) -> None: + """Attach a :class:`StreamStore` used to relay streaming job output.""" + self._stream_store = stream_store + + def get_job_status(self, job_id: str) -> Optional[dict]: + """Lightweight status lookup that never removes the job (used by /stream).""" + job = self.job_store.get_job(job_id) + if job is None: + return None + return {"id": job_id, "status": getattr(job.status, "value", job.status)} def set_queue_size(self, job_function: callable, queue_size: int = 500) -> None: self.queue_sizes[job_function.__name__] = queue_size @@ -69,7 +85,7 @@ def _add_job(self, job_function: callable, job_params: Optional[dict] = None) -> return job job.status = JOB_STATUS.QUEUED - job.queued_at = datetime.now(timezone.utc) + job.metrics.queued_at = datetime.now(timezone.utc) self.job_store.add_to_queue(job) if not self.worker_thread.is_alive(): @@ -88,12 +104,20 @@ def _create_job(self, job_function: callable, job_params: Optional[dict] = None) def _process_job(self, job: T) -> None: try: - job.execution_started_at = datetime.now(timezone.utc) + job.metrics.started_at = datetime.now(timezone.utc) job.status = JOB_STATUS.PROCESSING self._inject_job_progress(job) - job.result = job.job_function(**job.job_params) + result = job.job_function(**job.job_params) + + # Streaming endpoints return a StreamProducer instead of a value: the + # worker relays chunks into the stream store (consumed via /stream) + # and keeps the aggregated result for /status. + if isinstance(result, StreamProducer): + result = self._run_stream(job, result) + + job.result = result job.job_progress.set_status(1.0) self._complete_job(job=job, final_state=JOB_STATUS.FINISHED) @@ -106,10 +130,40 @@ def _process_job(self, job: T) -> None: print(f"Job {job.id} failed: {str(e)}") traceback.print_exc() # Writes full traceback to stderr + def _run_stream(self, job: T, producer: StreamProducer): + """Drive a :class:`StreamProducer`: relay chunks into the stream store + (if configured) while collecting raw items to build the full result. + + Without a stream store the chunks are simply aggregated, so the job still + returns a complete result via /status (streaming silently degrades). + """ + store = self._stream_store + collected = [] + + if store is None: + for item in producer.raw_chunks: + collected.append(item) + return producer.aggregate(collected) + + job.status = JOB_STATUS.STREAMING + store.open_stream(job.id) + try: + for item in producer.raw_chunks: + collected.append(item) + store.write_chunk(job.id, producer.to_chunk(item)) + for closing_chunk in producer.closing: + store.write_chunk(job.id, closing_chunk) + store.close_stream(job.id) + except Exception as e: + store.close_stream(job.id, error=str(e)) + raise + + return producer.aggregate(collected) + def _complete_job(self, job: T, final_state: JOB_STATUS) -> T: self.job_store.complete_job(job.id) # setting status here, because if this is done earlier, race conditions in get_job are the problem - job.execution_finished_at = datetime.now(timezone.utc) + job.metrics.finished_at = datetime.now(timezone.utc) job.status = final_state return job @@ -126,15 +180,8 @@ def cancel_job(self, job_id: str) -> None: # self._complete_job(job_id) def _inject_job_progress(self, job: T) -> T: - sig = inspect.signature(job.job_function) - - job_progress_params = [ - p for p in sig.parameters.values() - if p.name == "job_progress" or "FastJobProgress" in str(p.annotation) - ] - for job_progress_param in job_progress_params: - job.job_params[job_progress_param.name] = job.job_progress - + for param_name in job_progress_param_names(job.job_function): + job.job_params[param_name] = job.job_progress return job def _process_jobs_in_background(self) -> None: @@ -189,9 +236,9 @@ def _clean_up_orphan_jobs(self) -> None: return for job in self.job_store.completed_jobs: - if job.execution_finished_at is None: + if job.metrics.finished_at is None: self._remove_job(job) - elif (datetime.now(timezone.utc) - job.execution_finished_at).total_seconds() > self._delete_orphan_jobs_after_seconds: + elif (datetime.now(timezone.utc) - job.metrics.finished_at).total_seconds() > self._delete_orphan_jobs_after_seconds: self._remove_job(job) def _start_queued_jobs(self) -> None: diff --git a/apipod/engine/queue/job_queue_interface.py b/apipod/engine/queue/job_queue_interface.py index b45e070..c1dd2c6 100644 --- a/apipod/engine/queue/job_queue_interface.py +++ b/apipod/engine/queue/job_queue_interface.py @@ -13,22 +13,45 @@ class JobQueueInterface(Generic[T], ABC): Both :meth:`add_job` (submission) and :meth:`get_job_result` (status polling) return a :class:`JobResult`. The default implementations convert via ``JobResultFactory.from_base_job``; queue backends that store richer - job types (e.g. Redis ``ServiceJob``) override to add extra fields. + job types override to add extra fields. + + The transport layer (router) passes presentation context through this + port — ``supports_streaming`` (may the job's output be consumed via + ``GET /stream/{job_id}``) and ``link_prefix`` (mount prefix for hypermedia + links) — so it never needs to know which queue implementation it holds. """ - def add_job(self, job_function: Callable, job_params: Optional[dict] = None) -> JobResult: + def add_job( + self, + job_function: Callable, + job_params: Optional[dict] = None, + *, + supports_streaming: bool = False, + link_prefix: str = "", + ) -> JobResult: """Create a job, enqueue it, and return the public :class:`JobResult`. Subclasses **must** override :meth:`_add_job` (raw enqueue logic) and may override this method to customise the ``JobResult`` conversion. """ job = self._add_job(job_function, job_params) - return JobResultFactory.from_base_job(job) + job.supports_streaming = supports_streaming + return JobResultFactory.from_base_job( + job, + include_stream_link=supports_streaming, + link_prefix=link_prefix, + ) - def get_job_result(self, job_id: str) -> Optional[JobResult]: + def get_job_result(self, job_id: str, *, link_prefix: str = "") -> Optional[JobResult]: """Resolve job status for API responses.""" job = self.get_job(job_id) - return JobResultFactory.from_base_job(job) if job is not None else None + if job is None: + return None + return JobResultFactory.from_base_job( + job, + include_stream_link=bool(getattr(job, "supports_streaming", False)), + link_prefix=link_prefix, + ) @abstractmethod def _add_job(self, job_function: Callable, job_params: Optional[dict] = None) -> T: @@ -45,7 +68,14 @@ def get_job(self, job_id: str) -> Optional[T]: ... @abstractmethod - def cancel_job(self, job_id: str) -> None: + def cancel_job(self, job_id: str) -> Optional[dict]: + """Cancel a job. + + Returns an optional cancellation summary dict (``id`` / ``status`` / + ``message``) for the HTTP response; ``None`` means "cancelled, use the + default summary". Raise :class:`NotImplementedError` when the queue + does not support cancellation. + """ ... @abstractmethod diff --git a/apipod/engine/signatures/analysis.py b/apipod/engine/signatures/analysis.py new file mode 100644 index 0000000..dc344cc --- /dev/null +++ b/apipod/engine/signatures/analysis.py @@ -0,0 +1,134 @@ +""" +Backend-neutral analysis of function signatures and return types. +""" + +import ast +import inspect +import textwrap +from collections.abc import AsyncIterator, Iterator +from types import UnionType +from typing import Callable, Union, get_args, get_origin, get_type_hints + + +def _return_type_includes_iterator(return_type) -> bool: + """True when *return_type* is or contains ``Iterator`` / ``AsyncIterator``.""" + if return_type is None: + return False + origin = get_origin(return_type) + if origin in (Union, UnionType): + return any(_return_type_includes_iterator(arg) for arg in get_args(return_type)) + resolved = origin or return_type + return inspect.isclass(resolved) and issubclass(resolved, (Iterator, AsyncIterator)) + + +def is_streaming_endpoint(func: Callable, schema_binding=None) -> bool: + """Backend-neutral: True when *func* can produce a streaming response. + + Detection order: + - generator / async-generator function; + - return annotation contains ``Iterator`` / ``AsyncIterator`` (incl. inside a ``Union``); + - schema endpoint whose request model has a ``stream`` field and the function body + returns a generator (AST: ``yield``, generator expression, or under ``if request.stream``). + + ``schema_binding`` is resolved from *func* when not provided by the caller. + """ + target = inspect.unwrap(func) + if inspect.isgeneratorfunction(target) or inspect.isasyncgenfunction(target): + return True + try: + return_type = get_type_hints(target).get("return") + except Exception: + return_type = None + if _return_type_includes_iterator(return_type): + return True + + if schema_binding is None: + # Local import: schema_resolve depends on backend modules that in turn + # use this analysis module. + from apipod.engine.backend.schema_resolve import get_schema_binding + + try: + schema_binding = get_schema_binding(func) + except Exception: + schema_binding = None + + if schema_binding is not None and "stream" in schema_binding.request_model.model_fields: + return ast_suggests_request_stream(target) + return False + + +def _is_stream_attr_test(node: ast.AST) -> bool: + """True when *node* tests a ``.stream`` attribute (e.g. ``request.stream``).""" + if isinstance(node, ast.Attribute) and node.attr == "stream": + return True + if isinstance(node, ast.UnaryOp) and isinstance(node.op, ast.Not): + return _is_stream_attr_test(node.operand) + return False + + +def _is_streaming_return_value(node: ast.AST | None) -> bool: + """True when a ``return`` hands back a generator (expression, yield, or async yield).""" + if node is None: + return False + return isinstance(node, (ast.GeneratorExp, ast.Yield, ast.YieldFrom)) + + +def _statements_suggest_streaming(stmts: list[ast.stmt]) -> bool: + """Walk *stmts* for generator returns, yields, or a ``.stream`` branch that streams.""" + for node in ast.walk(ast.Module(body=stmts, type_ignores=[])): + if isinstance(node, ast.Return) and _is_streaming_return_value(node.value): + return True + if isinstance(node, (ast.Yield, ast.YieldFrom)): + return True + if isinstance(node, ast.If) and _is_stream_attr_test(node.test): + if _statements_suggest_streaming(node.body) or _statements_suggest_streaming(node.orelse): + return True + return False + + +def _function_ast_node(func: Callable) -> ast.FunctionDef | ast.AsyncFunctionDef | None: + """Parse the source of *func* into its top-level function AST node.""" + try: + source = textwrap.dedent(inspect.getsource(func)) + tree = ast.parse(source) + except (OSError, TypeError, SyntaxError, ValueError): + return None + + expected_name = func.__name__ + for node in tree.body: + if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) and node.name == expected_name: + return node + return None + + +def ast_suggests_request_stream(func: Callable) -> bool: + """True when the function body conditionally or unconditionally returns a generator. + + Detects patterns such as ``if request.stream: return (t for t in tokens)`` without + requiring a return-type annotation. Returns ``False`` when source is unavailable + (REPL, dynamic ``exec``) or the body has no streaming return path. + """ + fn_node = _function_ast_node(inspect.unwrap(func)) + if fn_node is None: + return False + return _statements_suggest_streaming(fn_node.body) + + +def is_injected_progress_param(param: inspect.Parameter) -> bool: + """True if the parameter is a framework-injected :class:`JobProgress`.""" + return param.name == "job_progress" or "JobProgress" in str(param.annotation) + + +def job_progress_param_names(func: Callable) -> list[str]: + """Names of the parameters of *func* that should receive a :class:`JobProgress`. + + A parameter qualifies when it is literally named ``job_progress`` or when its + annotation refers to a ``JobProgress`` type. This single detection is shared + by every injection site (queue worker, RunPod handler, direct FastAPI path) + so they can never disagree about what counts as a progress parameter. + """ + try: + params = inspect.signature(func).parameters.values() + except (TypeError, ValueError): + return [] + return [p.name for p in params if is_injected_progress_param(p)] diff --git a/apipod/engine/signatures/policies.py b/apipod/engine/signatures/policies.py index ce4c6d9..af0982e 100644 --- a/apipod/engine/signatures/policies.py +++ b/apipod/engine/signatures/policies.py @@ -1,10 +1,18 @@ -from types import UnionType -from typing import Any, Union, get_args, get_origin - import inspect +from typing import Any, get_args, get_origin + from fastapi import Body, Form -from apipod.common.schemas import SUPPORTED_LLM_REQUEST_SCHEMAS +from apipod.engine.backend.schema_resolve import ( + SCHEMA_REGISTRY, + resolve_request_model, + source_request_model, +) + +# Request schemas that are interpreted as JSON bodies by router decorators, +# even when endpoint authors do not specify Body(...). Derived from the +# schema registry so the two can never drift apart. +SUPPORTED_REQUEST_SCHEMAS = tuple(SCHEMA_REGISTRY) class FastAPISignaturePolicies: @@ -26,33 +34,30 @@ def is_fastapi_dependency(parameter: inspect.Parameter) -> bool: return module.startswith("fastapi") or module.startswith("starlette") @staticmethod - def is_supported_llm_request_schema(annotation: Any) -> bool: + def is_supported_request_schema(annotation: Any) -> bool: """ - True if annotation is a supported APIPod LLM request schema. - Supports direct and optional union annotations. + True if annotation is (or contains, for Optional/Union annotations) a + registered APIPod request schema or a subclass of one. """ - if annotation in SUPPORTED_LLM_REQUEST_SCHEMAS: + if resolve_request_model(annotation) is not None: return True - origin = get_origin(annotation) - if origin in (Union, UnionType): + if origin is not None: return any( - arg in SUPPORTED_LLM_REQUEST_SCHEMAS + FastAPISignaturePolicies.is_supported_request_schema(arg) for arg in get_args(annotation) - if arg is not type(None) ) - - return False + return inspect.isclass(annotation) and source_request_model(annotation) is not annotation @classmethod def build_non_file_default(cls, annotation: Any, default: Any, is_optional: bool): """ Select FastAPI parameter defaults for non-file fields. - Supported LLM request schemas are mapped to JSON body semantics. All other + Registered request schemas are mapped to JSON body semantics. All other non-file parameters keep form semantics for backward compatibility. """ normalized_default = None if is_optional else default - if cls.is_supported_llm_request_schema(annotation): + if cls.is_supported_request_schema(annotation): return Body(default=normalized_default) return Form(default=normalized_default) diff --git a/apipod/engine/streaming/__init__.py b/apipod/engine/streaming/__init__.py new file mode 100644 index 0000000..3c3ab69 --- /dev/null +++ b/apipod/engine/streaming/__init__.py @@ -0,0 +1,21 @@ +from apipod.engine.streaming.stream_store import StreamStore +from apipod.engine.streaming.local_stream_store import LocalStreamStore +from apipod.engine.streaming.stream_producer import StreamProducer +from apipod.engine.streaming.stream_serializer import ( + as_sync_iter, + build_stream_producer, + encode_chunk, + is_streaming_result, + store_chunk, +) + +__all__ = [ + "StreamStore", + "LocalStreamStore", + "StreamProducer", + "as_sync_iter", + "build_stream_producer", + "encode_chunk", + "is_streaming_result", + "store_chunk", +] diff --git a/apipod/engine/streaming/local_stream_store.py b/apipod/engine/streaming/local_stream_store.py new file mode 100644 index 0000000..5a8ec03 --- /dev/null +++ b/apipod/engine/streaming/local_stream_store.py @@ -0,0 +1,142 @@ +""" +Adapter: LocalStreamStore + +In-process implementation of the :class:`StreamStore` port. This is the default +backend APIPod uses on localhost to emulate how streaming behaves once a service +is deployed (e.g. on Socaity, where a Redis-backed store relays chunks from the +worker to the gateway). + +Everything lives in process memory, guarded by a lock: + - the in-process worker thread is the producer (``open_stream`` / + ``write_chunk`` / ``close_stream``); + - the FastAPI ``GET /stream/{job_id}`` route is the consumer + (``read_chunks``). + +It is intentionally simple — no external dependencies, no durability. A real +deployment swaps in a distributed store; the endpoint code stays the same. +""" + +import asyncio +import logging +import threading +import time +from typing import AsyncIterator, Dict, List, Optional + +from apipod.engine.streaming.stream_store import StreamStore + +logger = logging.getLogger(__name__) + + +class _LocalStream: + """State for a single in-memory stream.""" + + def __init__(self, ttl_seconds: int): + self.chunks: List[str] = [] + self.closed: bool = False + self.error: Optional[str] = None + self.expires_at: float = time.monotonic() + ttl_seconds + + +class LocalStreamStore(StreamStore): + """Thread-safe, in-memory :class:`StreamStore` for local development/testing.""" + + def __init__(self, poll_interval_s: float = 0.02): + self._streams: Dict[str, _LocalStream] = {} + self._lock = threading.Lock() + # How often the async consumer polls the in-memory buffer for new data. + self._poll_interval_s = poll_interval_s + + # -- Producer API --------------------------------------------------- + + def open_stream(self, job_id: str, ttl_seconds: int = 3600) -> None: + with self._lock: + self._streams[job_id] = _LocalStream(ttl_seconds) + logger.debug("Stream opened | job_id=%s ttl=%ss", job_id, ttl_seconds) + + def write_chunk(self, job_id: str, payload: str) -> None: + with self._lock: + stream = self._streams.get(job_id) + if stream is None: + # Be forgiving: auto-open if the producer forgot to. + stream = _LocalStream(ttl_seconds=3600) + self._streams[job_id] = stream + stream.chunks.append(payload) + + def close_stream(self, job_id: str, *, error: Optional[str] = None) -> None: + with self._lock: + stream = self._streams.get(job_id) + if stream is None: + stream = _LocalStream(ttl_seconds=300) + self._streams[job_id] = stream + stream.error = error + stream.closed = True + logger.debug("Stream closed | job_id=%s error=%s", job_id, error) + + # -- Consumer API --------------------------------------------------- + + async def read_chunks( + self, + job_id: str, + *, + block_ms: int = 5000, + batch_size: int = 50, + ) -> AsyncIterator[str]: + # The consumer may connect before the producer has opened the stream + # (the worker picks the job up slightly after the HTTP response). Wait a + # bounded time for the stream to appear before giving up. + if not await self._await_stream(job_id, timeout_s=max(block_ms / 1000.0, 5.0)): + return + + index = 0 + idle_s = 0.0 + keepalive_after_s = block_ms / 1000.0 + + while True: + with self._lock: + stream = self._streams.get(job_id) + if stream is None: + return + new_chunks = stream.chunks[index:index + batch_size] + index += len(new_chunks) + closed = stream.closed + error = stream.error + drained = index >= len(stream.chunks) + + if new_chunks: + idle_s = 0.0 + for chunk in new_chunks: + yield chunk + continue + + if closed and drained: + if error: + yield f'data: {{"error": {error!r}}}\n\n' + self.delete_stream(job_id) + return + + await asyncio.sleep(self._poll_interval_s) + idle_s += self._poll_interval_s + if idle_s >= keepalive_after_s: + idle_s = 0.0 + # SSE comment keeps proxies / browsers from timing out. + yield ": keepalive\n\n" + + async def _await_stream(self, job_id: str, timeout_s: float) -> bool: + deadline = time.monotonic() + timeout_s + while time.monotonic() < deadline: + with self._lock: + if job_id in self._streams: + return True + await asyncio.sleep(self._poll_interval_s) + return job_id in self._streams + + # -- Lifecycle ------------------------------------------------------ + + def delete_stream(self, job_id: str) -> None: + with self._lock: + self._streams.pop(job_id, None) + logger.debug("Stream deleted | job_id=%s", job_id) + + def stream_exists(self, job_id: str) -> bool: + with self._lock: + return job_id in self._streams diff --git a/apipod/engine/streaming/stream_producer.py b/apipod/engine/streaming/stream_producer.py new file mode 100644 index 0000000..0ab8880 --- /dev/null +++ b/apipod/engine/streaming/stream_producer.py @@ -0,0 +1,33 @@ +""" +StreamProducer: the bridge between a streaming endpoint and the stream store. + +When a streaming endpoint runs under a job queue, its result cannot be sent to +the client directly (the HTTP response already returned a ``JobResult``). +Instead the worker produces the stream into a :class:`StreamStore` while the +client consumes it from ``GET /stream/{job_id}``. + +The router knows *how* to serialize a given result (a ChatCompletion token +stream, encoded media bytes, or a plain generator); the queue worker only knows +the lifecycle. ``StreamProducer`` carries that router-built knowledge to the +worker without leaking schema details into the queue: + + - ``raw_chunks`` – the user generator's raw items (tokens / bytes / objects) + - ``to_chunk`` – serialize one raw item into a store/SSE chunk (str) + - ``closing`` – chunks appended after the stream (e.g. finish + ``[DONE]``) + - ``media_type`` – content type for the direct (non-queued) StreamingResponse + - ``aggregate`` – build the full ``/status`` result from the raw items, so a + client polling status (instead of streaming) gets the + complete result once the job finishes. +""" + +from dataclasses import dataclass, field +from typing import Any, Callable, Iterator, List + + +@dataclass +class StreamProducer: + raw_chunks: Iterator[Any] + to_chunk: Callable[[Any], str] + aggregate: Callable[[List[Any]], Any] + closing: List[str] = field(default_factory=list) + media_type: str = "text/event-stream" diff --git a/apipod/engine/streaming/stream_serializer.py b/apipod/engine/streaming/stream_serializer.py new file mode 100644 index 0000000..5e5997f --- /dev/null +++ b/apipod/engine/streaming/stream_serializer.py @@ -0,0 +1,160 @@ +""" +Backend-neutral stream serialization shared by all transport adapters. + +This module is the single place that knows how to: + - detect a streaming runtime result (``is_streaming_result``); + - bridge an async generator into a sync iterator (``as_sync_iter``); + - encode raw chunks into JSON-safe strings (``encode_chunk`` for RunPod, + ``store_chunk`` for the FastAPI stream store); + - aggregate all raw chunks back into a full result for ``/status`` polling; + - assemble a :class:`StreamProducer` for the FastAPI queued/direct path. + +Both :class:`SocaityFastAPIRouter` and :class:`SocaityRunpodRouter` import +from here instead of each maintaining their own copy. +""" + +import asyncio +import base64 +import inspect +from typing import Any, Iterator, Optional + +from media_toolkit import MediaFile + +from apipod.engine.backend.schema_resolve import ( + SchemaBinding, + SchemaStreamSerializer, + SSE_DONE, + SSE_STREAM_TAGS, + STREAM_CHUNK_SPECS, + wrap_schema_response, +) +from apipod.engine.jobs.job_result import JobResultFactory +from apipod.engine.streaming.stream_producer import StreamProducer + + +# --------------------------------------------------------------------------- +# Detection +# --------------------------------------------------------------------------- + +def is_streaming_result(result: Any) -> bool: + """Return True if *result* is a sync or async generator.""" + return inspect.isgenerator(result) or inspect.isasyncgen(result) + + +# --------------------------------------------------------------------------- +# Sync/async bridging +# --------------------------------------------------------------------------- + +def as_sync_iter(result: Any) -> Iterator: + """Return a synchronous iterator over a sync or async generator.""" + return _drain_async_gen(result) if inspect.isasyncgen(result) else result + + +def _drain_async_gen(agen) -> Iterator: + """Consume an async generator synchronously (for worker / sync contexts).""" + loop = asyncio.new_event_loop() + try: + while True: + try: + yield loop.run_until_complete(agen.__anext__()) + except StopAsyncIteration: + return + finally: + loop.close() + + +# --------------------------------------------------------------------------- +# Chunk encoding +# --------------------------------------------------------------------------- + +def _to_base64(chunk: Any) -> Optional[str]: + """Return a base64 string when *chunk* is bytes / bytearray / MediaFile; else None.""" + if isinstance(chunk, MediaFile): + chunk = chunk.to_bytes() + if isinstance(chunk, (bytes, bytearray)): + return base64.b64encode(bytes(chunk)).decode("ascii") + return None + + +def encode_chunk(chunk: Any) -> str: + """JSON-safe encoding without SSE framing (for RunPod transport). + + Binary / MediaFile → base64 string; str → unchanged; other → str(). + """ + b64 = _to_base64(chunk) + return b64 if b64 is not None else (chunk if isinstance(chunk, str) else str(chunk)) + + +def store_chunk(chunk: Any) -> str: + """SSE-framed encoding for the FastAPI stream store. + + Binary / MediaFile → ``data: {base64}\\n\\n``; str → unchanged (passthrough); + other → ``data: {chunk}\\n\\n``. + """ + b64 = _to_base64(chunk) + if b64 is not None: + return f"data: {b64}\n\n" + return chunk if isinstance(chunk, str) else f"data: {chunk}\n\n" + + +# --------------------------------------------------------------------------- +# Aggregation (full /status result from streamed items) +# --------------------------------------------------------------------------- + +def aggregate_plain(items: list) -> Any: + """Join text chunks or serialize each item for the /status result.""" + if not items: + return items + if all(isinstance(i, str) for i in items): + return "".join(items) + if all(isinstance(i, (bytes, bytearray)) for i in items): + return b"".join(bytes(i) for i in items) + return [JobResultFactory._serialize_result(i) for i in items] + + +def aggregate_schema_tokens(items: list, binding: SchemaBinding) -> Any: + """Reconstruct the schema response from streamed token items.""" + return JobResultFactory._serialize_result( + wrap_schema_response("".join(str(i) for i in items), binding) + ) + + +# --------------------------------------------------------------------------- +# StreamProducer construction (FastAPI queued + direct path) +# --------------------------------------------------------------------------- + +def build_stream_producer(result: Any, binding: Optional[SchemaBinding]) -> Optional[StreamProducer]: + """ + Build a :class:`StreamProducer` from a streamable endpoint result, or + return ``None`` when the result is not a generator. + + Schema endpoints with a registered chunk model (e.g. chat) wrap tokens + into the standardized ``ChatCompletionChunk`` SSE stream; all other + generators produce SSE-framed text / base64-encoded binary chunks. + """ + if not is_streaming_result(result): + return None + + raw_chunks = as_sync_iter(result) + + if binding is not None and binding.tag in STREAM_CHUNK_SPECS: + serializer = SchemaStreamSerializer(binding) + return StreamProducer( + raw_chunks=raw_chunks, + to_chunk=serializer.delta, + closing=[serializer.finish(), SSE_DONE], + media_type="text/event-stream", + aggregate=lambda items, b=binding: aggregate_schema_tokens(items, b), + ) + + media_type = ( + "text/event-stream" + if binding is None or binding.tag in SSE_STREAM_TAGS + else "application/octet-stream" + ) + return StreamProducer( + raw_chunks=raw_chunks, + to_chunk=store_chunk, + media_type=media_type, + aggregate=aggregate_plain, + ) diff --git a/apipod/engine/streaming/stream_store.py b/apipod/engine/streaming/stream_store.py new file mode 100644 index 0000000..fa0feb0 --- /dev/null +++ b/apipod/engine/streaming/stream_store.py @@ -0,0 +1,110 @@ +""" +Port: StreamStore + +Abstract interface for reading and writing streaming data chunks associated +with a job. Implementations may keep the chunks in process memory (the local +test backend), in Redis Streams, in Kafka, or in any other append-only log. + +In a real deployment the producer and the consumer live in different +processes: + + - The **worker** (producer) calls :meth:`open_stream` / :meth:`write_chunk` + as result chunks (ChatCompletion deltas, encoded ``AudioFile`` / + ``VideoFile`` bytes, or chunks of any generator endpoint) are produced. + - The **gateway** (consumer) calls :meth:`read_chunks` to relay those + chunks to the client over Server-Sent Events. + - Both sides use :meth:`close_stream` / :meth:`delete_stream` for lifecycle + management. + +APIPod on localhost emulates that split inside a single process: the in-process +worker thread produces, while the FastAPI ``GET /stream/{job_id}`` route +consumes. The default backend is :class:`LocalStreamStore`; deployments swap in +their own implementation (e.g. a Redis-backed store) without touching the +endpoint code. + +Design decisions: + - :meth:`read_chunks` is an **async generator** so the gateway can yield + chunks straight into a ``StreamingResponse`` without buffering. + - :meth:`write_chunk` / :meth:`open_stream` / :meth:`close_stream` are + **synchronous** because the worker writes from a (sync) worker context. + - :meth:`close_stream` signals completion (optionally with an error) so the + consumer's read loop exits cleanly. +""" + +from abc import ABC, abstractmethod +from typing import AsyncIterator, Optional + + +class StreamStore(ABC): + """Port for append-only stream storage keyed by ``job_id``.""" + + # -- Producer API (worker side) ------------------------------------- + + @abstractmethod + def open_stream(self, job_id: str, ttl_seconds: int = 3600) -> None: + """ + Prepare a stream for writing. + + Implementations should create the underlying resource and set a + safety-net TTL to prevent leaks if neither side cleans up. + """ + ... + + @abstractmethod + def write_chunk(self, job_id: str, payload: str) -> None: + """ + Append a single chunk (typically one SSE ``data:`` line) to the stream. + + Must be safe to call from a synchronous (worker) context. + """ + ... + + @abstractmethod + def close_stream(self, job_id: str, *, error: Optional[str] = None) -> None: + """ + Signal that the producer is done. + + Writes a terminal marker so the consumer knows the stream is complete. + If *error* is provided, the marker carries the error payload. + """ + ... + + # -- Consumer API (gateway side) ------------------------------------ + + @abstractmethod + async def read_chunks( + self, + job_id: str, + *, + block_ms: int = 5000, + batch_size: int = 50, + ) -> AsyncIterator[str]: + """ + Async generator that yields chunks until the terminal marker is read. + + Parameters: + block_ms: how long to wait for new data before emitting a + keep-alive (used for timeout / disconnect checks). + batch_size: max chunks returned per internal read. + """ + ... + # Make this an async generator at the type level. + if False: # pragma: no cover + yield "" + + # -- Lifecycle ------------------------------------------------------ + + @abstractmethod + def delete_stream(self, job_id: str) -> None: + """ + Remove the stream resource entirely. + + Called once the consumer has read the final chunk, or by a periodic + cleanup for abandoned streams. + """ + ... + + @abstractmethod + def stream_exists(self, job_id: str) -> bool: + """Return ``True`` if the stream key exists.""" + ... diff --git a/apipod/engine/utils.py b/apipod/engine/utils.py index 2b18c92..6840fc3 100644 --- a/apipod/engine/utils.py +++ b/apipod/engine/utils.py @@ -84,3 +84,11 @@ def normalize_name(name: str, preserve_paths: bool = False) -> str: if len(name) == 0: return "no_name" return name + + +def normalize_mount_prefix(prefix: str) -> str: + """Normalize a router mount prefix to a leading-slash path segment.""" + prefix = (prefix or "").strip() + if not prefix: + return "" + return prefix if prefix.startswith("/") else f"/{prefix}" diff --git a/docs/AGENTS.md b/docs/AGENTS.md new file mode 100644 index 0000000..9628794 --- /dev/null +++ b/docs/AGENTS.md @@ -0,0 +1,86 @@ +# Development instructions + +## 1. Think Before Coding + +**Don't assume. Don't hide confusion. Surface tradeoffs.** + +Before implementing: +- State your assumptions explicitly. If uncertain, ask. +- If multiple interpretations exist, present them. Don't pick silently. +- If a simpler approach exists, say so. Push back when warranted. +- If something is unclear, stop. Name what's confusing. Ask. +- Use thinking methods like Elon Musk's first principle reasoning or design-thinking for complex tasks. + +## 2. Simplicity First + +**Minimum code that solves the problem. Nothing speculative.** +- No features beyond what was asked. +- No abstractions for single-use code. +- No "flexibility" or "configurability" that wasn't requested. +- No error handling for impossible scenarios. +- If you write 200 lines and it could be 50, rewrite it. +- Don't repeat yourself. + +Ask yourself: "Would a senior engineer say this is overcomplicated?" If yes, simplify. + +## 3. Surgical Changes + +**Touch only what you must. Clean up only your own mess.** + +When editing existing code: +- Don't "improve" adjacent code, comments, or formatting. +- Match existing style, even if you'd do it differently. +- If you notice unrelated dead code, mention it - don't delete it. + +When your changes create orphans: +- Remove imports/variables/functions that YOUR changes made unused. +- Don't remove pre-existing dead code unless asked. + +The test: Every changed line should trace directly to the user's request. + +## 4. Code quality and style +- Write high-quality, developer-friendly and clean-code, use design-patterns (Refactoring Guru) where helpful. +- Include docstrings and comments to explain the why of a non-obvious method or code. No decorative comments. +- Write memory and compute efficient code. +- Write pythonic code and use builtins when it simplifies (generators, list-comprehensions, functools). +- Reduce complexity and boilerplate. Remove unused code. +- State unclear variable names and rename for better readability and clearity. +- Avoid local imports +- Do not write "deprecated" or backward compability code. +- Update the README.md if there's user related important changes. Update TECHNICAL_README.md for architectural, or project maintainer important changes. Keep the .md update brief use technical correct terms. +- Write ruff formatted code with ignores E203, W503, W501, W293, W291 and lineLenght 120. +- Avoid using direct cmd python calls and pip installs. +- Dev tools are make, ruff, mypy, pytest. +- Always ask before writing a test file. +- When executing python code, always use the local venv of the corresponding project. + +## 5. Reason and thinking style +Do thinking and reasoning like a smart caveman. +- Cut all filler, keep technical substance. +- Drop articles (a, an, the), filler (just, really, basically, actually). +- Drop pleasantries (sure, certainly, happy to). +- No hedging. Fragments fine. Short synonyms. +- Technical terms stay exact. Code blocks unchanged. +- Pattern: [thing] [action] [reason]. [next step] + +## 6. Public facing texts: Readmes, prints, logs. +- Write as technically brilliant European expert who gives you straight answers, backs claims with data, and genuinely wants you to succeed. +- Ttone spectrum Formality 35% (lean casual) · Humor 25% (dry, never flippant) · Complexity 40% (lean technical) · Confidence 80% (very confident). +- Direct without being cold. Say what you mean, no corporate padding. Warm without being informal. Complete sentences, measured enthusiasm. Committed, not hedging. +- Committed, not hedging. "This works", not "this might work". +- "We" for the platform, "you" for the reader. Reserve "I" for rare authored guides only. +- Developer-first, technical correct terms. +- No em-dashes. The character `—` (U+2014) must not appear in ANYWHERE. Use a period, comma, parentheses, or colon. +- No marketing adverbs like seamlessly, effortlessly, robust, powerful, cutting-edge, automatically, intelligently, revolutionary, game-changing, world-class, best-in-class. E.g. three steps, ~X minutes" beats "easy onboarding". +- No bullet-everything. Bullets only for true enumerations (statuses, regions, tiers). +- Subtle transmit socaity's vision and use emotional-intelligence and marketing techniques for end-user intent based messages. +- Use google docstring format. +- The four brand hues (each tied to an ICP) with Tonal scales (50/300/500/700/900). Apply where makes sense. +| Name | Hex | Meaning / ICP | +| ---------------------------- | ----------- | ------------------------------------------------------------------------- | +| **Neon Green** | `#5A8F00` | Developer. Active system health, engineering precision, technical truth. | +| **Silicon Blue** | `#0A86BF` | SMB CTO. Enterprise stability, secure infrastructure, professional trust. | +| **Creator Violet** | `#7C3AED` | Creator. Generative potential, creative momentum, warmth. | +| **Signal Pink (Rose)** | `#EC4899` | Alert. Warnings, critical status, high-priority attention. | + + diff --git a/docs/EXAMPLE_DOCKERFILE_AZURE b/docs/EXAMPLE_DOCKERFILE_AZURE deleted file mode 100644 index d9d8aac..0000000 --- a/docs/EXAMPLE_DOCKERFILE_AZURE +++ /dev/null @@ -1,32 +0,0 @@ -FROM python:3.10-slim - -# Set the working directory -WORKDIR /app - -# ffmpeg needed for media-toolkit video -# Install FFmpeg and dependencies required to install ffmpeg -RUN apt-get update && apt-get install -y \ - software-properties-common \ - apt-transport-https \ - ca-certificates \ - ffmpeg \ - && apt-get clean && rm -rf /var/lib/apt/lists/* - -# Copy the shared connectors module -COPY . /app - -# Install dependencies -RUN pip install --upgrade pip -RUN pip install . - -# Configure environment variables -ENV HOST = "0.0.0.0" - -# Expose the port -ARG port=8000 -ENV PORT=$port -EXPOSE $port - - -# Set the entrypoint with explicit host and port binding -CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"] diff --git a/docs/EXAMPLE_DOCKERFILE_RUNPOD b/docs/EXAMPLE_DOCKERFILE_RUNPOD deleted file mode 100644 index c302cd0..0000000 --- a/docs/EXAMPLE_DOCKERFILE_RUNPOD +++ /dev/null @@ -1,40 +0,0 @@ -FROM runpod/pytorch:2.4.0-py3.11-cuda12.4.1-devel-ubuntu22.04 - -SHELL ["/bin/bash", "-c"] - -# Install dependencies -# ffmpeg needed for mediatoolkit video -# Install system dependencies including CUDA and cuDNN -RUN apt-get update && apt-get install -y \ - ffmpeg \ - libcudnn9-cuda-12 \ - libcudnn9-dev-cuda-12 \ - && apt-get clean - -# Ensure CUDA and cuDNN libraries are in the library path -ENV LD_LIBRARY_PATH="/usr/local/cuda/lib64:${LD_LIBRARY_PATH}" - -# create the folder for face2face -WORKDIR / - -# Install Python dependencies (Worker Template) -# added runpod directly in dependencies -# added other libs before which keep failing in requirements file for unknown reasons. -# note that fairseq need to be installed differently in windows and linux -RUN pip install runpod>=1.7.7 && \ - pip install . --no-cache - -# Configure APIPod for RunPod serverless deployment - -ENV APIPOD_COMPUTE="serverless" -ENV APIPOD_PROVIDER="runpod" - -ARG port=8080 -ENV APIPOD_PORT=$port -# allows the docker container to use the port -EXPOSE $port -# allows any IP from the computer to connect to the host -ENV APIPOD_HOST="0.0.0.0" - -# Set the entrypoint with explicit host and port binding -CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"] \ No newline at end of file diff --git a/docs/README_TECH.md b/docs/README_TECH.md deleted file mode 100644 index 7b9c79c..0000000 --- a/docs/README_TECH.md +++ /dev/null @@ -1,20 +0,0 @@ - -# Queue mixin, Job Queue and Job lifecycle - -The fastapi router backend uses the _queue_mixin to manage jobs. -Every call to an task_endpoint will submit the job via the mixin to the job_queue. - - -Jobs are managed by the job_queue. -When a job is created in, it goes through a series of stages before it is completed. - -The stages are: -- validate_job_before_add: Used to validate parameters, user permissions, etc. -- add_job: job gets added to job store. -- create_job: literally creates the job. Can be overridden to use a specialized job class. -- process_job: The main job processing function. This is where the actual work is done. -- complete_job: Called when the job is completed. This is where the job is marked as completed and the result is stored. -- remove_job: - - Called when the job is removed from the JobStore and RAM. - - This happens when the user gets the result of the job or when the job was never collected (orphant job). - diff --git a/docs/TECHNICAL_README.md b/docs/TECHNICAL_README.md new file mode 100644 index 0000000..ac815ee --- /dev/null +++ b/docs/TECHNICAL_README.md @@ -0,0 +1,170 @@ +# APIPod — Technical Guide + +This document explains how APIPod works internally: its architecture, the request lifecycle, and the design principles behind it. It is aimed at developers maintaining or extending the package. For usage-oriented documentation see the [main README](../README.md). + +## What APIPod is + +APIPod is the standardized way to build AI services for the [socaity.ai](https://www.socaity.ai) catalog. It wraps FastAPI with the batteries an AI service needs — media file handling, job queues with progress reporting, serverless routing, standardized request/response schemas — while keeping the developer experience of plain FastAPI: you write a function, annotate its parameters, and decorate it with `@app.endpoint`. + +It sits in an ecosystem of three packages: + +| Package | Role | +| --- | --- | +| **APIPod** (this repo) | Server side: define and deploy AI service endpoints | +| [media-toolkit](https://github.com/SocAIty/media-toolkit) | Shared media types (`ImageFile`, `AudioFile`, `VideoFile`, …) used for I/O on both ends | +| [fastSDK](https://github.com/SocAIty/fastSDK) | Client side: generated SDKs that handle uploads, polling and streaming | + +## Package layout + +``` +apipod/ +├── api.py # APIPod() factory: resolves intent → backend instance +├── common/ +│ ├── settings.py # Env-driven config (APIPOD_SIMULATE / _DIRECT / _COMPUTE / _PROVIDER, cert, host, port) +│ ├── constants.py # Enums: COMPUTE, PROVIDER, SERVER_HEALTH +│ └── schemas/ +│ └── __init__.py # Re-exports from socaity-schemas (OpenAI-compatible request/response + FileModel) +├── engine/ +│ ├── base_backend.py # Shared backend base (title, version, health) +│ ├── endpoint_config.py # EndpointExecutionPlan + configurator (how to register an endpoint) +│ ├── backend/ +│ │ ├── fastapi/ # SocaityFastAPIRouter + file handling, streaming mixin, exceptions +│ │ └── runpod/ # SocaityRunpodRouter (serverless, path-based) +│ ├── files/ # _BaseFileHandlingMixin + parse_schema_media_fields (inputs → MediaFile) +│ ├── jobs/ # BaseJob, JobProgress, JobResult (+ factory) +│ ├── queue/ # JobQueue, JobStore, _QueueMixin (enqueue instead of block) +│ ├── streaming/ # StreamStore port + LocalStreamStore + StreamProducer +│ └── signatures/ # Signature inspection policies (media params, Body vs Form, …) +└── deploy/ # `apipod build`: Dockerfile generation, dependency/CUDA detection +``` + +## Core principles + +1. **One decorator, many execution modes.** Developers only write `@app.endpoint(...)`. The router inspects the function and decides how to run it. They never choose a "mode" explicitly — the signature is the contract. +2. **Deployment is intent, not code.** The same service file runs as a plain FastAPI app, a queued FastAPI app, or a RunPod serverless worker. `APIPod()` is a factory that picks the backend from a single *intent* (`simulate` / `direct`) locally, or from the `APIPOD_COMPUTE` / `APIPOD_PROVIDER` env vars Socaity injects in a managed deployment (see [Orchestrator, compute, provider](#orchestrator-compute-provider-and-simulate)). +3. **Media files are objects, not bytes.** Endpoint authors annotate parameters with media-toolkit types (`ImageFile`, `AudioFile`, …) and receive parsed, ready-to-use objects — regardless of whether the client sent a multipart upload, a URL, or base64. +4. **Standardized, OpenAI-compatible schemas.** Common AI payloads (chat, completion, embeddings, image/video/audio/3D generation, vision) have canonical pydantic schemas whose wire format mirrors the OpenAI API. The shape is provider-agnostic: any model can serve them; clients written against OpenAI-compatible tooling work without translation. +5. **Long-running work returns a job, not a blocked connection.** With a queue configured, endpoints return a `JobResult` immediately; clients poll `/status/{job_id}` (fastSDK does this automatically) and can receive progress updates via `JobProgress`. + +## Orchestrator, compute, provider, and simulate + +Three concepts describe *where and how* a service runs. Historically developers had to set all three; they no longer do. Understanding them still helps when reading the code. + +- **Orchestrator** — *who routes and queues requests*. There is exactly one: **Socaity**. It is the implicit orchestrator that fronts the service, distributes jobs, and handles scaling/auth. It is no longer a flag. `--native` is the one escape hatch: it *skips* Socaity to talk to a provider's native backend instead (e.g. RunPod's own serverless worker). +- **Compute** (`COMPUTE`) — *the shape of the machine*: `serverless` (scale-to-zero, job-queue semantics) or `dedicated` (an always-on box, plain request/response). +- **Provider** (`PROVIDER`) — *the cloud the compute lives on*: `runpod`, `azure`, `scaleway`, `localhost`, … Not every provider supports every compute (e.g. Azure has no serverless; APIPod warns and falls back to the job-queue emulation). + +A developer never assembles this matrix by hand. They express a single **intent**: + +- **Development** (`APIPod()` / `apipod start`) — plain FastAPI, the fastest loop. +- **Simulation** (`APIPod(simulate="{compute}-{provider}")` / `apipod simulate ...`) — emulate a deployment **locally**, no code changes. The target string collapses compute + provider (`serverless`, `serverless-runpod`, `dedicated-azure`); compute defaults to `serverless`. `direct=True` emulates the provider's native worker instead of the Socaity queue. +- **Managed deployment** — when the service runs on the platform, Socaity sets `SOCAITY_DEPLOYMENT_CERT` (SHA1 of a shared secret). When that cert verifies (`IS_MANAGED_DEPLOYMENT`), `simulate`/`direct` are ignored and the **real** backend is selected from the `APIPOD_COMPUTE` / `APIPOD_PROVIDER` env vars Socaity injects — so the serverless-RunPod path runs the *real* worker, not the emulator. + +## Backend resolution + +`APIPod()` in `api.py` is not a class — it is a factory. It resolves the intent (managed → `_resolve_managed`, otherwise → `_resolve_intent`) into a `(backend_class, use_job_queue, runpod_simulate)` triple and returns one of: + +- **`SocaityFastAPIRouter`** — an `APIRouter` subclass bound to a `FastAPI` app. Used for development and dedicated compute (no queue), and for the serverless emulation (paired with an in-memory `JobQueue` + `LocalStreamStore` and a background worker thread). +- **`SocaityRunpodRouter`** — a path-based dispatcher for RunPod serverless. There is no HTTP layer: RunPod delivers a JSON job whose `input.path` selects the registered function; the router converts files, injects `JobProgress`, executes, and returns a serialized `JobResult` (or a generator for streaming). It can also synthesize an OpenAPI schema by replaying the FastAPI signature conversion, so fastSDK clients can be generated against serverless deployments too. Its `simulate` flag chooses between RunPod's local API emulator (`apipod simulate serverless-runpod --native`) and the real worker (managed deployment). + +## The endpoint pipeline (FastAPI backend) + +When a function is decorated with `@app.endpoint(path)`, the router asks the `FastApiEndpointConfigurator` to build an immutable `EndpointExecutionPlan`. The plan records whether the signature contains a registered request schema, whether the function itself streams, and whether queueing is enabled. + +The function then goes through the normal APIPod decorator pipeline: + +1. **Schema binding detection** — a parameter (any name) annotated with a standardized request schema (e.g. `ChatCompletionRequest`) or a subclass becomes `plan.schema_binding`. This does not register a separate route. It only tells the downstream decorators to parse that JSON body into the schema object and wrap the final result into the registered response model. +2. **Streaming endpoint decorator** — for generator functions, output is bridged into a `StreamingResponse` with SSE-friendly headers. +3. **Task endpoint decorator** — when a job queue is configured (and `use_queue` is not `False`), the call enqueues a job and immediately returns a `JobResult`. Schema requests with `stream=true` are the one request-level exception: the task decorator executes them inline and streams the result instead of queueing. +4. **Standard endpoint decorator** — direct execution. It also handles non-queued schema endpoints: parse the request schema, execute the user function, wrap the result into the schema response model, and serialize non-schema media results through `JobResultFactory`. + +In all cases the function then passes through the **file-handling preparation** (next section) before being handed to FastAPI's `api_route`. + +### File handling: two layers + +File support is split into a *signature* layer and a *runtime* layer. + +- **Signature rewriting** (`engine/backend/fastapi/file_handling_mixin.py`): media-toolkit annotations in the user's signature are rewritten so FastAPI/OpenAPI understand them. `image: ImageFile` becomes `Union[LimitedUploadFile, ImageFileModel, str]` — the client may send a multipart upload, a `FileModel` JSON object (`{file_name, content_type, content}` where content is base64 or a URL), or a plain URL/base64 string. `MediaList[...]` maps to list variants. Upload size limits are enforced via a dynamically subclassed `LimitedUploadFile`. `JobProgress` parameters are stripped from the public signature (and a dummy is injected when no queue runs). +- **Runtime conversion** (`engine/files/base_file_mixin.py`): before the user function executes, every media-annotated argument is converted to the annotated media-toolkit type via `media_from_any` — whatever the client actually sent. The function body always receives real `MediaFile` objects. This layer is backend-agnostic and reused by the RunPod router. + +On the way out, `JobResultFactory._serialize_result` converts returned `MediaFile`/`MediaList`/pydantic objects back into JSON-safe `FileModel` payloads. + +### Standardized schemas + +`socaity-schemas` (re-exported at ``apipod.common.schemas``) defines request/response pairs for: chat completion, text completion, embeddings, image generation, video generation, transcription, speech (TTS), voice creation, voice conversion, 3D generation, vision, and multimodal embeddings. `model` is optional on every request, since an APIPod service typically serves exactly one model. The audio API is split per use case, mirroring OpenAI: `TranscriptionRequest` (STT), `SpeechRequest` (TTS, with `voice` as a named voice or a cloning reference file), `CreateVoiceRequest` (voice cloning → embedding) and `VoiceConversionRequest` (voice2voice). + +`SCHEMA_REGISTRY` in `engine/schema_extension/schema_mixin.py` is the single source of truth: it maps each request schema to a `SchemaEndpointSpec` (response model, tag). Everything else derives from it — `engine/signatures/policies.py` builds `SUPPORTED_REQUEST_SCHEMAS` from the registry keys (schema-annotated parameters are read from the JSON body while plain parameters stay form-encoded), and both routers detect schema endpoints with `get_schema_binding`, which finds the schema-typed parameter by annotation regardless of its name (subclasses of registered schemas are also detected, so services can extend a schema with custom fields). Schema endpoints may not declare additional user parameters; put extra inputs on the schema or a schema subclass. + +**Nested media files**: request schemas may declare `FileModel`-typed fields (`ImageGenerationRequest.image`, `TranscriptionRequest.audio`, …). Pydantic accepts uploads, FileModel JSON objects, URLs and plain base64 strings for those fields; before the endpoint function runs, `parse_schema_media_fields` (in `engine/files/base_file_mixin.py`) replaces them with parsed media-toolkit objects — the endpoint receives a ready-to-use `ImageFile`/`AudioFile`, exactly like method-level `def endpoint(image: ImageFile)` parameters. + +**Response wrapping** (`wrap_schema_response`): if the function already returns the response model, it passes through. Schema helpers also accept convenient raw results (a plain string for chat/completion/transcription, raw vectors for embeddings). A bare ``None`` or dict with ``None`` fields is coerced to the same empty shorthand (empty text or required-field defaults from the response model's pydantic JSON schema) before validation. Everything else shares one generic path: a uniform envelope (`created`, `model`) merged with the result dict and validated by pydantic — a returned media-toolkit file is lifted into the `data` list automatically. Response IDs are not generated by APIPod schemas; IDs belong to the platform `JobResult`/socaity.ai layer. + +### Jobs, queue and lifecycle + +The local `JobQueue` (in-memory, threaded) drives the serverless emulation and dedicated-with-queue modes. A job passes through these stages: + +- `validate_job_before_add` — parameter/permission validation +- `add_job` — persisted in the `JobStore`, returns immediately +- `create_job` / `process_job` — the worker picks it up and runs the user function +- `complete_job` — result stored, status set +- `remove_job` — cleaned up once collected (or orphaned) + +`JobProgress` is the in-band progress channel: if the user function declares a `job_progress` parameter, the backend injects an implementation (`JobProgress` locally, `JobProgressRunpod` on RunPod) and `set_status(progress, message)` updates surface in `/status` polls. `JobResult` is the unified public envelope: `job_id`, `status` (`pending/processing/completed/failed`), `result`, `progress`, `message`, timing `metrics`, and hypermedia `links` (status/cancel/stream). + +Standard routes registered automatically: `GET /status/{job_id}`, `POST /cancel/{job_id}`, `GET /health`, and — when a stream store is configured — `GET /stream/{job_id}` (SSE). + +### Streaming + +Three streaming paths exist today: + +1. **Generator endpoints** — any endpoint function that is a (async) generator, or whose return annotation declares an iterator, is served as an SSE `StreamingResponse` (FastAPI) or as a native generator handed back to RunPod (`return_aggregate_stream`). Detection is annotation/inspect-based — no source-code heuristics. +2. **Schema endpoints with `stream=true`** — the request schema carries the `stream` flag (chat, completion, transcription, speech, video generation). OpenAPI lists that field only when `is_streaming_endpoint` classifies the handler as streaming (generator function, ``Iterator`` return annotation, or AST detection of a generator return under ``if request.stream``). Non-streaming handlers register a sibling model where ``stream`` uses ``SkipJsonSchema``: hidden from OpenAPI but still accepted in validation so clients can send ``stream: false``. Streaming bypasses the queue; what gets streamed depends on what the function returns: + - a **generator of raw tokens**: for tags with a registered chunk model (`STREAM_CHUNK_SPECS`, currently `chat`), `SchemaStreamSerializer` wraps each token into the standardized chunk model (`ChatCompletionChunk`) as an SSE event — APIPod generates the stable chunk `id`, the `created` timestamp and the `object` discriminator, then closes with a final delta and the `[DONE]` sentinel. The endpoint only yields text. Other token-delta tags without a chunk model stream their tokens as-is (SSE), and non-SSE tags stream raw bytes; + - an **`AudioFile`/`VideoFile`**: its encoded bytes are chunked into a `StreamingResponse` with the file's media content type — raw audio chunks, not SSE, matching OpenAI's `stream_format="audio"` behavior; + - anything else: the regular wrapped JSON response (the endpoint cannot stream). + On RunPod the same logic applies (`_as_native_stream`): the request is validated and media-parsed by `prepare_schema_call`, the response is wrapped by `wrap_schema_response`, and stream chunks are base64-encoded when binary because RunPod transports JSON. +3. **Job streaming (serverless emulation)** — when a queue is configured, a streaming endpoint does **not** stream on the request connection. The job is enqueued and the response returns a `JobResult` immediately (with a `stream` link). The worker (producer) then relays the chunks into a **stream store**, and the client (consumer) reads them from `GET /stream/{job_id}` as SSE. A client that prefers to wait can poll `GET /status/{job_id}` instead and receive the **full aggregated result** once the job finishes. This mirrors a real deployment, where producer (worker) and consumer (gateway) live in different processes. + +### The stream store + +The **stream store** is the pluggable backend that buffers a job's chunks between the worker and the gateway. Its key idea is to decouple *producing* a stream (sync, worker side) from *consuming* it (async SSE, gateway side), so the same endpoint code streams identically whether it runs on localhost or on a real platform. + +- **`StreamStore`** (`engine/streaming/stream_store.py`) is the port (abstract base class). Producer methods (`open_stream`, `write_chunk`, `close_stream`) are synchronous because the worker writes from a sync context; the consumer method `read_chunks` is an async generator that yields straight into a `StreamingResponse`. `delete_stream` / `stream_exists` cover lifecycle. +- **`LocalStreamStore`** (`engine/streaming/local_stream_store.py`) is the default in-memory implementation APIPod uses on localhost — thread-safe, no external dependencies. It exists purely to *emulate* deployment behavior; a real platform (e.g. Socaity) injects its own implementation (such as a Redis Streams store) via the `stream_store` constructor argument, without changing any endpoint code. +- **`StreamProducer`** (`engine/streaming/stream_producer.py`) is the bridge the router hands to the worker: it carries the raw chunk iterator, how to serialize each chunk for the store (ChatCompletion deltas, base64-framed media bytes, or plain tokens), the closing chunks (`finish` + `[DONE]`), and how to aggregate the raw chunks into the full `/status` result. + +`APIPod()` wires a `LocalStreamStore` automatically whenever a job queue is in use (serverless-localhost and dedicated-with-queue). In plain FastAPI mode (no queue) there is no stream store: streaming happens directly on the request connection (path 1/2 above). + +### Deployment + +`apipod build` (see `deploy/`) scans the project (entrypoint, dependencies, CUDA requirements) and generates a Dockerfile from compatible templates. The resulting container runs unchanged on dedicated hosts, on socaity.ai, or on RunPod serverless — only the env vars differ: Socaity injects `SOCAITY_DEPLOYMENT_CERT` plus `APIPOD_COMPUTE` / `APIPOD_PROVIDER` to select the real backend, while locally you drive the same paths with `APIPOD_SIMULATE` / `APIPOD_NATIVE` (set for you by `apipod simulate`). + +## Request lifecycle, end to end + +A client calls `POST /tts` with a JSON body. FastAPI validates it against the rewritten signature and calls the outermost wrapper. The file-handling layer converts any media inputs into media-toolkit objects. If the endpoint is queued, the queue mixin stores the job and returns `{job_id, status: "queued", links}` — the worker thread later executes the real function, feeding `JobProgress` updates into the store. The client (typically fastSDK) polls `/status/{job_id}` until `finished` and receives the result, with any returned `AudioFile` serialized as a `FileModel`. Without a queue the same conversion happens inline and the response returns directly. On RunPod, the identical user function is reached through `handler → _router(path) → file handling → execute`, proving the core principle: the function is written once, the backend decides how it runs. + +## Testing and CI + +The suite (`test/`) is built so endpoint definitions live apart from test logic, and one helper boots them under any run intent. + +``` +test/ +├── conftest.py # build_service (in-process TestClient), live_service (real subprocess via CLI), config matrix, file paths +├── services/ # reusable services: register(app) callbacks + a runnable entrypoint +│ ├── core_service.py # scalars, custom model, mixed media, JobProgress, file upload +│ ├── schema_service.py # one endpoint per standardized schema, an extended schema, raw/typed mapping CASES +│ └── exec_service.py # runnable service for fastSDK + streaming (predict, echo_image, chat, text, video) +├── files/ # media assets for upload/download tests +├── test_config.py # intent -> backend resolution + /openapi.json loads for every backend +├── test_core.py # core endpoint plumbing (types, model, files, queue lifecycle) +├── test_schemas.py # standardized schema endpoints + response-model normalization +├── test_cli.py # scan / build / simulate produce the right artifacts +└── test_execution.py # fastSDK end-to-end (subprocess) + SSE streaming (in-process) +``` + +Run `pytest`. Two primitives keep tests DRY: `build_service(register, **config)` yields a `TestClient` for an app built from a `register(app)` callback under any `APIPod(**config)` intent; `live_service(simulate=...)` boots a service as a real subprocess through `apipod start` / `simulate` and yields its URL. Backends are parametrized from `FASTAPI_CONFIGS`, so a single test asserts behaviour across development, serverless, dedicated and runpod. + +The fastSDK end-to-end tests skip automatically when the installed fastsdk lacks the `connect` API (e.g. a local in-progress build); CI installs one that has it. Pytest config (`pythonpath` in `pyproject.toml`) resolves `import apipod` to the in-repo source and the `conftest`/`services` helpers by name, so no `sys.path` shims are needed and each file is also runnable directly from an IDE via its `__main__` block. + +CI (`.github/workflows/publish.yml`) runs the `test` job on every push and pull request (installing `.[test]`, which includes fastSDK). The `publish` job `needs: test`, so the patch version bump and PyPI upload happen only on a push to `main` after tests pass. ruff and mypy run as soft checks (`continue-on-error`): they surface warnings without gating the build. + diff --git a/pyproject.toml b/pyproject.toml index 091e93a..9f48e42 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -17,6 +17,7 @@ dependencies = [ "fastapi", 'python-multipart>=0.0.20', "media-toolkit>=0.02.22", + "socaity-schemas>=0.0.1", "jinja2", "toml" ] @@ -28,20 +29,34 @@ apipod = "apipod.cli:main" runpod = [ "runpod>=1.7.7" ] +test = [ + "pytest", + "httpx", + "ruff", + "mypy", +] [project.urls] Repository = "https://github.com/SocAIty/apipod" Homepage = "https://www.socaity.ai" -[tool.setuptools.packages.find] -where = ["."] -include = ["apipod*"] - -[tool.setuptools.package-data] -apipod = [ - "deploy/*.j2", - "deploy/*.txt", - "deploy/starter_README.md", -] - +[tool.pytest.ini_options] +testpaths = ["test"] +# Resolve in-repo apipod and sibling packages in local dev (CI installs from pip). +pythonpath = [".", "test", "../fastsdk", "../socaity-schemas"] + +[tool.ruff] +line-length = 120 +exclude = ["venv", "dist", "build", "*.egg-info"] + +[tool.ruff.lint] +# Mirror the previous flake8 config: ignore stylistic rules that conflict with +# black/our formatting conventions. +ignore = ["E203", "E261", "E501", "W293"] + +[tool.mypy] +# Soft static typing: a warning signal, never a build gate (see CI). +ignore_missing_imports = true +follow_imports = "silent" +warn_unused_ignores = false diff --git a/test/__init__.py b/test/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/test/conftest.py b/test/conftest.py new file mode 100644 index 0000000..9aebd4d --- /dev/null +++ b/test/conftest.py @@ -0,0 +1,141 @@ +""" +Shared test infrastructure for the APIPod suite. + +The suite is built on a few small primitives so individual test files stay +declarative and DRY: + +- ``FASTAPI_CONFIGS`` / ``QUEUE_CONFIGS`` — the canonical matrix of APIPod run + intents (development, serverless, dedicated, runpod, ...). Parametrize over + it to assert behaviour holds across every backend. +- ``build_service(register, **config)`` — build an app from a callback that + registers endpoints and hand back a wired FastAPI ``TestClient`` (in-process, + no socket). Used by the config / openapi / core / schema tests. +- ``live_service(simulate=...)`` — boot the example service in a subprocess via + the real ``apipod`` CLI (the same ``start`` / ``simulate`` a user runs), + yield its base URL, and shut it down afterwards. Used by the fastSDK + end-to-end execution tests. + +A registrar is just ``def register(app): ...``; keeping endpoints out of the +helpers lets one helper serve config, openapi, schema and execution tests alike. +""" + +import contextlib +import socket +import subprocess +import sys +import time +import urllib.error +import urllib.request +from pathlib import Path + +import pytest +from fastapi.testclient import TestClient + +from apipod import APIPod + +REPO_ROOT = Path(__file__).resolve().parents[1] + +# --------------------------------------------------------------------------- # +# Test asset files (used by the media upload/download tests) +# --------------------------------------------------------------------------- # +FILES_DIR = Path(__file__).parent / "files" +IMAGE_FILE = FILES_DIR / "test_image.png" +AUDIO_FILE = FILES_DIR / "test_audio.wav" +VIDEO_FILE = FILES_DIR / "test_video.mp4" + +# Entrypoint booted by the fastSDK end-to-end execution tests. +INFERENCE_SERVICE = Path(__file__).parent / "services" / "core_service.py" + + +# --------------------------------------------------------------------------- # +# Run-intent matrix. Each entry is the kwargs passed to APIPod(...). +# --------------------------------------------------------------------------- # +# FastAPI-backed intents resolve to SocaityFastAPIRouter and expose a real HTTP +# app (the only ones that can serve /openapi.json or run under TestClient). +FASTAPI_CONFIGS = [ + pytest.param({}, id="development"), + pytest.param({"simulate": "serverless"}, id="serverless"), + pytest.param({"simulate": "dedicated"}, id="dedicated"), + pytest.param({"simulate": "dedicated-azure"}, id="dedicated-azure"), + pytest.param({"simulate": "serverless-runpod"}, id="serverless-runpod"), + pytest.param({"simulate": "serverless-azure"}, id="serverless-azure-fallback"), +] + +# Intents that boot a job queue (serverless emulation): endpoints return a +# JobResult processed by the in-process worker. +QUEUE_CONFIGS = [ + pytest.param({"simulate": "serverless"}, id="serverless"), + pytest.param({"simulate": "serverless-runpod"}, id="serverless-runpod"), +] + + +def build_app(register, **config): + """Create an APIPod app for ``config`` and register endpoints via callback.""" + app = APIPod(**config) + register(app) + return app + + +@contextlib.contextmanager +def build_service(register, **config): + """ + Yield a ``TestClient`` for an app built from ``register`` under ``config``. + + The client is entered as a context manager so the app lifespan runs: that + starts the in-process queue worker for serverless intents. + """ + app = build_app(register, **config) + fastapi_app = app.app + fastapi_app.include_router(app) + with TestClient(fastapi_app) as client: + yield client + + +# --------------------------------------------------------------------------- # +# Live server (real subprocess via the apipod CLI) for fastSDK tests +# --------------------------------------------------------------------------- # +def _free_port() -> int: + with contextlib.closing(socket.socket(socket.AF_INET, socket.SOCK_STREAM)) as s: + s.bind(("127.0.0.1", 0)) + return s.getsockname()[1] + + +def _wait_until_healthy(url: str, proc: subprocess.Popen, timeout: float = 40.0): + """Poll GET /health until the service answers, or fail if the process dies.""" + deadline = time.time() + timeout + while time.time() < deadline: + if proc.poll() is not None: + raise RuntimeError(f"Service process exited early with code {proc.returncode}.") + try: + with urllib.request.urlopen(f"{url}/health", timeout=2) as resp: + if resp.status == 200: + return + except (urllib.error.URLError, ConnectionError, OSError): + time.sleep(0.2) + raise TimeoutError(f"Service at {url} did not become healthy within {timeout}s.") + + +@contextlib.contextmanager +def live_service(simulate: str = "serverless", entrypoint: Path = INFERENCE_SERVICE, port: int = 0): + """ + Boot ``entrypoint`` in a subprocess through the real apipod CLI and yield its URL. + + ``simulate=None`` runs ``apipod start`` (development); otherwise it runs + ``apipod simulate