Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
131 changes: 131 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,131 @@
# AGENTS.md

## Project Overview

Tilebox IaC - reusable Pulumi components for auto-scaling Tilebox runner clusters on AWS, Azure, and GCP.

Clusters use the public `ghcr.io/tilebox/runner:latest` image. This repository consumes that image but does not build or publish runner images. Custom images are outside this library's scope.

## Commands

- **Install dependencies**: `uv sync`
- **Lint**: `ruff check .`
- **Format**: `ruff format .`
- **Type check**: `pyright`
- **Pulumi preview**: `pulumi preview`
- **Pulumi deploy**: `pulumi up`

## Architecture

This is a Pulumi Python library providing reusable infrastructure components, organized into cloud-specific submodules:

### Package Structure

```
tilebox_iac/
├── __init__.py # Exports aws, azure, and gcp submodules
├── release_runner.py # Official GHCR runner image default
├── aws/ # Amazon Web Services components
│ ├── __init__.py
│ ├── auto_scaling_cluster.py
│ ├── cloud-init.yaml
│ ├── iam_role.py
│ ├── network.py
│ └── secrets.py
├── azure/ # Microsoft Azure components
│ ├── __init__.py
│ ├── auto_scaling_cluster.py
│ ├── cloud-init.yaml
│ ├── identity.py
│ ├── network.py
│ └── secrets.py
└── gcp/ # Google Cloud Platform components
├── __init__.py
├── auto_scaling_cluster.py
├── cloud-init.yaml
├── network.py
├── secrets.py
└── service_account.py
```

### GCP Components (`tilebox_iac.gcp`)

- `AutoScalingCluster` - Managed Instance Group with Spot VMs and CPU-based autoscaling
- `Network` - VPC with Private Google Access and optional NAT Gateway
- `ServiceAccount` - GCP IAM service accounts with role bindings
- `Secret` - GCP Secret Manager wrapper

### AWS Components (`tilebox_iac.aws`)

- `AutoScalingCluster` - Auto Scaling Group with Spot instances and a Launch Template
- `Network` - VPC with private subnets, optional NAT Gateway, and S3 VPC Gateway Endpoint
- `IAMRole` - IAM roles with instance profiles, bucket policies, secrets access
- `Secret` - AWS Secrets Manager wrapper

### Azure Components (`tilebox_iac.azure`)

- `AutoScalingCluster` - Virtual Machine Scale Set with CPU-based autoscaling and optional Spot instances
- `Network` - VNet with a private subnet and optional NAT Gateway
- `ManagedIdentity` - Azure managed identity with role assignments
- `Secret` - Azure Key Vault secret wrapper

## Usage

```python
from tilebox_iac import gcp

runner_environment = {
"TILEBOX_API_KEY": tilebox_api_key_secret,
}

cluster = gcp.AutoScalingCluster(
"my-cluster",
environment_variables=runner_environment,
...,
)
```

AWS, Azure, and GCP use the same runner image and environment variable contract. `TILEBOX_API_KEY` is required; `TILEBOX_CLUSTER` is optional and defaults to the account's default cluster. In the upstream Tilebox repository, keep `ghcr.io/tilebox/runner:latest` as the built-in image and do not add image configuration, image builders, publishing, or private registry authentication. Forks may customize the image behavior for their own deployments.

## Code Style

- Use `ComponentResource` pattern for Pulumi components
- Use `TypedDict` for configuration dictionaries
- Use Jinja2 for cloud-init templates
- Follow ruff ALL rules (see pyproject.toml for ignored rules)
- Use type hints everywhere
- Import order: stdlib, blank line, external, blank line, internal (enforced by ruff isort)
- Format with ruff format

## Key Patterns

### ComponentResource Structure
```python
class MyComponent(ComponentResource):
def __init__(self, name: str, ..., opts: ResourceOptions | None = None) -> None:
super().__init__("tilebox:cloud:ComponentName", name, opts=opts)
# Create resources with ResourceOptions(parent=self)
self.register_outputs({...})
```

### TypedDict for Configs
```python
class MyConfigDict(TypedDict):
required_field: str
optional_field: NotRequired[str]
```

### Cloud-init Templates
- GCP: `gcp/cloud-init.yaml` - Uses GCP metadata server for secrets
- AWS: `aws/cloud-init.yaml` - Uses AWS CLI/IMDS for secrets
- Azure: `azure/cloud-init.yaml` - Uses a managed identity to fetch Key Vault secrets
- All templates anonymously pull the official GHCR runner image
- AWS self-reports persistent runner failures; GCP and Azure use provider health probes

## Dependencies

- `pulumi` - Core Pulumi SDK
- `pulumi-gcp` - GCP provider
- `pulumi-aws` - AWS provider
- `pulumi-azure-native` - Azure provider
- `jinja2` - Cloud-init templating
51 changes: 45 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,50 @@
# IaC for a auto-scaling cluster utilizing GCP Spot instances
# Tilebox infrastructure components

This library provides the following [Pulumi](https://www.pulumi.com/) resources:
This library provides reusable [Pulumi](https://www.pulumi.com/) components for running auto-scaling Tilebox runners on AWS, Azure, and GCP.

- A `LocalBuildTrigger` ressource that runs a local command to build a Docker image on code changes and pushes it to a Google Artifact Registry repository
- A `Secret` resource to manage GCP secrets
- A `AutoScalingGCPCluster` resource to manage the auto-scaling cluster
- A `GCPNetwork` resource to manage networking within the Cluster, optionally enabling Private Google Access (PGA) and a router for outbound internet access
## Tilebox runner

A [Tilebox runner](https://docs.tilebox.com/workflows/concepts/runners) watches one cluster and executes Python workflow releases deployed to it. It downloads release artifacts, starts their Python runtimes with `uv`, and updates its task registrations when deployments change. Workflow releases can therefore be deployed or rolled back without rebuilding the runner image or restarting the runner fleet.

Tilebox publishes the runner image for Linux AMD64 and ARM64 at [`ghcr.io/tilebox/runner`](https://github.com/tilebox/runner/pkgs/container/runner). It includes `uv`, Python 3.12–3.14, Git, Git LFS, SSH support, and the matching Tilebox CLI release. Every cluster runs `ghcr.io/tilebox/runner:latest`; instances pull the current image when they start. The image starts the runner with:

```bash
tilebox runner start
```

The default runner requires no image configuration:

```python
from tilebox_iac import gcp

cluster = gcp.AutoScalingCluster(
"tilebox-release-runners",
environment_variables={
"TILEBOX_API_KEY": tilebox_api_key_secret,
"TILEBOX_CLUSTER": tilebox_cluster_slug,
},
# Cloud, network, machine, and scaling configuration omitted.
)
```

AWS, Azure, and GCP use the same official image and environment variable contract. This library consumes the Tilebox image but does not build or publish runner images.

## Deployment requirements

Each release runner deployment needs:

- A `TILEBOX_API_KEY`, injected through the cloud provider's secret manager rather than baked into the image.
- Optionally, a Tilebox cluster slug created with `tilebox cluster create`, passed as `TILEBOX_CLUSTER`. The [default cluster](https://docs.tilebox.com/workflows/concepts/clusters#default-cluster) is used when it is omitted.
- Optionally, a non-default Tilebox API endpoint passed as `TILEBOX_API_URL`.
- Outbound network access for the Tilebox API, workflow release artifacts, Python package indexes, and any data sources used by the workflows.
- Cloud credentials, network access, system libraries, disk space, and CPU or GPU hardware required by the deployed workflows.
- At least one workflow release deployed to the same cluster with `tilebox workflow deploy-release` before the runner can execute its tasks.

[Release runners](https://docs.tilebox.com/guides/workflows/deploy-to-your-compute) currently execute Python workflow projects. The runner image is shared across workflows; workflow code and Python dependencies belong in each immutable workflow release.

## Components

The `tilebox_iac.aws`, `tilebox_iac.azure`, and `tilebox_iac.gcp` modules provide cloud-specific auto-scaling clusters, networking, identities, and secret integrations.

## Dev setup

Expand Down
6 changes: 3 additions & 3 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,17 +1,17 @@
[project]
name = "tilebox_iac"
version = "0.1.0"
description = "IaC for auto-scaling clusters utilizing GCP and AWS Spot instances."
description = "Pulumi components for auto-scaling Tilebox release runner clusters on AWS, Azure, and GCP."
readme = "README.md"
authors = [{ name = "Tilebox, Inc.", email = "support@tilebox.com" }]
requires-python = ">=3.10"
dependencies = [
"dirhash>=0.5.0",
"jinja2>=3.1.6",
"pulumi>=3.186.0",
"pulumi-aws>=6.60.0",
"pulumi-command>=1.1.0",
"pulumi-azure-native>=3.0.0",
"pulumi-gcp>=8.39.0",
"typing-extensions>=4.0",
]

[dependency-groups]
Expand Down
3 changes: 2 additions & 1 deletion tilebox_iac/__init__.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
from tilebox_iac import aws, gcp
from tilebox_iac import aws, azure, gcp

__all__ = [
"aws",
"azure",
"gcp",
]
2 changes: 0 additions & 2 deletions tilebox_iac/aws/__init__.py
Original file line number Diff line number Diff line change
@@ -1,13 +1,11 @@
from tilebox_iac.aws.auto_scaling_cluster import AutoScalingCluster
from tilebox_iac.aws.iam_role import IAMRole
from tilebox_iac.aws.image_builder import LocalBuildTrigger
from tilebox_iac.aws.network import Network
from tilebox_iac.aws.secrets import Secret

__all__ = [
"AutoScalingCluster",
"IAMRole",
"LocalBuildTrigger",
"Network",
"Secret",
]
60 changes: 37 additions & 23 deletions tilebox_iac/aws/auto_scaling_cluster.py
Original file line number Diff line number Diff line change
@@ -1,48 +1,42 @@
import base64
import json
from collections.abc import Sequence
from pathlib import Path
from typing import Any, TypedDict
from typing import Any

import pulumi_aws as aws
from jinja2 import Environment, FileSystemLoader
from pulumi import ComponentResource, Input, Output, ResourceOptions
from pulumi_aws import autoscaling as aws_autoscaling
from pulumi_aws import ec2 as aws_ec2
from typing_extensions import NotRequired
from pulumi_aws import iam as aws_iam

from tilebox_iac.aws.iam_role import IAMRole, IAMRoleConfigDict
from tilebox_iac.aws.secrets import Secret
from tilebox_iac.release_runner import RUNNER_IMAGE

env = Environment(loader=FileSystemLoader(Path(__file__).parent), autoescape=True)
template = env.get_template("cloud-init.yaml")


def _get_cloud_init(kwargs: dict[str, Any]) -> str:
"""Render the cloud-init config for the AWS VMs."""
image: str = kwargs["image"]
tag: str = kwargs["tag"] or "latest" # Default empty string to "latest"
environment_variables: dict[str, str] = kwargs["environment_variables"]
secrets: dict[str, str] = kwargs["secrets"]
secret_versions: dict[str, str] = kwargs["secret_versions"]

return template.render(
CONTAINER_IMAGE=f"{image}:{tag}",
REGISTRY_HOSTNAME=image.split("/", maxsplit=1)[0],
CONTAINER_IMAGE=RUNNER_IMAGE,
SECRETS=secrets,
SECRET_VERSIONS=secret_versions,
ENVIRONMENT_VARS=environment_variables,
)


class ContainerConfig(TypedDict):
image: Input[str]
tag: NotRequired[Input[str]]


class AutoScalingCluster(ComponentResource):
def __init__( # noqa: PLR0913
self,
name: str,
container: ContainerConfig,
instance_type: str,
cpu_target: float,
cluster_enabled: bool,
Expand All @@ -59,7 +53,6 @@ def __init__( # noqa: PLR0913

Args:
name: Name of the cluster.
container: Container image to run (ECR image URL).
instance_type: EC2 instance type to use.
cpu_target: CPU target for autoscaling (0.0 to 1.0).
cluster_enabled: Whether the cluster is enabled.
Expand Down Expand Up @@ -90,13 +83,6 @@ def __init__( # noqa: PLR0913
# Copy to avoid mutating caller's config (could cause side effects if reused)
iam_config_copy: IAMRoleConfigDict = dict(iam_config) if iam_config else {} # type: ignore[assignment]

# ECR read access is required for cloud-init to docker pull the container image
managed_policies = list(iam_config_copy.get("managed_policies", []))
ecr_policy = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
if ecr_policy not in managed_policies:
managed_policies.append(ecr_policy)
iam_config_copy["managed_policies"] = managed_policies # type: ignore[typeddict-item]

secrets_access = list(iam_config_copy.get("secrets_access", []))
secrets_access.extend(
{"secret_slug": secret.resource_name, "secret_arn": secret.arn} for secret in used_secrets.values()
Expand All @@ -111,6 +97,36 @@ def __init__( # noqa: PLR0913
opts=ResourceOptions(depends_on=[*list(used_secrets.values())], parent=self),
)

asg_arn = Output.all(
partition=aws.get_partition_output().partition,
region=aws.get_region_output().name,
account_id=aws.get_caller_identity_output().account_id,
).apply(
lambda values: (
f"arn:{values['partition']}:autoscaling:{values['region']}:{values['account_id']}:"
f"autoScalingGroup:*:autoScalingGroupName/{name}-*"
)
)
health_policy = aws_iam.RolePolicy(
f"{name}-set-instance-health",
role=iam_role.role.name,
policy=asg_arn.apply(
lambda arn: json.dumps(
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "autoscaling:SetInstanceHealth",
"Resource": arn,
}
],
}
)
),
opts=ResourceOptions(parent=self),
)

secrets: dict[str, Input[str]] = {}
# Include version IDs so secret value changes trigger Launch Template updates
secret_versions: dict[str, Input[str]] = {}
Expand All @@ -119,8 +135,6 @@ def __init__( # noqa: PLR0913
secret_versions[secret_env_var] = secret.latest_version

cloud_init_config = Output.all(
image=container["image"],
tag=container.get("tag", "latest"),
environment_variables=envs,
secrets=secrets,
secret_versions=secret_versions,
Expand Down Expand Up @@ -170,7 +184,7 @@ def __init__( # noqa: PLR0913
tags={"Name": f"{name}-instance"},
),
],
opts=ResourceOptions(depends_on=[iam_role], parent=self),
opts=ResourceOptions(depends_on=[iam_role, health_policy], parent=self),
)

if cluster_enabled:
Expand Down
Loading
Loading