Use official Tilebox release runners across all clouds#7
Conversation
Amp-Thread-ID: https://ampcode.com/threads/T-019e5927-b912-7498-bfc1-dd39fe936065 Co-authored-by: Amp <amp@ampcode.com>
- Run the published Tilebox runner image across AWS, Azure, and GCP - Add provider-native runner health checks and instance replacement - Remove local image builders and private registry integration Amp-Thread-ID: https://ampcode.com/threads/T-019f60ba-e218-745b-8fa8-ae9c31e00fbf Co-authored-by: Amp <amp@ampcode.com>
📝 WalkthroughWalkthroughThe repository now documents and exports multi-cloud Tilebox runner infrastructure. AWS and GCP clusters use a shared runner image with cloud-specific health handling, while Azure gains network, identity, secret, VM scale-set, autoscaling, and cloud-init components. ChangesRunner contract and package surface
AWS runner health integration
Azure runner infrastructure
GCP runner health integration
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant Pulumi
participant AzureAutoScalingCluster
participant AzureManagedIdentity
participant AzureVMScaleSet
participant CloudInitServices
Pulumi->>AzureAutoScalingCluster: create cluster component
AzureAutoScalingCluster->>AzureManagedIdentity: create identity and dependencies
AzureAutoScalingCluster->>AzureVMScaleSet: provision VM scale set with custom_data
AzureVMScaleSet->>CloudInitServices: execute cloud-init
CloudInitServices->>CloudInitServices: fetch secrets and start runner
CloudInitServices->>CloudInitServices: expose health endpoint
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (1)
tilebox_iac/gcp/auto_scaling_cluster.py (1)
106-124: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAvoid mutating the input
rolesdictionary in place.If the caller passes a shared
rolesdictionary, mutating it here by appendingsecret_rolesand updatingroleswill cause those modifications to leak and compound if the dictionary is reused for other clusters. Consider creating a shallow copy of the dictionary before modifying it.♻️ Proposed refactor
if roles is None: roles = {"roles": list(required_roles)} else: + roles = roles.copy() configured_roles = set(roles.get("roles", [])) roles["roles"] = list(required_roles | configured_roles)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tilebox_iac/gcp/auto_scaling_cluster.py` around lines 106 - 124, The role-building logic mutates the caller’s roles dictionary in place, causing changes to leak across reused configurations. In the surrounding function, create a shallow copy of the provided roles dictionary before updating "roles" and extending "secret_roles", while preserving the existing defaults and merged role behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tilebox_iac/azure/auto_scaling_cluster.py`:
- Around line 54-63: Validate that environment_variables contains the required
TILEBOX_API_KEY before provisioning, and reject the cluster configuration when
it is absent. Keep TILEBOX_CLUSTER optional and preserve the existing
secret-versus-regular environment variable handling in the loop.
- Around line 13-14: Update the Jinja Environment initialization in
auto_scaling_cluster.py to disable HTML autoescaping for the cloud-init
template, while preserving the existing FileSystemLoader and cloud-init.yaml
loading behavior.
- Around line 102-108: Update the VMSS configuration’s compute.UpgradePolicyArgs
in the cluster definition to include automatic_repairs_policy using
compute.AutomaticRepairsPolicyArgs with enabled=True, grace_period="PT10M", and
repair_action="Replace".
In `@tilebox_iac/azure/cloud-init.yaml`:
- Around line 19-31: Update get_access_token and fetch_secret in cloud-init.yaml
to fail when IMDS or Key Vault requests return errors or when the expected
token/value fields are missing or null. Use curl’s failure mode and jq
validation, then ensure invalid results are rejected before secrets are appended
to /tmp/secrets.env or the runner starts.
---
Nitpick comments:
In `@tilebox_iac/gcp/auto_scaling_cluster.py`:
- Around line 106-124: The role-building logic mutates the caller’s roles
dictionary in place, causing changes to leak across reused configurations. In
the surrounding function, create a shallow copy of the provided roles dictionary
before updating "roles" and extending "secret_roles", while preserving the
existing defaults and merged role behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 48d4fa47-fe4f-4334-b1c3-cd671a91833c
⛔ Files ignored due to path filters (1)
uv.lockis excluded by!**/*.lock
📒 Files selected for processing (19)
AGENTS.mdREADME.mdpyproject.tomltilebox_iac/__init__.pytilebox_iac/aws/__init__.pytilebox_iac/aws/auto_scaling_cluster.pytilebox_iac/aws/cloud-init.yamltilebox_iac/aws/image_builder.pytilebox_iac/azure/__init__.pytilebox_iac/azure/auto_scaling_cluster.pytilebox_iac/azure/cloud-init.yamltilebox_iac/azure/identity.pytilebox_iac/azure/network.pytilebox_iac/azure/secrets.pytilebox_iac/gcp/__init__.pytilebox_iac/gcp/auto_scaling_cluster.pytilebox_iac/gcp/cloud-init.yamltilebox_iac/gcp/image_builder.pytilebox_iac/release_runner.py
💤 Files with no reviewable changes (4)
- tilebox_iac/aws/init.py
- tilebox_iac/gcp/image_builder.py
- tilebox_iac/gcp/init.py
- tilebox_iac/aws/image_builder.py
| env = Environment(loader=FileSystemLoader(Path(__file__).parent), autoescape=True) | ||
| template = env.get_template("cloud-init.yaml") |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf 'Repository files:\n'
git ls-files 'tilebox_iac/azure/auto_scaling_cluster.py' 'tilebox_iac/azure/cloud-init.yaml' || true
printf '\nAST outline for auto_scaling_cluster.py:\n'
ast-grep outline tilebox_iac/azure/auto_scaling_cluster.py --view expanded || true
printf '\nLine count for cloud-init.yaml:\n'
wc -l tilebox_iac/azure/cloud-init.yaml || true
printf '\nRelevant snippets:\n'
sed -n '1,220p' tilebox_iac/azure/auto_scaling_cluster.py
printf '\n--- cloud-init.yaml ---\n'
sed -n '1,220p' tilebox_iac/azure/cloud-init.yamlRepository: tilebox/tilebox-iac
Length of output: 14955
Remove HTML autoescaping from the cloud-init template. autoescape=True will encode YAML and systemd values like &, <, and quotes into HTML entities, breaking the rendered cloud-init output. Use a non-HTML Jinja environment here.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tilebox_iac/azure/auto_scaling_cluster.py` around lines 13 - 14, Update the
Jinja Environment initialization in auto_scaling_cluster.py to disable HTML
autoescaping for the cloud-init template, while preserving the existing
FileSystemLoader and cloud-init.yaml loading behavior.
| used_secrets: dict[str, Secret] = {} | ||
| envs: dict[str, Input[str]] = {} | ||
| if environment_variables is not None: | ||
| for key in sorted(environment_variables): | ||
| value = environment_variables[key] | ||
| if isinstance(value, Secret): | ||
| used_secrets[key] = value | ||
| else: | ||
| envs[key] = value | ||
|
|
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Reject clusters without TILEBOX_API_KEY.
The current optional dictionary permits deployment without the required runner credential, producing instances that cannot service work. Validate key presence before provisioning; continue allowing TILEBOX_CLUSTER to be absent.
Proposed validation
used_secrets: dict[str, Secret] = {}
envs: dict[str, Input[str]] = {}
+ if environment_variables is None or "TILEBOX_API_KEY" not in environment_variables:
+ raise ValueError("environment_variables must include TILEBOX_API_KEY")
+
if environment_variables is not None:As per coding guidelines, TILEBOX_API_KEY must be required while TILEBOX_CLUSTER remains optional.
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| used_secrets: dict[str, Secret] = {} | |
| envs: dict[str, Input[str]] = {} | |
| if environment_variables is not None: | |
| for key in sorted(environment_variables): | |
| value = environment_variables[key] | |
| if isinstance(value, Secret): | |
| used_secrets[key] = value | |
| else: | |
| envs[key] = value | |
| used_secrets: dict[str, Secret] = {} | |
| envs: dict[str, Input[str]] = {} | |
| if environment_variables is None or "TILEBOX_API_KEY" not in environment_variables: | |
| raise ValueError("environment_variables must include TILEBOX_API_KEY") | |
| if environment_variables is not None: | |
| for key in sorted(environment_variables): | |
| value = environment_variables[key] | |
| if isinstance(value, Secret): | |
| used_secrets[key] = value | |
| else: | |
| envs[key] = value |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tilebox_iac/azure/auto_scaling_cluster.py` around lines 54 - 63, Validate
that environment_variables contains the required TILEBOX_API_KEY before
provisioning, and reject the cluster configuration when it is absent. Keep
TILEBOX_CLUSTER optional and preserve the existing secret-versus-regular
environment variable handling in the loop.
Source: Coding guidelines
| upgrade_policy=compute.UpgradePolicyArgs( | ||
| mode=compute.UpgradeMode.ROLLING, | ||
| rolling_upgrade_policy=compute.RollingUpgradePolicyArgs( | ||
| # New VMs rerun cloud-init; quota fallback does not. | ||
| max_surge=True, | ||
| ), | ||
| ), |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n## outline\n'
ast-grep outline tilebox_iac/azure/auto_scaling_cluster.py --view expanded || true
printf '\n## relevant lines\n'
nl -ba tilebox_iac/azure/auto_scaling_cluster.py | sed -n '1,220p'
printf '\n## search repair policy usage\n'
rg -n "automatic_repairs_policy|automatic repairs|repair.*policy|repair_policy" tilebox_iac -S || trueRepository: tilebox/tilebox-iac
Length of output: 550
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "## auto_scaling_cluster.py"
sed -n '1,240p' tilebox_iac/azure/auto_scaling_cluster.py
echo
echo "## repair-policy search"
rg -n "automatic_repairs_policy|automaticRepairsPolicy|repair.*policy|health_extension|ApplicationHealth" tilebox_iac -S || trueRepository: tilebox/tilebox-iac
Length of output: 11578
🌐 Web query:
Azure VMSS Application Health extension automatic repairs policy replacement pulumi azure native 3.x AutomaticRepairsPolicyArgs
💡 Result:
In Azure Virtual Machine Scale Sets (VMSS), the Application Health extension and the Automatic Repairs policy are not replacements for one another; rather, they are complementary components that must work together to enable automatic instance repairs [1][2]. The Application Health extension serves as the monitoring mechanism that determines whether an instance is healthy or unhealthy [1][2]. The Automatic Repairs policy, configured via automaticRepairsPolicy (or AutomaticRepairsPolicyArgs in Pulumi's azure-native 3.x provider), acts as the orchestrator that performs a specified repair action (such as replace, restart, or reimage) when an instance is flagged as unhealthy [1][3]. To implement automatic repairs in your Pulumi azure-native infrastructure code, you must configure both: 1. Application Health Extension: You must attach this extension to your VMSS so that it can monitor and report the health status of instances [1][2]. 2. Automatic Repairs Policy: You define this policy in your VirtualMachineScaleSet resource using the automaticRepairsPolicy property [4][1]. In Pulumi azure-native 3.x, the AutomaticRepairsPolicyArgs object accepts the following parameters [4][3]: - enabled (boolean): Specifies whether automatic repairs are enabled (default is false) [5][3]. - gracePeriod (string): The duration (ISO 8601 format, e.g., "PT10M") to suspend repairs after a state change, allowing the instance time to settle [5][3]. - repairAction (string): The action to take when an instance is unhealthy (e.g., "replace", "restart", "reimage") [5][1][3]. If you find that an instance is reported as unhealthy but not being repaired, ensure that the automaticRepairsPolicy is enabled and that the service state is set to Running [6]. You cannot replace the policy itself with the extension; they are distinct requirements for the automatic repair feature to function [1][7][8].
Citations:
- 1: https://github.com/MicrosoftDocs/azure-compute-docs/blob/main/articles/virtual-machine-scale-sets/virtual-machine-scale-sets-automatic-instance-repairs.md
- 2: https://learn.microsoft.com/en-us/azure/virtual-machine-scale-sets/virtual-machine-scale-sets-health-extension
- 3: https://learn.microsoft.com/en-us/javascript/api/@azure/arm-compute/automaticrepairspolicy?view=azure-node-latest
- 4: https://cdn.jsdelivr.net/npm/@pulumi/azure-native@3.8.0/compute/virtualMachineScaleSet.d.ts
- 5: https://www.pulumi.com/registry/packages/azure-native/api-docs/compute/getvirtualmachinescaleset/
- 6: https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machine-scale-sets/restart-stop/instances-not-repaired
- 7: https://www.pulumi.com/registry/packages/azure/api-docs/compute/linuxvirtualmachinescaleset/
- 8: https://www.pulumi.com/registry/packages/azure/api-docs/compute/windowsvirtualmachinescaleset/
🌐 Web query:
Pulumi azure-native Python AutomaticRepairsPolicyArgs enabled grace_period repair_action Replace
💡 Result:
In the Pulumi azure-native Python provider, AutomaticRepairsPolicyArgs is used to configure automatic instance repairs for a Virtual Machine Scale Set (VMSS) [1][2]. To use it, you pass an instance of this class to the automatic_repairs_policy argument within the VirtualMachineScaleSet resource properties [1][2]. The AutomaticRepairsPolicyArgs class typically includes the following properties: enabled (bool): Specifies whether automatic repairs should be enabled on the VMSS. The default is false [1]. grace_period (str): The duration for which automatic repairs are suspended after a VM state change to prevent premature repairs. It must be specified in ISO 8601 format (e.g., PT10M for 10 minutes) [1]. The allowed range is between 10 minutes (PT10M, which is the default) and 90 minutes (PT90M) [1]. repair_action (str): The action to perform on unhealthy VMs. Valid values are 'Replace', 'Restart', or 'Reimage'. The default value is 'Replace' [1][3][4]. Example usage pattern: import pulumi_azure_native.compute as compute vmss_repairs = compute.VirtualMachineScaleSet( "vmss", properties=compute.VirtualMachineScaleSetPropertiesArgs( automatic_repairs_policy=compute.AutomaticRepairsPolicyArgs( enabled=True, grace_period="PT15M", repair_action="Replace"), #... other required properties), #... other required parameters)
Citations:
- 1: https://learn.microsoft.com/en-us/azure/templates/microsoft.compute/2024-11-01/virtualmachinescalesets
- 2: https://cdn.jsdelivr.net/npm/@pulumi/azure-native@3.8.0/compute/virtualMachineScaleSet.d.ts
- 3: https://learn.microsoft.com/en-us/powershell/module/az.compute/update-azvmss?view=azps-16.0.0
- 4: https://learn.microsoft.com/en-us/javascript/api/@azure/arm-compute/automaticrepairspolicy?view=azure-node-latest
Enable automatic repairs on the VMSS. The Application Health extension only reports health; add automatic_repairs_policy=compute.AutomaticRepairsPolicyArgs(enabled=True, grace_period="PT10M", repair_action="Replace") so unhealthy instances are replaced automatically.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tilebox_iac/azure/auto_scaling_cluster.py` around lines 102 - 108, Update the
VMSS configuration’s compute.UpgradePolicyArgs in the cluster definition to
include automatic_repairs_policy using compute.AutomaticRepairsPolicyArgs with
enabled=True, grace_period="PT10M", and repair_action="Replace".
| get_access_token() { | ||
| curl -s -H "Metadata: true" \ | ||
| "http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fvault.azure.net{% if CLIENT_ID %}&client_id={{ CLIENT_ID }}{% endif %}" \ | ||
| | jq -r ".access_token" | ||
| } | ||
|
|
||
| fetch_secret() { | ||
| local secret_url="$1" | ||
| local access_token="$2" | ||
|
|
||
| curl -s -H "Authorization: Bearer ${access_token}" \ | ||
| "${secret_url}?api-version=7.4" \ | ||
| | jq -r ".value" |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Locate and inspect the Azure cloud-init template.
git ls-files 'tilebox_iac/azure/cloud-init.yaml' 'tilebox_iac/**/cloud-init.yaml' | sed -n '1,20p'
echo '--- outline ---'
ast-grep outline tilebox_iac/azure/cloud-init.yaml --view expanded
echo '--- relevant lines ---'
nl -ba tilebox_iac/azure/cloud-init.yaml | sed -n '1,220p'Repository: tilebox/tilebox-iac
Length of output: 383
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Search for shell safety flags and how failures are handled in related cloud-init templates.
rg -n --hidden --glob 'tilebox_iac/**/*.yaml' --glob '!**/.git/**' 'set -e|set -u|pipefail|curl -s|fail-with-body|retry|jq -r ".value"|ExecStartPre|systemd' tilebox_iacRepository: tilebox/tilebox-iac
Length of output: 2296
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the surrounding service/unit generation if present.
rg -n --hidden --glob 'tilebox_iac/**' 'ExecStartPre|ExecStart|systemd|health probe|Application Health|ManagedIdentity|vault.azure.net' tilebox_iacRepository: tilebox/tilebox-iac
Length of output: 3376
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Read the Azure cloud-init template around the secret-fetch logic and the service unit.
sed -n '1,170p' tilebox_iac/azure/cloud-init.yaml | cat -nRepository: tilebox/tilebox-iac
Length of output: 5201
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the Azure script body and how its output is consumed.
sed -n '1,150p' tilebox_iac/azure/cloud-init.yaml | python3 - <<'PY'
import sys
for i, line in enumerate(sys.stdin, 1):
if 10 <= i <= 45 or 85 <= i <= 130:
print(f"{i:4d}: {line}", end="")
PYRepository: tilebox/tilebox-iac
Length of output: 161
🌐 Web query:
Azure IMDS token endpoint error response JSON access_token missing jq null Key Vault secret endpoint error response JSON value missing
💡 Result:
When interacting with Azure REST APIs like IMDS or Key Vault, a missing JSON field (such as access_token or value) in a response is typically the result of an error state rather than a standard success response [1][2]. Azure IMDS Token Endpoint A successful request to the Azure Instance Metadata Service (IMDS) managed identity token endpoint (http://169.254.169.254/metadata/identity/oauth2/token) returns a JSON object containing the access_token [1]. If the access_token field is missing or null, it indicates the token request failed [3]. Common causes include: - Missing Metadata header: The Metadata: true header is mandatory to prevent Server-Side Request Forgery (SSRF) attacks [1]. - Missing or invalid identity: The resource has no assigned identity or the requested resource audience (via the resource parameter) is incorrect [1][2]. - Insufficient permissions: The identity lacks the required role assignments to access the target service [1]. To handle this in scripts using jq, you should verify the token exists before proceeding: TOKEN=$(curl -s -H "Metadata: true" "http://169.254.169.254/metadata/identity/oauth2/token?..." | jq -r '.access_token') if [[ -z "$TOKEN" || "$TOKEN" == "null" ]]; then echo "Error: Failed to obtain token" exit 1 fi Azure Key Vault Secret Endpoint When a request to the Azure Key Vault secret endpoint returns a missing value field, it is also typically because the request failed, often returning a 4xx error (e.g., 401 Unauthorized or 403 Forbidden) [2][4]. Azure Key Vault does not return the secret value in an error response; instead, it returns a JSON object containing an error code and message [5][4]. Example error response structure: { "error": { "code": "Forbidden", "message": "The user does not have secrets get permission..." } } If your automation (such as Ansible or a script) attempts to access secret_res.json["value"] when an error occurs, it will trigger a KeyError or null value error [2]. Always check the HTTP status code or verify the presence of the error key before attempting to access the value field [4]. If the jq utility is unavailable, consider using | python -m json.tool for debugging [6].
Citations:
- 1: https://insightcrunch.com/2023/01/23/fix-managed-identity-token-error/
- 2: KeyError: 'value' on ret.append(secret_res.json()["value"]) ansible-collections/azure#1297
- 3: https://gist.github.com/chrismeistre/df67038a7b64ea125b3641dc279dc70b
- 4: https://learn.microsoft.com/en-us/azure/key-vault/general/common-parameters-and-headers
- 5: https://learn.microsoft.com/en-us/rest/api/keyvault/secrets/get-secret/get-secret?view=rest-keyvault-secrets-7.4
- 6: https://learn.microsoft.com/en-us/azure/virtual-machines/instance-metadata-service
Fail on missing IMDS/Key Vault fields tilebox_iac/azure/cloud-init.yaml:19-31
curl -s | jq -r can turn error responses into null, which writes invalid values into /tmp/secrets.env and lets the runner start with broken credentials. Use curl --fail-with-body and/or jq -e with explicit null checks before appending.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tilebox_iac/azure/cloud-init.yaml` around lines 19 - 31, Update
get_access_token and fetch_secret in cloud-init.yaml to fail when IMDS or Key
Vault requests return errors or when the expected token/value fields are missing
or null. Use curl’s failure mode and jq validation, then ensure invalid results
are rejected before secrets are appended to /tmp/secrets.env or the runner
starts.
1 participant
Summary
ghcr.io/tilebox/runner:latestimage across every providerVerification
uv run ruff check .uv run ruff format --check .uv run pyrightuv lock --checkgit diff --checkbash -nSummary by CodeRabbit