Layered Terraform IaC and batch scripts cleanup

.gitignore

@@ -9,12 +9,14 @@ checkpoints/
 experiments/
 plan/
 
-# Terraform
-terraform/.terraform/
-terraform/*.tfstate
-terraform/*.tfstate.*
-terraform/.terraform.lock.hcl
-terraform/terraform.tfvars
+# Terraform — covers terraform/ (legacy) and infra/terraform/ (layered)
+# State + provider cache: never commit (state can contain sensitive values)
+*.tfstate
+*.tfstate.*
+.terraform/
+# Per-account config: copy from the *.example files
+terraform.tfvars
+backend.hcl
 
 # Byte-compiled / optimized / DLL files
 __pycache__/

infra/terraform/README.md

@@ -0,0 +1,203 @@
+# Terraform Infrastructure
+
+Infrastructure as code for the bridge classification batch inference pipeline.
+Three independent stacks, applied in order: bootstrap, then foundation, then app.
+Bootstrap and foundation are **optional** when using existing infrastructure - only the app stack is required.
+When using an existing VPC and IAM roles, skip bootstrap and foundation entirely.
+
+Each stack reads two local config files that are gitignored and created from committed `.example` templates.
+
+- `terraform.tfvars` - input variables, copied from `terraform.tfvars.example`
+- `backend.hcl` - remote state config, copied from `backend.hcl.example`
+
+## Prerequisites
+
+- Terraform >= 1.14, AWS provider ~> 6.0 (pinned per stack).
+- AWS CLI with a profile for the target account.
+- Permissions to create S3, IAM, VPC, Batch, ECR, and CloudWatch resources.
+
+## Layout
+
+```
+infra/terraform/
+├── README.md
+├── bootstrap/                     state bucket stack, apply once
+│   ├── main.tf                    S3 state bucket - versioning, AES256 encryption, public access block, account-restricted TLS-only policy
+│   ├── outputs.tf                 bucket_name, bucket_arn, region
+│   ├── providers.tf               AWS provider config, default tags (incl. optional team/poc)
+│   ├── terraform.tf               version constraints, S3 backend block, first-time setup notes
+│   ├── variables.tf               allowed_account_id, project_name, region, team, poc
+│   ├── backend.hcl.example        remote state backend template
+│   ├── terraform.tfvars.example   input variable template
+│   └── .terraform.lock.hcl        provider version lock (generated, committed)
+├── foundation/                    persistent infra stack - networking only
+│   ├── networking.tf              VPC, public/private subnets, IGW, NAT gateway, S3 endpoint, optional ECR/CloudWatch Logs interface endpoints, VPCE SG
+│   ├── outputs.tf                 vpc_id, private_subnet_ids, vpce_security_group_id
+│   ├── providers.tf               AWS provider config, default tags (incl. optional team/poc)
+│   ├── terraform.tf               version constraints, S3 backend block
+│   ├── variables.tf               create_networking, CIDRs, enable_nat_gateway, create_vpc_endpoints, existing_* fallbacks
+│   ├── backend.hcl.example        remote state backend template
+│   ├── terraform.tfvars.example   input variable template
+│   └── .terraform.lock.hcl        provider version lock (generated, committed)
+└── app/                           application stack, ok to destroy and recreate
+    ├── batch.tf                   launch template (IMDSv2, encrypted EBS), Batch compute env (GPU SPOT), job queue, job definition
+    ├── cloudwatch.tf              log group (configurable retention + optional KMS encryption)
+    ├── data.tf                    aws_partition, aws_caller_identity data sources
+    ├── ecr.tf                     ECR repo + lifecycle policy (optional, gated by create_ecr)
+    ├── iam.tf                     create_iam toggle, Batch IAM roles + instance profile, existing_* fallbacks
+    ├── outputs.tf                 ECR/image repo, Batch, CloudWatch, S3 outputs
+    ├── providers.tf               AWS provider config, default tags (incl. optional team/poc)
+    ├── security_groups.tf         Batch SG + optional VPC endpoint ingress rule
+    ├── terraform.tf               version constraints, S3 backend block
+    ├── variables.tf               shared + IAM + container registry + compute + inference variables
+    ├── backend.hcl.example        remote state backend template
+    ├── terraform.tfvars.example   input variable template
+    └── .terraform.lock.hcl        provider version lock (generated, committed)
+```
+
+## Stacks
+
+### Bootstrap
+
+Creates the S3 bucket that holds Terraform remote state for the other two stacks.
+Apply once per AWS account.
+
+Bootstrap is **optional**.
+If you have an existing S3 bucket for state storage, skip this stack and set `bucket` in foundation and app `backend.hcl` to that bucket name.
+
+### Foundation
+
+Persistent networking infrastructure that survives an app stack destroy/recreate.
+**Optional** when using an existing VPC - skip this stack entirely and pass values directly to the app stack.
+
+Creates:
+- VPC with public subnets (NAT gateway placement only, no workloads) and private subnets (all workloads)
+- NAT gateway for private subnet internet access
+- S3 gateway endpoint (free, attached to both route tables)
+- Optional ECR API, ECR DKR, and CloudWatch Logs interface endpoints (for no-NAT deployments)
+- VPC endpoint security group (app stack adds ingress rules when `vpce_security_group_id` is provided)
+
+### App
+
+Application infrastructure, safe to destroy and recreate.
+
+Creates:
+- AWS Batch GPU SPOT compute environment, job queue, job definition (with launch template for IMDSv2 + encrypted EBS)
+- ECR repo with scan-on-push and lifecycle policy (optional, gated by `create_ecr`)
+- CloudWatch log group (configurable retention, optional KMS encryption)
+- Batch IAM roles + instance profile (optional, gated by `create_iam`)
+- Batch security group + optional VPC endpoint ingress rule
+
+IAM is toggleable via `create_iam`.
+ECR is toggleable via `create_ecr` - set to `false` when using an external registry like GHCR.
+When `create_ecr = false`, the image repository is provided via `inference_image_repo`.
+Security groups are always created by this stack.
+
+## Tags
+
+All three stacks apply default tags to every resource:
+
+| Tag | Source | Required |
+|---|---|---|
+| `ManagedBy` | hardcoded `"Terraform"` | Always |
+| `Project` | `var.project_name` | Always |
+| `Stack` | hardcoded per stack | Always |
+| `Team` | `var.team` | Optional (omitted if empty) |
+| `POC` | `var.poc` | Optional (omitted if empty) |
+
+## Toggles
+
+| Toggle | Stack | Default | Controls |
+|---|---|---|---|
+| `create_networking` | foundation | `true` | VPC, subnets, IGW, NAT gateway, VPC endpoints, VPCE SG |
+| `enable_nat_gateway` | foundation | `true` | NAT gateway + private subnet default route |
+| `create_vpc_endpoints` | foundation | `false` | ECR + CloudWatch Logs interface endpoints + VPCE SG |
+| `create_iam` | app | `true` | Batch IAM roles + instance profile |
+| `create_ecr` | app | `true` | ECR repo + lifecycle policy |
+| `create_batch_service_linked_role` | app | `true` | Account-global AWSServiceRoleForBatch |
+
+**NAT-off warning.**
+Disabling `enable_nat_gateway` without enabling `create_vpc_endpoints` leaves private subnets with no route to ECR or CloudWatch Logs.
+Batch jobs will fail at image pull.
+
+## Fresh deployment
+
+Set the AWS profile and confirm the account ID before touching any stack.
+Bootstrap and foundation are optional. If using existing networking, skip to step 3 (App).
+
+```bash
+export AWS_PROFILE=<your-profile>
+aws sts get-caller-identity --query Account --output text
+```
+
+### 1. Bootstrap
+
+```bash
+cd infra/terraform/bootstrap
+cp terraform.tfvars.example terraform.tfvars
+cp backend.hcl.example backend.hcl
+# edit both: account ID, state bucket name
+
+# Step 1: comment out `backend "s3" {}` in terraform.tf
+terraform init
+terraform apply
+
+# Step 2: uncomment `backend "s3" {}`
+terraform init -backend-config=backend.hcl -migrate-state
+
+# Step 3: delete local state (now in S3)
+rm terraform.tfstate terraform.tfstate.backup
+```
+
+### 2. Foundation (optional - skip if you have an existing VPC)
+
+```bash
+cd infra/terraform/foundation
+cp terraform.tfvars.example terraform.tfvars
+cp backend.hcl.example backend.hcl
+# edit both: account ID, state bucket name
+
+terraform init -backend-config=backend.hcl
+terraform plan
+terraform apply
+```
+
+### 3. App
+
+```bash
+cd infra/terraform/app
+cp terraform.tfvars.example terraform.tfvars
+cp backend.hcl.example backend.hcl
+
+# If foundation was deployed, pull its outputs into terraform.tfvars:
+#   terraform -chdir=../foundation output
+# If using existing infra, fill in vpc_id, private_subnet_ids from your environment
+
+terraform init -backend-config=backend.hcl
+terraform plan
+terraform apply
+```
+
+If the account already has the Batch service-linked role, set `create_batch_service_linked_role = false`.
+
+## Foundation outputs
+
+| Output | Provides |
+|---|---|
+| `vpc_id` | VPC ID (created, or existing VPC ID passed through) |
+| `private_subnet_ids` | Private subnet IDs for all workloads |
+| `vpce_security_group_id` | VPC endpoint SG ID (empty if not created) |
+
+## App outputs
+
+| Output | Provides |
+|---|---|
+| `inference_image_repo` | Image repository (ECR URL or external registry) |
+| `job_queue_name` | Batch job queue name |
+| `job_definition_name` | Batch job definition name |
+| `compute_environment_name` | Batch compute environment name |
+| `log_group_name` | CloudWatch log group name |
+| `s3_manifest_uri` | S3 manifest URI (passthrough) |
+| `aws_region` | AWS region (passthrough) |
+| `s3_bucket` | S3 data bucket (passthrough) |
+| `s3_output_prefix` | S3 output prefix (passthrough) |

infra/terraform/app/backend.hcl.example

@@ -0,0 +1,16 @@
+# Copy to backend.hcl and fill in. backend.hcl is git-ignored.
+#
+# Option A: dedicated state bucket (created by bootstrap layer)
+#   bucket = "bridge-classifier-terraform-state-<ACCOUNT_ID>"
+#   key    = "app/terraform.tfstate"
+#
+# Option B: existing bucket (skip bootstrap, use a key prefix to isolate state)
+#   bucket = "my-existing-bucket"
+#   key    = "some-prefix/terraform-state/app/terraform.tfstate"
+#
+bucket              = "<BUCKET_NAME>"
+key                 = "<KEY_PATH>/app/terraform.tfstate"
+region              = "us-east-1"
+use_lockfile        = true
+encrypt             = true
+allowed_account_ids = ["<ACCOUNT_ID>"]

infra/terraform/app/batch.tf

@@ -0,0 +1,130 @@
+# ----- Launch template (IMDSv2, encrypted EBS) -----
+resource "aws_launch_template" "batch" {
+  name_prefix = "${var.project_name}-batch-"
+
+  metadata_options {
+    http_tokens = "required"
+  }
+
+  block_device_mappings {
+    device_name = "/dev/xvda"
+    ebs {
+      encrypted = true
+    }
+  }
+}
+
+# ----- Compute environment (SPOT or on-demand) -----
+resource "aws_batch_compute_environment" "gpu" {
+  name         = "${var.project_name}-gpu-${var.use_spot ? "spot" : "ec2"}"
+  type         = "MANAGED"
+  state        = "ENABLED"
+  service_role = local.batch_service_role_arn
+
+  compute_resources {
+    type                = var.use_spot ? "SPOT" : "EC2"
+    allocation_strategy = var.use_spot ? "SPOT_CAPACITY_OPTIMIZED" : "BEST_FIT_PROGRESSIVE"
+    min_vcpus           = 0
+    max_vcpus           = var.max_vcpus
+    desired_vcpus       = 0
+    instance_type       = var.instance_types
+
+    subnets             = var.private_subnet_ids
+    security_group_ids  = [aws_security_group.batch.id]
+    instance_role       = local.batch_instance_profile_arn
+    spot_iam_fleet_role = var.use_spot ? local.spot_fleet_role_arn : null
+
+    launch_template {
+      launch_template_id = aws_launch_template.batch.id
+      version            = "$Latest"
+    }
+
+    # Batch launches these instances at runtime, outside Terraform, so provider
+    # default_tags don't reach them -replicate them here for cost/ownership tagging.
+    tags = merge({
+      ManagedBy = "Terraform"
+      Project   = var.project_name
+      Stack     = "app"
+    }, local.optional_tags)
+  }
+
+  lifecycle {
+    create_before_destroy = true
+    ignore_changes        = [compute_resources[0].desired_vcpus]
+  }
+}
+
+# ----- Job queue -----
+resource "aws_batch_job_queue" "inference" {
+  name     = "${var.project_name}-inference-queue"
+  state    = "ENABLED"
+  priority = 1
+
+  compute_environment_order {
+    order               = 1
+    compute_environment = aws_batch_compute_environment.gpu.arn
+  }
+}
+
+# ----- Job definition -----
+resource "aws_batch_job_definition" "inference" {
+  name                  = "${var.project_name}-inference"
+  type                  = "container"
+  propagate_tags        = true
+  platform_capabilities = ["EC2"]
+
+  timeout {
+    attempt_duration_seconds = var.job_timeout_seconds
+  }
+
+  retry_strategy {
+    attempts = var.retry_attempts
+
+    evaluate_on_exit {
+      action           = "RETRY"
+      on_status_reason = "Host EC2*"
+    }
+    evaluate_on_exit {
+      action    = "EXIT"
+      on_reason = "*"
+    }
+  }
+
+  container_properties = jsonencode({
+    image      = "${local.inference_image_repo}:${var.image_tag}"
+    vcpus      = var.job_vcpus
+    memory     = var.job_memory
+    jobRoleArn = local.batch_job_role_arn
+    command    = ["python", "/app/scripts/batch_entrypoint.py"]
+
+    resourceRequirements = [
+      {
+        type  = "GPU"
+        value = "1"
+      }
+    ]
+
+    linuxParameters = {
+      sharedMemorySize = var.shared_memory_size
+    }
+
+    logConfiguration = {
+      logDriver = "awslogs"
+      options = {
+        "awslogs-group"         = aws_cloudwatch_log_group.batch.name
+        "awslogs-region"        = var.region
+        "awslogs-stream-prefix" = "inference"
+      }
+    }
+
+    environment = [
+      { name = "S3_BUCKET", value = var.s3_bucket },
+      { name = "S3_INPUT_PREFIX", value = var.s3_input_prefix },
+      { name = "S3_MANIFEST_URI", value = var.s3_manifest_uri },
+      { name = "S3_MODEL_URI", value = var.s3_model_uri },
+      { name = "S3_OUTPUT_PREFIX", value = var.s3_output_prefix },
+      { name = "INFERENCE_MODE", value = var.inference_mode },
+      { name = "BRIDGE_TIMEOUT", value = tostring(var.bridge_timeout) },
+    ]
+  })
+}