Update docs for IaC rework and GHCR, strip notebook outputs

docs/aws-batch-inference.md

@@ -10,11 +10,12 @@ Each array child downloads the manifest and model, computes its chunk, then proc
 | File                                 | Purpose                                                            |
 | ------------------------------------ | ------------------------------------------------------------------ |
 | `infra/terraform/bootstrap/`         | S3 bucket for Terraform remote state (once per account)            |
-| `infra/terraform/foundation/`        | IAM roles + networking (VPC or reference existing)                 |
-| `infra/terraform/app/`               | Workload: ECR, compute env, queue, job definition                  |
+| `infra/terraform/foundation/`        | Networking (VPC, subnets - optional, skip if using existing VPC)   |
+| `infra/terraform/app/`               | Workload: IAM, ECR (optional), compute env, queue, job definition  |
 | `infra/terraform/app/terraform.tfvars` | All configurable values - gitignored; copy from `.tfvars.example`|
 | `infra/terraform/app/terraform.tfvars.example` | Template with placeholder values for new setups          |
-| `scripts/build_and_push.sh`          | Build Docker image and push to ECR                                 |
+| `.github/workflows/build-dev-images.yml` | Build and publish image to GHCR (automatic on push to main)    |
+| `scripts/build_and_push.sh`          | Build Docker image and push to ECR (manual, when `create_ecr = true`) |
 | `scripts/submit_batch_job.py`        | Submit single or array batch jobs                                  |
 | `scripts/batch_entrypoint.py`        | Container entrypoint - per-bridge processing loop                  |
 | `scripts/audit_outputs.py`           | Post-run verification - checks all expected outputs exist in S3    |
@@ -66,7 +67,7 @@ Each child processes bridges **one at a time** in a loop:
 
 ## Prerequisites
 
-- AWS account with IAM permissions for Batch, ECR, S3
+- AWS account with IAM permissions for Batch and S3 (ECR permissions also needed when `create_ecr = true`)
 - [Terraform](https://developer.hashicorp.com/terraform/install) installed
 - Docker installed (for building inference images)
 - Python environment for management scripts (job submission, audit, reporting):
@@ -88,15 +89,15 @@ Each child processes bridges **one at a time** in a loop:
 
 ## AWS Profile Configuration
 
-All scripts use `AWS_PROFILE` as the primary credential source. Set it to the account where infrastructure was deployed (Batch, ECR, CloudWatch):
+All scripts use `AWS_PROFILE` as the primary credential source. Set it to the account where infrastructure was deployed (Batch, CloudWatch):
 
 ```bash
 export AWS_PROFILE=my-profile
 ```
 
 **Single account** (infra and data in the same account) - this is all you need. Every script falls back to `AWS_PROFILE` for all AWS access.
 
-**Cross-account** (S3 data in a different account than Batch infra) - pass `--profile` to specify the S3 data profile. `AWS_PROFILE` still controls Batch/ECR/CloudWatch access:
+**Cross-account** (S3 data in a different account than Batch infra) - pass `--profile` to specify the S3 data profile. `AWS_PROFILE` still controls Batch/CloudWatch access:
 
 ```bash
 # Submit: Batch uses AWS_PROFILE, manifest is read via --profile
@@ -108,7 +109,7 @@ python scripts/post_run_report.py --bucket my-bucket --output-prefix my-output-p
 
 | Script | `AWS_PROFILE` | `--profile` | `--batch-profile` |
 |--------|---------------|-------------|-------------------|
-| `build_and_push.sh` | ECR login + push | - | - |
+| `build_and_push.sh` | ECR login + push (when using ECR) | - | - |
 | `submit_batch_job.py` | Batch job submission | S3 manifest access (optional) | - |
 | `audit_outputs.py` | - | S3 output checks | - |
 | `post_run_report.py` | - | S3 audit | Batch/CloudWatch queries (optional) |
@@ -119,7 +120,7 @@ python scripts/post_run_report.py --bucket my-bucket --output-prefix my-output-p
 
 ### 1. Configure & Deploy Infrastructure
 
-Follow [`infra/terraform/README.md`](../infra/terraform/README.md) to deploy all three layers (bootstrap → foundation → app). Each layer has a `backend.hcl.example` and `terraform.tfvars.example` - copy both and fill in your values. Bootstrap and foundation are applied once per account; for day-to-day config changes (S3 paths, model URI, instance types), only the app layer needs re-applying:
+Follow [`infra/terraform/README.md`](../infra/terraform/README.md) to deploy infrastructure. Only the app layer is required; bootstrap and foundation are optional (see Terraform README for when to use each). Each layer has a `backend.hcl.example` and `terraform.tfvars.example` - copy both and fill in your values. For day-to-day config changes (S3 paths, model URI, instance types), only the app layer needs re-applying:
 
 ```bash
 cd infra/terraform/app && terraform plan && terraform apply
@@ -129,6 +130,12 @@ See [Configuration Reference](#configuration-reference) for all app-layer variab
 
 ### 2. Build and Push Docker Image
 
+**GHCR (default):** images are published automatically on push to main by the `build-dev-images` GitHub Actions workflow.
+Tags: `sha-<short>` (immutable) + `dev` (floating).
+No manual steps needed.
+
+**ECR (manual, when `create_ecr = true`):**
+
 ```bash
 export AWS_PROFILE=my-profile
 export AWS_REGION=us-east-1
@@ -273,15 +280,15 @@ Use `--skip-timing` for a faster report without per-bridge p50/p95 stats.
 
 ### 7. Cleanup
 
-To tear down all Batch infrastructure, destroy layers in reverse order:
+To tear down Batch infrastructure, destroy layers in reverse order:
 
 ```bash
-cd infra/terraform/app && terraform destroy        # workload (ECR, Batch)
-cd ../foundation && terraform destroy              # IAM roles + networking
+cd infra/terraform/app && terraform destroy        # workload (IAM, ECR if created, Batch)
+cd ../foundation && terraform destroy              # networking (if deployed)
 cd ../bootstrap && terraform destroy               # state bucket (optional - safe to keep)
 ```
 
-Destroying `app` alone is usually sufficient (removes ECR, compute env, queue, job def). Foundation and bootstrap are shared infrastructure rarely torn down. S3 data is not affected.
+Destroying `app` alone is usually sufficient (removes IAM, compute env, queue, job def, and ECR if `create_ecr = true`). Foundation and bootstrap are shared infrastructure rarely torn down. S3 data is not affected.
 
 ---
 
@@ -368,7 +375,7 @@ One bridge per line. Extensionless or with extension:
 11010009/bridge_1234567_USGS_Some_Dataset.las
 ```
 
-If a line has no extension, the entrypoint probes S3 for `.laz` first, then `.las` via `head_object`. .
+If a line has no extension, the entrypoint probes S3 for `.laz` first, then `.las` via `head_object`.
 
 The split manifest produced by `utils/split_data.py` (`split_test_ids.txt`) is directly usable.
 
@@ -444,7 +451,7 @@ All Batch resources are tagged with `Project = bridge-classifier`. Tags propagat
 
 **GPU out of memory**: Large bridges with dense point clouds can exceed GPU memory. Use a larger instance or increase `--voxel-size` (coarser voxels = fewer voxels = less memory).
 
-**S3 permission denied**: The Batch job IAM role is managed by the foundation layer and scoped to the `data_bucket`. Verify that `s3_bucket` in the app tfvars matches `data_bucket` in the foundation tfvars.
+**S3 permission denied**: The Batch job IAM role is managed by the app layer and scoped to the `data_bucket`. Verify that `s3_bucket` matches `data_bucket` in the app tfvars.
 
 **SPOT instance interruptions**: The job definition auto-retries up to `retry_attempts` times on SPOT interruption. Combined with skip-if-exists, retries are cheap. For critical runs with no tolerance for delay, set `use_spot = false`.
 

docs/deployment-runbook.md

@@ -36,10 +36,10 @@ flowchart LR
 
 ## 2. Deploy Infrastructure
 
-Follow the [Terraform README](https://github.com/NGWPC/bridge-classification/blob/main/infra/terraform/README.md) to deploy the three layers:
+Follow the [Terraform README](https://github.com/NGWPC/bridge-classification/blob/main/infra/terraform/README.md) to deploy:
 
 ```
-bootstrap (optional) -> foundation -> app
+bootstrap (optional) -> foundation (optional) -> app
 ```
 
 Each layer has `backend.hcl.example` and `terraform.tfvars.example` - copy both and fill in your values.
@@ -48,10 +48,11 @@ Each layer has `backend.hcl.example` and `terraform.tfvars.example` - copy both
 
 | Scenario | What to do |
 |----------|-----------|
-| Fresh account, no existing resources | Run all three layers with `create_iam = true`, `create_networking = true` |
+| Fresh account, no existing resources | Run all layers: `create_networking = true` in foundation, `create_iam = true` in app |
 | Existing S3 bucket for state | Skip bootstrap, point `backend.hcl` at your bucket with a key prefix |
-| Existing IAM roles and VPC | Set `create_iam = false`, `create_networking = false` in foundation, provide existing ARNs/IDs |
-| Account already has ECR repo | `terraform import aws_ecr_repository.inference <repo-name>` |
+| Existing VPC and IAM roles | Skip foundation, set `create_iam = false` in app, provide existing ARNs/IDs and VPC/subnet IDs |
+| Using GHCR (default) | Set `create_ecr = false` and `inference_image_repo` in app |
+| Using ECR | Set `create_ecr = true` in app (default) |
 
 Set your AWS profile before running any commands:
 
@@ -136,19 +137,24 @@ See [API Reference: Inference & Cloud](api/inference-cloud.md) for registry func
 
 ## 5. Build & Push Docker Image
 
+**GHCR (default):** images are published automatically on push to main by the `build-dev-images` GitHub Actions workflow.
+Tags: `sha-<short>` (immutable) + `dev` (floating).
+No manual steps needed.
+
+**ECR (manual, when `create_ecr = true`):**
+
 ```bash
 export AWS_PROFILE=<infra-profile>
 chmod +x ./scripts/build_and_push.sh
 ./scripts/build_and_push.sh
 ```
 
-Reads ECR URL and region from terraform outputs.
-Tags with `:latest` and `:git-<sha>` for rollback.
+Reads image repo and region from terraform outputs.
 
 **When to rebuild:** only when you change code (`src/`, `scripts/`, or `Dockerfile`).
 Changing S3 paths, model URI, or inference config in `terraform.tfvars` does NOT require a rebuild - those are environment variables in the job definition.
 
-See [AWS Batch Inference: Build and Push](aws-batch-inference.md#3-build-and-push-docker-image) for details.
+See [AWS Batch Inference: Build and Push](aws-batch-inference.md#2-build-and-push-docker-image) for details.
 
 ---
 
@@ -171,7 +177,7 @@ python scripts/submit_batch_job.py \
     --profile <data-profile>
 ```
 
-**Cross-account setup:** use `--profile` for S3 data access and `AWS_PROFILE` for Batch/ECR operations.
+**Cross-account setup:** use `--profile` for S3 data access and `AWS_PROFILE` for Batch operations.
 Override output location with `--env S3_OUTPUT_PREFIX=<path>`.
 
 ### Monitor
@@ -180,7 +186,7 @@ Override output location with `--env S3_OUTPUT_PREFIX=<path>`.
 aws logs tail /aws/batch/bridge-classifier --follow --profile <infra-profile>
 ```
 
-See [AWS Batch Inference: Submit a Job](aws-batch-inference.md#4-submit-a-job) and [Monitor](aws-batch-inference.md#5-monitor) for full options.
+See [AWS Batch Inference: Submit a Job](aws-batch-inference.md#3-submit-a-job) and [Monitor](aws-batch-inference.md#4-monitor) for full options.
 
 ---
 
@@ -199,7 +205,7 @@ python scripts/post_run_report.py \
 Do not include a trailing slash on `--output-prefix`.
 The report auto-discovers `--bucket` from terraform outputs.
 
-See [AWS Batch Inference: Post-Run Report](aws-batch-inference.md#7-post-run-report) for details.
+See [AWS Batch Inference: Post-Run Report](aws-batch-inference.md#6-post-run-report) for details.
 
 ### Re-run failed or missing bridges
 
@@ -352,10 +358,10 @@ No manual intervention needed unless all retries are exhausted.
 |-------|-------|-----|
 | `NoSuchBucket` | `s3_bucket` in terraform.tfvars is a prefix, not a bucket name | Set to the actual S3 bucket name (e.g. `fimc-data`, not `bridge-classification`) |
 | `AccessDenied` on S3 | Cross-account access not configured | The data bucket policy must grant access to the Batch job role from the infra account |
-| `RepositoryAlreadyExistsException` | ECR repo exists in this account | `terraform import aws_ecr_repository.inference <repo-name>` |
+| `RepositoryAlreadyExistsException` | ECR repo exists (when `create_ecr = true`) | `terraform import aws_ecr_repository.inference <repo-name>` |
 | `Backend configuration changed` | Switching between accounts | `terraform init -backend-config=backend.hcl -reconfigure` |
 | `JOB_DEF_NAME` missing | App layer not fully applied | Run `terraform apply` in `infra/terraform/app` |
-| `timeout` not found (macOS) | `build_and_push.sh` uses GNU timeout | Use the version on `feature/iac-rework` branch which removed the timeout dependency |
+| `timeout` not found (macOS) | Older `build_and_push.sh` used GNU timeout | Fixed on main - current version does not use timeout |
 
 ### Reading CloudWatch logs
 
@@ -378,18 +384,18 @@ Key events to search for:
 
 ### Rolling back a bad image
 
-Every image push creates a `:git-<sha>` tag alongside `:latest`:
+Every build creates an immutable `sha-<short>` tag.
+Pin a known-good tag by updating `image_tag` in `terraform.tfvars` and re-applying:
 
 ```bash
-# Check available tags
-aws ecr list-images --repository-name bridge-classifier --profile <profile>
-
-# Roll back to a known-good image
-docker pull <ecr-url>:git-abc1234
-docker tag <ecr-url>:git-abc1234 <ecr-url>:latest
-docker push <ecr-url>:latest
+# Set image_tag = "sha-abc1234" in terraform.tfvars
+cd infra/terraform/app
+terraform plan && terraform apply
 ```
 
+To list available GHCR tags, check the repository's Packages page on GitHub.
+For ECR, use `aws ecr list-images --repository-name bridge-classifier --profile <profile>`.
+
 ---
 
 ## 10. S3 Data Layout

infra/terraform/app/terraform.tfvars.example

@@ -14,7 +14,8 @@ private_subnet_ids = ["subnet-...", "subnet-..."]
 # --- Container registry (ECR by default, GHCR alternative) ---
 # create_ecr           = true   # false to skip ECR repo (use GHCR or other registry)
 # Required when create_ecr = false:
-# inference_image_repo = "ghcr.io/noaa-owp/bridge-classification"
+# inference_image_repo = "ghcr.io/ngwpc/bridge-classification/inference"
+# image_tag           = "dev"
 
 # --- IAM toggle (optional, defaults shown) ---
 # create_iam                       = true
@@ -44,7 +45,6 @@ s3_output_prefix = "bridge-classification/runs/<run>/predictions"
 # use_spot            = true
 
 # --- Job definition (optional, defaults shown) ---
-# image_tag           = "dev"
 # inference_mode      = "masked"
 # bridge_timeout      = 150
 # job_vcpus           = 3

infra/terraform/app/variables.tf

@@ -154,7 +154,7 @@ variable "create_ecr" {
 }
 
 variable "inference_image_repo" {
-  description = "Image repository for inference (required when create_ecr = false, e.g. ghcr.io/noaa-owp/bridge-classification)"
+  description = "Image repository for inference (required when create_ecr = false, e.g. ghcr.io/ngwpc/bridge-classification/inference)"
   type        = string
   default     = ""