Enable automated backups and take a baseline snapshot on the shared RDS instance

[ale210]

Overview

We need to enable automated backups and capture a baseline manual snapshot on the shared incubator-prod-database RDS instance, because it currently runs with backup_retention_period = 0 and has zero snapshots — there is no recovery point for any project's data. It also has to land before the PostgreSQL 13 → 15 upgrade, which has no safe rollback without one.

Action Items

• Confirm the current state before changing anything, so the work is done against reality rather than this ticket's description: aws rds describe-db-instances --db-instance-identifier incubator-prod-database --region us-west-2 --query 'DBInstances[0].BackupRetentionPeriod' (expect 0) and aws rds describe-db-snapshots --db-instance-identifier incubator-prod-database --region us-west-2 --query 'DBSnapshots[]' (expect []).
• In terraform/database.tf, change backup_retention_period from 0 to 7. It is on line 12 as of writing — line numbers drift, so find it by the attribute name if it has moved.
• Schedule and announce a short outage window with the project teams before merging. Changing backup retention from 0 to a non-zero value restarts the RDS instance — this is not a zero-downtime change, and every project shares this one instance (vrms, civic-tech-index, civic-tech-jobs, home-unite-us, people-depot, access-the-data).
• Time the merge to that window. Merging the PR is the deploy: .github/workflows/terraform-apply.yaml runs terraform apply with auto_approve: true on every push to main touching a .tf file, and the instance has apply_immediately = true. There is no separate apply step to schedule afterwards.
• Open the PR and check the terraform-plan output shows an in-place update to aws_db_instance.default and nothing else. A plan proposing to replace this resource must not be merged — skip_final_snapshot = true and deletion_protection = false mean nothing would stop a destroy.
• After the PR merges, confirm the apply succeeded and retention is live by re-running the describe-db-instances query above; expect 7.
• After the PR merges, take the baseline manual snapshot: aws rds create-db-snapshot --db-instance-identifier incubator-prod-database --db-snapshot-identifier incubator-prod-database-pre-pg15 --region us-west-2. A manual snapshot is needed in addition to automated backups, because automated snapshots are deleted along with the instance — the manual one is the rollback artifact for the upgrade.
• After the snapshot finishes, verify it reports Status: available: aws rds describe-db-snapshots --db-snapshot-identifier incubator-prod-database-pre-pg15 --region us-west-2.
• After the PR merges, confirm the ECS services in the incubator-prod cluster reconnected following the restart, checking the two production ones first: vrms-backend-prod and cti-backend-prod.

Resources/Instructions

• terraform/database.tf — the RDS instance declaration.
• .github/workflows/terraform-apply.yaml — the apply-on-merge workflow.
• AWS: account 035866691871 (incubator), region us-west-2, instance incubator-prod-database. Read-only checks can be run with any incubator credentials; the snapshot creation needs write access.
• 7 days retention is a starting proposal, not a researched requirement. If HfLA has a data retention standard, use that instead and say so on this issue.
• Working with backups in Amazon RDS