diff --git a/.azure/plan.md b/.azure/plan.md new file mode 100644 index 0000000..3f6f3c0 --- /dev/null +++ b/.azure/plan.md @@ -0,0 +1,744 @@ +# Convolens Mystira Azure Go-Live Plan + +Status: Draft - pending approval +Date: 2026-07-16 +Target subscription: bb4e3882-2079-4bab-8974-611bc0b8bb58 +Target tenant: 9530cd32-9e33-47f0-9247-ed964730b580 +Target region: southafricanorth wherever the selected Azure service supports it; use the nearest approved fallback only when a service is unavailable there +Target hostname: convolens.neuralliquid.ai + +## 1. Objective + +Put Convolens live on the Mystira Azure subscription as a production service that can ingest WhatsApp conversation data and create actionable Baton intake items from that data. + +The go-live is not just hosting the current app. The production target must include: + +- a public web experience; +- a durable API runtime; +- production storage and database behavior; +- secure secret and identity management; +- observable health, logs, and alerts; +- a verified WhatsApp ingestion path; +- a verified Baton task/intake path; +- CI/CD with preview, validation, deployment, and rollback steps. + +## 2. Current Evidence + +Repository shape: + +- Monorepo with `apps/web`, `apps/api`, `apps/chrome-extension`, shared packages, tests, and Terraform. +- Existing Terraform is under `infra/terraform/env/dev`. +- Existing dev environment provisions resource group, Log Analytics, Application Insights, Storage, Cosmos DB, Key Vault, Container Apps, Static Web App, optional Redis, optional Azure OpenAI, and optional budget alerts. +- Existing dev Azure resources include `nl-dev-convolens-rg` in `eastus2`. +- Convolens should stay under the NeuralLiquid `nl-*` resource naming family even though it is deployed in the Mystira Azure subscription. +- The go-live target is straight to `prod`; there is no separate dev environment for this deployment cycle. For now, `prod` is the effective development and production environment. +- Mystira production resources in the active subscription are primarily in `southafricanorth`; Convolens should also use `southafricanorth` wherever possible. + +Important code findings: + +- API database config is currently SQLite (`database.sqlite`) with production migrations enabled, which is not a production Azure data plan. +- Terraform provisions Cosmos DB, but the API does not currently use Cosmos through TypeORM or a repository layer. +- Chat export upload and Chrome extension ingestion parse/validate WhatsApp data, but both still have TODOs for durable persistence and summarization queueing. +- Current Puppeteer-based WhatsApp Web client is not production-ready as-is: it launches non-headless and depends on interactive QR/session state. +- Storage service can use Azure Blob Storage, but it uses direct REST/shared key style paths that should be replaced or hardened with Azure SDK and managed identity for production. +- Extension selector reports are held in memory; production needs persistent storage and admin access control. + +Baton consultation: + +- Baton org-wide health/search endpoints returned 502 during planning. +- Narrow task search worked. +- Existing WhatsApp-related Mystira tasks found were completed: + - `a86ec55e-834d-4c6a-810e-0052728f4cfc` - Analyze Mystira decision-maker WhatsApp export. + - `ec2b4119-7464-406d-bd77-f196502fa075` - Triage WhatsApp support business group behavior. + - `7ca8db3c-135c-4692-8fdf-2bfe14beee5c` - Ship WhatsApp support link in PWA. +- No open Convolens tracker was found by title search. +- No existing Baton intake tracker was found by `intake` or `Baton intake` title search. + +## 3. Target Architecture + +Recommended production shape: + +- Frontend: Azure App Service is the recommended target because the app is a Next.js application with auth/runtime needs; Static Web Apps remains acceptable only if the production build is confirmed to be fully static or SWA-compatible. +- API: Azure Container Apps, with min replicas greater than zero for production. +- Database: PostgreSQL is the recommended production source of truth for the first go-live because the API already uses TypeORM entities/migrations and needs durable relational workflow state for ingestion, review, Baton publishing, and audit. Cosmos DB remains useful later for high-volume message/document storage if needed, but using it first would require a bigger repository-layer rewrite. +- Storage: Azure Blob Storage for uploaded WhatsApp exports, normalized artifacts, and processing outputs. +- Queue: Azure Service Bus or Storage Queue for ingestion and Baton task creation work. +- Secrets: Key Vault with managed identity access from API/worker and GitHub OIDC for deployments. +- Telemetry: Application Insights and Log Analytics with app health, ingestion failures, Baton API failures, queue depth, and auth failures. +- Auth: Mystira Identity / Entra OIDC for web/operator login, following the already-used sibling-service pattern. +- Budget and cost tags: production budget enabled, cost tags aligned with NeuralLiquid/Mystira cost tracking. +- DNS: `convolens.neuralliquid.ai`, with API CORS locked to that origin and any required callback origin. + +## 4. Production Environment Work + +Create a `prod` Terraform environment and deploy directly to it: + +- Add `infra/terraform/env/prod`. +- Use `env = "prod"` and `org = "nl"`. +- Use `southafricanorth` wherever possible. +- Use a separate Terraform state key, for example `convolens-prod.tfstate`, under the existing remote-state pattern or the Mystira shared tfstate standard. +- Ensure production Key Vault has purge protection enabled. +- Enable budget alerts with a real admin email. +- Create a dedicated resource group `nl-prod-convolens-rg` for blast-radius and cost clarity. +- Add deployment outputs for web URL, API URL, Key Vault, App Insights, and resource group. + +Production Terraform gaps to close: + +- Replace placeholder API image default with a required input or CI-produced image. +- Add ACR or reuse the existing shared ACR if policy allows; keep resource names and tags `nl-prod-convolens-*` where Convolens owns the resource. +- Add managed identity role assignments for storage, Key Vault, queue, and database. +- Disable or reduce shared-key dependency where possible. +- Tighten public network defaults where supported. +- Add Container Apps secret references from Key Vault or secure app secrets. +- Add health probes and startup/liveness behavior for Container Apps. +- Add autoscale rules that include HTTP concurrency and queue depth once queue processing exists. + +## 5. Code Readiness Work + +Blockers before production deployment: + +- Replace SQLite production data path with Cosmos or PostgreSQL. +- Add Mystira Identity OIDC integration: + - register Convolens as an OIDC relying party/client; + - configure callback/logout redirect URIs for `https://convolens.neuralliquid.ai`; + - store client secret in Key Vault or GitHub environment secret as appropriate; + - require authenticated sessions for import/review/publish flows. +- Implement durable save for `/api/chat-export/upload`. +- Implement durable save for `/api/chat-export/extension`. +- Add idempotent ingestion records keyed by user, chat id/export hash, source, and time window. +- Add persistence for extension selector reports. +- Harden auth around admin-only endpoints (`/api/extension/selectors/reports`, selector update). +- Add production CORS configuration for final web/API hosts. +- Add structured health endpoints: + - `/health/live` for process health. + - `/health/ready` for database/storage/queue readiness. + - `/health/dependencies` protected or internal for detailed dependency status. +- Add migration/seed strategy for the selected database. +- Add a production build smoke test for API Docker image and frontend build. + +## 6. WhatsApp Intake Into Baton + +This is a first-class go-live feature, not a post-launch nicety. + +Target workflow: + +1. Chrome extension sends structured chat data, with manual export upload retained as a fallback/admin import path. +2. API validates, sanitizes, hashes, and persists the raw artifact and normalized messages. +3. API creates an ingestion job. +4. Worker summarizes/extracts actionable items from the conversation. +5. Worker maps extracted items into Baton-ready task candidates and recommends the target Baton project. +6. Human review gate approves, edits, rejects, or bulk-publishes candidates. +7. Approved candidates are created in Baton with source metadata, trace id, chat/date range, confidence, and audit note. +8. Baton task ids are written back to the Convolens ingestion record. + +Required Baton integration behavior: + +- Route Baton intakes into the relevant Baton project rather than one global inbox. +- Use the active Mystira project (`7ebf10db-5261-408a-9db2-8ba14d4110b7`) as the default when a candidate is product/Mystira-specific or the project is ambiguous. +- Route clear platform/service items to their project when identifiable, for example Sluice, Docket, Baton, Retort, HOV, or Cognitive Mesh. +- Stage candidates for review before creating Baton tasks. Direct-create should be opt-in later for trusted extraction classes only. +- Define task fields: + - title + - description + - source chat/export id + - source timestamp range + - priority + - owner type + - labels + - parent task or epic relation when known + - trace id +- Add duplicate prevention across Baton by checking for existing tasks with the same source hash or normalized title before creation. +- Add failure handling for Baton 502/timeout responses: + - retry with backoff; + - leave job in `pending_baton`; + - expose a retry UI/admin action; + - never lose the extracted candidate. + +Minimal go-live slice for WhatsApp-to-Baton: + +- Chrome extension extraction works against the selected WhatsApp source and submits to production API. +- Manual export upload works as fallback. +- Messages are persisted. +- Candidate extraction produces reviewable items. +- Reviewer can publish selected candidates to Baton. +- Baton write result is persisted and auditable. +- Failed Baton calls are retryable. + +Do not rely on the Puppeteer WhatsApp Web client for production go-live unless a separate security/session/runbook decision is made. The Chrome extension path is the recommended first-class ingestion path; manual export is the fallback. + +## 7. Security and Compliance + +Production requirements: + +- No hardcoded secrets. +- GitHub Actions uses OIDC, not long-lived Azure credentials. +- API uses managed identity to access Azure resources where possible. +- Web and API auth is through Mystira Identity. +- Key Vault purge protection enabled. +- Storage containers private. +- Uploaded chat exports treated as sensitive data. +- Define retention policy for raw WhatsApp exports, normalized messages, and summaries. +- Add data deletion path by user/export. +- Add audit trail for who imported, reviewed, and published Baton tasks. +- Add rate limits for upload and extension endpoints. +- Add file size and content validation already exists; expand with malware/content scanning decision if uploads become public-facing. +- Review WhatsApp terms and support posture before any automated WhatsApp Web monitoring. + +## 8. CI/CD and Release Flow + +Required pipelines: + +- PR validation: + - install dependencies; + - lint; + - typecheck; + - unit tests; + - API Docker build; + - frontend build; + - Terraform fmt/validate; + - Terraform plan/what-if comment for prod. +- Main branch: + - build and push API image; + - deploy/provision infra after approval; + - deploy API; + - deploy frontend; + - run smoke tests. +- Production environment: + - protected GitHub environment; + - required reviewers; + - OIDC federated credential for `repo:neuralliquid/convolens:environment:prod`; + - rollback instructions. + +Recommended release gates: + +- No open critical/high security findings. +- API image tested locally or in CI before deployment. +- Terraform plan reviewed. +- App health checks pass. +- One WhatsApp import smoke test passes. +- One Baton candidate publish smoke test passes. +- Logs/alerts verified. + +## 9. Observability + +Minimum dashboards and alerts: + +- API availability and 5xx rate. +- API latency. +- Container App restarts. +- Queue depth and oldest message age. +- WhatsApp ingestion success/failure count. +- Baton publish success/failure count. +- Baton retry backlog. +- Database RU/throughput or PostgreSQL CPU/connections, depending on chosen database. +- Storage errors. +- Auth failures. +- Budget threshold alerts. + +## 10. Go-Live Runbook + +Preflight: + +- Confirm subscription and tenant. +- Confirm target resource group naming. +- Confirm DNS names. +- Confirm `convolens.neuralliquid.ai` DNS ownership and target record type. +- Confirm GitHub OIDC prod credential. +- Confirm Mystira Identity client registration and redirect URIs. +- Confirm Baton project-routing rules and auth path. +- Confirm PostgreSQL as the first production database unless a concrete blocker appears. +- Confirm retention policy. +- Confirm admin email for budgets/alerts. + +Build: + +- Implement production data path. +- Implement WhatsApp-to-Baton intake slice. +- Add prod Terraform environment. +- Add CI/CD changes. +- Add runbook and smoke scripts. + +Validation: + +- Run unit and integration tests. +- Build frontend. +- Build API image. +- Run API image locally if Docker is available. +- Run Terraform fmt/validate. +- Run Terraform plan for prod. +- Run Azure deployment preview/what-if. +- Run security review on secrets and public endpoints. + +Deployment: + +- Provision/update prod infrastructure. +- Push API image. +- Deploy API. +- Deploy frontend. +- Configure DNS and CORS. +- Configure alerts. + +Smoke: + +- `GET /health/live` returns 200. +- `GET /health/ready` returns 200. +- Web app loads over production domain. +- Auth flow works through Mystira Identity. +- Chrome extension submits one known WhatsApp extraction. +- Upload one known WhatsApp export as fallback. +- Confirm raw artifact and normalized messages persisted. +- Confirm intake candidates are created. +- Publish one candidate to Baton. +- Confirm Baton task id is written back. +- Confirm failed Baton call retry path can be exercised without data loss. + +Rollback: + +- Keep previous API image tag. +- Use Container Apps revision rollback. +- Revert frontend deployment to previous build. +- Do not destroy production data resources during rollback. +- Pause Baton publishing if duplicate or bad candidate creation is detected. + +## 11. Proposed Baton Work Packages + +Baton tracker created on 2026-07-16 in the active Mystira project (`7ebf10db-5261-408a-9db2-8ba14d4110b7`). Continue to treat this plan as the delivery source of truth, and keep the Baton tracker synchronized as phases start and close. + +Root task: + +- Title: Convolens Mystira Azure production go-live +- Task id: `27d59d1e-0e0b-40e7-80de-fe1aba94de85` +- Project: active Mystira project unless a Convolens-specific Baton project is created later +- Priority: high + +Child tasks: + +- Phase 0 coordination baseline and approvals: `06658c74-a86e-4613-9a37-d9d4558b01a7`. +- Phase 1 production Azure foundation: `fccd0995-b045-42bc-baca-dc991060e4d8`. +- Phase 2 Mystira Identity auth and runtime shell: `a7004cd8-1097-4779-a22c-e8c1a686c0f7`. +- Phase 3 Postgres production data model: `0dea1455-4e56-40da-8d50-c77c6e102d18`. +- Phase 4 WhatsApp extension and manual intake persistence: `1b07ade8-409e-4384-ad3b-17f90513ff85`. +- Phase 5 candidate extraction and review UI: `4811f3fd-5759-43b4-9e26-52d0c8d6fe8e`. +- Phase 6 Baton publish integration with retry and idempotency: `725f1c46-cc0b-40da-bfd3-38a343ab0c46`. +- Phase 7 CI/CD and deployment readiness: `09443dfc-aa2f-4732-9e05-20c8119b09ac`. +- Phase 8 observability alerts and runbook: `788dc618-9060-41bf-a244-dfc0a2f91e8a`. +- Phase 9 serial go-live and handoff: `3d62321c-663b-44bf-954d-1e2dc9581ef2`. + +## 12. Decisions and Recommendations + +- Naming: use `nl-prod-convolens-*` in the Mystira subscription. This matches ownership and the user's corrected direction. +- Region: use `southafricanorth` wherever available. Use another region only for an explicitly unsupported service or capacity issue. +- Environment strategy: straight to `prod`; do not build a separate `dev` environment right now. Treat `prod` as the live working environment and keep deployment gates tight. +- Database: use PostgreSQL first. Reason: existing TypeORM entities/migrations map naturally to Postgres, and the go-live data model needs relational workflow state for ingestion, review, audit, and Baton publish records. Add Cosmos later only if message-document scale or query shape justifies it. +- Frontend hosting: use App Service for the Next.js web app unless a build spike proves Static Web Apps supports every runtime/auth requirement cleanly. App Service reduces risk for Mystira Identity callback/session behavior. +- Domain: use `convolens.neuralliquid.ai`. +- Auth: use Mystira Identity / Entra OIDC for web/operator access. Convolens should be a relying-party client, similar to the HOV/Sluice/Docket direction already used elsewhere. +- Baton project routing: route each intake to the relevant project. Default to the active Mystira project (`7ebf10db-5261-408a-9db2-8ba14d4110b7`) when classification is unclear; route obvious platform/service items to Sluice, Docket, Baton, Retort, HOV, Cognitive Mesh, etc. after a lookup. +- Baton publishing: review-gated at launch. Reason: WhatsApp extraction can overproduce noisy or ambiguous tasks, and wrong-project task creation creates durable cleanup work. +- Raw export retention: recommended default is 30 days for raw WhatsApp exports, 90 days for normalized extracted messages/candidates, and indefinite retention only for approved Baton task references/audit metadata. Add user/admin deletion support. +- Primary ingestion path: Chrome extension should be the go-live path because it can preserve structure and avoid manual export friction. Manual export upload stays as fallback and admin/debug path. +- Automated WhatsApp Web/Puppeteer monitoring: out of scope for initial production go-live. It has session, QR, browser automation, and WhatsApp policy risk. Revisit only after the extension/manual-import path is stable. + +## 13. Approval Checkpoint + +No implementation or deployment should happen until this plan is reviewed and approved. + +After approval, the next step is to convert this plan into implementation work: + +1. Create/reuse Baton root tracker and subtasks. +2. Create production Terraform environment. +3. Implement the production persistence and Baton intake slice. +4. Run Azure validation. +5. Deploy only after validation passes. + +## 14. Phased Execution Plan + +This work should run as coordinated phases with parallel lanes where dependencies allow it. The critical path is: production foundation -> data/auth readiness -> WhatsApp intake -> Baton publish -> validation -> go-live. + +### Phase 0 - Coordination and Baseline + +Goal: freeze the target shape and create durable coordination artifacts before implementation. + +Inputs: + +- Approved `.azure/plan.md`. +- Current repo state on the intended branch. +- Live Azure subscription context. +- Baton availability. + +Work: + +- Create or reuse a Baton root tracker for `Convolens Mystira Azure production go-live`. +- Create child tasks for infra, auth, data, frontend, API, extension, Baton integration, CI/CD, observability, and DNS. +- Confirm resource naming: `nl-prod-convolens-*`. +- Confirm target hostname: `convolens.neuralliquid.ai`. +- Confirm production-only delivery model: no separate dev environment for now. +- Confirm that `southafricanorth` is the default region. + +Exit criteria: + +- Baton tracker exists with work split into reviewable tasks. +- Implementation branch exists. +- Current worktree state is understood and unrelated local dirt is isolated. + +Parallelism: + +- Coordinator lane can create Baton tasks while infra/auth/data leads read code and prepare implementation notes. + +### Phase 1 - Production Azure Foundation + +Goal: create the production infrastructure plan without deploying application behavior yet. + +Implementation note: + +- The first Terraform profile is cost-minimized for internal evaluation: PostgreSQL, Redis, and dedicated ACR are disabled by default; the API scales to zero; frontend App Service uses the Free tier; storage uses LRS; telemetry is capped at 1 GB/day. The API target port defaults to `80` for the Azure helloworld placeholder image; set `api_target_port = 3001` when deploying the real API image. Flip `enable_postgres = true` and choose paid frontend/API sizing before treating the environment as durable production. + +Work: + +- Add `infra/terraform/env/prod`. +- Set `org = "nl"`, `env = "prod"`, `location = "southafricanorth"`. +- Add or wire: + - `nl-prod-convolens-rg`; + - Log Analytics; + - Application Insights; + - Key Vault with purge protection; + - Storage account and private containers; + - PostgreSQL Flexible Server or equivalent production Postgres service; + - Container Apps API; + - frontend App Service; + - queue for ingestion/Baton publishing; + - budget alerts; + - managed identity role assignments; + - outputs for URLs, identities, and telemetry. +- Decide whether to reuse shared ACR or create `nl-prod-convolens-acr`. +- Add DNS readiness notes for `convolens.neuralliquid.ai`. + +Exit criteria: + +- Terraform formats and validates. +- `terraform plan` is reviewed. +- No live deployment yet unless separately approved. + +Parallelism: + +- Infra lead owns Terraform. +- CI/CD lead can update workflow design in parallel once resource names are stable. +- Observability lead can define alert queries in parallel. + +### Phase 2 - Auth and Runtime Shell + +Goal: make Convolens production-authenticated through Mystira Identity and ready for the prod hostname. + +Work: + +- Register Convolens as a Mystira Identity OIDC relying party/client. +- Add app settings/env vars for: + - issuer; + - client id; + - client secret reference; + - callback/logout URLs; + - session secret; + - allowed origins. +- Wire Next.js auth flow for web/operator access. +- Protect import, review, publish, and admin routes. +- Update API auth validation if API receives identity tokens directly. +- Lock CORS to `https://convolens.neuralliquid.ai` and any required callback/dev-only temporary origins. + +Exit criteria: + +- Auth flow can be smoked locally or in a temporary environment. +- Protected routes fail closed. +- No auth secrets are committed. + +Parallelism: + +- Auth lead can work while infra lead builds Terraform. +- Frontend lead can wire UI route protection in parallel with API token validation. + +### Phase 3 - Production Data Model + +Goal: remove SQLite as the production dependency and persist all ingestion state durably. + +Work: + +- Add PostgreSQL driver/config and production connection handling. +- Update TypeORM config for Postgres in production. +- Define entities/tables for: + - imports/extractions; + - raw artifact metadata; + - normalized chats; + - normalized messages; + - candidate tasks; + - review decisions; + - Baton publish attempts; + - Baton task links; + - audit events. +- Add migrations. +- Add idempotency keys/hashes for export and extension ingestion. +- Add deletion/retention fields. +- Add tests for duplicate extraction and persistence. + +Exit criteria: + +- API can start against Postgres. +- Migrations apply cleanly. +- Ingested data survives restart. +- Duplicate ingest does not duplicate candidates or Baton publishes. + +Parallelism: + +- Data lead owns schema/migrations. +- API intake lead can implement against repository interfaces once schemas are agreed. +- Retention/audit lead can write tests and policy docs in parallel. + +### Phase 4 - WhatsApp Intake + +Goal: make Chrome extension ingestion the primary go-live path, with manual export upload as fallback. + +Work: + +- Confirm Chrome extension production endpoint/config flow. +- Persist `/api/chat-export/extension` submissions. +- Persist `/api/chat-export/upload` fallback submissions. +- Store raw artifacts in Blob Storage. +- Store normalized messages in Postgres. +- Add ingestion queue job after persistence. +- Add selector-report persistence and admin access control. +- Add ingestion status APIs for review UI. +- Add smoke fixture for a known WhatsApp chat. + +Exit criteria: + +- Extension can submit a real extraction to the API. +- Manual upload works. +- Raw and normalized data are persisted. +- Ingestion status is visible. + +Parallelism: + +- Extension lead owns extension config and extraction submit flow. +- API intake lead owns endpoints/persistence. +- Storage lead owns Blob upload/download and managed identity access. + +### Phase 5 - Candidate Extraction and Review + +Goal: produce reviewable Baton task candidates from WhatsApp data without polluting Baton. + +Work: + +- Implement extraction pipeline from normalized messages to candidate actions. +- Add simple deterministic first pass before LLM dependence where possible: + - explicit "todo/follow up/please" phrasing; + - named owner/date; + - project keywords; + - decision/action markers. +- Add optional LLM summarization/extraction behind config and cost controls. +- Add candidate confidence and evidence spans. +- Add review UI/API: + - accept; + - edit; + - reject; + - choose target Baton project; + - bulk publish selected. +- Add project recommendation rules: + - default Mystira project; + - Sluice/Docket/Baton/Retort/HOV/Cognitive Mesh keyword/project routing. + +Exit criteria: + +- Known fixture produces sensible candidates. +- Reviewer can edit and select target project. +- No task is created in Baton before approval. + +Parallelism: + +- Extraction lead owns candidate generation. +- Frontend lead owns review UI. +- Baton lead owns project lookup/routing rules. + +### Phase 6 - Baton Publish Integration + +Goal: publish reviewed candidates into the correct Baton projects with auditability and retries. + +Work: + +- Implement Baton client wrapper. +- Add task creation with: + - title; + - description; + - priority; + - context; + - trace id; + - triggeredBy; + - source metadata. +- Add pre-publish duplicate check by source hash and normalized title. +- Add retry/backoff for Baton 502/timeouts. +- Persist publish attempts and results. +- Persist Baton task id and project id back to Convolens. +- Add admin retry action for `pending_baton` or `failed_baton`. + +Exit criteria: + +- One approved candidate creates one Baton task in the intended project. +- Duplicate publish is blocked. +- Baton outage does not lose candidates. +- Publish audit trail is visible. + +Parallelism: + +- Baton lead works after candidate schema is stable. +- API lead can implement retry worker in parallel with frontend publish controls. + +### Phase 7 - CI/CD and Deployment Readiness + +Goal: make the release repeatable and gated. + +Work: + +- Add production GitHub environment. +- Add OIDC federated credential for `repo:neuralliquid/convolens:environment:prod`. +- Build and push API image. +- Build frontend. +- Run test/lint/typecheck/build gates. +- Add Terraform fmt/validate/plan gate. +- Add production deployment workflow with manual approval. +- Add smoke test workflow: + - health; + - auth redirect; + - API readiness; + - extension config; + - sample ingest; + - Baton publish dry run or controlled real publish. + +Exit criteria: + +- CI passes. +- Deployment workflow is ready but not run without approval. +- Rollback procedure is documented. + +Parallelism: + +- CI/CD lead can start once infra names and app build requirements are known. +- QA lead can build smoke fixtures in parallel. + +### Phase 8 - Observability and Runbook + +Goal: make production operable before exposing it. + +Work: + +- Add dashboards/queries for: + - API availability; + - 5xx rate; + - latency; + - Container App restarts; + - App Service health; + - queue depth; + - oldest queue message age; + - ingestion failures; + - Baton publish failures; + - duplicate publish blocks; + - auth failures; + - DB health; + - storage failures. +- Add alerts for critical conditions. +- Add go-live and rollback runbook. +- Add data retention/deletion runbook. + +Exit criteria: + +- Alerts exist. +- Runbook has concrete commands/URLs. +- Smoke evidence has a place to be recorded. + +Parallelism: + +- Observability lead can work after resource names and telemetry conventions are stable. +- Runbook lead can draft from Phase 1 onward and refine at each phase. + +### Phase 9 - Go-Live + +Goal: deploy, verify, and record the live state. + +Work: + +- Run final Terraform plan/preview. +- Apply approved infrastructure. +- Deploy API. +- Deploy frontend. +- Bind DNS for `convolens.neuralliquid.ai`. +- Configure CORS/callback URLs to final host. +- Run smoke tests: + - web loads; + - Mystira Identity login works; + - API health works; + - Chrome extension ingestion works; + - manual upload works; + - candidates are created; + - one approved candidate publishes to Baton; + - Baton task link is written back; + - telemetry captures the flow. +- Record Baton closeout/handoff with: + - URLs; + - deployed revisions/images; + - smoke results; + - known limitations; + - rollback steps. + +Exit criteria: + +- Production URL is usable. +- WhatsApp-to-Baton flow is verified. +- Baton handoff is recorded. +- No unresolved go-live blocker remains. + +## 15. Parallel Agent Workstreams + +Use concurrent subagents where the work is separable and each lane has clear inputs/outputs. Do not run parallel agents against the same files without coordination. + +Recommended lanes: + +- Coordinator/Baton agent: + - create/reuse Baton tracker; + - maintain phase status; + - write closeout evidence; + - avoid implementation files. +- Azure infra agent: + - own Terraform prod environment; + - own managed identity, Key Vault, storage, Postgres, Container Apps, App Service, budgets; + - coordinate with CI/CD on resource names and outputs. +- Auth agent: + - own Mystira Identity client requirements; + - own web/API auth config; + - coordinate redirect/CORS settings with infra and frontend. +- Data/API agent: + - own Postgres migration from SQLite; + - own persistence entities/repositories; + - own ingestion status APIs. +- WhatsApp extension agent: + - own extension endpoint config; + - own extraction submit flow; + - own extension smoke fixture. +- Baton integration agent: + - own candidate-to-Baton publish API; + - own project routing and duplicate detection; + - own retry/pending states. +- Frontend/review UI agent: + - own authenticated screens; + - own import status and candidate review UI; + - coordinate with API contracts. +- CI/CD agent: + - own GitHub Actions, OIDC environment, image build, deployment, smoke workflows. +- Observability/QA agent: + - own test matrix, smoke scripts, telemetry queries, alerts, runbook proof. + +Concurrency rules: + +- Phase 0 must complete before task creation spreads across lanes. +- Phase 1, Phase 2, and Phase 3 can run concurrently after decisions are locked, but shared config contracts must be reviewed daily. +- Phase 4 can start once the data repository interfaces are agreed; it does not need all infra to exist. +- Phase 5 can start with fixtures while Phase 4 persists real data. +- Phase 6 starts after candidate schema is stable. +- Phase 7 starts early but cannot finalize until infra outputs and app build paths are stable. +- Phase 8 starts early and follows resource naming/telemetry conventions. +- Phase 9 is serial and should be run by one coordinator with supporting agents on standby. diff --git a/.github/workflows/infrastructure.yml b/.github/workflows/infrastructure.yml index c74718b..0785d9d 100644 --- a/.github/workflows/infrastructure.yml +++ b/.github/workflows/infrastructure.yml @@ -31,8 +31,8 @@ on: type: choice options: - validate - - what-if - - deploy + - plan + - apply permissions: id-token: write @@ -40,47 +40,44 @@ permissions: pull-requests: write env: - AZURE_RESOURCE_GROUP_DEV: rg-whatssummarize-dev - AZURE_RESOURCE_GROUP_STAGING: rg-whatssummarize-staging - AZURE_RESOURCE_GROUP_PROD: rg-whatssummarize-prod - AZURE_LOCATION: eastus + TF_VERSION: '1.14.7' + TF_IN_AUTOMATION: 'true' + TF_INPUT: '0' + ARM_USE_OIDC: 'true' jobs: # ============================================================================= - # Validate Bicep Templates + # Validate Terraform # ============================================================================= validate: - name: Validate Templates + name: Validate Terraform runs-on: ubuntu-latest steps: - name: Checkout uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7 - - name: Install Bicep CLI - run: | - az bicep install - az config set bicep.use_binary_from_path=false + - name: Setup Terraform + uses: hashicorp/setup-terraform@b9cd54a3c349d3f38e8881555d616ced269862dd # v3.1.2 + with: + terraform_version: ${{ env.TF_VERSION }} - - name: Validate Bicep Syntax - run: | - echo "Validating Bicep files..." - az bicep build --file infra/bicep/main.bicep --stdout > /dev/null - echo "✅ Bicep syntax validation passed" + - name: Terraform Format Check + run: terraform -chdir=infra/terraform/env/dev fmt -check -recursive - - name: Lint Bicep Files - run: | - echo "Linting Bicep files..." - # Install bicep linter if available - az bicep lint --file infra/bicep/main.bicep 2>/dev/null || echo "Linting not available, skipping..." + - name: Terraform Init (no backend) + run: terraform -chdir=infra/terraform/env/dev init -backend=false + + - name: Terraform Validate + run: terraform -chdir=infra/terraform/env/dev validate # ============================================================================= - # Preview Changes (What-If) + # Plan Changes (PR preview / workflow_dispatch=plan) # ============================================================================= - preview: - name: Preview Changes (${{ matrix.environment }}) + plan: + name: Plan (${{ matrix.environment }}) runs-on: ubuntu-latest needs: validate - if: github.event_name == 'pull_request' || (github.event_name == 'workflow_dispatch' && github.event.inputs.action == 'what-if') + if: github.event_name == 'pull_request' || (github.event_name == 'workflow_dispatch' && github.event.inputs.action == 'plan') strategy: matrix: environment: ${{ github.event_name == 'workflow_dispatch' && fromJson(format('["{0}"]', github.event.inputs.environment)) || fromJson('["dev"]') }} @@ -90,6 +87,11 @@ jobs: - name: Checkout uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7 + - name: Setup Terraform + uses: hashicorp/setup-terraform@b9cd54a3c349d3f38e8881555d616ced269862dd # v3.1.2 + with: + terraform_version: ${{ env.TF_VERSION }} + - name: Parse Azure Credentials id: azure-creds env: @@ -99,104 +101,91 @@ jobs: CLIENT_ID=$(echo "$AZURE_CREDENTIALS" | jq -r '.clientId') TENANT_ID=$(echo "$AZURE_CREDENTIALS" | jq -r '.tenantId') SUBSCRIPTION_ID=$(echo "$AZURE_CREDENTIALS" | jq -r '.subscriptionId') - - # Validate that parsing succeeded - if [ "$CLIENT_ID" = "null" ] || [ -z "$CLIENT_ID" ]; then - echo "Error: Failed to parse clientId from AZURE_CREDENTIALS" - exit 1 - fi - if [ "$TENANT_ID" = "null" ] || [ -z "$TENANT_ID" ]; then - echo "Error: Failed to parse tenantId from AZURE_CREDENTIALS" - exit 1 - fi - if [ "$SUBSCRIPTION_ID" = "null" ] || [ -z "$SUBSCRIPTION_ID" ]; then - echo "Error: Failed to parse subscriptionId from AZURE_CREDENTIALS" - exit 1 - fi - - echo "client-id=$CLIENT_ID" >> $GITHUB_OUTPUT - echo "tenant-id=$TENANT_ID" >> $GITHUB_OUTPUT - echo "subscription-id=$SUBSCRIPTION_ID" >> $GITHUB_OUTPUT - - name: Azure Login + for v in "$CLIENT_ID" "$TENANT_ID" "$SUBSCRIPTION_ID"; do + if [ "$v" = "null" ] || [ -z "$v" ]; then + echo "Error: AZURE_CREDENTIALS missing clientId/tenantId/subscriptionId" >&2 + exit 1 + fi + done + echo "client-id=$CLIENT_ID" >> "$GITHUB_OUTPUT" + echo "tenant-id=$TENANT_ID" >> "$GITHUB_OUTPUT" + echo "subscription-id=$SUBSCRIPTION_ID" >> "$GITHUB_OUTPUT" + + - name: Azure Login (OIDC) uses: azure/login@cb79c773a3cfa27f31f25eb3f677781210c9ce3d # v1.6.1 with: client-id: ${{ steps.azure-creds.outputs.client-id }} tenant-id: ${{ steps.azure-creds.outputs.tenant-id }} subscription-id: ${{ steps.azure-creds.outputs.subscription-id }} - - name: Ensure Resource Group - run: | - RG_NAME="rg-whatssummarize-${{ matrix.environment }}" - if ! az group show --name "$RG_NAME" &>/dev/null; then - echo "Creating resource group: $RG_NAME" - az group create --name "$RG_NAME" --location "${{ env.AZURE_LOCATION }}" - fi + - name: Terraform Init + env: + ARM_CLIENT_ID: ${{ steps.azure-creds.outputs.client-id }} + ARM_TENANT_ID: ${{ steps.azure-creds.outputs.tenant-id }} + ARM_SUBSCRIPTION_ID: ${{ steps.azure-creds.outputs.subscription-id }} + run: terraform -chdir=infra/terraform/env/${{ matrix.environment }} init -backend-config=backend.hcl - - name: What-If Analysis - id: whatif + - name: Terraform Plan + id: plan + env: + ARM_CLIENT_ID: ${{ steps.azure-creds.outputs.client-id }} + ARM_TENANT_ID: ${{ steps.azure-creds.outputs.tenant-id }} + ARM_SUBSCRIPTION_ID: ${{ steps.azure-creds.outputs.subscription-id }} run: | - RG_NAME="rg-whatssummarize-${{ matrix.environment }}" - echo "Running what-if analysis for ${{ matrix.environment }}..." - - az deployment group what-if \ - --resource-group "$RG_NAME" \ - --template-file infra/bicep/main.bicep \ - --parameters infra/parameters/${{ matrix.environment }}.bicepparam \ - --name "whatif-${{ github.run_id }}" \ - > whatif-output.txt 2>&1 || true - - cat whatif-output.txt - - # Save for PR comment - echo "whatif<> $GITHUB_OUTPUT - cat whatif-output.txt >> $GITHUB_OUTPUT - echo "EOF" >> $GITHUB_OUTPUT + terraform -chdir=infra/terraform/env/${{ matrix.environment }} plan -no-color -out=tfplan | tee plan-output.txt + { + echo "plan<> "$GITHUB_OUTPUT" - name: Comment on PR if: github.event_name == 'pull_request' uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7.0.1 + env: + PLAN_OUTPUT: ${{ steps.plan.outputs.plan }} with: script: | - const output = `### Infrastructure What-If Analysis (${{ matrix.environment }}) - -
- Click to expand - - \`\`\` - ${{ steps.whatif.outputs.whatif }} - \`\`\` - -
- - *Run ID: ${{ github.run_id }}*`; - - github.rest.issues.createComment({ + const planOutput = process.env.PLAN_OUTPUT || ''; + const truncated = planOutput.length > 60000 + ? planOutput.slice(0, 60000) + '\n\n[…output truncated…]' + : planOutput; + const body = `### Terraform Plan (\`${{ matrix.environment }}\`)\n\n
\nClick to expand\n\n\`\`\`\n${truncated}\n\`\`\`\n\n
\n\n*Run ID: ${{ github.run_id }}*`; + await github.rest.issues.createComment({ issue_number: context.issue.number, owner: context.repo.owner, repo: context.repo.repo, - body: output + body }); # ============================================================================= - # Deploy Infrastructure + # Apply (push to main / workflow_dispatch=apply) # ============================================================================= - deploy: - name: Deploy (${{ matrix.environment }}) + apply: + name: Apply (${{ matrix.environment }}) runs-on: ubuntu-latest needs: validate if: | (github.event_name == 'push' && github.ref == 'refs/heads/main') || - (github.event_name == 'workflow_dispatch' && github.event.inputs.action == 'deploy') + (github.event_name == 'workflow_dispatch' && github.event.inputs.action == 'apply') strategy: matrix: environment: ${{ github.event_name == 'workflow_dispatch' && fromJson(format('["{0}"]', github.event.inputs.environment)) || fromJson('["dev"]') }} max-parallel: 1 environment: ${{ matrix.environment }} + outputs: + container_app_url: ${{ steps.outputs.outputs.container_app_url }} steps: - name: Checkout uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7 + - name: Setup Terraform + uses: hashicorp/setup-terraform@b9cd54a3c349d3f38e8881555d616ced269862dd # v3.1.2 + with: + terraform_version: ${{ env.TF_VERSION }} + terraform_wrapper: false + - name: Parse Azure Credentials id: azure-creds env: @@ -206,148 +195,73 @@ jobs: CLIENT_ID=$(echo "$AZURE_CREDENTIALS" | jq -r '.clientId') TENANT_ID=$(echo "$AZURE_CREDENTIALS" | jq -r '.tenantId') SUBSCRIPTION_ID=$(echo "$AZURE_CREDENTIALS" | jq -r '.subscriptionId') - - # Validate that parsing succeeded - if [ "$CLIENT_ID" = "null" ] || [ -z "$CLIENT_ID" ]; then - echo "Error: Failed to parse clientId from AZURE_CREDENTIALS" - exit 1 - fi - if [ "$TENANT_ID" = "null" ] || [ -z "$TENANT_ID" ]; then - echo "Error: Failed to parse tenantId from AZURE_CREDENTIALS" - exit 1 - fi - if [ "$SUBSCRIPTION_ID" = "null" ] || [ -z "$SUBSCRIPTION_ID" ]; then - echo "Error: Failed to parse subscriptionId from AZURE_CREDENTIALS" - exit 1 - fi - - echo "client-id=$CLIENT_ID" >> $GITHUB_OUTPUT - echo "tenant-id=$TENANT_ID" >> $GITHUB_OUTPUT - echo "subscription-id=$SUBSCRIPTION_ID" >> $GITHUB_OUTPUT - - name: Azure Login + for v in "$CLIENT_ID" "$TENANT_ID" "$SUBSCRIPTION_ID"; do + if [ "$v" = "null" ] || [ -z "$v" ]; then + echo "Error: AZURE_CREDENTIALS missing clientId/tenantId/subscriptionId" >&2 + exit 1 + fi + done + echo "client-id=$CLIENT_ID" >> "$GITHUB_OUTPUT" + echo "tenant-id=$TENANT_ID" >> "$GITHUB_OUTPUT" + echo "subscription-id=$SUBSCRIPTION_ID" >> "$GITHUB_OUTPUT" + + - name: Azure Login (OIDC) uses: azure/login@cb79c773a3cfa27f31f25eb3f677781210c9ce3d # v1.6.1 with: client-id: ${{ steps.azure-creds.outputs.client-id }} tenant-id: ${{ steps.azure-creds.outputs.tenant-id }} subscription-id: ${{ steps.azure-creds.outputs.subscription-id }} - - name: Ensure Resource Group - run: | - RG_NAME="rg-whatssummarize-${{ matrix.environment }}" - if ! az group show --name "$RG_NAME" &>/dev/null; then - echo "Creating resource group: $RG_NAME" - az group create \ - --name "$RG_NAME" \ - --location "${{ env.AZURE_LOCATION }}" \ - --tags project=convolens environment=${{ matrix.environment }} managedBy=github-actions - fi - - - name: Deploy Infrastructure - id: deploy - run: | - RG_NAME="rg-whatssummarize-${{ matrix.environment }}" - DEPLOYMENT_NAME="deploy-${{ github.run_id }}-${{ github.run_attempt }}" - - echo "Deploying to ${{ matrix.environment }}..." - - az deployment group create \ - --resource-group "$RG_NAME" \ - --template-file infra/bicep/main.bicep \ - --parameters infra/parameters/${{ matrix.environment }}.bicepparam \ - --name "$DEPLOYMENT_NAME" + - name: Terraform Init + env: + ARM_CLIENT_ID: ${{ steps.azure-creds.outputs.client-id }} + ARM_TENANT_ID: ${{ steps.azure-creds.outputs.tenant-id }} + ARM_SUBSCRIPTION_ID: ${{ steps.azure-creds.outputs.subscription-id }} + run: terraform -chdir=infra/terraform/env/${{ matrix.environment }} init -backend-config=backend.hcl - # Get outputs - echo "Getting deployment outputs..." - az deployment group show \ - --resource-group "$RG_NAME" \ - --name "$DEPLOYMENT_NAME" \ - --query "properties.outputs" \ - -o json > deployment-outputs.json + - name: Terraform Apply + env: + ARM_CLIENT_ID: ${{ steps.azure-creds.outputs.client-id }} + ARM_TENANT_ID: ${{ steps.azure-creds.outputs.tenant-id }} + ARM_SUBSCRIPTION_ID: ${{ steps.azure-creds.outputs.subscription-id }} + run: terraform -chdir=infra/terraform/env/${{ matrix.environment }} apply -auto-approve - cat deployment-outputs.json + - name: Capture Outputs + id: outputs + env: + ARM_CLIENT_ID: ${{ steps.azure-creds.outputs.client-id }} + ARM_TENANT_ID: ${{ steps.azure-creds.outputs.tenant-id }} + ARM_SUBSCRIPTION_ID: ${{ steps.azure-creds.outputs.subscription-id }} + run: | + terraform -chdir=infra/terraform/env/${{ matrix.environment }} output -json > tf-outputs.json + CONTAINER_APP_URL=$(jq -r '.container_app_url.value // ""' tf-outputs.json) + echo "container_app_url=$CONTAINER_APP_URL" >> "$GITHUB_OUTPUT" - - name: Upload Deployment Outputs + - name: Upload Outputs uses: actions/upload-artifact@834a144ee995460fba8ed112a2fc961b36a5ec5a # v4.3.6 with: - name: deployment-outputs-${{ matrix.environment }} - path: deployment-outputs.json + name: tf-outputs-${{ matrix.environment }} + path: infra/terraform/env/${{ matrix.environment }}/tf-outputs.json retention-days: 30 - - name: Validate Deployment - run: | - echo "Validating deployed resources..." - chmod +x infra/scripts/validate-resources.sh - ./infra/scripts/validate-resources.sh ${{ matrix.environment }} || true - # ============================================================================= - # Post-Deployment Verification + # Post-Apply Health Check # ============================================================================= verify: name: Verify Deployment runs-on: ubuntu-latest - needs: deploy - if: success() + needs: apply + if: success() && needs.apply.outputs.container_app_url != '' environment: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.environment || 'dev' }} steps: - - name: Checkout - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7 - - - name: Download Outputs - uses: actions/download-artifact@fa0a91b85d4f404e444e00e005971372dc801d16 # v4.1.8 - with: - name: deployment-outputs-${{ github.event_name == 'workflow_dispatch' && github.event.inputs.environment || 'dev' }} - - - name: Parse Azure Credentials - id: azure-creds - env: - AZURE_CREDENTIALS: ${{ secrets.AZURE_CREDENTIALS }} - run: | - set -euo pipefail - CLIENT_ID=$(echo "$AZURE_CREDENTIALS" | jq -r '.clientId') - TENANT_ID=$(echo "$AZURE_CREDENTIALS" | jq -r '.tenantId') - SUBSCRIPTION_ID=$(echo "$AZURE_CREDENTIALS" | jq -r '.subscriptionId') - - # Validate that parsing succeeded - if [ "$CLIENT_ID" = "null" ] || [ -z "$CLIENT_ID" ]; then - echo "Error: Failed to parse clientId from AZURE_CREDENTIALS" - exit 1 - fi - if [ "$TENANT_ID" = "null" ] || [ -z "$TENANT_ID" ]; then - echo "Error: Failed to parse tenantId from AZURE_CREDENTIALS" - exit 1 - fi - if [ "$SUBSCRIPTION_ID" = "null" ] || [ -z "$SUBSCRIPTION_ID" ]; then - echo "Error: Failed to parse subscriptionId from AZURE_CREDENTIALS" - exit 1 - fi - - echo "client-id=$CLIENT_ID" >> $GITHUB_OUTPUT - echo "tenant-id=$TENANT_ID" >> $GITHUB_OUTPUT - echo "subscription-id=$SUBSCRIPTION_ID" >> $GITHUB_OUTPUT - - name: Azure Login - uses: azure/login@cb79c773a3cfa27f31f25eb3f677781210c9ce3d # v1.6.1 - with: - client-id: ${{ steps.azure-creds.outputs.client-id }} - tenant-id: ${{ steps.azure-creds.outputs.tenant-id }} - subscription-id: ${{ steps.azure-creds.outputs.subscription-id }} - - name: Health Check run: | - echo "Running health checks..." - - # Get API URL from outputs - API_URL=$(cat deployment-outputs.json | jq -r '.containerAppsApiUrl.value // empty') - - if [ -n "$API_URL" ]; then - echo "Checking API health: $API_URL/health" - curl -sf "$API_URL/health" || echo "API health check failed (may not be deployed yet)" - else - echo "API URL not available yet" - fi + API_URL="${{ needs.apply.outputs.container_app_url }}" + echo "Probing $API_URL/health" + curl -sfS --max-time 10 "$API_URL/health" \ + || echo "API health check failed (placeholder image returns 200 / — non-critical)" - name: Notify Success - if: success() run: | - echo "✅ Deployment and verification completed successfully!" - echo "Environment: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.environment || 'dev' }}" + echo "Apply + verify completed: ${{ needs.apply.outputs.container_app_url }}" diff --git a/.github/workflows/release-validation.yml b/.github/workflows/release-validation.yml index e459933..622a7da 100644 --- a/.github/workflows/release-validation.yml +++ b/.github/workflows/release-validation.yml @@ -89,8 +89,11 @@ jobs: id: validate run: | ENV="${{ matrix.environment }}" - PROJECT="whatssummarize" - RG="rg-${PROJECT}-${ENV}" + ORG="nl" + PROJECT="convolens" + # Naming pattern per ADR-0027 (no region suffix): {org}-{env}-{project}-{type} + BASE="${ORG}-${ENV}-${PROJECT}" + RG="${BASE}-rg" echo "==============================================" echo "Validating Azure Resources for Release" @@ -112,8 +115,7 @@ jobs: fi # Check Key Vault - KV_NAME="kv${PROJECT}${ENV}" - KV_NAME="${KV_NAME//-/}" + KV_NAME="${BASE}-kv" echo "Checking Key Vault: $KV_NAME..." if ! az keyvault show --name "$KV_NAME" &>/dev/null; then echo "❌ Key Vault not found" @@ -133,9 +135,8 @@ jobs: done fi - # Check Storage Account - STORAGE_NAME="st${PROJECT}${ENV}" - STORAGE_NAME="${STORAGE_NAME//-/}" + # Check Storage Account (alphanumeric only) + STORAGE_NAME="${ORG}${ENV}${PROJECT}st" echo "Checking Storage Account: $STORAGE_NAME..." if ! az storage account show --name "$STORAGE_NAME" --resource-group "$RG" &>/dev/null; then echo "❌ Storage account not found" @@ -146,7 +147,7 @@ jobs: fi # Check Azure OpenAI - OPENAI_NAME="oai-${PROJECT}-${ENV}" + OPENAI_NAME="${BASE}-oai" echo "Checking Azure OpenAI: $OPENAI_NAME..." if ! az cognitiveservices account show --name "$OPENAI_NAME" --resource-group "$RG" &>/dev/null; then echo "⚠️ Azure OpenAI not found (optional)" @@ -164,7 +165,7 @@ jobs: fi # Check Cosmos DB - COSMOS_NAME="cosmos-${PROJECT}-${ENV}" + COSMOS_NAME="${BASE}-cosmos" echo "Checking Cosmos DB: $COSMOS_NAME..." if ! az cosmosdb show --name "$COSMOS_NAME" --resource-group "$RG" &>/dev/null; then echo "⚠️ Cosmos DB not found (optional)" @@ -174,7 +175,7 @@ jobs: fi # Check Redis - REDIS_NAME="redis-${PROJECT}-${ENV}" + REDIS_NAME="${BASE}-redis" echo "Checking Redis Cache: $REDIS_NAME..." if ! az redis show --name "$REDIS_NAME" --resource-group "$RG" &>/dev/null; then echo "⚠️ Redis Cache not found (optional)" @@ -184,7 +185,7 @@ jobs: fi # Check App Insights - APPI_NAME="appi-${PROJECT}-${ENV}" + APPI_NAME="${BASE}-appi" echo "Checking Application Insights: $APPI_NAME..." if ! az monitor app-insights component show --app "$APPI_NAME" --resource-group "$RG" &>/dev/null; then echo "❌ Application Insights not found" @@ -194,8 +195,8 @@ jobs: echo "✅ Application Insights exists" fi - # Check Container Apps - CAE_NAME="cae-${PROJECT}-${ENV}" + # Check Container Apps (names follow ${BASE}-{cae,api} per infra/terraform/env/{env}/main.tf) + CAE_NAME="${BASE}-cae" echo "Checking Container Apps Environment: $CAE_NAME..." if ! az containerapp env show --name "$CAE_NAME" --resource-group "$RG" &>/dev/null; then echo "❌ Container Apps Environment not found" @@ -205,10 +206,10 @@ jobs: echo "✅ Container Apps Environment exists" # Check API app - CA_NAME="ca-${PROJECT}-${ENV}-api" + CA_NAME="${BASE}-api" if ! az containerapp show --name "$CA_NAME" --resource-group "$RG" &>/dev/null; then echo "❌ API Container App not found" - MISSING="${MISSING}container-app-api," + MISSING="${MISSING}container-apps-api," PASSED="false" else echo "✅ API Container App exists" diff --git a/.gitignore b/.gitignore index 2c1d477..0ec8db6 100644 --- a/.gitignore +++ b/.gitignore @@ -68,3 +68,4 @@ logs .tmp actionlint infra/bicep/*.json +**/.terraform/ diff --git a/apps/api/package.json b/apps/api/package.json index a0108ad..8eae26b 100644 --- a/apps/api/package.json +++ b/apps/api/package.json @@ -34,6 +34,7 @@ "jsonwebtoken": "^9.0.2", "morgan": "^1.10.1", "multer": "^2.0.2", + "pg": "^8.16.3", "puppeteer": "^21.0.0", "qrcode-terminal": "^0.12.0", "reflect-metadata": "^0.2.2", diff --git a/apps/api/src/config/database.ts b/apps/api/src/config/database.ts index 2e0aa63..dfa9e88 100644 --- a/apps/api/src/config/database.ts +++ b/apps/api/src/config/database.ts @@ -1,28 +1,59 @@ import { DataSource } from 'typeorm'; +import type { DataSourceOptions } from 'typeorm'; import { Message } from '../db/entities/Message'; import { Group } from '../db/entities/Group'; import { User } from '../db/entities/User'; import { logger } from '../utils/logger'; -export const AppDataSource = new DataSource({ - type: 'sqlite', - database: 'database.sqlite', - synchronize: process.env.NODE_ENV !== 'production', // Auto-create tables in dev/test - logging: process.env.NODE_ENV === 'development', +const isProduction = process.env.NODE_ENV === 'production'; +const dbType = process.env.DB_TYPE || 'sqlite'; +const migrationsRun = process.env.DB_MIGRATIONS_RUN === undefined + ? isProduction + : process.env.DB_MIGRATIONS_RUN === 'true'; + +const commonOptions = { + synchronize: !isProduction && process.env.DB_SYNCHRONIZE !== 'false', + logging: process.env.DB_LOGGING === 'true' || process.env.NODE_ENV === 'development', entities: [Message, Group, User], migrations: ['dist/db/migrations/*.js'], - migrationsRun: process.env.NODE_ENV === 'production', + migrationsRun, subscribers: [], cache: { - duration: 1000 * 30, // 30 seconds + duration: 1000 * 30, }, - // Enable WAL mode for better concurrency +}; + +const sqliteOptions: DataSourceOptions = { + ...commonOptions, + type: 'sqlite', + database: process.env.DATABASE_PATH || 'database.sqlite', extra: { connection: { pragma: 'journal_mode = WAL', }, }, -}); +}; + +const postgresOptions: DataSourceOptions = { + ...commonOptions, + type: 'postgres', + host: process.env.DB_HOST, + port: Number.parseInt(process.env.DB_PORT || '5432', 10), + username: process.env.DB_USERNAME, + password: process.env.DB_PASSWORD, + database: process.env.DB_NAME || 'convolens', + ssl: process.env.DB_SSL === 'true' ? { rejectUnauthorized: false } : false, +}; + +const getDataSourceOptions = (): DataSourceOptions => { + if (dbType === 'postgres') { + return postgresOptions; + } + + return sqliteOptions; +}; + +export const AppDataSource = new DataSource(getDataSourceOptions()); export const initializeDatabase = async (): Promise => { if (AppDataSource.isInitialized) { diff --git a/apps/api/src/config/datasource.ts b/apps/api/src/config/datasource.ts index 7c0d96a..5f91478 100644 --- a/apps/api/src/config/datasource.ts +++ b/apps/api/src/config/datasource.ts @@ -1,4 +1,5 @@ import { DataSource } from 'typeorm'; +import type { DataSourceOptions } from 'typeorm'; import { User } from '../db/entities/User'; import { Group } from '../db/entities/Group'; import { Message } from '../db/entities/Message'; @@ -9,28 +10,58 @@ import dotenv from 'dotenv'; dotenv.config(); const isProduction = process.env.NODE_ENV === 'production'; +const dbType = process.env.DB_TYPE || 'sqlite'; +const migrationsRun = process.env.DB_MIGRATIONS_RUN === undefined + ? isProduction + : process.env.DB_MIGRATIONS_RUN === 'true'; -// Database configuration -export const AppDataSource = new DataSource({ - type: 'sqlite', - database: process.env.DATABASE_PATH || 'database.sqlite', - synchronize: process.env.NODE_ENV !== 'production', // Auto-create tables in dev/test - logging: process.env.NODE_ENV === 'development' ? 'all' : ['error', 'warn'], +const commonOptions = { + synchronize: !isProduction && process.env.DB_SYNCHRONIZE !== 'false', + logging: process.env.DB_LOGGING === 'true' || process.env.NODE_ENV === 'development' + ? 'all' + : ['error', 'warn'], logger: isProduction ? 'file' : 'debug', entities: [User, Group, Message], migrations: ['dist/db/migrations/*.js'], - migrationsRun: process.env.NODE_ENV === 'production', + migrationsRun, migrationsTableName: 'migrations', cache: { - duration: 1000 * 30, // 30 seconds + duration: 1000 * 30, }, - // Enable WAL mode for better concurrency in SQLite +} satisfies Partial; + +const sqliteOptions: DataSourceOptions = { + ...commonOptions, + type: 'sqlite', + database: process.env.DATABASE_PATH || 'database.sqlite', extra: { connection: { pragma: 'journal_mode = WAL', }, }, -}); +}; + +const postgresOptions: DataSourceOptions = { + ...commonOptions, + type: 'postgres', + host: process.env.DB_HOST, + port: Number.parseInt(process.env.DB_PORT || '5432', 10), + username: process.env.DB_USERNAME, + password: process.env.DB_PASSWORD, + database: process.env.DB_NAME || 'convolens', + ssl: process.env.DB_SSL === 'true' ? { rejectUnauthorized: false } : false, +}; + +const getDataSourceOptions = (): DataSourceOptions => { + if (dbType === 'postgres') { + return postgresOptions; + } + + return sqliteOptions; +}; + +// Database configuration +export const AppDataSource = new DataSource(getDataSourceOptions()); // Initialize the data source export const initializeDataSource = async (): Promise => { diff --git a/apps/api/src/db/entities/Group.ts b/apps/api/src/db/entities/Group.ts index e1688d3..880de18 100644 --- a/apps/api/src/db/entities/Group.ts +++ b/apps/api/src/db/entities/Group.ts @@ -73,12 +73,12 @@ export class Group { @Column({ type: 'simple-json', nullable: true }) metadata?: GroupMetadata; - @CreateDateColumn({ type: 'datetime' }) + @CreateDateColumn() createdAt: Date; - @UpdateDateColumn({ type: 'datetime' }) + @UpdateDateColumn() updatedAt: Date; - @Column({ type: 'datetime', nullable: true }) + @Column({ nullable: true }) archivedAt?: Date; } diff --git a/apps/api/src/db/entities/Message.ts b/apps/api/src/db/entities/Message.ts index 95af676..b95b94d 100644 --- a/apps/api/src/db/entities/Message.ts +++ b/apps/api/src/db/entities/Message.ts @@ -61,12 +61,12 @@ export class Message { @Column({ type: 'uuid' }) groupId: string; - @CreateDateColumn({ type: 'datetime' }) + @CreateDateColumn() createdAt: Date; - @UpdateDateColumn({ type: 'datetime' }) + @UpdateDateColumn() updatedAt: Date; - @Column({ type: 'datetime', nullable: true }) + @Column({ nullable: true }) deletedAt?: Date; } diff --git a/apps/api/src/db/entities/User.ts b/apps/api/src/db/entities/User.ts index 034e3de..a2cb8b3 100644 --- a/apps/api/src/db/entities/User.ts +++ b/apps/api/src/db/entities/User.ts @@ -8,6 +8,7 @@ import { BeforeUpdate, OneToMany } from 'typeorm'; +import type { Relation } from 'typeorm'; import bcrypt from 'bcryptjs'; import { Group } from './Group'; import { Message } from './Message'; @@ -47,15 +48,18 @@ export class User { lastLogin?: Date; @OneToMany(() => Group, group => group.owner) - ownedGroups: Group[]; + ownedGroups: Relation; + + @OneToMany(() => Group, group => group.members) + groups: Relation; @OneToMany(() => Message, message => message.sender) - messages: Message[]; + messages: Relation; - @CreateDateColumn({ type: 'datetime' }) + @CreateDateColumn() createdAt: Date; - @UpdateDateColumn({ type: 'datetime' }) + @UpdateDateColumn() updatedAt: Date; @BeforeInsert() diff --git a/docs/AZURE_SETUP.md b/docs/AZURE_SETUP.md index 7848ce9..1f36aaf 100644 --- a/docs/AZURE_SETUP.md +++ b/docs/AZURE_SETUP.md @@ -393,11 +393,9 @@ az role assignment list --assignee # List federated credentials az ad app federated-credential list --id -# Test deployment (dry run) -az deployment group what-if \ - --resource-group rg-whatssummarize-dev \ - --template-file infra/bicep/main.bicep \ - --parameters infra/parameters/dev.bicepparam +# Test deployment (dry run) — Terraform +terraform -chdir=infra/terraform/env/dev init -backend-config=backend.hcl +terraform -chdir=infra/terraform/env/dev plan ``` ## Migration from Legacy Credentials diff --git a/infra/README.md b/infra/README.md index 19bc222..097e1a2 100644 --- a/infra/README.md +++ b/infra/README.md @@ -1,315 +1,182 @@ # ConvoLens Infrastructure -Azure infrastructure as code using Bicep templates for ConvoLens. +Azure infrastructure as code for ConvoLens, managed with **Terraform** (AzureRM provider). -> **Note:** Azure resource names (e.g. `rg-whatssummarize-*`, `kvwhatssummarizedev`, -> `cosmos-whatssummarize-dev`, `oai-whatssummarize-dev`) intentionally still -> reference the legacy project name — renaming live Azure resources is a -> separate, user-driven decision tracked in PR4 / cross-repo follow-ups. +> Migrated from Bicep on 2026-05-11. Naming follows the NL Azure Naming +> Standard with [ADR-0027](https://github.com/JustAGhosT/mystira-workspace/blob/main/docs/architecture/adr/0027-azure-resource-naming-convention.md) +> (no region suffix): `{org}-{env}-{project}-{type}` → +> `nl-dev-convolens-rg`, `nl-dev-convolens-kv`, etc. -## Overview - -This directory contains all infrastructure configuration for deploying ConvoLens to Azure: +## Layout ``` infra/ -├── bicep/ -│ ├── main.bicep # Main orchestration template -│ └── modules/ -│ ├── key-vault.bicep # Azure Key Vault -│ ├── storage.bicep # Azure Blob Storage -│ ├── openai.bicep # Azure OpenAI Service -│ ├── cosmos-db.bicep # Azure Cosmos DB -│ ├── redis.bicep # Azure Redis Cache -│ ├── app-insights.bicep # Application Insights -│ ├── container-apps.bicep# Azure Container Apps -│ └── static-web-app.bicep# Azure Static Web Apps -├── parameters/ -│ ├── dev.bicepparam # Development environment -│ ├── staging.bicepparam # Staging environment (create as needed) -│ └── prod.bicepparam # Production environment -└── scripts/ - ├── deploy.ps1 # PowerShell deployment (recommended) - ├── deploy.sh # Bash deployment script - ├── validate-resources.ps1 # PowerShell validation - └── validate-resources.sh # Bash validation script +├── terraform/ +│ ├── env/ +│ │ └── dev/ +│ │ ├── terraform.tf # Provider versions + remote backend +│ │ ├── main.tf # All resources (RG, KV, Storage, Cosmos, ACA, SWA, …) +│ │ ├── variables.tf # Input variables +│ │ ├── outputs.tf # Output values (endpoints, names, deploy tokens) +│ │ ├── backend.hcl # Backend init args (state location) +│ │ └── terraform.tfvars # Dev values (auto-loaded by Terraform) +│ └── .gitignore # .terraform/, *.tfstate, etc. +├── scripts/ # (empty — legacy bicep scripts removed) +└── README.md # This file ``` -## Azure Resources - -| Resource | Purpose | Required | -|----------|---------|----------| -| Resource Group | Container for all resources | Yes | -| Key Vault | Secrets management | Yes | -| Storage Account | Blob storage for exports | Yes | -| Azure OpenAI | AI summarization | No* | -| Cosmos DB | NoSQL database | No* | -| Redis Cache | Distributed caching | No* | -| Application Insights | Monitoring & logging | Yes | -| Container Apps | API hosting | Yes | -| Static Web Apps | Frontend hosting | Yes | - -*Optional but recommended for production. - -## Prerequisites - -1. **Azure CLI** installed and logged in - ```bash - az login - az account set --subscription "" - ``` - -2. **Bicep CLI** (usually included with Azure CLI) - ```bash - az bicep install - az bicep version - ``` +## Resources provisioned -3. **Required permissions**: - - Contributor access to the subscription/resource group - - Key Vault Administrator (for managing secrets) - - Cognitive Services Contributor (for OpenAI) +The `dev` environment ships these by default (toggle via `enable_*` variables): -4. **GitHub Actions Setup** (for CI/CD deployments): - - See [Azure Setup Guide](../docs/AZURE_SETUP.md) for configuring OIDC authentication - - Requires configuring three GitHub secrets: `AZURE_CLIENT_ID`, `AZURE_TENANT_ID`, `AZURE_SUBSCRIPTION_ID` +| Resource | Name | Notes | +|---|---|---| +| Resource Group | `nl-dev-convolens-rg` | All resources scoped to this RG | +| Log Analytics workspace | `nl-dev-convolens-law` | 30-day retention (90 in prod) | +| Application Insights | `nl-dev-convolens-appi` | Linked to LAW | +| Storage Account | `nldevconvolensst` | LRS, hot tier, blob soft-delete 7 days | +| Blob containers | `chat-exports`, `user-uploads`, `summaries` | Private access | +| Cosmos DB account | `nl-dev-convolens-cosmos` | Serverless, single region, no AZ redundancy | +| Cosmos SQL DB | `convolens` | 3 containers: `users` `/id`, `chats` `/userId`, `summaries` `/chatId` | +| Key Vault | `nl-dev-convolens-kv` | Access-policy mode, soft-delete 7 days | +| Container Apps Environment | `nl-dev-convolens-cae` | Consumption profile only | +| Container App | `nl-dev-convolens-api` | Placeholder helloworld image; the API CD workflow overrides `container_image_api` | +| Static Web App | `nl-dev-convolens-swa` | Free tier | +| KV secrets | `cosmos-db-endpoint`, `cosmos-db-key`, `cosmos-db-connection-string`, `storage-connection-string`, `appinsights-connection-string` | Created by Terraform after the deployer access policy lands | -## Quick Start +Off by default in dev (set the variable to enable): -### Deploy Development Environment (PowerShell - Recommended) - -```powershell -# From repository root -cd infra/scripts - -# Deploy (what-if runs automatically first, then prompts for confirmation) -./deploy.ps1 -Environment dev - -# Skip what-if and deploy directly -./deploy.ps1 -Environment dev -SkipWhatIf -Force - -# Only run what-if analysis -./deploy.ps1 -Environment dev -WhatIfOnly - -# Validate templates only -./deploy.ps1 -Environment dev -ValidateOnly -``` +| Variable | Resource | +|---|---| +| `enable_redis = true` | Azure Cache for Redis Basic C0 (~$15-30/mo) | +| `enable_openai = true` | Azure OpenAI account (Foundry is configured separately today) | +| `enable_budget_alerts = true` + non-empty `admin_email` | RG-scoped consumption budget with 80% actual / 100% forecasted alerts | -### Deploy Production Environment +## Remote state -```powershell -# Deploy to production (what-if + confirmation by default) -./deploy.ps1 -Environment prod +Terraform state lives in Azure Storage. **Bootstrap once per workspace** (see `Bootstrap` below); never recreate by hand. -# Preview production changes only -./deploy.ps1 -Environment prod -WhatIfOnly ``` - -### Bash Alternative - -```bash -./deploy.sh dev # Deploy to dev -./deploy.sh prod --what-if # Preview production changes -./deploy.sh staging --validate-only # Validate staging templates +Storage account: nltfstateconvolens +Resource group: nl-tfstate-rg +Container: tfstate +State key (dev): convolens-dev.tfstate ``` -## Deployment Options - -### Using Scripts (Recommended) - -```powershell -# PowerShell (what-if is automatic) -./scripts/deploy.ps1 -Environment # Full deploy with what-if -./scripts/deploy.ps1 -Environment -WhatIfOnly # Preview only -./scripts/deploy.ps1 -Environment -ValidateOnly # Validate templates -./scripts/validate-resources.ps1 -Environment # Validate resources -``` +Backend config is in `infra/terraform/env/dev/backend.hcl`. Passed to `init` via `-backend-config=backend.hcl`. The default `azurerm` backend uses access keys; local users and the GitHub Actions SP (`convolens-sp`) both need Contributor on the tfstate storage account, which their subscription-scoped Contributor already grants. -```bash -# Bash alternative -./scripts/deploy.sh # Deploy -./scripts/deploy.sh --what-if # Preview -./scripts/deploy.sh --validate-only # Validate -./scripts/validate-resources.sh # Validate resources -``` +## Deploying -### Using Azure CLI Directly +### Locally ```bash -# Create resource group -az group create \ - --name rg-whatssummarize-dev \ - --location eastus - -# Deploy -az deployment group create \ - --resource-group rg-whatssummarize-dev \ - --template-file bicep/main.bicep \ - --parameters parameters/dev.bicepparam +cd infra/terraform/env/dev +terraform init -backend-config=backend.hcl +terraform plan +terraform apply ``` -### Using GitHub Actions - -1. **Configure Azure credentials** (see [Azure Setup Guide](../docs/AZURE_SETUP.md) for detailed instructions): - - Create Azure AD application (service principal) - - Configure federated credentials for OIDC - - Set up GitHub repository secrets: - - `AZURE_CLIENT_ID` - Service principal client ID - - `AZURE_TENANT_ID` - Azure AD tenant ID - - `AZURE_SUBSCRIPTION_ID` - Azure subscription ID +You need to be logged into Azure with permissions to manage `nl-dev-convolens-rg` and read `nl-tfstate-rg/nltfstateconvolens` keys. -2. **Trigger deployment**: - - Push to `main` branch (auto-deploys to dev) - - Use workflow dispatch for staging/prod - - PRs trigger what-if analysis +### Via CI -## Configuration +The `Infrastructure` GitHub Actions workflow (`.github/workflows/infrastructure.yml`) runs: -### Environment Parameters +- **Validate** — `terraform fmt -check` + `terraform validate` on every push and PR +- **Plan** — `terraform plan` with a posted PR comment on pull requests +- **Apply** — `terraform apply` on push to `main` (or `workflow_dispatch` → `apply`) +- **Verify** — health-checks the Container App URL after apply -Edit `parameters/.bicepparam` to customize: +Auth is OpenID Connect via the AAD app `convolens-sp` and the federated credential for `repo:neuralliquid/convolens:environment:dev`. The `AZURE_CREDENTIALS` env-scoped secret (env `dev`) provides `clientId`/`tenantId`/`subscriptionId` for the workflow. -```bicep -param environment = 'dev' -param projectName = 'whatssummarize' -param location = 'eastus' +## Bootstrap (new tenant / first time) -// Enable/disable optional resources -param enableOpenAI = true -param enableCosmosDB = true -param enableRedis = true +These steps were done for the active tenant (`9530cd32-9e33-47f0-9247-ed964730b580`, sub `bb4e3882-2079-4bab-8974-611bc0b8bb58`) on 2026-05-10. Keep here as a reference for future environments or tenant moves: -// OpenAI model deployments -param openAIDeployments = [ - { name: 'gpt-4', model: 'gpt-4', version: '0613', capacity: 10 } -] +```bash +# 1. AAD app + service principal + federated cred +az ad app create --display-name convolens-sp +APP_OBJECT_ID=$(az ad app list --display-name convolens-sp --query '[0].id' -o tsv) +APP_CLIENT_ID=$(az ad app list --display-name convolens-sp --query '[0].appId' -o tsv) +az ad sp create --id "$APP_CLIENT_ID" +SP_OBJECT_ID=$(az ad sp list --display-name convolens-sp --query '[0].id' -o tsv) + +az ad app federated-credential create --id "$APP_OBJECT_ID" --parameters '{ + "name": "github-actions-dev", + "issuer": "https://token.actions.githubusercontent.com", + "subject": "repo:neuralliquid/convolens:environment:dev", + "audiences": ["api://AzureADTokenExchange"] +}' + +# 2. Sub-scoped Contributor (lets the SP create RGs and provision resources) +az role assignment create \ + --assignee-object-id "$SP_OBJECT_ID" \ + --assignee-principal-type ServicePrincipal \ + --role Contributor \ + --scope /subscriptions/ + +# 3. Remote-state storage (one-time) +az group create --name nl-tfstate-rg --location eastus2 +az storage account create \ + --name nltfstateconvolens \ + --resource-group nl-tfstate-rg \ + --location eastus2 \ + --sku Standard_LRS \ + --kind StorageV2 \ + --min-tls-version TLS1_2 \ + --allow-blob-public-access false +SA_KEY=$(az storage account keys list --account-name nltfstateconvolens --resource-group nl-tfstate-rg --query "[0].value" -o tsv) +az storage container create --name tfstate --account-name nltfstateconvolens --account-key "$SA_KEY" + +# 4. Set the env-scoped GitHub secret +gh secret set AZURE_CREDENTIALS --env dev --repo neuralliquid/convolens --body "$(jq -nc \ + --arg c "$APP_CLIENT_ID" \ + --arg t "" \ + --arg s "" \ + '{clientId:$c, tenantId:$t, subscriptionId:$s}')" ``` -### Resource Naming Convention - -Resources follow this naming pattern: -- `--` - -Examples: -- `rg-whatssummarize-dev` (Resource Group) -- `kvwhatssummarizedev` (Key Vault - no hyphens) -- `cosmos-whatssummarize-dev` (Cosmos DB) +## Adding staging or prod -## CI/CD Workflows +1. Copy `infra/terraform/env/dev/` to `infra/terraform/env/{env}/`. +2. Update `terraform.tfvars` (env name, location, sizing flags, admin email). +3. Update `backend.hcl` → `key = "convolens-{env}.tfstate"`. +4. Create a federated credential matching `repo:neuralliquid/convolens:environment:{env}` on the AAD app. +5. Create the GitHub Environment `{env}` and set its `AZURE_CREDENTIALS` secret. +6. PR + merge — the workflow's `apply` job picks up the new environment. -### Infrastructure Workflow +## Cost (dev, today) -Triggered by: -- Push to `main` (auto-deploy to dev) -- Pull requests (what-if analysis) -- Manual dispatch (any environment) +Rough monthly run-rate at idle: -### Release Validation +- Cosmos DB serverless: $0 baseline; pay-per-request (RU-based). Empty DB ~free. +- Static Web App Free: $0. +- Storage Account LRS, hot tier, empty: <$1. +- App Insights + LAW: $0 until you ingest meaningful telemetry (per-GB pricing). +- Container Apps Consumption + 0 min replicas: $0 when idle. +- Key Vault Standard: ~$0.03 per 10k operations. -Runs before releases to verify: -- All required Azure resources exist -- Secrets are configured in Key Vault -- Application builds successfully -- Tests pass +Expect <$5/mo at idle. The big drivers when active: Container Apps execution time, Cosmos RU consumption, and App Insights ingestion. -## Post-Deployment +If you flip `enable_redis = true`, add ~$15-30/mo for Basic C0. `enable_openai = true` is metered separately by token usage. -After deployment, the script generates `.env.azure` with: +## Drift / state recovery -```bash -# Azure Provider -AZURE_PROVIDER_ENABLED=true - -# Azure OpenAI -AZURE_OPENAI_ENDPOINT=https://oai-whatssummarize-dev.openai.azure.com -AZURE_OPENAI_DEPLOYMENT=gpt-4 - -# Azure Cosmos DB -AZURE_COSMOS_ENDPOINT=https://cosmos-whatssummarize-dev.documents.azure.com - -# etc... -``` - -Secrets are stored in Key Vault and should be retrieved at runtime. - -## Validation - -Validate resources exist before release: +If `terraform apply` errors mid-deploy and a resource exists in Azure but is missing from state: ```bash -# Run validation script -./scripts/validate-resources.sh prod - -# Or use strict mode (fails on any missing required resource) -./scripts/validate-resources.sh prod --strict +cd infra/terraform/env/dev +MSYS_NO_PATHCONV=1 terraform import +terraform apply ``` -## Cost Optimization - -### Development -- Uses serverless Cosmos DB -- Basic Redis tier -- Minimal OpenAI capacity - -### Production -- Standard Redis tier -- Higher OpenAI capacity -- Zone redundancy recommended - -### Cost Estimates (approximate) +Common cause: transient ARM API hiccup between Azure responding and Terraform receiving the response. The resource shows up in `az resource list` but `terraform state list` is missing it. Import to recover. -| Environment | Monthly Cost | -|-------------|--------------| -| Dev | ~$50-100 | -| Staging | ~$100-200 | -| Production | ~$300-500+ | +## Removing the dev environment -*Costs vary based on usage. OpenAI costs scale with token usage.* - -## Troubleshooting - -### Deployment Fails - -1. Check Azure CLI login: - ```bash - az account show - ``` - -2. Validate templates: - ```bash - az bicep build --file bicep/main.bicep - ``` - -3. Check resource quotas: - ```bash - az vm list-usage --location eastus - ``` - -### OpenAI Not Available - -Azure OpenAI has limited regional availability. Update `location` parameter if needed. - -### Key Vault Access - -Ensure your identity has Key Vault access: ```bash -az keyvault set-policy \ - --name kvwhatssummarizedev \ - --upn your@email.com \ - --secret-permissions get list set delete +cd infra/terraform/env/dev +terraform destroy ``` -## Security Considerations - -- Secrets stored in Key Vault, never in code -- Managed identities for service-to-service auth -- HTTPS enforced everywhere -- Private endpoints available for production -- Regular key rotation recommended - -## Related Documentation - -- [Azure Bicep](https://docs.microsoft.com/azure/azure-resource-manager/bicep/) -- [Azure OpenAI](https://docs.microsoft.com/azure/cognitive-services/openai/) -- [Azure Container Apps](https://docs.microsoft.com/azure/container-apps/) -- [Azure Static Web Apps](https://docs.microsoft.com/azure/static-web-apps/) +This tears down everything in `nl-dev-convolens-rg`. The Key Vault is soft-deleted (7-day window in dev); the provider's `purge_soft_delete_on_destroy = true` setting purges it cleanly. diff --git a/infra/bicep/main.bicep b/infra/bicep/main.bicep deleted file mode 100644 index f66f211..0000000 --- a/infra/bicep/main.bicep +++ /dev/null @@ -1,271 +0,0 @@ -// ============================================================================= -// ConvoLens - Azure Infrastructure (Main Template) -// ============================================================================= -// This template deploys all Azure resources required for ConvoLens. -// Use with appropriate parameter files for each environment. -// -// Resources deployed: -// - Azure OpenAI Service (AI Foundry) -// - Azure Cosmos DB (NoSQL database) -// - Azure Blob Storage (file storage) -// - Azure Redis Cache (distributed caching) -// - Azure AD B2C (authentication) - requires manual setup -// - Azure Application Insights (monitoring) -// - Azure Key Vault (secrets management) -// - Azure Container Apps (API hosting) -// - Azure Static Web Apps (frontend hosting) -// - Azure Budget Alerts (cost management) -// ============================================================================= - -targetScope = 'resourceGroup' - -// ============================================================================= -// Parameters -// ============================================================================= - -@description('Environment name (dev, staging, prod)') -@allowed(['dev', 'staging', 'prod']) -param environment string = 'dev' - -@description('Azure region for resources') -param location string = resourceGroup().location - -@description('Project name used for resource naming. Resource names retain the legacy "whatssummarize" prefix until a separate user-driven Azure rename decision lands.') -@minLength(3) -@maxLength(15) -param projectName string = 'whatssummarize' - -@description('Logical database name used for Cosmos DB (and future Azure SQL). Decoupled from projectName so the brand rename can land without renaming Azure resources. Override at deploy time only when migrating data.') -@minLength(3) -@maxLength(40) -param databaseName string = 'convolens' - -@description('Enable Azure OpenAI deployment') -param enableOpenAI bool = true - -@description('Enable Cosmos DB deployment') -param enableCosmosDB bool = true - -@description('Enable Redis Cache deployment') -param enableRedis bool = true - -@description('Enable Container Apps deployment') -param enableContainerApps bool = true - -@description('Enable Static Web Apps deployment') -param enableStaticWebApps bool = true - -@description('Administrator email for alerts') -param adminEmail string = '' - -@description('Enable budget alerts') -param enableBudgetAlerts bool = true - -@description('Monthly budget amount in USD') -param monthlyBudgetAmount int = environment == 'prod' ? 500 : environment == 'staging' ? 200 : 100 - -@description('OpenAI model deployments') -param openAIDeployments array = [ - { - name: 'gpt-4' - model: 'gpt-4' - version: '0613' - capacity: 10 - } - { - name: 'gpt-35-turbo' - model: 'gpt-35-turbo' - version: '0613' - capacity: 30 - } -] - -@description('Tags to apply to all resources') -param tags object = { - project: projectName - environment: environment - managedBy: 'bicep' -} - -// ============================================================================= -// Variables -// ============================================================================= - -var resourcePrefix = '${projectName}-${environment}' -var resourcePrefixClean = replace(resourcePrefix, '-', '') - -// SKUs based on environment -// Redis: Basic (no SLA) for dev, Standard (SLA) for prod -var redisSku = environment == 'prod' ? 'Standard' : 'Basic' -var redisFamily = environment == 'prod' ? 'C' : 'C' -var redisCapacity = environment == 'prod' ? 1 : 0 - -// Storage: Standard for dev, Premium for prod (optional) -var storageSkuName = environment == 'prod' ? 'Standard_GRS' : 'Standard_LRS' - -// Container Apps: Production uses more resources -var containerAppCpu = environment == 'prod' ? '1.0' : '0.5' -var containerAppMemory = environment == 'prod' ? '2.0Gi' : '1.0Gi' -var containerAppMinReplicas = environment == 'prod' ? 2 : 0 -var containerAppMaxReplicas = environment == 'prod' ? 10 : 3 - -// ============================================================================= -// Modules -// ============================================================================= - -// Key Vault (deployed first for secret management) -module keyVault 'modules/key-vault.bicep' = { - name: 'keyVault-${environment}' - params: { - name: 'kv-${resourcePrefixClean}' - location: location - tags: tags - enableSoftDelete: environment == 'prod' - enablePurgeProtection: environment == 'prod' - } -} - -// Storage Account -module storage 'modules/storage.bicep' = { - name: 'storage-${environment}' - params: { - name: 'st${resourcePrefixClean}' - location: location - tags: tags - sku: storageSkuName - containerNames: ['chat-exports', 'user-uploads', 'summaries'] - enableVersioning: environment == 'prod' - } -} - -// Azure OpenAI -module openAI 'modules/openai.bicep' = if (enableOpenAI) { - name: 'openai-${environment}' - params: { - name: 'oai-${resourcePrefix}' - location: location // Note: OpenAI has limited region availability - tags: tags - deployments: openAIDeployments - keyVaultName: keyVault.outputs.name - } -} - -// Cosmos DB -module cosmosDB 'modules/cosmos-db.bicep' = if (enableCosmosDB) { - name: 'cosmosdb-${environment}' - params: { - name: 'cosmos-${resourcePrefix}' - location: location - tags: tags - databaseName: databaseName - containers: [ - { - name: 'users' - partitionKey: '/id' - } - { - name: 'chats' - partitionKey: '/userId' - } - { - name: 'summaries' - partitionKey: '/chatId' - } - ] - keyVaultName: keyVault.outputs.name - } -} - -// Redis Cache -module redis 'modules/redis.bicep' = if (enableRedis) { - name: 'redis-${environment}' - params: { - name: 'redis-${resourcePrefix}' - location: location - tags: tags - sku: redisSku - family: redisFamily - capacity: redisCapacity - keyVaultName: keyVault.outputs.name - } -} - -// Application Insights -module appInsights 'modules/app-insights.bicep' = { - name: 'appinsights-${environment}' - params: { - name: 'appi-${resourcePrefix}' - location: location - tags: tags - } -} - -// Container Apps Environment & API -module containerApps 'modules/container-apps.bicep' = if (enableContainerApps) { - name: 'containerapps-${environment}' - params: { - environmentName: 'cae-${resourcePrefix}' - apiAppName: 'ca-${resourcePrefix}-api' - location: location - tags: tags - appInsightsConnectionString: appInsights.outputs.connectionString - keyVaultName: keyVault.outputs.name - cosmosDbEndpoint: enableCosmosDB ? cosmosDB.outputs.endpoint : '' - redisHostname: enableRedis ? redis.outputs.hostname : '' - storageAccountName: storage.outputs.name - openAIEndpoint: enableOpenAI ? openAI.outputs.endpoint : '' - // Environment-specific scaling - apiCpu: containerAppCpu - apiMemory: containerAppMemory - minReplicas: containerAppMinReplicas - maxReplicas: containerAppMaxReplicas - } -} - -// Static Web Apps (Frontend) -module staticWebApp 'modules/static-web-app.bicep' = if (enableStaticWebApps) { - name: 'staticwebapp-${environment}' - params: { - name: 'stapp-${resourcePrefix}' - location: location - tags: tags - apiUrl: enableContainerApps ? containerApps.outputs.apiUrl : '' - } -} - -// Budget Alerts (Cost Management) -module budgetAlerts 'modules/budget-alerts.bicep' = if (enableBudgetAlerts && !empty(adminEmail)) { - name: 'budget-${environment}' - params: { - name: 'budget-${resourcePrefix}' - amount: monthlyBudgetAmount - contactEmails: [adminEmail] - environment: environment - tags: tags - } -} - -// ============================================================================= -// Outputs -// ============================================================================= - -output resourceGroupName string = resourceGroup().name -output keyVaultName string = keyVault.outputs.name -output keyVaultUri string = keyVault.outputs.uri -output storageAccountName string = storage.outputs.name -output storageBlobEndpoint string = storage.outputs.blobEndpoint - -output openAIEndpoint string = enableOpenAI ? openAI.outputs.endpoint : '' -output openAIDeploymentNames array = enableOpenAI ? openAI.outputs.deploymentNames : [] - -output cosmosDBEndpoint string = enableCosmosDB ? cosmosDB.outputs.endpoint : '' -output cosmosDBDatabaseName string = enableCosmosDB ? cosmosDB.outputs.databaseName : '' - -output redisHostname string = enableRedis ? redis.outputs.hostname : '' -output redisPort int = enableRedis ? redis.outputs.port : 0 - -output appInsightsConnectionString string = appInsights.outputs.connectionString -output appInsightsInstrumentationKey string = appInsights.outputs.instrumentationKey - -output containerAppsApiUrl string = enableContainerApps ? containerApps.outputs.apiUrl : '' -output staticWebAppUrl string = enableStaticWebApps ? staticWebApp.outputs.url : '' diff --git a/infra/bicep/modules/app-insights.bicep b/infra/bicep/modules/app-insights.bicep deleted file mode 100644 index 24d3a64..0000000 --- a/infra/bicep/modules/app-insights.bicep +++ /dev/null @@ -1,82 +0,0 @@ -// ============================================================================= -// Azure Application Insights Module -// ============================================================================= - -@description('Name of the Application Insights resource') -param name string - -@description('Azure region') -param location string - -@description('Resource tags') -param tags object - -@description('Application type') -@allowed(['web', 'other']) -param applicationType string = 'web' - -@description('Retention in days') -@minValue(30) -@maxValue(730) -param retentionInDays int = 90 - -@description('Daily cap in GB (0 = no cap)') -param dailyCapGb int = 0 - -// ============================================================================= -// Resources -// ============================================================================= - -resource logAnalyticsWorkspace 'Microsoft.OperationalInsights/workspaces@2022-10-01' = { - name: '${name}-workspace' - location: location - tags: tags - properties: { - sku: { - name: 'PerGB2018' - } - retentionInDays: retentionInDays - publicNetworkAccessForIngestion: 'Enabled' - publicNetworkAccessForQuery: 'Enabled' - } -} - -resource appInsights 'Microsoft.Insights/components@2020-02-02' = { - name: name - location: location - tags: tags - kind: applicationType - properties: { - Application_Type: applicationType - WorkspaceResourceId: logAnalyticsWorkspace.id - RetentionInDays: retentionInDays - IngestionMode: 'LogAnalytics' - publicNetworkAccessForIngestion: 'Enabled' - publicNetworkAccessForQuery: 'Enabled' - } -} - -// Set daily cap if specified -resource dailyCap 'Microsoft.Insights/components/CurrentBillingFeatures@2015-05-01' = if (dailyCapGb > 0) { - name: '${appInsights.name}/CurrentBillingFeatures' - properties: { - CurrentBillingFeatures: ['Basic'] - DataVolumeCap: { - Cap: dailyCapGb - ResetTime: 0 - StopSendNotificationWhenHitCap: false - StopSendNotificationWhenHitThreshold: false - WarningThreshold: 90 - } - } -} - -// ============================================================================= -// Outputs -// ============================================================================= - -output id string = appInsights.id -output name string = appInsights.name -output instrumentationKey string = appInsights.properties.InstrumentationKey -output connectionString string = appInsights.properties.ConnectionString -output workspaceId string = logAnalyticsWorkspace.id diff --git a/infra/bicep/modules/budget-alerts.bicep b/infra/bicep/modules/budget-alerts.bicep deleted file mode 100644 index bdcb85e..0000000 --- a/infra/bicep/modules/budget-alerts.bicep +++ /dev/null @@ -1,140 +0,0 @@ -// ============================================================================= -// Budget Alerts Module -// ============================================================================= -// Creates Azure Cost Management budgets with alerts for cost monitoring. -// Sends notifications when spending approaches or exceeds thresholds. -// ============================================================================= - -@description('Name of the budget') -param name string - -@description('Budget amount in USD') -param amount int - -@description('Email addresses to receive alerts') -param contactEmails array - -@description('Resource group name for scope') -param resourceGroupName string = resourceGroup().name - -@description('Environment (dev, staging, prod)') -@allowed(['dev', 'staging', 'prod']) -param environment string - -@description('Tags to apply to the budget') -param tags object = {} - -@description('Budget time grain (Monthly, Quarterly, Annually)') -@allowed(['Monthly', 'Quarterly', 'Annually']) -param timeGrain string = 'Monthly' - -@description('Current date for budget calculation (passed from parent to ensure consistency)') -param currentDate string = utcNow('yyyy-MM-01') - -// Derive startDate from the single currentDate value to avoid race conditions -var startDate = currentDate - -// ============================================================================= -// Variables -// ============================================================================= - -// Environment-specific thresholds -var thresholds = environment == 'prod' ? { - warning: 50 - alert: 75 - critical: 90 - exceeded: 100 -} : environment == 'staging' ? { - warning: 60 - alert: 80 - critical: 95 - exceeded: 100 -} : { - warning: 70 - alert: 85 - critical: 100 - exceeded: 110 -} - -// ============================================================================= -// Resources -// ============================================================================= - -resource budget 'Microsoft.Consumption/budgets@2023-05-01' = { - name: name - properties: { - category: 'Cost' - amount: amount - timeGrain: timeGrain - timePeriod: { - startDate: startDate - } - filter: { - dimensions: { - name: 'ResourceGroupName' - operator: 'In' - values: [resourceGroupName] - } - } - notifications: { - // Warning notification (50-70% depending on env) - warningNotification: { - enabled: true - operator: 'GreaterThan' - threshold: thresholds.warning - thresholdType: 'Actual' - contactEmails: contactEmails - contactRoles: ['Owner', 'Contributor'] - locale: 'en-us' - } - // Alert notification (75-85% depending on env) - alertNotification: { - enabled: true - operator: 'GreaterThan' - threshold: thresholds.alert - thresholdType: 'Actual' - contactEmails: contactEmails - contactRoles: ['Owner', 'Contributor'] - locale: 'en-us' - } - // Critical notification (90-100% depending on env) - criticalNotification: { - enabled: true - operator: 'GreaterThan' - threshold: thresholds.critical - thresholdType: 'Actual' - contactEmails: contactEmails - contactRoles: ['Owner'] - locale: 'en-us' - } - // Exceeded notification (100-110% depending on env) - exceededNotification: { - enabled: true - operator: 'GreaterThan' - threshold: thresholds.exceeded - thresholdType: 'Actual' - contactEmails: contactEmails - contactRoles: ['Owner'] - locale: 'en-us' - } - // Forecasted threshold notification - forecastedNotification: { - enabled: true - operator: 'GreaterThan' - threshold: 100 - thresholdType: 'Forecasted' - contactEmails: contactEmails - contactRoles: ['Owner', 'Contributor'] - locale: 'en-us' - } - } - } -} - -// ============================================================================= -// Outputs -// ============================================================================= - -output budgetName string = budget.name -output budgetAmount int = amount -output thresholds object = thresholds diff --git a/infra/bicep/modules/container-apps.bicep b/infra/bicep/modules/container-apps.bicep deleted file mode 100644 index 2e53d0e..0000000 --- a/infra/bicep/modules/container-apps.bicep +++ /dev/null @@ -1,189 +0,0 @@ -// ============================================================================= -// Azure Container Apps Module -// ============================================================================= - -@description('Name of the Container Apps Environment') -param environmentName string - -@description('Name of the API Container App') -param apiAppName string - -@description('Azure region') -param location string - -@description('Resource tags') -param tags object - -@description('Application Insights connection string') -param appInsightsConnectionString string = '' - -@description('Key Vault name for secrets') -param keyVaultName string = '' - -@description('Cosmos DB endpoint') -param cosmosDbEndpoint string = '' - -@description('Redis hostname') -param redisHostname string = '' - -@description('Storage account name') -param storageAccountName string = '' - -@description('Azure OpenAI endpoint') -param openAIEndpoint string = '' - -@description('Container image for API') -param apiImage string = 'mcr.microsoft.com/azuredocs/containerapps-helloworld:latest' - -@description('CPU cores for API container') -param apiCpu string = '0.5' - -@description('Memory for API container') -param apiMemory string = '1Gi' - -@description('Minimum replicas') -param minReplicas int = 0 - -@description('Maximum replicas') -param maxReplicas int = 3 - -// ============================================================================= -// Resources -// ============================================================================= - -resource environment 'Microsoft.App/managedEnvironments@2023-05-01' = { - name: environmentName - location: location - tags: tags - properties: { - daprAIConnectionString: appInsightsConnectionString - zoneRedundant: false - workloadProfiles: [ - { - name: 'Consumption' - workloadProfileType: 'Consumption' - } - ] - } -} - -resource keyVault 'Microsoft.KeyVault/vaults@2023-07-01' existing = if (!empty(keyVaultName)) { - name: keyVaultName -} - -resource apiApp 'Microsoft.App/containerApps@2023-05-01' = { - name: apiAppName - location: location - tags: tags - identity: { - type: 'SystemAssigned' - } - properties: { - managedEnvironmentId: environment.id - workloadProfileName: 'Consumption' - configuration: { - activeRevisionsMode: 'Single' - ingress: { - external: true - targetPort: 3001 - transport: 'http' - corsPolicy: { - allowedOrigins: ['*'] - allowedMethods: ['GET', 'POST', 'PUT', 'DELETE', 'OPTIONS'] - allowedHeaders: ['*'] - } - } - secrets: [ - { - name: 'app-insights-connection-string' - value: appInsightsConnectionString - } - ] - } - template: { - containers: [ - { - name: 'api' - image: apiImage - resources: { - cpu: json(apiCpu) - memory: apiMemory - } - env: [ - { - name: 'NODE_ENV' - value: 'production' - } - { - name: 'PORT' - value: '3001' - } - { - name: 'APPLICATIONINSIGHTS_CONNECTION_STRING' - secretRef: 'app-insights-connection-string' - } - { - name: 'AZURE_COSMOS_ENDPOINT' - value: cosmosDbEndpoint - } - { - name: 'AZURE_REDIS_HOSTNAME' - value: redisHostname - } - { - name: 'AZURE_STORAGE_ACCOUNT_NAME' - value: storageAccountName - } - { - name: 'AZURE_OPENAI_ENDPOINT' - value: openAIEndpoint - } - { - name: 'AZURE_KEY_VAULT_NAME' - value: keyVaultName - } - ] - } - ] - scale: { - minReplicas: minReplicas - maxReplicas: maxReplicas - rules: [ - { - name: 'http-rule' - http: { - metadata: { - concurrentRequests: '100' - } - } - } - ] - } - } - } -} - -// Grant Key Vault access to the Container App -resource keyVaultAccessPolicy 'Microsoft.KeyVault/vaults/accessPolicies@2023-07-01' = if (!empty(keyVaultName)) { - name: '${keyVaultName}/add' - properties: { - accessPolicies: [ - { - tenantId: subscription().tenantId - objectId: apiApp.identity.principalId - permissions: { - secrets: ['get', 'list'] - } - } - ] - } -} - -// ============================================================================= -// Outputs -// ============================================================================= - -output environmentId string = environment.id -output apiAppId string = apiApp.id -output apiUrl string = 'https://${apiApp.properties.configuration.ingress.fqdn}' -output apiPrincipalId string = apiApp.identity.principalId diff --git a/infra/bicep/modules/cosmos-db.bicep b/infra/bicep/modules/cosmos-db.bicep deleted file mode 100644 index 8e8db10..0000000 --- a/infra/bicep/modules/cosmos-db.bicep +++ /dev/null @@ -1,137 +0,0 @@ -// ============================================================================= -// Azure Cosmos DB Module -// ============================================================================= - -@description('Name of the Cosmos DB account') -param name string - -@description('Azure region') -param location string - -@description('Resource tags') -param tags object - -@description('Database name') -param databaseName string - -@description('Container configurations') -param containers array = [] - -@description('Key Vault name for storing secrets') -param keyVaultName string = '' - -@description('Enable free tier (only one per subscription)') -param enableFreeTier bool = false - -@description('Default consistency level') -@allowed(['Eventual', 'ConsistentPrefix', 'Session', 'BoundedStaleness', 'Strong']) -param consistencyLevel string = 'Session' - -// ============================================================================= -// Resources -// ============================================================================= - -resource cosmosAccount 'Microsoft.DocumentDB/databaseAccounts@2023-11-15' = { - name: name - location: location - tags: tags - kind: 'GlobalDocumentDB' - properties: { - databaseAccountOfferType: 'Standard' - enableFreeTier: enableFreeTier - consistencyPolicy: { - defaultConsistencyLevel: consistencyLevel - } - locations: [ - { - locationName: location - failoverPriority: 0 - isZoneRedundant: false - } - ] - capabilities: [ - { - name: 'EnableServerless' - } - ] - publicNetworkAccess: 'Enabled' - enableAutomaticFailover: false - enableMultipleWriteLocations: false - } -} - -resource database 'Microsoft.DocumentDB/databaseAccounts/sqlDatabases@2023-11-15' = { - parent: cosmosAccount - name: databaseName - properties: { - resource: { - id: databaseName - } - } -} - -resource cosmosContainers 'Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers@2023-11-15' = [for container in containers: { - parent: database - name: container.name - properties: { - resource: { - id: container.name - partitionKey: { - paths: [container.partitionKey] - kind: 'Hash' - } - indexingPolicy: { - indexingMode: 'consistent' - automatic: true - includedPaths: [ - { - path: '/*' - } - ] - excludedPaths: [ - { - path: '/"_etag"/?' - } - ] - } - } - } -}] - -// Store connection info in Key Vault -resource keyVault 'Microsoft.KeyVault/vaults@2023-07-01' existing = if (!empty(keyVaultName)) { - name: keyVaultName -} - -resource cosmosKeySecret 'Microsoft.KeyVault/vaults/secrets@2023-07-01' = if (!empty(keyVaultName)) { - parent: keyVault - name: 'cosmos-db-key' - properties: { - value: cosmosAccount.listKeys().primaryMasterKey - } -} - -resource cosmosEndpointSecret 'Microsoft.KeyVault/vaults/secrets@2023-07-01' = if (!empty(keyVaultName)) { - parent: keyVault - name: 'cosmos-db-endpoint' - properties: { - value: cosmosAccount.properties.documentEndpoint - } -} - -resource cosmosConnectionStringSecret 'Microsoft.KeyVault/vaults/secrets@2023-07-01' = if (!empty(keyVaultName)) { - parent: keyVault - name: 'cosmos-db-connection-string' - properties: { - value: cosmosAccount.listConnectionStrings().connectionStrings[0].connectionString - } -} - -// ============================================================================= -// Outputs -// ============================================================================= - -output id string = cosmosAccount.id -output name string = cosmosAccount.name -output endpoint string = cosmosAccount.properties.documentEndpoint -output databaseName string = database.name diff --git a/infra/bicep/modules/key-vault.bicep b/infra/bicep/modules/key-vault.bicep deleted file mode 100644 index 99e06b1..0000000 --- a/infra/bicep/modules/key-vault.bicep +++ /dev/null @@ -1,58 +0,0 @@ -// ============================================================================= -// Azure Key Vault Module -// ============================================================================= - -@description('Name of the Key Vault') -param name string - -@description('Azure region') -param location string - -@description('Resource tags') -param tags object - -@description('Enable soft delete') -param enableSoftDelete bool = true - -@description('Enable purge protection') -param enablePurgeProtection bool = false - -@description('Soft delete retention in days') -param softDeleteRetentionInDays int = 7 - -// ============================================================================= -// Resources -// ============================================================================= - -resource keyVault 'Microsoft.KeyVault/vaults@2023-07-01' = { - name: name - location: location - tags: tags - properties: { - sku: { - family: 'A' - name: 'standard' - } - tenantId: subscription().tenantId - enabledForDeployment: true - enabledForDiskEncryption: false - enabledForTemplateDeployment: true - enableSoftDelete: enableSoftDelete - enablePurgeProtection: enablePurgeProtection ? true : null - softDeleteRetentionInDays: enableSoftDelete ? softDeleteRetentionInDays : null - enableRbacAuthorization: true - publicNetworkAccess: 'Enabled' - networkAcls: { - bypass: 'AzureServices' - defaultAction: 'Allow' - } - } -} - -// ============================================================================= -// Outputs -// ============================================================================= - -output id string = keyVault.id -output name string = keyVault.name -output uri string = keyVault.properties.vaultUri diff --git a/infra/bicep/modules/openai.bicep b/infra/bicep/modules/openai.bicep deleted file mode 100644 index 97cbd57..0000000 --- a/infra/bicep/modules/openai.bicep +++ /dev/null @@ -1,91 +0,0 @@ -// ============================================================================= -// Azure OpenAI Service Module -// ============================================================================= - -@description('Name of the Azure OpenAI resource') -param name string - -@description('Azure region (Note: Limited availability)') -param location string - -@description('Resource tags') -param tags object - -@description('Model deployments configuration') -param deployments array = [] - -@description('Key Vault name for storing secrets') -param keyVaultName string = '' - -@description('SKU name') -@allowed(['S0']) -param sku string = 'S0' - -// ============================================================================= -// Resources -// ============================================================================= - -resource openAI 'Microsoft.CognitiveServices/accounts@2023-10-01-preview' = { - name: name - location: location - tags: tags - kind: 'OpenAI' - sku: { - name: sku - } - properties: { - customSubDomainName: name - publicNetworkAccess: 'Enabled' - networkAcls: { - defaultAction: 'Allow' - } - } -} - -// Deploy models -resource modelDeployments 'Microsoft.CognitiveServices/accounts/deployments@2023-10-01-preview' = [for deployment in deployments: { - parent: openAI - name: deployment.name - sku: { - name: 'Standard' - capacity: deployment.capacity - } - properties: { - model: { - format: 'OpenAI' - name: deployment.model - version: deployment.version - } - raiPolicyName: 'Microsoft.Default' - } -}] - -// Store API key in Key Vault if provided -resource keyVault 'Microsoft.KeyVault/vaults@2023-07-01' existing = if (!empty(keyVaultName)) { - name: keyVaultName -} - -resource openAIKeySecret 'Microsoft.KeyVault/vaults/secrets@2023-07-01' = if (!empty(keyVaultName)) { - parent: keyVault - name: 'azure-openai-api-key' - properties: { - value: openAI.listKeys().key1 - } -} - -resource openAIEndpointSecret 'Microsoft.KeyVault/vaults/secrets@2023-07-01' = if (!empty(keyVaultName)) { - parent: keyVault - name: 'azure-openai-endpoint' - properties: { - value: openAI.properties.endpoint - } -} - -// ============================================================================= -// Outputs -// ============================================================================= - -output id string = openAI.id -output name string = openAI.name -output endpoint string = openAI.properties.endpoint -output deploymentNames array = [for (deployment, i) in deployments: deployment.name] diff --git a/infra/bicep/modules/redis.bicep b/infra/bicep/modules/redis.bicep deleted file mode 100644 index 89ed330..0000000 --- a/infra/bicep/modules/redis.bicep +++ /dev/null @@ -1,88 +0,0 @@ -// ============================================================================= -// Azure Redis Cache Module -// ============================================================================= - -@description('Name of the Redis Cache') -param name string - -@description('Azure region') -param location string - -@description('Resource tags') -param tags object - -@description('Redis SKU') -@allowed(['Basic', 'Standard', 'Premium']) -param sku string = 'Basic' - -@description('Redis family') -@allowed(['C', 'P']) -param family string = 'C' - -@description('Redis capacity (0-6 for C family, 1-5 for P family)') -@minValue(0) -@maxValue(6) -param capacity int = 0 - -@description('Key Vault name for storing secrets') -param keyVaultName string = '' - -@description('Enable non-SSL port (not recommended)') -param enableNonSslPort bool = false - -@description('Minimum TLS version') -@allowed(['1.0', '1.1', '1.2']) -param minimumTlsVersion string = '1.2' - -// ============================================================================= -// Resources -// ============================================================================= - -resource redis 'Microsoft.Cache/redis@2023-08-01' = { - name: name - location: location - tags: tags - properties: { - sku: { - name: sku - family: family - capacity: capacity - } - enableNonSslPort: enableNonSslPort - minimumTlsVersion: minimumTlsVersion - publicNetworkAccess: 'Enabled' - redisConfiguration: { - 'maxmemory-policy': 'volatile-lru' - } - } -} - -// Store connection info in Key Vault -resource keyVault 'Microsoft.KeyVault/vaults@2023-07-01' existing = if (!empty(keyVaultName)) { - name: keyVaultName -} - -resource redisPasswordSecret 'Microsoft.KeyVault/vaults/secrets@2023-07-01' = if (!empty(keyVaultName)) { - parent: keyVault - name: 'redis-password' - properties: { - value: redis.listKeys().primaryKey - } -} - -resource redisConnectionStringSecret 'Microsoft.KeyVault/vaults/secrets@2023-07-01' = if (!empty(keyVaultName)) { - parent: keyVault - name: 'redis-connection-string' - properties: { - value: '${redis.properties.hostName}:${redis.properties.sslPort},password=${redis.listKeys().primaryKey},ssl=True,abortConnect=False' - } -} - -// ============================================================================= -// Outputs -// ============================================================================= - -output id string = redis.id -output name string = redis.name -output hostname string = redis.properties.hostName -output port int = redis.properties.sslPort diff --git a/infra/bicep/modules/static-web-app.bicep b/infra/bicep/modules/static-web-app.bicep deleted file mode 100644 index a29f66d..0000000 --- a/infra/bicep/modules/static-web-app.bicep +++ /dev/null @@ -1,78 +0,0 @@ -// ============================================================================= -// Azure Static Web Apps Module -// ============================================================================= - -@description('Name of the Static Web App') -param name string - -@description('Azure region') -param location string - -@description('Resource tags') -param tags object - -@description('API backend URL') -param apiUrl string = '' - -@description('SKU name') -@allowed(['Free', 'Standard']) -param sku string = 'Free' - -@description('GitHub repository URL (for linked deployment)') -param repositoryUrl string = '' - -@description('GitHub branch') -param branch string = 'main' - -@description('App location (path to app source)') -param appLocation string = 'apps/web' - -@description('API location (path to API source, if using Azure Functions)') -param apiLocation string = '' - -@description('Output location (build output path)') -param outputLocation string = '.next' - -// ============================================================================= -// Resources -// ============================================================================= - -resource staticWebApp 'Microsoft.Web/staticSites@2022-09-01' = { - name: name - location: location - tags: tags - sku: { - name: sku - tier: sku - } - properties: { - repositoryUrl: !empty(repositoryUrl) ? repositoryUrl : null - branch: !empty(repositoryUrl) ? branch : null - buildProperties: !empty(repositoryUrl) ? { - appLocation: appLocation - apiLocation: apiLocation - outputLocation: outputLocation - } : null - stagingEnvironmentPolicy: 'Enabled' - allowConfigFileUpdates: true - enterpriseGradeCdnStatus: 'Disabled' - } -} - -// Configure app settings -resource staticWebAppSettings 'Microsoft.Web/staticSites/config@2022-09-01' = { - parent: staticWebApp - name: 'appsettings' - properties: { - NEXT_PUBLIC_API_URL: apiUrl - } -} - -// ============================================================================= -// Outputs -// ============================================================================= - -output id string = staticWebApp.id -output name string = staticWebApp.name -output url string = 'https://${staticWebApp.properties.defaultHostname}' -output deploymentToken string = staticWebApp.listSecrets().properties.apiKey diff --git a/infra/bicep/modules/storage.bicep b/infra/bicep/modules/storage.bicep deleted file mode 100644 index 22f7ec8..0000000 --- a/infra/bicep/modules/storage.bicep +++ /dev/null @@ -1,100 +0,0 @@ -// ============================================================================= -// Azure Storage Account Module -// ============================================================================= - -@description('Name of the Storage Account (3-24 chars, lowercase alphanumeric)') -@minLength(3) -@maxLength(24) -param name string - -@description('Azure region') -param location string - -@description('Resource tags') -param tags object - -@description('Storage account SKU') -@allowed(['Standard_LRS', 'Standard_GRS', 'Standard_RAGRS', 'Standard_ZRS', 'Premium_LRS']) -param sku string = 'Standard_LRS' - -@description('Blob container names to create') -param containerNames array = [] - -@description('Enable blob versioning') -param enableVersioning bool = false - -@description('Enable soft delete for blobs') -param enableBlobSoftDelete bool = true - -@description('Blob soft delete retention days') -param blobSoftDeleteRetentionDays int = 7 - -// ============================================================================= -// Resources -// ============================================================================= - -resource storageAccount 'Microsoft.Storage/storageAccounts@2023-01-01' = { - name: name - location: location - tags: tags - sku: { - name: sku - } - kind: 'StorageV2' - properties: { - accessTier: 'Hot' - allowBlobPublicAccess: false - allowSharedKeyAccess: true - minimumTlsVersion: 'TLS1_2' - supportsHttpsTrafficOnly: true - encryption: { - services: { - blob: { - enabled: true - } - file: { - enabled: true - } - } - keySource: 'Microsoft.Storage' - } - networkAcls: { - bypass: 'AzureServices' - defaultAction: 'Allow' - } - } -} - -resource blobService 'Microsoft.Storage/storageAccounts/blobServices@2023-01-01' = { - parent: storageAccount - name: 'default' - properties: { - isVersioningEnabled: enableVersioning - deleteRetentionPolicy: { - enabled: enableBlobSoftDelete - days: blobSoftDeleteRetentionDays - } - containerDeleteRetentionPolicy: { - enabled: enableBlobSoftDelete - days: blobSoftDeleteRetentionDays - } - } -} - -resource containers 'Microsoft.Storage/storageAccounts/blobServices/containers@2023-01-01' = [for containerName in containerNames: { - parent: blobService - name: containerName - properties: { - publicAccess: 'None' - } -}] - -// ============================================================================= -// Outputs -// ============================================================================= - -output id string = storageAccount.id -output name string = storageAccount.name -output blobEndpoint string = storageAccount.properties.primaryEndpoints.blob -output primaryKey string = storageAccount.listKeys().keys[0].value -output connectionString string = 'DefaultEndpointsProtocol=https;AccountName=${storageAccount.name};AccountKey=${storageAccount.listKeys().keys[0].value};EndpointSuffix=${environment().suffixes.storage}' diff --git a/infra/parameters/dev.bicepparam b/infra/parameters/dev.bicepparam deleted file mode 100644 index caa0c41..0000000 --- a/infra/parameters/dev.bicepparam +++ /dev/null @@ -1,29 +0,0 @@ -// ============================================================================= -// Development Environment Parameters -// ============================================================================= -using '../bicep/main.bicep' - -param environment = 'dev' -param projectName = 'whatssummarize' -param location = 'eastus' - -// Enable services for development -// Note: OpenAI disabled - configure Azure AI Foundry separately via Azure Portal -param enableOpenAI = false -param enableCosmosDB = true -param enableRedis = true -param enableContainerApps = true -param enableStaticWebApps = true - -// OpenAI deployments (only used if enableOpenAI = true) -// Configure Azure AI Foundry manually and set AZURE_OPENAI_* env vars -param openAIDeployments = [] - -param adminEmail = '' - -param tags = { - project: 'whatssummarize' - environment: 'dev' - managedBy: 'bicep' - costCenter: 'development' -} diff --git a/infra/parameters/prod.bicepparam b/infra/parameters/prod.bicepparam deleted file mode 100644 index 5ae0fcd..0000000 --- a/infra/parameters/prod.bicepparam +++ /dev/null @@ -1,30 +0,0 @@ -// ============================================================================= -// Production Environment Parameters -// ============================================================================= -using '../bicep/main.bicep' - -param environment = 'prod' -param projectName = 'whatssummarize' -param location = 'eastus' - -// Enable services for production -// Note: OpenAI disabled - configure Azure AI Foundry separately via Azure Portal -param enableOpenAI = false -param enableCosmosDB = true -param enableRedis = true -param enableContainerApps = true -param enableStaticWebApps = true - -// OpenAI deployments (only used if enableOpenAI = true) -// Configure Azure AI Foundry manually and set AZURE_OPENAI_* env vars -param openAIDeployments = [] - -param adminEmail = 'admin@whatssummarize.com' - -param tags = { - project: 'whatssummarize' - environment: 'prod' - managedBy: 'bicep' - costCenter: 'production' - criticalityLevel: 'high' -} diff --git a/infra/parameters/staging.bicepparam b/infra/parameters/staging.bicepparam deleted file mode 100644 index e4ea669..0000000 --- a/infra/parameters/staging.bicepparam +++ /dev/null @@ -1,34 +0,0 @@ -// ============================================================================= -// Staging Environment Parameters -// ============================================================================= -// Staging mirrors production configuration with reduced capacity for cost -// optimization while maintaining feature parity for pre-production testing. -// ============================================================================= -using '../bicep/main.bicep' - -param environment = 'staging' -param projectName = 'whatssummarize' -param location = 'eastus' - -// Enable services for staging (mirrors production) -// Note: OpenAI disabled - configure Azure AI Foundry separately via Azure Portal -param enableOpenAI = false -param enableCosmosDB = true -param enableRedis = true -param enableContainerApps = true -param enableStaticWebApps = true - -// OpenAI deployments (only used if enableOpenAI = true) -// Configure Azure AI Foundry manually and set AZURE_OPENAI_* env vars -param openAIDeployments = [] - -param adminEmail = 'staging-alerts@whatssummarize.com' - -param tags = { - project: 'whatssummarize' - environment: 'staging' - managedBy: 'bicep' - costCenter: 'staging' - criticalityLevel: 'medium' - purpose: 'pre-production-testing' -} diff --git a/infra/scripts/deploy.ps1 b/infra/scripts/deploy.ps1 deleted file mode 100644 index 8398568..0000000 --- a/infra/scripts/deploy.ps1 +++ /dev/null @@ -1,429 +0,0 @@ -# ============================================================================= -# Azure Infrastructure Deployment Script (PowerShell) -# ============================================================================= -# Deploys or updates Azure infrastructure using Bicep templates. -# What-if analysis runs automatically before deployment unless skipped. -# -# Usage: -# ./deploy.ps1 -Environment dev -# ./deploy.ps1 -Environment prod -SkipWhatIf -# ./deploy.ps1 -Environment dev -WhatIfOnly -# ./deploy.ps1 -Environment dev -ValidateOnly -# -# ============================================================================= - -[CmdletBinding()] -param( - [Parameter(Mandatory = $false)] - [ValidateSet('dev', 'staging', 'prod')] - [string]$Environment = 'dev', - - [Parameter(Mandatory = $false)] - [switch]$SkipWhatIf, - - [Parameter(Mandatory = $false)] - [switch]$WhatIfOnly, - - [Parameter(Mandatory = $false)] - [switch]$ValidateOnly, - - [Parameter(Mandatory = $false)] - [switch]$Force -) - -# ============================================================================= -# Configuration -# ============================================================================= - -$ErrorActionPreference = 'Stop' -$ProjectName = 'whatssummarize' -$Location = 'eastus' - -$ScriptDir = Split-Path -Parent $MyInvocation.MyCommand.Path -$InfraDir = Split-Path -Parent $ScriptDir -$BicepDir = Join-Path $InfraDir 'bicep' -$ParamsDir = Join-Path $InfraDir 'parameters' - -$ResourceGroup = "rg-$ProjectName-$Environment" - -# ============================================================================= -# Utility Functions -# ============================================================================= - -function Write-Info { param([string]$Message) Write-Host "[INFO] $Message" -ForegroundColor Cyan } -function Write-SuccessMessage { param([string]$Message) Write-Host "[SUCCESS] $Message" -ForegroundColor Green } -function Write-WarningMessage { param([string]$Message) Write-Host "[WARNING] $Message" -ForegroundColor Yellow } -function Write-ErrorMessage { param([string]$Message) Write-Host "[ERROR] $Message" -ForegroundColor Red } - -# Helper for null coalescing (PowerShell 5.1 compatible) -function Get-ValueOrDefault { - param($Value, $Default) - if ($null -eq $Value -or $Value -eq '') { return $Default } - return $Value -} - -function Write-Banner { - Write-Host "" - Write-Host "==============================================" -ForegroundColor Magenta - Write-Host " Azure Infrastructure Deployment" -ForegroundColor Magenta - Write-Host " Environment: $Environment" -ForegroundColor Magenta - Write-Host " Project: $ProjectName" -ForegroundColor Magenta - Write-Host "==============================================" -ForegroundColor Magenta - Write-Host "" -} - -# ============================================================================= -# Pre-flight Checks -# ============================================================================= - -function Test-Prerequisites { - Write-Info "Running pre-flight checks..." - - # Check Azure CLI - try { - $null = az version 2>$null - } - catch { - Write-ErrorMessage "Azure CLI not found. Please install: https://docs.microsoft.com/cli/azure/install-azure-cli" - exit 1 - } - - # Check Bicep - $bicepVersion = az bicep version 2>$null - if (-not $bicepVersion) { - Write-Info "Installing Bicep CLI..." - az bicep install - } - - # Check login - $account = az account show 2>$null | ConvertFrom-Json - if (-not $account) { - Write-ErrorMessage "Not logged into Azure. Run: az login" - exit 1 - } - Write-Info "Logged in as: $($account.user.name)" - Write-Info "Subscription: $($account.name)" - - # Check parameter file exists - $paramFile = Join-Path $ParamsDir "$Environment.bicepparam" - if (-not (Test-Path $paramFile)) { - Write-ErrorMessage "Parameter file not found: $paramFile" - exit 1 - } - - # Check main.bicep exists - $mainBicep = Join-Path $BicepDir 'main.bicep' - if (-not (Test-Path $mainBicep)) { - Write-ErrorMessage "Main Bicep file not found: $mainBicep" - exit 1 - } - - Write-SuccessMessage "Pre-flight checks passed" -} - -# ============================================================================= -# Validation -# ============================================================================= - -function Test-Templates { - Write-Info "Validating Bicep templates..." - - $mainBicep = Join-Path $BicepDir 'main.bicep' - $result = az bicep build --file $mainBicep --stdout 2>&1 - - if ($LASTEXITCODE -ne 0) { - Write-ErrorMessage "Template validation failed" - Write-Host $result - exit 1 - } - - Write-SuccessMessage "Template validation passed" -} - -# ============================================================================= -# Resource Group -# ============================================================================= - -function Ensure-ResourceGroup { - Write-Info "Ensuring resource group exists: $ResourceGroup" - - $exists = az group exists --name $ResourceGroup 2>$null - if ($exists -eq 'true') { - Write-Info "Resource group already exists" - } - else { - Write-Info "Creating resource group..." - az group create ` - --name $ResourceGroup ` - --location $Location ` - --tags project=$ProjectName environment=$Environment managedBy=bicep | Out-Null - Write-SuccessMessage "Resource group created" - } -} - -# ============================================================================= -# What-If Analysis -# ============================================================================= - -function Invoke-WhatIfAnalysis { - param([string]$DeploymentName) - - Write-Host "" - Write-Host "==============================================" -ForegroundColor Yellow - Write-Host " WHAT-IF ANALYSIS" -ForegroundColor Yellow - Write-Host " Previewing changes before deployment..." -ForegroundColor Yellow - Write-Host "==============================================" -ForegroundColor Yellow - Write-Host "" - - $paramFile = Join-Path $ParamsDir "$Environment.bicepparam" - $mainBicep = Join-Path $BicepDir 'main.bicep' - - az deployment group what-if ` - --resource-group $ResourceGroup ` - --template-file $mainBicep ` - --parameters $paramFile ` - --name $DeploymentName - - if ($LASTEXITCODE -ne 0) { - Write-ErrorMessage "What-if analysis failed" - exit 1 - } - - Write-Host "" - Write-SuccessMessage "What-if analysis complete" - - return $true -} - -# ============================================================================= -# Deployment -# ============================================================================= - -function Invoke-Deployment { - param([string]$DeploymentName) - - $paramFile = Join-Path $ParamsDir "$Environment.bicepparam" - $mainBicep = Join-Path $BicepDir 'main.bicep' - - if ($ValidateOnly) { - Write-Info "Validating deployment (no changes will be made)..." - az deployment group validate ` - --resource-group $ResourceGroup ` - --template-file $mainBicep ` - --parameters $paramFile ` - --name $DeploymentName - - if ($LASTEXITCODE -ne 0) { - Write-ErrorMessage "Deployment validation failed" - exit 1 - } - - Write-SuccessMessage "Deployment validation passed" - return - } - - Write-Host "" - Write-Host "==============================================" -ForegroundColor Green - Write-Host " DEPLOYING INFRASTRUCTURE" -ForegroundColor Green - Write-Host "==============================================" -ForegroundColor Green - Write-Host "" - - Write-Info "Starting deployment: $DeploymentName" - Write-Info "Environment: $Environment" - Write-Info "Resource Group: $ResourceGroup" - Write-Host "" - - az deployment group create ` - --resource-group $ResourceGroup ` - --template-file $mainBicep ` - --parameters $paramFile ` - --name $DeploymentName ` - --verbose - - if ($LASTEXITCODE -ne 0) { - Write-ErrorMessage "Deployment failed" - exit 1 - } - - Write-Host "" - Write-SuccessMessage "Deployment complete!" - - # Show outputs - Write-Host "" - Write-Info "Deployment Outputs:" - az deployment group show ` - --resource-group $ResourceGroup ` - --name $DeploymentName ` - --query "properties.outputs" ` - -o json -} - -# ============================================================================= -# Confirmation Prompt -# ============================================================================= - -function Get-DeploymentConfirmation { - if ($Force) { - return $true - } - - Write-Host "" - Write-Host "==============================================" -ForegroundColor Cyan - Write-Host " READY TO DEPLOY" -ForegroundColor Cyan - Write-Host "==============================================" -ForegroundColor Cyan - Write-Host "" - Write-Host "Environment: " -NoNewline; Write-Host $Environment -ForegroundColor Yellow - Write-Host "Resource Group: " -NoNewline; Write-Host $ResourceGroup -ForegroundColor Yellow - Write-Host "" - - $response = Read-Host "Proceed with deployment? (y/N)" - return $response -eq 'y' -or $response -eq 'Y' -} - -# ============================================================================= -# Post-Deployment -# ============================================================================= - -function Invoke-PostDeployment { - param([string]$DeploymentName) - - Write-Info "Running post-deployment tasks..." - - # Run validation script - $validateScript = Join-Path $ScriptDir 'validate-resources.ps1' - if (Test-Path $validateScript) { - Write-Info "Validating deployed resources..." - try { - & $validateScript -Environment $Environment - } - catch { - Write-WarningMessage "Resource validation had issues: $_" - } - } - - # Generate .env file for local development - if ($Environment -eq 'dev') { - Write-Info "Generating development environment file..." - New-EnvironmentFile -DeploymentName $DeploymentName - } - - Write-SuccessMessage "Post-deployment tasks complete" -} - -function New-EnvironmentFile { - param([string]$DeploymentName) - - $envFile = Join-Path $InfraDir '..' 'apps' 'api' '.env.azure' - $kvName = "kv$ProjectName$Environment" -replace '-', '' - - try { - $outputs = az deployment group show ` - --resource-group $ResourceGroup ` - --name $DeploymentName ` - --query "properties.outputs" ` - -o json 2>$null | ConvertFrom-Json - - if ($outputs) { - $content = @" -# =========================================== -# Auto-generated Azure Environment Variables -# Generated: $(Get-Date -Format 'yyyy-MM-dd HH:mm:ss') -# Environment: $Environment -# =========================================== - -# Azure Provider -AZURE_PROVIDER_ENABLED=true - -# Azure OpenAI / AI Foundry -# Configure via Azure Portal, API version auto-detected -AZURE_OPENAI_ENDPOINT=$($outputs.openAIEndpoint.value) -AZURE_OPENAI_DEPLOYMENT=gpt-4 - -# Azure Cosmos DB -AZURE_COSMOS_ENDPOINT=$($outputs.cosmosDBEndpoint.value) -AZURE_COSMOS_DATABASE=$($outputs.cosmosDBDatabaseName.value) - -# Azure Redis -AZURE_REDIS_HOSTNAME=$($outputs.redisHostname.value) -AZURE_REDIS_PORT=$(Get-ValueOrDefault $outputs.redisPort.value '6380') - -# Azure Storage -AZURE_STORAGE_ACCOUNT_NAME=$($outputs.storageAccountName.value) - -# Azure App Insights -AZURE_APP_INSIGHTS_CONNECTION_STRING=$($outputs.appInsightsConnectionString.value) - -# Key Vault (for retrieving secrets) -AZURE_KEY_VAULT_NAME=$($outputs.keyVaultName.value) - -# Note: Sensitive values should be retrieved from Key Vault at runtime -# Use: az keyvault secret show --vault-name `$AZURE_KEY_VAULT_NAME --name -"@ - - $content | Out-File -FilePath $envFile -Encoding UTF8 - Write-SuccessMessage "Generated: $envFile" - } - } - catch { - Write-WarningMessage "Could not retrieve deployment outputs: $_" - } -} - -# ============================================================================= -# Main Execution -# ============================================================================= - -function Main { - Write-Banner - - # Pre-flight checks - Test-Prerequisites - Write-Host "" - - # Validate templates - Test-Templates - Write-Host "" - - # Ensure resource group exists (needed for what-if) - if (-not $ValidateOnly) { - Ensure-ResourceGroup - Write-Host "" - } - - # Generate deployment name - $timestamp = Get-Date -Format 'yyyyMMdd-HHmmss' - $deploymentName = "$ProjectName-$Environment-$timestamp" - - # What-if is the DEFAULT first step (unless skipped or validate-only) - if (-not $ValidateOnly -and -not $SkipWhatIf) { - Invoke-WhatIfAnalysis -DeploymentName "$deploymentName-whatif" - Write-Host "" - - # If what-if only, stop here - if ($WhatIfOnly) { - Write-SuccessMessage "What-if analysis complete. No changes were made." - return - } - - # Prompt for confirmation before actual deployment - if (-not (Get-DeploymentConfirmation)) { - Write-WarningMessage "Deployment cancelled by user" - return - } - } - - # Deploy - Invoke-Deployment -DeploymentName $deploymentName - Write-Host "" - - # Post-deployment (only for actual deployments) - if (-not $ValidateOnly) { - Invoke-PostDeployment -DeploymentName $deploymentName - Write-Host "" - } - - Write-SuccessMessage "All tasks completed successfully!" -} - -# Run main -Main diff --git a/infra/scripts/deploy.sh b/infra/scripts/deploy.sh deleted file mode 100755 index d17db3f..0000000 --- a/infra/scripts/deploy.sh +++ /dev/null @@ -1,440 +0,0 @@ -#!/bin/bash -# ============================================================================= -# Azure Infrastructure Deployment Script -# ============================================================================= -# Deploys or updates Azure infrastructure using Bicep templates. -# -# Usage: -# ./deploy.sh [environment] [options] -# -# Arguments: -# environment - dev, staging, or prod (default: dev) -# -# Options: -# --what-if - Preview changes without deploying -# --validate-only - Only validate templates, don't deploy -# --skip-what-if - Skip what-if preview (not recommended) -# --force - Skip confirmation prompts -# -h, --help - Show this help message -# -# Requirements: -# - Azure CLI installed and logged in -# - Bicep CLI installed (usually comes with Azure CLI) -# - jq installed (for JSON parsing) -# ============================================================================= - -set -e -set -o pipefail - -# Colors for output -RED='\033[0;31m' -GREEN='\033[0;32m' -YELLOW='\033[1;33m' -BLUE='\033[0;34m' -NC='\033[0m' # No Color - -# Default configuration -ENVIRONMENT="dev" -WHAT_IF=false -VALIDATE_ONLY=false -SKIP_WHAT_IF=false -FORCE=false -PROJECT_NAME="whatssummarize" -LOCATION="eastus" - -# Parse arguments -while [[ $# -gt 0 ]]; do - case $1 in - --what-if) - WHAT_IF=true - shift - ;; - --validate-only) - VALIDATE_ONLY=true - shift - ;; - --skip-what-if) - SKIP_WHAT_IF=true - shift - ;; - --force|-f) - FORCE=true - shift - ;; - -h|--help) - echo "Usage: $0 [environment] [options]" - echo "" - echo "Arguments:" - echo " environment - dev, staging, or prod (default: dev)" - echo "" - echo "Options:" - echo " --what-if - Preview changes without deploying" - echo " --validate-only - Only validate templates, don't deploy" - echo " --skip-what-if - Skip automatic what-if preview (not recommended)" - echo " --force, -f - Skip confirmation prompts" - echo " -h, --help - Show this help message" - echo "" - echo "Examples:" - echo " $0 dev # Deploy to dev (what-if first)" - echo " $0 prod --what-if # Preview production changes only" - echo " $0 staging --validate-only # Validate staging templates" - echo " $0 prod --skip-what-if # Deploy to prod without preview" - exit 0 - ;; - -*) - echo "Unknown option: $1" >&2 - echo "Use --help for usage information" >&2 - exit 1 - ;; - *) - # Positional argument (environment) - if [[ "$1" =~ ^(dev|staging|prod)$ ]]; then - ENVIRONMENT="$1" - else - echo "Invalid environment: $1" >&2 - echo "Valid options: dev, staging, prod" >&2 - exit 1 - fi - shift - ;; - esac -done - -# Paths -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -INFRA_DIR="$(dirname "$SCRIPT_DIR")" -BICEP_DIR="${INFRA_DIR}/bicep" -PARAMS_DIR="${INFRA_DIR}/parameters" - -# Resource naming -RESOURCE_GROUP="rg-${PROJECT_NAME}-${ENVIRONMENT}" - -# ============================================================================= -# Utility Functions -# ============================================================================= - -log_info() { - echo -e "${BLUE}[INFO]${NC} $1" -} - -log_success() { - echo -e "${GREEN}[SUCCESS]${NC} $1" -} - -log_warning() { - echo -e "${YELLOW}[WARNING]${NC} $1" -} - -log_error() { - echo -e "${RED}[ERROR]${NC} $1" -} - -# ============================================================================= -# Pre-flight Checks -# ============================================================================= - -preflight_checks() { - log_info "Running pre-flight checks..." - - # Check Azure CLI - if ! command -v az &>/dev/null; then - log_error "Azure CLI not found. Please install: https://docs.microsoft.com/cli/azure/install-azure-cli" - exit 1 - fi - - # Check jq (required for JSON parsing) - if ! command -v jq &>/dev/null; then - log_error "jq not found. Please install: apt-get install jq (Linux) or brew install jq (macOS)" - exit 1 - fi - - # Check Bicep - if ! az bicep version &>/dev/null; then - log_info "Installing Bicep CLI..." - az bicep install - fi - - # Check login - if ! az account show &>/dev/null; then - log_error "Not logged into Azure. Run: az login" - exit 1 - fi - - # Check parameter file exists - local param_file="${PARAMS_DIR}/${ENVIRONMENT}.bicepparam" - if [ ! -f "$param_file" ]; then - log_error "Parameter file not found: $param_file" - exit 1 - fi - - # Check main.bicep exists - if [ ! -f "${BICEP_DIR}/main.bicep" ]; then - log_error "Main Bicep file not found: ${BICEP_DIR}/main.bicep" - exit 1 - fi - - log_success "Pre-flight checks passed" -} - -# ============================================================================= -# Validation -# ============================================================================= - -validate_templates() { - log_info "Validating Bicep templates..." - - # Build all Bicep files to check for errors - az bicep build --file "${BICEP_DIR}/main.bicep" --stdout > /dev/null - - log_success "Template validation passed" -} - -# ============================================================================= -# Resource Group -# ============================================================================= - -ensure_resource_group() { - log_info "Ensuring resource group exists: $RESOURCE_GROUP" - - if az group show --name "$RESOURCE_GROUP" &>/dev/null; then - log_info "Resource group already exists" - else - log_info "Creating resource group..." - az group create \ - --name "$RESOURCE_GROUP" \ - --location "$LOCATION" \ - --tags project="$PROJECT_NAME" environment="$ENVIRONMENT" managedBy="bicep" - log_success "Resource group created" - fi -} - -# ============================================================================= -# Deployment -# ============================================================================= - -run_what_if() { - local param_file="${PARAMS_DIR}/${ENVIRONMENT}.bicepparam" - local deployment_name="whatssummarize-${ENVIRONMENT}-whatif-$(date +%Y%m%d-%H%M%S)" - - echo "" - echo -e "${YELLOW}==============================================" - echo " WHAT-IF ANALYSIS" - echo " Previewing changes..." - echo "==============================================${NC}" - echo "" - - az deployment group what-if \ - --resource-group "$RESOURCE_GROUP" \ - --template-file "${BICEP_DIR}/main.bicep" \ - --parameters "$param_file" \ - --name "$deployment_name" - - log_success "What-if analysis complete" -} - -confirm_deployment() { - if [[ "$FORCE" = true ]]; then - return 0 - fi - - echo "" - echo -e "${YELLOW}==============================================" - echo " READY TO DEPLOY" - echo "==============================================${NC}" - echo "" - echo "Environment: $ENVIRONMENT" - echo "Resource Group: $RESOURCE_GROUP" - echo "" - - # Extra warning for production - if [[ "$ENVIRONMENT" = "prod" ]]; then - echo -e "${RED}⚠️ WARNING: You are about to deploy to PRODUCTION!${NC}" - echo "" - read -p "Type 'yes' to confirm production deployment: " confirm - if [[ "$confirm" != "yes" ]]; then - log_warning "Deployment cancelled by user" - exit 0 - fi - else - read -p "Proceed with deployment? (y/N): " confirm - if [[ "$confirm" != "y" && "$confirm" != "Y" ]]; then - log_warning "Deployment cancelled by user" - exit 0 - fi - fi -} - -deploy_infrastructure() { - local param_file="${PARAMS_DIR}/${ENVIRONMENT}.bicepparam" - local deployment_name="whatssummarize-${ENVIRONMENT}-$(date +%Y%m%d-%H%M%S)" - - log_info "Starting deployment: $deployment_name" - log_info "Environment: $ENVIRONMENT" - log_info "Resource Group: $RESOURCE_GROUP" - log_info "Parameter File: $param_file" - echo "" - - if [[ "$VALIDATE_ONLY" = true ]]; then - log_info "Validating deployment..." - az deployment group validate \ - --resource-group "$RESOURCE_GROUP" \ - --template-file "${BICEP_DIR}/main.bicep" \ - --parameters "$param_file" \ - --name "$deployment_name" - - log_success "Deployment validation passed" - return - fi - - if [[ "$WHAT_IF" = true ]]; then - run_what_if - return - fi - - # Default behavior: run what-if first, then deploy - if [[ "$SKIP_WHAT_IF" = false ]]; then - run_what_if - confirm_deployment - fi - - echo "" - echo -e "${GREEN}==============================================" - echo " DEPLOYING INFRASTRUCTURE" - echo "==============================================${NC}" - echo "" - - log_info "Deploying infrastructure..." - az deployment group create \ - --resource-group "$RESOURCE_GROUP" \ - --template-file "${BICEP_DIR}/main.bicep" \ - --parameters "$param_file" \ - --name "$deployment_name" \ - --verbose - - log_success "Deployment complete!" - - # Show outputs - echo "" - log_info "Deployment Outputs:" - az deployment group show \ - --resource-group "$RESOURCE_GROUP" \ - --name "$deployment_name" \ - --query "properties.outputs" \ - -o json -} - -# ============================================================================= -# Post-deployment -# ============================================================================= - -post_deployment() { - log_info "Running post-deployment tasks..." - - # Run validation script - if [ -x "${SCRIPT_DIR}/validate-resources.sh" ]; then - log_info "Validating deployed resources..." - "${SCRIPT_DIR}/validate-resources.sh" "$ENVIRONMENT" || true - fi - - # Generate .env file for local development - if [ "$ENVIRONMENT" = "dev" ]; then - log_info "Generating development environment file..." - generate_env_file - fi - - log_success "Post-deployment tasks complete" -} - -generate_env_file() { - local env_file="${INFRA_DIR}/../apps/api/.env.azure" - local kv_name="kv${PROJECT_NAME}${ENVIRONMENT}" - kv_name="${kv_name//-/}" - - log_info "Generating .env.azure file..." - - # Get outputs from deployment - local outputs=$(az deployment group show \ - --resource-group "$RESOURCE_GROUP" \ - --name "$(az deployment group list --resource-group "$RESOURCE_GROUP" --query "[0].name" -o tsv)" \ - --query "properties.outputs" \ - -o json 2>/dev/null) - - if [ -n "$outputs" ]; then - cat > "$env_file" << EOF -# =========================================== -# Auto-generated Azure Environment Variables -# Generated: $(date) -# Environment: $ENVIRONMENT -# =========================================== - -# Azure Provider -AZURE_PROVIDER_ENABLED=true - -# Azure OpenAI -AZURE_OPENAI_ENDPOINT=$(echo "$outputs" | jq -r '.openAIEndpoint.value // ""') -AZURE_OPENAI_DEPLOYMENT=gpt-4 -AZURE_OPENAI_API_VERSION=2024-02-15-preview - -# Azure Cosmos DB -AZURE_COSMOS_ENDPOINT=$(echo "$outputs" | jq -r '.cosmosDBEndpoint.value // ""') -AZURE_COSMOS_DATABASE=$(echo "$outputs" | jq -r '.cosmosDBDatabaseName.value // ""') - -# Azure Redis -AZURE_REDIS_HOSTNAME=$(echo "$outputs" | jq -r '.redisHostname.value // ""') -AZURE_REDIS_PORT=$(echo "$outputs" | jq -r '.redisPort.value // "6380"') - -# Azure Storage -AZURE_STORAGE_ACCOUNT_NAME=$(echo "$outputs" | jq -r '.storageAccountName.value // ""') - -# Azure App Insights -AZURE_APP_INSIGHTS_CONNECTION_STRING=$(echo "$outputs" | jq -r '.appInsightsConnectionString.value // ""') - -# Key Vault (for retrieving secrets) -AZURE_KEY_VAULT_NAME=$(echo "$outputs" | jq -r '.keyVaultName.value // ""') - -# Note: Sensitive values should be retrieved from Key Vault at runtime -# Use: az keyvault secret show --vault-name \$AZURE_KEY_VAULT_NAME --name -EOF - log_success "Generated: $env_file" - else - log_warning "Could not retrieve deployment outputs" - fi -} - -# ============================================================================= -# Main Execution -# ============================================================================= - -main() { - echo "" - echo "==============================================" - echo " Azure Infrastructure Deployment" - echo " Environment: $ENVIRONMENT" - echo " Project: $PROJECT_NAME" - echo "==============================================" - echo "" - - preflight_checks - echo "" - - validate_templates - echo "" - - if [[ "$VALIDATE_ONLY" = false ]]; then - ensure_resource_group - echo "" - fi - - deploy_infrastructure - echo "" - - if [[ "$WHAT_IF" = false && "$VALIDATE_ONLY" = false ]]; then - post_deployment - echo "" - fi - - log_success "All tasks completed successfully!" -} - -# Run main function -main diff --git a/infra/scripts/validate-resources.ps1 b/infra/scripts/validate-resources.ps1 deleted file mode 100644 index 034716f..0000000 --- a/infra/scripts/validate-resources.ps1 +++ /dev/null @@ -1,303 +0,0 @@ -# ============================================================================= -# Azure Resource Validation Script (PowerShell) -# ============================================================================= -# Validates that required Azure resources exist before deployment or release. -# -# Usage: -# ./validate-resources.ps1 -Environment dev -# ./validate-resources.ps1 -Environment prod -Strict -# -# ============================================================================= - -[CmdletBinding()] -param( - [Parameter(Mandatory = $false)] - [ValidateSet('dev', 'staging', 'prod')] - [string]$Environment = 'dev', - - [Parameter(Mandatory = $false)] - [switch]$Strict -) - -# ============================================================================= -# Configuration -# ============================================================================= - -$ErrorActionPreference = 'Continue' -$ProjectName = 'whatssummarize' -$ResourceGroup = "rg-$ProjectName-$Environment" - -# Track validation results -$script:ValidationPassed = $true -$script:MissingResources = @() -$script:WarningResources = @() - -# ============================================================================= -# Utility Functions -# ============================================================================= - -function Write-Info { param([string]$Message) Write-Host "[INFO] $Message" -ForegroundColor Cyan } -function Write-Success { param([string]$Message) Write-Host "[OK] $Message" -ForegroundColor Green } -function Write-Warning { param([string]$Message) Write-Host "[WARN] $Message" -ForegroundColor Yellow } -function Write-Fail { param([string]$Message) Write-Host "[FAIL] $Message" -ForegroundColor Red } - -function Test-ResourceExists { - param( - [string]$ResourceName, - [string]$ResourceType, - [scriptblock]$CheckScript, - [bool]$Required = $true - ) - - Write-Host " Checking $ResourceType`: " -NoNewline - - try { - $result = & $CheckScript 2>$null - if ($result -or $LASTEXITCODE -eq 0) { - Write-Host $ResourceName -ForegroundColor Green -NoNewline - Write-Host " [OK]" -ForegroundColor Green - return $true - } - } - catch { - # Resource doesn't exist - } - - if ($Required) { - Write-Host $ResourceName -ForegroundColor Red -NoNewline - Write-Host " [MISSING]" -ForegroundColor Red - $script:ValidationPassed = $false - $script:MissingResources += $ResourceType - } - else { - Write-Host $ResourceName -ForegroundColor Yellow -NoNewline - Write-Host " [NOT CONFIGURED]" -ForegroundColor Yellow - $script:WarningResources += $ResourceType - } - - return $false -} - -# ============================================================================= -# Resource Checks -# ============================================================================= - -function Test-AllResources { - Write-Host "" - Write-Host "==============================================" -ForegroundColor Cyan - Write-Host " Azure Resource Validation" -ForegroundColor Cyan - Write-Host " Environment: $Environment" -ForegroundColor Cyan - Write-Host " Resource Group: $ResourceGroup" -ForegroundColor Cyan - Write-Host "==============================================" -ForegroundColor Cyan - Write-Host "" - - # Required Resources - Write-Host "Required Resources:" -ForegroundColor White - Write-Host "-------------------" -ForegroundColor Gray - - # Resource Group - Test-ResourceExists ` - -ResourceName $ResourceGroup ` - -ResourceType "Resource Group" ` - -CheckScript { az group show --name $ResourceGroup -o json } ` - -Required $true - - # Key Vault - $kvName = "kv$ProjectName$Environment" -replace '-', '' - Test-ResourceExists ` - -ResourceName $kvName ` - -ResourceType "Key Vault" ` - -CheckScript { az keyvault show --name $kvName -o json } ` - -Required $true - - # Storage Account - $storageName = "st$ProjectName$Environment" -replace '-', '' - Test-ResourceExists ` - -ResourceName $storageName ` - -ResourceType "Storage Account" ` - -CheckScript { az storage account show --name $storageName --resource-group $ResourceGroup -o json } ` - -Required $true - - # Application Insights - $appiName = "appi-$ProjectName-$Environment" - Test-ResourceExists ` - -ResourceName $appiName ` - -ResourceType "Application Insights" ` - -CheckScript { az monitor app-insights component show --app $appiName --resource-group $ResourceGroup -o json } ` - -Required $true - - # Container Apps Environment - $caeName = "cae-$ProjectName-$Environment" - Test-ResourceExists ` - -ResourceName $caeName ` - -ResourceType "Container Apps Environment" ` - -CheckScript { az containerapp env show --name $caeName --resource-group $ResourceGroup -o json } ` - -Required $true - - Write-Host "" - - # Optional Resources - Write-Host "Optional Resources:" -ForegroundColor White - Write-Host "-------------------" -ForegroundColor Gray - - # Azure OpenAI / AI Foundry - $openaiName = "oai-$ProjectName-$Environment" - Test-ResourceExists ` - -ResourceName $openaiName ` - -ResourceType "Azure OpenAI" ` - -CheckScript { az cognitiveservices account show --name $openaiName --resource-group $ResourceGroup -o json } ` - -Required $false - - # Cosmos DB - $cosmosName = "cosmos-$ProjectName-$Environment" - Test-ResourceExists ` - -ResourceName $cosmosName ` - -ResourceType "Cosmos DB" ` - -CheckScript { az cosmosdb show --name $cosmosName --resource-group $ResourceGroup -o json } ` - -Required $false - - # Redis Cache - $redisName = "redis-$ProjectName-$Environment" - Test-ResourceExists ` - -ResourceName $redisName ` - -ResourceType "Redis Cache" ` - -CheckScript { az redis show --name $redisName --resource-group $ResourceGroup -o json } ` - -Required $false - - # Static Web App - $swaName = "swa-$ProjectName-$Environment" - Test-ResourceExists ` - -ResourceName $swaName ` - -ResourceType "Static Web App" ` - -CheckScript { az staticwebapp show --name $swaName --resource-group $ResourceGroup -o json } ` - -Required $false - - Write-Host "" -} - -# ============================================================================= -# Secrets Check -# ============================================================================= - -function Test-KeyVaultSecrets { - $kvName = "kv$ProjectName$Environment" -replace '-', '' - - Write-Host "Key Vault Secrets:" -ForegroundColor White - Write-Host "------------------" -ForegroundColor Gray - - # Check if Key Vault exists first - $kvExists = az keyvault show --name $kvName -o json 2>$null - if (-not $kvExists) { - Write-Warning " Cannot check secrets - Key Vault does not exist" - return - } - - $secrets = @( - @{ Name = "azure-openai-api-key"; Required = $false }, - @{ Name = "cosmos-db-key"; Required = $false }, - @{ Name = "redis-key"; Required = $false } - ) - - foreach ($secret in $secrets) { - Write-Host " Checking secret: " -NoNewline - - try { - $result = az keyvault secret show --vault-name $kvName --name $secret.Name -o json 2>$null - if ($result) { - Write-Host $secret.Name -ForegroundColor Green -NoNewline - Write-Host " [OK]" -ForegroundColor Green - } - else { - throw "Not found" - } - } - catch { - if ($secret.Required) { - Write-Host $secret.Name -ForegroundColor Red -NoNewline - Write-Host " [MISSING]" -ForegroundColor Red - $script:ValidationPassed = $false - } - else { - Write-Host $secret.Name -ForegroundColor Yellow -NoNewline - Write-Host " [NOT SET]" -ForegroundColor Yellow - } - } - } - - Write-Host "" -} - -# ============================================================================= -# Summary -# ============================================================================= - -function Write-Summary { - Write-Host "==============================================" -ForegroundColor Cyan - Write-Host " Validation Summary" -ForegroundColor Cyan - Write-Host "==============================================" -ForegroundColor Cyan - Write-Host "" - - if ($script:ValidationPassed) { - Write-Host " Status: " -NoNewline - Write-Host "PASSED" -ForegroundColor Green - } - else { - Write-Host " Status: " -NoNewline - Write-Host "FAILED" -ForegroundColor Red - } - - if ($script:MissingResources.Count -gt 0) { - Write-Host "" - Write-Host " Missing Required Resources:" -ForegroundColor Red - foreach ($resource in $script:MissingResources) { - Write-Host " - $resource" -ForegroundColor Red - } - } - - if ($script:WarningResources.Count -gt 0) { - Write-Host "" - Write-Host " Optional Resources Not Configured:" -ForegroundColor Yellow - foreach ($resource in $script:WarningResources) { - Write-Host " - $resource" -ForegroundColor Yellow - } - } - - Write-Host "" - - if (-not $script:ValidationPassed) { - Write-Host " To deploy missing resources, run:" -ForegroundColor Cyan - Write-Host " ./deploy.ps1 -Environment $Environment" -ForegroundColor White - Write-Host "" - } -} - -# ============================================================================= -# Main -# ============================================================================= - -function Main { - # Check Azure CLI login - $account = az account show -o json 2>$null | ConvertFrom-Json - if (-not $account) { - Write-Fail "Not logged into Azure. Run: az login" - exit 1 - } - - Test-AllResources - Test-KeyVaultSecrets - Write-Summary - - if ($Strict -and -not $script:ValidationPassed) { - Write-Fail "Strict mode: Validation failed due to missing required resources" - exit 1 - } - - if ($script:ValidationPassed) { - exit 0 - } - else { - exit 1 - } -} - -Main diff --git a/infra/scripts/validate-resources.sh b/infra/scripts/validate-resources.sh deleted file mode 100755 index 0d592ab..0000000 --- a/infra/scripts/validate-resources.sh +++ /dev/null @@ -1,365 +0,0 @@ -#!/bin/bash -# ============================================================================= -# Azure Resource Validation Script -# ============================================================================= -# Validates that all required Azure resources exist and are properly configured -# before deployment or release. -# -# Usage: -# ./validate-resources.sh [--strict] -# -# Arguments: -# environment - dev, staging, or prod -# --strict - Fail on any missing or misconfigured resource -# -# Requirements: -# - Azure CLI installed and logged in -# - jq installed for JSON parsing -# ============================================================================= - -set -e - -# Colors for output -RED='\033[0;31m' -GREEN='\033[0;32m' -YELLOW='\033[1;33m' -BLUE='\033[0;34m' -NC='\033[0m' # No Color - -# Configuration -ENVIRONMENT="${1:-dev}" -STRICT_MODE="${2:-}" -PROJECT_NAME="whatssummarize" -RESOURCE_PREFIX="${PROJECT_NAME}-${ENVIRONMENT}" - -# Counters -TOTAL_CHECKS=0 -PASSED_CHECKS=0 -FAILED_CHECKS=0 -WARNING_CHECKS=0 - -# ============================================================================= -# Utility Functions -# ============================================================================= - -log_info() { - echo -e "${BLUE}[INFO]${NC} $1" -} - -log_success() { - echo -e "${GREEN}[PASS]${NC} $1" - ((PASSED_CHECKS++)) -} - -log_warning() { - echo -e "${YELLOW}[WARN]${NC} $1" - ((WARNING_CHECKS++)) -} - -log_error() { - echo -e "${RED}[FAIL]${NC} $1" - ((FAILED_CHECKS++)) -} - -check_resource() { - local resource_type="$1" - local resource_name="$2" - local resource_group="$3" - local required="$4" # true or false - - ((TOTAL_CHECKS++)) - - if az resource show --resource-type "$resource_type" --name "$resource_name" --resource-group "$resource_group" &>/dev/null; then - log_success "$resource_type: $resource_name exists" - return 0 - else - if [ "$required" = "true" ]; then - log_error "$resource_type: $resource_name NOT FOUND (required)" - return 1 - else - log_warning "$resource_type: $resource_name not found (optional)" - return 0 - fi - fi -} - -# ============================================================================= -# Validation Functions -# ============================================================================= - -validate_resource_group() { - local rg_name="rg-${RESOURCE_PREFIX}" - - log_info "Checking Resource Group: $rg_name" - ((TOTAL_CHECKS++)) - - if az group show --name "$rg_name" &>/dev/null; then - log_success "Resource Group exists: $rg_name" - RESOURCE_GROUP="$rg_name" - return 0 - else - log_error "Resource Group NOT FOUND: $rg_name" - return 1 - fi -} - -validate_key_vault() { - local kv_name="kv${PROJECT_NAME}${ENVIRONMENT}" - kv_name="${kv_name//-/}" # Remove hyphens - - log_info "Checking Key Vault..." - check_resource "Microsoft.KeyVault/vaults" "$kv_name" "$RESOURCE_GROUP" "true" - - # Check for required secrets - if az keyvault show --name "$kv_name" &>/dev/null; then - log_info "Checking Key Vault secrets..." - - local required_secrets=("azure-openai-api-key" "cosmos-db-key" "redis-password") - for secret in "${required_secrets[@]}"; do - ((TOTAL_CHECKS++)) - if az keyvault secret show --vault-name "$kv_name" --name "$secret" &>/dev/null; then - log_success "Secret exists: $secret" - else - log_warning "Secret missing: $secret" - fi - done - fi -} - -validate_storage() { - local storage_name="st${PROJECT_NAME}${ENVIRONMENT}" - storage_name="${storage_name//-/}" - - log_info "Checking Storage Account..." - check_resource "Microsoft.Storage/storageAccounts" "$storage_name" "$RESOURCE_GROUP" "true" - - # Check containers - if az storage account show --name "$storage_name" &>/dev/null; then - log_info "Checking blob containers..." - - local required_containers=("chat-exports" "user-uploads" "summaries") - local connection_string=$(az storage account show-connection-string --name "$storage_name" --resource-group "$RESOURCE_GROUP" -o tsv 2>/dev/null) - - for container in "${required_containers[@]}"; do - ((TOTAL_CHECKS++)) - if az storage container show --name "$container" --connection-string "$connection_string" &>/dev/null; then - log_success "Container exists: $container" - else - log_warning "Container missing: $container" - fi - done - fi -} - -validate_openai() { - local openai_name="oai-${RESOURCE_PREFIX}" - - log_info "Checking Azure OpenAI..." - - ((TOTAL_CHECKS++)) - if az cognitiveservices account show --name "$openai_name" --resource-group "$RESOURCE_GROUP" &>/dev/null; then - log_success "Azure OpenAI exists: $openai_name" - - # Check deployments - log_info "Checking OpenAI model deployments..." - local deployments=$(az cognitiveservices account deployment list --name "$openai_name" --resource-group "$RESOURCE_GROUP" -o json 2>/dev/null) - - local required_models=("gpt-4" "gpt-35-turbo") - for model in "${required_models[@]}"; do - ((TOTAL_CHECKS++)) - if echo "$deployments" | jq -e ".[] | select(.name == \"$model\")" &>/dev/null; then - log_success "Model deployment exists: $model" - else - log_warning "Model deployment missing: $model" - fi - done - else - log_warning "Azure OpenAI not found: $openai_name (optional)" - fi -} - -validate_cosmos_db() { - local cosmos_name="cosmos-${RESOURCE_PREFIX}" - - log_info "Checking Cosmos DB..." - - ((TOTAL_CHECKS++)) - if az cosmosdb show --name "$cosmos_name" --resource-group "$RESOURCE_GROUP" &>/dev/null; then - log_success "Cosmos DB exists: $cosmos_name" - - # Check database and containers - log_info "Checking Cosmos DB containers..." - local containers=$(az cosmosdb sql container list --account-name "$cosmos_name" --database-name "$PROJECT_NAME" --resource-group "$RESOURCE_GROUP" -o json 2>/dev/null) - - local required_containers=("users" "chats" "summaries") - for container in "${required_containers[@]}"; do - ((TOTAL_CHECKS++)) - if echo "$containers" | jq -e ".[] | select(.name == \"$container\")" &>/dev/null; then - log_success "Container exists: $container" - else - log_warning "Container missing: $container" - fi - done - else - log_warning "Cosmos DB not found: $cosmos_name (optional)" - fi -} - -validate_redis() { - local redis_name="redis-${RESOURCE_PREFIX}" - - log_info "Checking Redis Cache..." - - ((TOTAL_CHECKS++)) - if az redis show --name "$redis_name" --resource-group "$RESOURCE_GROUP" &>/dev/null; then - log_success "Redis Cache exists: $redis_name" - - # Check status - local status=$(az redis show --name "$redis_name" --resource-group "$RESOURCE_GROUP" --query "provisioningState" -o tsv 2>/dev/null) - ((TOTAL_CHECKS++)) - if [ "$status" = "Succeeded" ]; then - log_success "Redis Cache status: $status" - else - log_warning "Redis Cache status: $status (expected: Succeeded)" - fi - else - log_warning "Redis Cache not found: $redis_name (optional)" - fi -} - -validate_app_insights() { - local appi_name="appi-${RESOURCE_PREFIX}" - - log_info "Checking Application Insights..." - check_resource "Microsoft.Insights/components" "$appi_name" "$RESOURCE_GROUP" "true" -} - -validate_container_apps() { - local env_name="cae-${RESOURCE_PREFIX}" - local app_name="ca-${RESOURCE_PREFIX}-api" - - log_info "Checking Container Apps Environment..." - - ((TOTAL_CHECKS++)) - if az containerapp env show --name "$env_name" --resource-group "$RESOURCE_GROUP" &>/dev/null; then - log_success "Container Apps Environment exists: $env_name" - - # Check API app - log_info "Checking API Container App..." - ((TOTAL_CHECKS++)) - if az containerapp show --name "$app_name" --resource-group "$RESOURCE_GROUP" &>/dev/null; then - log_success "API Container App exists: $app_name" - - # Check if running - local running=$(az containerapp show --name "$app_name" --resource-group "$RESOURCE_GROUP" --query "properties.runningStatus" -o tsv 2>/dev/null) - ((TOTAL_CHECKS++)) - if [ "$running" = "Running" ]; then - log_success "API Container App status: Running" - else - log_warning "API Container App status: $running" - fi - else - log_warning "API Container App not found: $app_name" - fi - else - log_warning "Container Apps Environment not found: $env_name (optional)" - fi -} - -validate_static_web_app() { - local swa_name="stapp-${RESOURCE_PREFIX}" - - log_info "Checking Static Web App..." - - ((TOTAL_CHECKS++)) - if az staticwebapp show --name "$swa_name" --resource-group "$RESOURCE_GROUP" &>/dev/null; then - log_success "Static Web App exists: $swa_name" - else - log_warning "Static Web App not found: $swa_name (optional)" - fi -} - -# ============================================================================= -# Main Execution -# ============================================================================= - -main() { - echo "" - echo "==============================================" - echo " Azure Resource Validation" - echo " Environment: $ENVIRONMENT" - echo " Project: $PROJECT_NAME" - echo "==============================================" - echo "" - - # Check Azure CLI - if ! command -v az &>/dev/null; then - log_error "Azure CLI not found. Please install: https://docs.microsoft.com/cli/azure/install-azure-cli" - exit 1 - fi - - # Check login - if ! az account show &>/dev/null; then - log_error "Not logged into Azure. Run: az login" - exit 1 - fi - - # Show current subscription - local subscription=$(az account show --query "name" -o tsv) - log_info "Using subscription: $subscription" - echo "" - - # Run validations - validate_resource_group || { log_error "Resource group validation failed. Cannot continue."; exit 1; } - echo "" - - validate_key_vault - echo "" - - validate_storage - echo "" - - validate_openai - echo "" - - validate_cosmos_db - echo "" - - validate_redis - echo "" - - validate_app_insights - echo "" - - validate_container_apps - echo "" - - validate_static_web_app - echo "" - - # Summary - echo "==============================================" - echo " Validation Summary" - echo "==============================================" - echo -e " Total checks: $TOTAL_CHECKS" - echo -e " ${GREEN}Passed:${NC} $PASSED_CHECKS" - echo -e " ${YELLOW}Warnings:${NC} $WARNING_CHECKS" - echo -e " ${RED}Failed:${NC} $FAILED_CHECKS" - echo "==============================================" - echo "" - - # Exit code - if [ "$STRICT_MODE" = "--strict" ] && [ $FAILED_CHECKS -gt 0 ]; then - log_error "Validation failed in strict mode" - exit 1 - elif [ $FAILED_CHECKS -gt 0 ]; then - log_warning "Some required resources are missing" - exit 1 - else - log_success "All validations passed" - exit 0 - fi -} - -# Run main function -main "$@" diff --git a/infra/terraform/.gitignore b/infra/terraform/.gitignore new file mode 100644 index 0000000..0765fc8 --- /dev/null +++ b/infra/terraform/.gitignore @@ -0,0 +1,14 @@ +.terraform/ +*.tfstate +*.tfstate.backup +crash.log +crash.*.log +override.tf +override.tf.json +*_override.tf +*_override.tf.json +.terraformrc +terraform.rc +*.tfvars.local +tfplan +*.tfplan diff --git a/infra/terraform/env/dev/.terraform.lock.hcl b/infra/terraform/env/dev/.terraform.lock.hcl new file mode 100644 index 0000000..372181d --- /dev/null +++ b/infra/terraform/env/dev/.terraform.lock.hcl @@ -0,0 +1,42 @@ +# This file is maintained automatically by "terraform init". +# Manual edits may be lost in future updates. + +provider "registry.terraform.io/hashicorp/azurerm" { + version = "4.72.0" + constraints = ">= 4.62.0" + hashes = [ + "h1:AymDoYag6FI+U2Q89QXJM9oJlHr8c/0d4laM4A/nnk8=", + "zh:073472587c3752e89738522814d2b4eb2fd69eb2cb19c5a5ead3c7d2eabdc279", + "zh:1950effc0c315b6002c8cb6327b94fe59bda210e699367d9727bc66490d651d2", + "zh:47c990db75658525de57c8955a05b4752b88f3a900fffac0e7661d4a749e94f2", + "zh:610f2cbd6fab76750d8b093f03beabbb7162dc8c6affe0109f534ce240b3ff0f", + "zh:6739d645fe548c5a489d711f7748f32368cf68d723d2c59d3f2e21456304d692", + "zh:78d5eefdd9e494defcb3c68d282b8f96630502cac21d1ea161f53cfe9bb483b3", + "zh:a277ab095cc8aff3aede9e43eca2a699936472ef90abb272adf3daa609eb9141", + "zh:b1fdcdaf926c86de0d884beda90d78cb94a42ddede03a1f0b92c36b321d4f07e", + "zh:c003f1f15e52c54e189301ae2c7d8dd65acb2e5a7527d201355f2757b5465ba9", + "zh:c45f2d2206c0f8f71f207cd39eec73da9619d35932bbe1a5b8be7679c50a151e", + "zh:d7040d8ec295481bc1d30346ed7f3075c40ede87c0fedf1db34dd91c1c367a10", + "zh:e595f0b870cd5fd5debdc926fc1740201d2b66188b9b132dc598bdd6444e7348", + ] +} + +provider "registry.terraform.io/hashicorp/random" { + version = "3.8.1" + constraints = ">= 3.6.0" + hashes = [ + "h1:osH3aBqEARwOz3VBJKdpFKJJCNIdgRC6k8vPojkLmlY=", + "zh:08dd03b918c7b55713026037c5400c48af5b9f468f483463321bd18e17b907b4", + "zh:0eee654a5542dc1d41920bbf2419032d6f0d5625b03bd81339e5b33394a3e0ae", + "zh:229665ddf060aa0ed315597908483eee5b818a17d09b6417a0f52fd9405c4f57", + "zh:2469d2e48f28076254a2a3fc327f184914566d9e40c5780b8d96ebf7205f8bc0", + "zh:37d7eb334d9561f335e748280f5535a384a88675af9a9eac439d4cfd663bcb66", + "zh:741101426a2f2c52dee37122f0f4a2f2d6af6d852cb1db634480a86398fa3511", + "zh:78d5eefdd9e494defcb3c68d282b8f96630502cac21d1ea161f53cfe9bb483b3", + "zh:a902473f08ef8df62cfe6116bd6c157070a93f66622384300de235a533e9d4a9", + "zh:b85c511a23e57a2147355932b3b6dce2a11e856b941165793a0c3d7578d94d05", + "zh:c5172226d18eaac95b1daac80172287b69d4ce32750c82ad77fa0768be4ea4b8", + "zh:dab4434dba34aad569b0bc243c2d3f3ff86dd7740def373f2a49816bd2ff819b", + "zh:f49fd62aa8c5525a5c17abd51e27ca5e213881d58882fd42fec4a545b53c9699", + ] +} diff --git a/infra/terraform/env/dev/backend.hcl b/infra/terraform/env/dev/backend.hcl new file mode 100644 index 0000000..df58126 --- /dev/null +++ b/infra/terraform/env/dev/backend.hcl @@ -0,0 +1,4 @@ +resource_group_name = "nl-tfstate-rg" +storage_account_name = "nltfstateconvolens" +container_name = "tfstate" +key = "convolens-dev.tfstate" diff --git a/infra/terraform/env/dev/main.tf b/infra/terraform/env/dev/main.tf new file mode 100644 index 0000000..0695d79 --- /dev/null +++ b/infra/terraform/env/dev/main.tf @@ -0,0 +1,415 @@ +data "azurerm_client_config" "current" {} + +locals { + prefix = "${var.org}-${var.env}-${var.projname}" + alphanumeric = lower(replace(local.prefix, "-", "")) + + rg_name = "${local.prefix}-rg" + law_name = "${local.prefix}-law" + ai_name = "${local.prefix}-appi" + storage_name = substr("${local.alphanumeric}st", 0, 24) + kv_name = substr("${local.prefix}-kv", 0, 24) + cosmos_name = "${local.prefix}-cosmos" + cae_name = "${local.prefix}-cae" + api_name = "${local.prefix}-api" + swa_name = "${local.prefix}-swa" + redis_name = "${local.prefix}-redis" + oai_name = "${local.prefix}-oai" + + base_tags = { + org = var.org + project = var.projname + environment = var.env + managedBy = "terraform" + } + tags = merge(local.base_tags, var.tags) + + blob_containers = ["chat-exports", "user-uploads", "summaries"] + + cosmos_containers = [ + { name = "users", partition_key = "/id" }, + { name = "chats", partition_key = "/userId" }, + { name = "summaries", partition_key = "/chatId" }, + ] + + is_prod = var.env == "prod" + + redis_capacity = local.is_prod ? 1 : 0 + redis_sku = local.is_prod ? "Standard" : "Basic" + + storage_sku = local.is_prod ? "Standard_GRS" : "Standard_LRS" + + ai_retention_days = local.is_prod ? 90 : 30 + + api_cpu = local.is_prod ? 1.0 : 0.5 + api_memory = local.is_prod ? "2.0Gi" : "1.0Gi" + api_min_replicas = local.is_prod ? 2 : 0 + api_max_replicas = local.is_prod ? 10 : 3 +} + +resource "azurerm_resource_group" "rg" { + name = local.rg_name + location = var.location + tags = local.tags +} + +resource "azurerm_log_analytics_workspace" "law" { + name = local.law_name + location = azurerm_resource_group.rg.location + resource_group_name = azurerm_resource_group.rg.name + sku = "PerGB2018" + retention_in_days = local.ai_retention_days + tags = local.tags +} + +resource "azurerm_application_insights" "ai" { + name = local.ai_name + location = azurerm_resource_group.rg.location + resource_group_name = azurerm_resource_group.rg.name + workspace_id = azurerm_log_analytics_workspace.law.id + application_type = "web" + retention_in_days = local.ai_retention_days + tags = local.tags +} + +resource "azurerm_storage_account" "st" { + name = local.storage_name + location = azurerm_resource_group.rg.location + resource_group_name = azurerm_resource_group.rg.name + account_tier = "Standard" + account_replication_type = local.is_prod ? "GRS" : "LRS" + account_kind = "StorageV2" + access_tier = "Hot" + min_tls_version = "TLS1_2" + https_traffic_only_enabled = true + allow_nested_items_to_be_public = false + shared_access_key_enabled = true + tags = local.tags + + blob_properties { + versioning_enabled = local.is_prod + + delete_retention_policy { + days = 7 + } + + container_delete_retention_policy { + days = 7 + } + } + + network_rules { + bypass = ["AzureServices"] + default_action = "Allow" + } +} + +resource "azurerm_storage_container" "containers" { + for_each = toset(local.blob_containers) + name = each.value + storage_account_id = azurerm_storage_account.st.id + container_access_type = "private" +} + +resource "azurerm_cosmosdb_account" "cosmos" { + count = var.enable_cosmos ? 1 : 0 + name = local.cosmos_name + location = azurerm_resource_group.rg.location + resource_group_name = azurerm_resource_group.rg.name + offer_type = "Standard" + kind = "GlobalDocumentDB" + free_tier_enabled = false + public_network_access_enabled = true + automatic_failover_enabled = false + tags = local.tags + + consistency_policy { + consistency_level = "Session" + } + + geo_location { + location = var.location + failover_priority = 0 + zone_redundant = false + } + + capabilities { + name = "EnableServerless" + } +} + +resource "azurerm_cosmosdb_sql_database" "db" { + count = var.enable_cosmos ? 1 : 0 + name = var.database_name + resource_group_name = azurerm_resource_group.rg.name + account_name = azurerm_cosmosdb_account.cosmos[0].name +} + +resource "azurerm_cosmosdb_sql_container" "containers" { + for_each = var.enable_cosmos ? { for c in local.cosmos_containers : c.name => c } : {} + name = each.value.name + resource_group_name = azurerm_resource_group.rg.name + account_name = azurerm_cosmosdb_account.cosmos[0].name + database_name = azurerm_cosmosdb_sql_database.db[0].name + partition_key_paths = [each.value.partition_key] + + indexing_policy { + indexing_mode = "consistent" + + included_path { + path = "/*" + } + + excluded_path { + path = "/\"_etag\"/?" + } + } +} + +resource "azurerm_key_vault" "kv" { + name = local.kv_name + location = azurerm_resource_group.rg.location + resource_group_name = azurerm_resource_group.rg.name + tenant_id = data.azurerm_client_config.current.tenant_id + sku_name = "standard" + enabled_for_deployment = true + enabled_for_template_deployment = true + rbac_authorization_enabled = false + purge_protection_enabled = local.is_prod + soft_delete_retention_days = local.is_prod ? 30 : 7 + public_network_access_enabled = true + tags = local.tags + + network_acls { + bypass = "AzureServices" + default_action = "Allow" + } +} + +resource "azurerm_key_vault_access_policy" "deployer" { + key_vault_id = azurerm_key_vault.kv.id + tenant_id = data.azurerm_client_config.current.tenant_id + object_id = data.azurerm_client_config.current.object_id + + secret_permissions = ["Get", "List", "Set", "Delete", "Purge", "Recover"] +} + +resource "azurerm_key_vault_secret" "cosmos_endpoint" { + count = var.enable_cosmos ? 1 : 0 + name = "cosmos-db-endpoint" + value = azurerm_cosmosdb_account.cosmos[0].endpoint + key_vault_id = azurerm_key_vault.kv.id + + depends_on = [azurerm_key_vault_access_policy.deployer] +} + +resource "azurerm_key_vault_secret" "cosmos_key" { + count = var.enable_cosmos ? 1 : 0 + name = "cosmos-db-key" + value = azurerm_cosmosdb_account.cosmos[0].primary_key + key_vault_id = azurerm_key_vault.kv.id + + depends_on = [azurerm_key_vault_access_policy.deployer] +} + +resource "azurerm_key_vault_secret" "cosmos_connection_string" { + count = var.enable_cosmos ? 1 : 0 + name = "cosmos-db-connection-string" + value = azurerm_cosmosdb_account.cosmos[0].primary_sql_connection_string + key_vault_id = azurerm_key_vault.kv.id + + depends_on = [azurerm_key_vault_access_policy.deployer] +} + +resource "azurerm_key_vault_secret" "appinsights_connection_string" { + name = "appinsights-connection-string" + value = azurerm_application_insights.ai.connection_string + key_vault_id = azurerm_key_vault.kv.id + + depends_on = [azurerm_key_vault_access_policy.deployer] +} + +resource "azurerm_key_vault_secret" "storage_connection_string" { + name = "storage-connection-string" + value = azurerm_storage_account.st.primary_connection_string + key_vault_id = azurerm_key_vault.kv.id + + depends_on = [azurerm_key_vault_access_policy.deployer] +} + +resource "azurerm_redis_cache" "redis" { + count = var.enable_redis ? 1 : 0 + name = local.redis_name + location = azurerm_resource_group.rg.location + resource_group_name = azurerm_resource_group.rg.name + capacity = local.redis_capacity + family = "C" + sku_name = local.redis_sku + minimum_tls_version = "1.2" + tags = local.tags +} + +resource "azurerm_key_vault_secret" "redis_password" { + count = var.enable_redis ? 1 : 0 + name = "redis-password" + value = azurerm_redis_cache.redis[0].primary_access_key + key_vault_id = azurerm_key_vault.kv.id + + depends_on = [azurerm_key_vault_access_policy.deployer] +} + +resource "azurerm_cognitive_account" "openai" { + count = var.enable_openai ? 1 : 0 + name = local.oai_name + location = azurerm_resource_group.rg.location + resource_group_name = azurerm_resource_group.rg.name + kind = "OpenAI" + sku_name = "S0" + custom_subdomain_name = local.oai_name + tags = local.tags +} + +resource "azurerm_container_app_environment" "cae" { + count = var.enable_container_apps ? 1 : 0 + name = local.cae_name + location = azurerm_resource_group.rg.location + resource_group_name = azurerm_resource_group.rg.name + log_analytics_workspace_id = azurerm_log_analytics_workspace.law.id + tags = local.tags +} + +resource "azurerm_container_app" "api" { + count = var.enable_container_apps ? 1 : 0 + name = local.api_name + container_app_environment_id = azurerm_container_app_environment.cae[0].id + resource_group_name = azurerm_resource_group.rg.name + revision_mode = "Single" + tags = local.tags + + identity { + type = "SystemAssigned" + } + + secret { + name = "appinsights-connection-string" + value = azurerm_application_insights.ai.connection_string + } + + ingress { + external_enabled = true + target_port = 3001 + transport = "http" + + traffic_weight { + percentage = 100 + latest_revision = true + } + + cors { + allowed_origins = ["*"] + allowed_methods = ["GET", "POST", "PUT", "DELETE", "OPTIONS"] + allowed_headers = ["*"] + } + } + + template { + min_replicas = local.api_min_replicas + max_replicas = local.api_max_replicas + + container { + name = "api" + image = var.container_image_api + cpu = local.api_cpu + memory = local.api_memory + + env { + name = "NODE_ENV" + value = "production" + } + env { + name = "PORT" + value = "3001" + } + env { + name = "APPLICATIONINSIGHTS_CONNECTION_STRING" + secret_name = "appinsights-connection-string" + } + env { + name = "AZURE_COSMOS_ENDPOINT" + value = var.enable_cosmos ? azurerm_cosmosdb_account.cosmos[0].endpoint : "" + } + env { + name = "AZURE_REDIS_HOSTNAME" + value = var.enable_redis ? azurerm_redis_cache.redis[0].hostname : "" + } + env { + name = "AZURE_STORAGE_ACCOUNT_NAME" + value = azurerm_storage_account.st.name + } + env { + name = "AZURE_OPENAI_ENDPOINT" + value = var.enable_openai ? azurerm_cognitive_account.openai[0].endpoint : "" + } + env { + name = "AZURE_KEY_VAULT_NAME" + value = azurerm_key_vault.kv.name + } + } + + http_scale_rule { + name = "http-rule" + concurrent_requests = "100" + } + } +} + +resource "azurerm_key_vault_access_policy" "container_app" { + count = var.enable_container_apps ? 1 : 0 + key_vault_id = azurerm_key_vault.kv.id + tenant_id = data.azurerm_client_config.current.tenant_id + object_id = azurerm_container_app.api[0].identity[0].principal_id + + secret_permissions = ["Get", "List"] +} + +resource "azurerm_static_web_app" "swa" { + count = var.enable_static_web_app ? 1 : 0 + name = local.swa_name + location = azurerm_resource_group.rg.location + resource_group_name = azurerm_resource_group.rg.name + sku_tier = "Free" + sku_size = "Free" + tags = local.tags +} + +resource "azurerm_consumption_budget_resource_group" "budget" { + count = var.enable_budget_alerts && var.admin_email != "" ? 1 : 0 + name = "${local.prefix}-budget" + resource_group_id = azurerm_resource_group.rg.id + amount = var.monthly_budget_amount + time_grain = "Monthly" + + time_period { + start_date = formatdate("YYYY-MM-01'T'00:00:00'Z'", timestamp()) + } + + notification { + enabled = true + operator = "GreaterThan" + threshold = 80 + threshold_type = "Actual" + contact_emails = [var.admin_email] + } + + notification { + enabled = true + operator = "GreaterThan" + threshold = 100 + threshold_type = "Forecasted" + contact_emails = [var.admin_email] + } + + lifecycle { + ignore_changes = [time_period] + } +} diff --git a/infra/terraform/env/dev/outputs.tf b/infra/terraform/env/dev/outputs.tf new file mode 100644 index 0000000..4076d85 --- /dev/null +++ b/infra/terraform/env/dev/outputs.tf @@ -0,0 +1,71 @@ +output "resource_group_name" { + value = azurerm_resource_group.rg.name + description = "Resource group containing all convolens dev resources." +} + +output "key_vault_name" { + value = azurerm_key_vault.kv.name + description = "Key Vault holding application secrets." +} + +output "key_vault_uri" { + value = azurerm_key_vault.kv.vault_uri + description = "Key Vault dataplane URI." +} + +output "storage_account_name" { + value = azurerm_storage_account.st.name + description = "Storage account name for blob containers." +} + +output "storage_blob_endpoint" { + value = azurerm_storage_account.st.primary_blob_endpoint + description = "Public blob endpoint." +} + +output "appinsights_connection_string" { + value = azurerm_application_insights.ai.connection_string + description = "Application Insights connection string. Forwarded to the API container as APPLICATIONINSIGHTS_CONNECTION_STRING." + sensitive = true +} + +output "log_analytics_workspace_id" { + value = azurerm_log_analytics_workspace.law.id + description = "Log Analytics workspace ARM ID (referenced by Container Apps env)." +} + +output "cosmos_endpoint" { + value = var.enable_cosmos ? azurerm_cosmosdb_account.cosmos[0].endpoint : "" + description = "Cosmos DB document endpoint, empty when enable_cosmos = false." +} + +output "cosmos_database_name" { + value = var.enable_cosmos ? azurerm_cosmosdb_sql_database.db[0].name : "" + description = "Cosmos SQL database name, empty when enable_cosmos = false." +} + +output "redis_hostname" { + value = var.enable_redis ? azurerm_redis_cache.redis[0].hostname : "" + description = "Redis hostname, empty when enable_redis = false." +} + +output "openai_endpoint" { + value = var.enable_openai ? azurerm_cognitive_account.openai[0].endpoint : "" + description = "Azure OpenAI endpoint, empty when enable_openai = false." +} + +output "container_app_url" { + value = var.enable_container_apps ? "https://${azurerm_container_app.api[0].latest_revision_fqdn}" : "" + description = "Public URL of the API container app, empty when enable_container_apps = false." +} + +output "static_web_app_default_host" { + value = var.enable_static_web_app ? "https://${azurerm_static_web_app.swa[0].default_host_name}" : "" + description = "Public URL of the Static Web App, empty when enable_static_web_app = false." +} + +output "static_web_app_api_key" { + value = var.enable_static_web_app ? azurerm_static_web_app.swa[0].api_key : "" + description = "SWA deployment token consumed by the GitHub Actions deploy workflow." + sensitive = true +} diff --git a/infra/terraform/env/dev/terraform.tf b/infra/terraform/env/dev/terraform.tf new file mode 100644 index 0000000..81417d7 --- /dev/null +++ b/infra/terraform/env/dev/terraform.tf @@ -0,0 +1,28 @@ +terraform { + required_version = ">= 1.14.0" + + required_providers { + azurerm = { + source = "hashicorp/azurerm" + version = ">= 4.62.0" + } + random = { + source = "hashicorp/random" + version = ">= 3.6.0" + } + } + + backend "azurerm" {} +} + +provider "azurerm" { + features { + key_vault { + purge_soft_delete_on_destroy = true + recover_soft_deleted_key_vaults = true + } + resource_group { + prevent_deletion_if_contains_resources = false + } + } +} diff --git a/infra/terraform/env/dev/terraform.tfvars b/infra/terraform/env/dev/terraform.tfvars new file mode 100644 index 0000000..0952fda --- /dev/null +++ b/infra/terraform/env/dev/terraform.tfvars @@ -0,0 +1,22 @@ +env = "dev" +org = "nl" +projname = "convolens" +location = "eastus2" + +database_name = "convolens" + +enable_cosmos = true +enable_openai = false +enable_redis = false +enable_container_apps = true +enable_static_web_app = true +enable_budget_alerts = false + +admin_email = "" +monthly_budget_amount = 100 + +container_image_api = "mcr.microsoft.com/azuredocs/containerapps-helloworld:latest" + +tags = { + costCenter = "development" +} diff --git a/infra/terraform/env/dev/variables.tf b/infra/terraform/env/dev/variables.tf new file mode 100644 index 0000000..f5cced7 --- /dev/null +++ b/infra/terraform/env/dev/variables.tf @@ -0,0 +1,103 @@ +variable "env" { + type = string + description = "Environment name (dev/staging/prod). Drives naming and SKU sizing." + default = "dev" + validation { + condition = contains(["dev", "staging", "prod"], var.env) + error_message = "env must be dev, staging, or prod." + } +} + +variable "org" { + type = string + description = "Organisation prefix per NL Azure Naming Standards (ADR-0027)." + default = "nl" + validation { + condition = contains(["nl", "pvc", "tws", "mys"], var.org) + error_message = "org must be one of nl, pvc, tws, mys." + } +} + +variable "projname" { + type = string + description = "Project name used for resource naming." + default = "convolens" + validation { + condition = length(var.projname) >= 3 && length(var.projname) <= 15 + error_message = "projname must be 3-15 characters." + } +} + +variable "location" { + type = string + description = "Primary Azure region for resources." + default = "eastus2" +} + +variable "database_name" { + type = string + description = "Cosmos DB SQL database name. Decoupled from projname so brand renames don't force data migration." + default = "convolens" +} + +variable "enable_cosmos" { + type = bool + description = "Provision Cosmos DB account, database, and containers." + default = true +} + +variable "enable_openai" { + type = bool + description = "Provision Azure OpenAI account. Disabled in dev because Azure AI Foundry is configured separately." + default = false +} + +variable "enable_redis" { + type = bool + description = "Provision Azure Cache for Redis. Off by default in dev to avoid the Basic-tier Redis cost (~$15-30/mo)." + default = false +} + +variable "enable_container_apps" { + type = bool + description = "Provision Container Apps Environment + API Container App." + default = true +} + +variable "enable_static_web_app" { + type = bool + description = "Provision Static Web App for the frontend." + default = true +} + +variable "enable_budget_alerts" { + type = bool + description = "Provision a resource-group-scoped consumption budget. Skipped automatically when admin_email is empty." + default = true +} + +variable "admin_email" { + type = string + description = "Email address that receives budget alerts. Empty disables budget alerts regardless of enable_budget_alerts." + default = "" +} + +variable "monthly_budget_amount" { + type = number + description = "Monthly budget in USD." + default = 100 +} + +variable "container_image_api" { + type = string + description = "Image for the API Container App. Defaults to the Microsoft helloworld placeholder; the API CD pipeline overrides this." + default = "mcr.microsoft.com/azuredocs/containerapps-helloworld:latest" +} + +variable "tags" { + type = map(string) + description = "Additional tags merged with the org/project/env defaults." + default = { + costCenter = "development" + } +} diff --git a/infra/terraform/env/prod/.terraform.lock.hcl b/infra/terraform/env/prod/.terraform.lock.hcl new file mode 100644 index 0000000..b9b3824 --- /dev/null +++ b/infra/terraform/env/prod/.terraform.lock.hcl @@ -0,0 +1,43 @@ +# This file is maintained automatically by "terraform init". +# Manual edits may be lost in future updates. + +provider "registry.terraform.io/hashicorp/azurerm" { + version = "4.81.0" + constraints = ">= 4.62.0" + hashes = [ + "h1:4BEgtU/crlkbMduT7tm8Sp3+C+1RBfrpUIlTBM/wtMc=", + "zh:0732e7b74264ddfa2b90ba69d01c283d3cbae9f72ed3e506c6ac92529fed7fd3", + "zh:12afb524e232fe4e3d6161927724af5dfa4831d71edd9c174917ca9b7377bfae", + "zh:169d619ae202c4145e02fb706fb7c3679445ab3e3ff722edbf89597517a8c92e", + "zh:6beb95a3ef2f2d9c76abaa48e5450e90686a3fb6a47f1cb0ff7c5e94b6960151", + "zh:705e075fb5ffc4bf66fd7cbabf1a65007a41621e80030a2c158a4c83b6046216", + "zh:78d5eefdd9e494defcb3c68d282b8f96630502cac21d1ea161f53cfe9bb483b3", + "zh:79a8d17fefe647040fcb9ee8821a4f09f395427c4fd49493489b9a93a9a1038e", + "zh:8cc3f900b3774c0ae37ae42365c4579a199cf9e5edf88e476fdf5ab1048f84ea", + "zh:dec373b9390fa95e257291acd018ed65a7d512b428645d35e22cdbe8b245a08b", + "zh:e60f1e9fb45df6defade2855ed6e68547409ea75d30655c556adb0c08579749b", + "zh:f901d12ec82f3f8b5880a27b5cbcd7bd0d97e60c9367a2d7ed82fdd1157b39ff", + "zh:facf68ea5bf0f2b8ba720e7fba5f86492e1d4c591100460bb91c3f79f391f4b6", + ] +} + +provider "registry.terraform.io/hashicorp/random" { + version = "3.9.0" + constraints = ">= 3.6.0" + hashes = [ + "h1:q/uaUTBdKgAmZESrwsoeDQff9uUA/cI/N5ZKNgVwa9c=", + "zh:161ad0bd9a75768c82f53fb6e7172a9d8be2d4889b012645a34795031aaf1bf1", + "zh:19dc9a5b17729725ccfc4f45b0500af0ee5bc6b6b160c7adb8f2bf617d2c80ea", + "zh:269eda8fe42daa7974d5a34d166c3ba9defe80cde86c01e4dadcfdf2e1f05e5f", + "zh:373f7c65566f8f2cc7f45d698654feb9d988996957e1266a69ca00c52d6d16d0", + "zh:5599d16804c41c83009ec621b6d6b6f74e102f5827678a4750f8809055546b61", + "zh:583be0440469a22bff70dcfa56593b01566860b29607437264adb51060cf46fc", + "zh:5f211d8ec3f2e1f414870d9584bfe26e6995560ef81c748f8447a48164767398", + "zh:78d5eefdd9e494defcb3c68d282b8f96630502cac21d1ea161f53cfe9bb483b3", + "zh:7b547fd16216761ef86efc3ed516ac5ac0c5c42b7c7eb24a08cef2d93f69ed5e", + "zh:7e7c0679daf2a382151d05068c8c3f0dae6b7b7dccf818827b73dd08638df2ef", + "zh:8089dec888a8038b9b4fb23b3df7e1057293dbc5b60b42cc47ff690d69d4b61b", + "zh:c51f15a031edfd6f23ce8ced3446ca7f8d8d647e2499890d7d5d10d5016d7257", + "zh:c94784f005708890dc6895afd53636ec00ec1e430b15d41e5aebfb1d4b39bd04", + ] +} diff --git a/infra/terraform/env/prod/backend.hcl b/infra/terraform/env/prod/backend.hcl new file mode 100644 index 0000000..c16987b --- /dev/null +++ b/infra/terraform/env/prod/backend.hcl @@ -0,0 +1,4 @@ +resource_group_name = "nl-tfstate-rg" +storage_account_name = "nltfstateconvolens" +container_name = "tfstate" +key = "convolens-prod.tfstate" diff --git a/infra/terraform/env/prod/main.tf b/infra/terraform/env/prod/main.tf new file mode 100644 index 0000000..c8ae079 --- /dev/null +++ b/infra/terraform/env/prod/main.tf @@ -0,0 +1,471 @@ +data "azurerm_client_config" "current" {} + +locals { + prefix = "${var.org}-${var.env}-${var.projname}" + alphanumeric = lower(replace(local.prefix, "-", "")) + + rg_name = "${local.prefix}-rg" + law_name = "${local.prefix}-law" + ai_name = "${local.prefix}-appi" + storage_name = substr("${local.alphanumeric}st", 0, 24) + kv_name = substr("${local.prefix}-kv", 0, 24) + postgres_name = "${local.prefix}-pg" + cae_name = "${local.prefix}-cae" + api_name = "${local.prefix}-api" + service_plan_name = "${local.prefix}-asp" + frontend_name = "${local.prefix}-web" + acr_name = substr("${local.alphanumeric}acr", 0, 50) + redis_name = "${local.prefix}-redis" + ingestion_queue_name = "ingestion" + publish_queue_name = "baton-publish" + + tags = merge( + { + org = var.org + project = var.projname + environment = var.env + managedBy = "terraform" + }, + var.tags, + ) + + blob_containers = [ + "chat-exports", + "raw-artifacts", + "normalized-artifacts", + "review-exports", + ] +} + +resource "azurerm_resource_group" "rg" { + name = local.rg_name + location = var.location + tags = local.tags +} + +resource "azurerm_log_analytics_workspace" "law" { + name = local.law_name + location = azurerm_resource_group.rg.location + resource_group_name = azurerm_resource_group.rg.name + sku = "PerGB2018" + retention_in_days = 30 + daily_quota_gb = 1 + tags = local.tags +} + +resource "azurerm_application_insights" "ai" { + name = local.ai_name + location = azurerm_resource_group.rg.location + resource_group_name = azurerm_resource_group.rg.name + workspace_id = azurerm_log_analytics_workspace.law.id + application_type = "web" + retention_in_days = 30 + daily_data_cap_in_gb = 1 + tags = local.tags +} + +resource "azurerm_key_vault" "kv" { + name = local.kv_name + location = azurerm_resource_group.rg.location + resource_group_name = azurerm_resource_group.rg.name + tenant_id = data.azurerm_client_config.current.tenant_id + sku_name = "standard" + purge_protection_enabled = true + soft_delete_retention_days = 90 + rbac_authorization_enabled = true + public_network_access_enabled = true + tags = local.tags + + network_acls { + bypass = "AzureServices" + default_action = "Allow" + } +} + +resource "azurerm_role_assignment" "deployer_key_vault_secrets_officer" { + scope = azurerm_key_vault.kv.id + role_definition_name = "Key Vault Secrets Officer" + principal_id = data.azurerm_client_config.current.object_id +} + +resource "azurerm_storage_account" "st" { + name = local.storage_name + location = azurerm_resource_group.rg.location + resource_group_name = azurerm_resource_group.rg.name + account_tier = "Standard" + account_replication_type = "LRS" + account_kind = "StorageV2" + access_tier = "Hot" + min_tls_version = "TLS1_2" + https_traffic_only_enabled = true + allow_nested_items_to_be_public = false + shared_access_key_enabled = false + tags = local.tags + + blob_properties { + versioning_enabled = true + + delete_retention_policy { + days = 7 + } + + container_delete_retention_policy { + days = 7 + } + } + +} + +resource "azurerm_storage_account_queue_properties" "queue" { + storage_account_id = azurerm_storage_account.st.id + + logging { + delete = true + read = true + write = true + version = "1.0" + retention_policy_days = 30 + } +} + +resource "azurerm_storage_container" "containers" { + for_each = toset(local.blob_containers) + name = each.value + storage_account_id = azurerm_storage_account.st.id + container_access_type = "private" +} + +resource "azurerm_storage_queue" "ingestion" { + name = local.ingestion_queue_name + storage_account_id = azurerm_storage_account.st.id +} + +resource "azurerm_storage_queue" "baton_publish" { + name = local.publish_queue_name + storage_account_id = azurerm_storage_account.st.id +} + +resource "random_password" "postgres_admin" { + length = 32 + special = true + override_special = "!#$%&*()-_=+[]{}<>:?" +} + +resource "azurerm_postgresql_flexible_server" "postgres" { + count = var.enable_postgres ? 1 : 0 + name = local.postgres_name + location = azurerm_resource_group.rg.location + resource_group_name = azurerm_resource_group.rg.name + version = "16" + administrator_login = var.postgres_admin_login + administrator_password = random_password.postgres_admin.result + sku_name = var.postgres_sku_name + storage_mb = var.postgres_storage_mb + backup_retention_days = var.postgres_backup_retention_days + public_network_access_enabled = true + zone = "1" + tags = local.tags +} + +resource "azurerm_postgresql_flexible_server_database" "app" { + count = var.enable_postgres ? 1 : 0 + name = var.database_name + server_id = azurerm_postgresql_flexible_server.postgres[0].id + charset = "UTF8" + collation = "en_US.utf8" +} + +resource "azurerm_postgresql_flexible_server_firewall_rule" "azure_services" { + count = var.enable_postgres ? 1 : 0 + name = "allow-azure-services" + server_id = azurerm_postgresql_flexible_server.postgres[0].id + start_ip_address = "0.0.0.0" + end_ip_address = "0.0.0.0" +} + +resource "azurerm_key_vault_secret" "postgres_password" { + count = var.enable_postgres ? 1 : 0 + name = "postgres-admin-password" + value = random_password.postgres_admin.result + key_vault_id = azurerm_key_vault.kv.id + + depends_on = [azurerm_role_assignment.deployer_key_vault_secrets_officer] +} + +resource "azurerm_key_vault_secret" "appinsights_connection_string" { + name = "appinsights-connection-string" + value = azurerm_application_insights.ai.connection_string + key_vault_id = azurerm_key_vault.kv.id + + depends_on = [azurerm_role_assignment.deployer_key_vault_secrets_officer] +} + +resource "azurerm_container_registry" "acr" { + count = var.enable_container_registry ? 1 : 0 + name = local.acr_name + location = azurerm_resource_group.rg.location + resource_group_name = azurerm_resource_group.rg.name + sku = "Basic" + admin_enabled = false + anonymous_pull_enabled = false + public_network_access_enabled = true + tags = local.tags +} + +resource "azurerm_container_app_environment" "cae" { + name = local.cae_name + location = azurerm_resource_group.rg.location + resource_group_name = azurerm_resource_group.rg.name + log_analytics_workspace_id = azurerm_log_analytics_workspace.law.id + tags = local.tags +} + +resource "azurerm_container_app" "api" { + name = local.api_name + container_app_environment_id = azurerm_container_app_environment.cae.id + resource_group_name = azurerm_resource_group.rg.name + revision_mode = "Single" + tags = local.tags + + identity { + type = "SystemAssigned" + } + + secret { + name = "appinsights-connection-string" + value = azurerm_application_insights.ai.connection_string + } + + secret { + name = "db-password" + value = random_password.postgres_admin.result + } + + ingress { + external_enabled = true + target_port = var.api_target_port + transport = "http" + + traffic_weight { + percentage = 100 + latest_revision = true + } + + cors { + allowed_origins = [var.allowed_origin] + allowed_methods = ["GET", "POST", "PUT", "DELETE", "OPTIONS"] + allowed_headers = ["content-type", "authorization", "x-correlation-id", "x-idempotency-key"] + exposed_headers = [] + } + } + + template { + min_replicas = 0 + max_replicas = 2 + + container { + name = "api" + image = var.container_image_api + cpu = 0.5 + memory = "1Gi" + args = [] + command = [] + + env { + name = "NODE_ENV" + value = "production" + } + env { + name = "PORT" + value = tostring(var.api_target_port) + } + env { + name = "FRONTEND_URL" + value = var.allowed_origin + } + env { + name = "CORS_ORIGIN" + value = var.allowed_origin + } + env { + name = "DB_TYPE" + value = var.enable_postgres ? "postgres" : "sqlite" + } + env { + name = "DB_HOST" + value = var.enable_postgres ? azurerm_postgresql_flexible_server.postgres[0].fqdn : "" + } + env { + name = "DB_PORT" + value = "5432" + } + env { + name = "DB_USERNAME" + value = var.postgres_admin_login + } + env { + name = "DB_PASSWORD" + secret_name = "db-password" + } + env { + name = "DB_NAME" + value = var.enable_postgres ? azurerm_postgresql_flexible_server_database.app[0].name : var.database_name + } + env { + name = "DB_SSL" + value = var.enable_postgres ? "true" : "false" + } + env { + name = "DB_MIGRATIONS_RUN" + value = var.enable_postgres ? "true" : "false" + } + env { + name = "DATABASE_PATH" + value = "/tmp/convolens-eval.sqlite" + } + env { + name = "APPLICATIONINSIGHTS_CONNECTION_STRING" + secret_name = "appinsights-connection-string" + } + env { + name = "AZURE_APP_INSIGHTS_CONNECTION_STRING" + secret_name = "appinsights-connection-string" + } + env { + name = "AZURE_STORAGE_ACCOUNT_NAME" + value = azurerm_storage_account.st.name + } + env { + name = "AZURE_STORAGE_CONTAINER" + value = "chat-exports" + } + env { + name = "AZURE_STORAGE_INGESTION_QUEUE" + value = azurerm_storage_queue.ingestion.name + } + env { + name = "AZURE_STORAGE_BATON_PUBLISH_QUEUE" + value = azurerm_storage_queue.baton_publish.name + } + env { + name = "AZURE_KEY_VAULT_NAME" + value = azurerm_key_vault.kv.name + } + } + + http_scale_rule { + name = "http-rule" + concurrent_requests = "100" + } + } +} + +resource "azurerm_service_plan" "frontend" { + name = local.service_plan_name + location = azurerm_resource_group.rg.location + resource_group_name = azurerm_resource_group.rg.name + os_type = "Linux" + sku_name = "F1" + tags = local.tags +} + +resource "azurerm_linux_web_app" "frontend" { + name = local.frontend_name + location = azurerm_resource_group.rg.location + resource_group_name = azurerm_resource_group.rg.name + service_plan_id = azurerm_service_plan.frontend.id + https_only = true + tags = local.tags + + identity { + type = "SystemAssigned" + } + + site_config { + always_on = false + + application_stack { + node_version = var.frontend_runtime_stack + } + } + + app_settings = { + NODE_ENV = "production" + NEXT_PUBLIC_API_URL = "https://${azurerm_container_app.api.ingress[0].fqdn}/api" + NEXT_PUBLIC_API_BASE_URL = "https://${azurerm_container_app.api.ingress[0].fqdn}" + APPLICATIONINSIGHTS_CONNECTION_STRING = azurerm_application_insights.ai.connection_string + AZURE_APP_INSIGHTS_CONNECTION_STRING = azurerm_application_insights.ai.connection_string + SCM_DO_BUILD_DURING_DEPLOYMENT = "true" + WEBSITE_RUN_FROM_PACKAGE = "1" + CONVOLENS_CANONICAL_HOSTNAME = var.custom_hostname + } +} + +resource "azurerm_redis_cache" "redis" { + count = var.enable_redis ? 1 : 0 + name = local.redis_name + location = azurerm_resource_group.rg.location + resource_group_name = azurerm_resource_group.rg.name + capacity = 1 + family = "C" + sku_name = "Standard" + minimum_tls_version = "1.2" + public_network_access_enabled = true + tags = local.tags +} + +resource "azurerm_role_assignment" "api_blob_contributor" { + scope = azurerm_storage_account.st.id + role_definition_name = "Storage Blob Data Contributor" + principal_id = azurerm_container_app.api.identity[0].principal_id +} + +resource "azurerm_role_assignment" "api_queue_contributor" { + scope = azurerm_storage_account.st.id + role_definition_name = "Storage Queue Data Contributor" + principal_id = azurerm_container_app.api.identity[0].principal_id +} + +resource "azurerm_role_assignment" "api_key_vault_secrets_user" { + scope = azurerm_key_vault.kv.id + role_definition_name = "Key Vault Secrets User" + principal_id = azurerm_container_app.api.identity[0].principal_id +} + +resource "azurerm_role_assignment" "frontend_key_vault_secrets_user" { + scope = azurerm_key_vault.kv.id + role_definition_name = "Key Vault Secrets User" + principal_id = azurerm_linux_web_app.frontend.identity[0].principal_id +} + +resource "azurerm_consumption_budget_resource_group" "budget" { + count = var.enable_budget_alerts && var.admin_email != "" ? 1 : 0 + name = "${local.prefix}-budget" + resource_group_id = azurerm_resource_group.rg.id + amount = var.monthly_budget_amount + time_grain = "Monthly" + + time_period { + start_date = formatdate("YYYY-MM-01'T'00:00:00'Z'", timestamp()) + } + + notification { + enabled = true + operator = "GreaterThan" + threshold = 80 + threshold_type = "Actual" + contact_emails = [var.admin_email] + } + + notification { + enabled = true + operator = "GreaterThan" + threshold = 100 + threshold_type = "Forecasted" + contact_emails = [var.admin_email] + } + + lifecycle { + ignore_changes = [time_period] + } +} diff --git a/infra/terraform/env/prod/outputs.tf b/infra/terraform/env/prod/outputs.tf new file mode 100644 index 0000000..25f4fd8 --- /dev/null +++ b/infra/terraform/env/prod/outputs.tf @@ -0,0 +1,70 @@ +output "resource_group_name" { + value = azurerm_resource_group.rg.name + description = "Resource group containing Convolens production resources." +} + +output "key_vault_name" { + value = azurerm_key_vault.kv.name + description = "Key Vault holding production secrets." +} + +output "key_vault_uri" { + value = azurerm_key_vault.kv.vault_uri + description = "Key Vault dataplane URI." +} + +output "storage_account_name" { + value = azurerm_storage_account.st.name + description = "Storage account for raw exports, artifacts, and queues." +} + +output "storage_blob_endpoint" { + value = azurerm_storage_account.st.primary_blob_endpoint + description = "Blob endpoint for production storage." +} + +output "ingestion_queue_name" { + value = azurerm_storage_queue.ingestion.name + description = "Queue for ingestion work." +} + +output "baton_publish_queue_name" { + value = azurerm_storage_queue.baton_publish.name + description = "Queue for Baton publish work." +} + +output "postgres_server_fqdn" { + value = var.enable_postgres ? azurerm_postgresql_flexible_server.postgres[0].fqdn : "" + description = "PostgreSQL Flexible Server FQDN." +} + +output "postgres_database_name" { + value = var.enable_postgres ? azurerm_postgresql_flexible_server_database.app[0].name : "" + description = "Application database name." +} + +output "application_insights_connection_string" { + value = azurerm_application_insights.ai.connection_string + description = "Application Insights connection string." + sensitive = true +} + +output "container_app_url" { + value = "https://${azurerm_container_app.api.ingress[0].fqdn}" + description = "Stable public URL of the API Container App." +} + +output "frontend_default_hostname" { + value = "https://${azurerm_linux_web_app.frontend.default_hostname}" + description = "Default App Service URL for the frontend before custom DNS binding." +} + +output "custom_hostname" { + value = var.custom_hostname + description = "Approved production hostname to bind during go-live." +} + +output "container_registry_login_server" { + value = var.enable_container_registry ? azurerm_container_registry.acr[0].login_server : "" + description = "Dedicated ACR login server, empty when ACR is disabled." +} diff --git a/infra/terraform/env/prod/terraform.tf b/infra/terraform/env/prod/terraform.tf new file mode 100644 index 0000000..8770aa3 --- /dev/null +++ b/infra/terraform/env/prod/terraform.tf @@ -0,0 +1,30 @@ +terraform { + required_version = ">= 1.14.0" + + required_providers { + azurerm = { + source = "hashicorp/azurerm" + version = ">= 4.62.0" + } + random = { + source = "hashicorp/random" + version = ">= 3.6.0" + } + } + + backend "azurerm" {} +} + +provider "azurerm" { + storage_use_azuread = true + + features { + key_vault { + purge_soft_delete_on_destroy = false + recover_soft_deleted_key_vaults = true + } + resource_group { + prevent_deletion_if_contains_resources = true + } + } +} diff --git a/infra/terraform/env/prod/terraform.tfvars b/infra/terraform/env/prod/terraform.tfvars new file mode 100644 index 0000000..9957660 --- /dev/null +++ b/infra/terraform/env/prod/terraform.tfvars @@ -0,0 +1,24 @@ +env = "prod" +org = "nl" +projname = "convolens" +location = "southafricanorth" + +database_name = "convolens" + +allowed_origin = "https://convolens.neuralliquid.ai" +custom_hostname = "convolens.neuralliquid.ai" + +enable_budget_alerts = true +enable_container_registry = false +enable_postgres = false +enable_redis = false + +admin_email = "" +monthly_budget_amount = 75 + +container_image_api = "mcr.microsoft.com/azuredocs/containerapps-helloworld:latest" +api_target_port = 80 + +tags = { + costCenter = "production" +} diff --git a/infra/terraform/env/prod/variables.tf b/infra/terraform/env/prod/variables.tf new file mode 100644 index 0000000..f65dd48 --- /dev/null +++ b/infra/terraform/env/prod/variables.tf @@ -0,0 +1,143 @@ +variable "env" { + type = string + description = "Environment name. Prod is intentionally the only supported value in this environment." + default = "prod" + + validation { + condition = var.env == "prod" + error_message = "This Terraform environment only supports env = prod." + } +} + +variable "org" { + type = string + description = "Organisation prefix per NL Azure naming standards." + default = "nl" + + validation { + condition = var.org == "nl" + error_message = "The approved production prefix is nl." + } +} + +variable "projname" { + type = string + description = "Project name used for resource naming." + default = "convolens" + + validation { + condition = var.projname == "convolens" + error_message = "This production environment is scoped to convolens." + } +} + +variable "location" { + type = string + description = "Primary Azure region for production resources." + default = "southafricanorth" +} + +variable "database_name" { + type = string + description = "PostgreSQL database name." + default = "convolens" +} + +variable "postgres_admin_login" { + type = string + description = "PostgreSQL administrator login. The password is generated and stored in Key Vault." + default = "convolensadmin" + sensitive = true +} + +variable "postgres_sku_name" { + type = string + description = "PostgreSQL Flexible Server SKU." + default = "B_Standard_B1ms" +} + +variable "postgres_storage_mb" { + type = number + description = "PostgreSQL storage size in MB." + default = 32768 +} + +variable "postgres_backup_retention_days" { + type = number + description = "PostgreSQL backup retention period." + default = 7 +} + +variable "enable_postgres" { + type = bool + description = "Provision PostgreSQL Flexible Server. Keep false for the internal eval apply to minimize recurring cost." + default = false +} + +variable "container_image_api" { + type = string + description = "Container image for the API. The deployment pipeline should override the placeholder." + default = "mcr.microsoft.com/azuredocs/containerapps-helloworld:latest" +} + +variable "api_target_port" { + type = number + description = "Container App ingress target port. Internal eval uses the Azure helloworld placeholder on port 80; set to 3001 when deploying the real API image." + default = 80 +} + +variable "frontend_runtime_stack" { + type = string + description = "Linux App Service runtime stack for the Next.js frontend." + default = "20-lts" +} + +variable "allowed_origin" { + type = string + description = "Canonical production origin for CORS and browser callbacks." + default = "https://convolens.neuralliquid.ai" +} + +variable "custom_hostname" { + type = string + description = "Canonical production hostname. DNS binding is a go-live step after validation." + default = "convolens.neuralliquid.ai" +} + +variable "admin_email" { + type = string + description = "Email address that receives budget alerts. Empty disables budget alerts." + default = "" +} + +variable "monthly_budget_amount" { + type = number + description = "Monthly resource-group budget in USD." + default = 75 +} + +variable "enable_budget_alerts" { + type = bool + description = "Provision a resource-group-scoped consumption budget when admin_email is set." + default = true +} + +variable "enable_container_registry" { + type = bool + description = "Create a dedicated production Azure Container Registry for Convolens images." + default = false +} + +variable "enable_redis" { + type = bool + description = "Provision Azure Cache for Redis for distributed cache/session workloads." + default = false +} + +variable "tags" { + type = map(string) + description = "Additional tags merged with org/project/env defaults." + default = { + costCenter = "production" + } +} diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 3833d64..9c6b83e 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -170,6 +170,9 @@ importers: multer: specifier: ^2.0.2 version: 2.0.2 + pg: + specifier: ^8.16.3 + version: 8.22.0 puppeteer: specifier: ^21.0.0 version: 21.11.0(typescript@5.8.3) @@ -187,7 +190,7 @@ importers: version: 5.1.7 typeorm: specifier: ^0.3.26 - version: 0.3.26(reflect-metadata@0.2.2)(sqlite3@5.1.7)(ts-node@10.9.2) + version: 0.3.26(pg@8.22.0)(reflect-metadata@0.2.2)(sqlite3@5.1.7)(ts-node@10.9.2) uuid: specifier: ^13.0.0 version: 13.0.0 @@ -10881,6 +10884,68 @@ packages: resolution: {integrity: sha512-F3asv42UuXchdzt+xXqfW1OGlVBe+mxa2mqI0pg5yAHZPvFmY3Y6drSf/GQ1A86WgWEN9Kzh/WrgKa6iGcHXLg==} dev: false + /pg-cloudflare@1.4.0: + resolution: {integrity: sha512-Vo7z/6rrQYxpNRylp4Tlob2elzbh+N/MOQbxFVWCxS7oEx6jF53GTJFxK2WWpKuBRkmiin4Mt+xofFDjx09R0A==} + requiresBuild: true + dev: false + optional: true + + /pg-connection-string@2.14.0: + resolution: {integrity: sha512-XwWDGcLRGCXAR8F/AM5bG7Q+A3Wm2s6QeEjlOKZLlH3UYcguiqCWKyWXVag5TLTIjR7oOJUY8kcADaZgWPyLeg==} + dev: false + + /pg-int8@1.0.1: + resolution: {integrity: sha512-WCtabS6t3c8SkpDBUlb1kjOs7l66xsGdKpIPZsg4wR+B3+u9UAum2odSsF9tnvxg80h4ZxLWMy4pRjOsFIqQpw==} + engines: {node: '>=4.0.0'} + dev: false + + /pg-pool@3.14.0(pg@8.22.0): + resolution: {integrity: sha512-gKtPkFdQPU3DksooVLi9LsjZxrsBUZIpa+7aVx+LV5pNh0KzP4Zleud2po+ConrxbuXGBJ6Hfer6hdgpIBpBaw==} + peerDependencies: + pg: '>=8.0' + dependencies: + pg: 8.22.0 + dev: false + + /pg-protocol@1.15.0: + resolution: {integrity: sha512-cq9sECI5s0+uPUXjbz8ioyPJni6RzsRib0US67i5IoTZKw8fNeYlVE7u8F4dG7vEJJtc5wdD1K189lCCUwqWTQ==} + dev: false + + /pg-types@2.2.0: + resolution: {integrity: sha512-qTAAlrEsl8s4OiEQY69wDvcMIdQN6wdz5ojQiOy6YRMuynxenON0O5oCpJI6lshc6scgAY8qvJ2On/p+CXY0GA==} + engines: {node: '>=4'} + dependencies: + pg-int8: 1.0.1 + postgres-array: 2.0.0 + postgres-bytea: 1.0.1 + postgres-date: 1.0.7 + postgres-interval: 1.2.0 + dev: false + + /pg@8.22.0: + resolution: {integrity: sha512-8wih1vVIBMxoUM2oB4soJsD9tDnDpLv4OXBJ+EJzFsvycD+lfyIreC2gGHq78f8jbLLt+bvlPTFdFZfJkOuzAA==} + engines: {node: '>= 16.0.0'} + peerDependencies: + pg-native: '>=3.0.1' + peerDependenciesMeta: + pg-native: + optional: true + dependencies: + pg-connection-string: 2.14.0 + pg-pool: 3.14.0(pg@8.22.0) + pg-protocol: 1.15.0 + pg-types: 2.2.0 + pgpass: 1.0.5 + optionalDependencies: + pg-cloudflare: 1.4.0 + dev: false + + /pgpass@1.0.5: + resolution: {integrity: sha512-FdW9r/jQZhSeohs1Z3sI1yxFQNFvMcnmfuj4WBMUTxOrAyLMaTcE1aAMBiTlbMNaXvBCQuVi0R7hd8udDSP7ug==} + dependencies: + split2: 4.2.0 + dev: false + /picocolors@1.1.1: resolution: {integrity: sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA==} @@ -11034,6 +11099,28 @@ packages: picocolors: 1.1.1 source-map-js: 1.2.1 + /postgres-array@2.0.0: + resolution: {integrity: sha512-VpZrUqU5A69eQyW2c5CA1jtLecCsN2U/bD6VilrFDWq5+5UIEVO7nazS3TEcHf1zuPYO/sqGvUvW62g86RXZuA==} + engines: {node: '>=4'} + dev: false + + /postgres-bytea@1.0.1: + resolution: {integrity: sha512-5+5HqXnsZPE65IJZSMkZtURARZelel2oXUEO8rH83VS/hxH5vv1uHquPg5wZs8yMAfdv971IU+kcPUczi7NVBQ==} + engines: {node: '>=0.10.0'} + dev: false + + /postgres-date@1.0.7: + resolution: {integrity: sha512-suDmjLVQg78nMK2UZ454hAG+OAW+HQPZ6n++TNDUX+L0+uUlLywnoxJKDou51Zm+zTCjrCl0Nq6J9C5hP9vK/Q==} + engines: {node: '>=0.10.0'} + dev: false + + /postgres-interval@1.2.0: + resolution: {integrity: sha512-9ZhXKM/rw350N1ovuWHbGxnGh/SNJ4cnxHiM0rxE4VN41wsg8P8zWn9hv/buK00RP4WvlOyr/RBDiptyxVbkZQ==} + engines: {node: '>=0.10.0'} + dependencies: + xtend: 4.0.2 + dev: false + /preact-render-to-string@5.2.6(preact@10.28.0): resolution: {integrity: sha512-JyhErpYOvBV1hEPwIxc/fHWXPfnEGdRKxc8gFdAZ7XV4tlzyzG847XAyEZqoDnynP88akM4eaHcSOzNcLWFguw==} peerDependencies: @@ -12540,6 +12627,11 @@ packages: whatwg-url: 7.1.0 dev: true + /split2@4.2.0: + resolution: {integrity: sha512-UcjcJOWknrNkF6PLX83qcHM6KHgVKNkV62Y8a5uYDVv9ydGQVwAHMKqHdJje1VTWpljG0WYpCDhrCdAOYH4TWg==} + engines: {node: '>= 10.x'} + dev: false + /sprintf-js@1.0.3: resolution: {integrity: sha512-D9cPgkvLlV3t3IzL0D0YLvGA9Ahk4PcvVwUbN0dSGr1aP0Nrt4AEnTUbuGvquEC0mA64Gqt1fzirlRs5ibXx8g==} dev: true @@ -13579,7 +13671,7 @@ packages: resolution: {integrity: sha512-/aCDEGatGvZ2BIk+HmLf4ifCJFwvKFNb9/JeZPMulfgFracn9QFcAf5GO8B/mweUjSoblS5In0cWhqpfs/5PQA==} dev: false - /typeorm@0.3.26(reflect-metadata@0.2.2)(sqlite3@5.1.7)(ts-node@10.9.2): + /typeorm@0.3.26(pg@8.22.0)(reflect-metadata@0.2.2)(sqlite3@5.1.7)(ts-node@10.9.2): resolution: {integrity: sha512-o2RrBNn3lczx1qv4j+JliVMmtkPSqEGpG0UuZkt9tCfWkoXKu8MZnjvp2GjWPll1SehwemQw6xrbVRhmOglj8Q==} engines: {node: '>=16.13.0'} hasBin: true @@ -13644,6 +13736,7 @@ packages: dedent: 1.7.0 dotenv: 16.6.1 glob: 10.5.0 + pg: 8.22.0 reflect-metadata: 0.2.2 sha.js: 2.4.12 sql-highlight: 6.1.0