From 3dd3bf942dfb3777c3205bf8ce379a6d187b511e Mon Sep 17 00:00:00 2001 From: Omri SirComp Date: Fri, 15 May 2026 18:54:48 +0300 Subject: [PATCH 1/3] docs(testing): align validation guide with current frameworks --- docs/contributing/testing-validation.md | 264 +++++++++++++----------- 1 file changed, 141 insertions(+), 123 deletions(-) diff --git a/docs/contributing/testing-validation.md b/docs/contributing/testing-validation.md index cf92de56c..86a19ed63 100644 --- a/docs/contributing/testing-validation.md +++ b/docs/contributing/testing-validation.md @@ -13,7 +13,7 @@ keywords: - terraform testing - bicep testing - pester - - terratest + - terraform test - checkov - security testing --- @@ -32,9 +32,9 @@ To maintain code quality and the OSSF Best Practices Badge, we enforce the follo | Technology | Framework | Minimum Requirement | |:---------------|:------------------------|:------------------------------------------------------| | **Terraform** | native `terraform test` | One `.tftest.hcl` per component with `command = plan` | -| **Rust** | `cargo test` | `#[cfg(test)]` module covering core logic | +| **Rust** | `cargo test` | Run from each service crate with a local `Cargo.toml` | | **.NET** | xUnit / NUnit | Test project covering business logic | -| **JavaScript** | vitest | Test file with 80% coverage threshold | +| **JavaScript** | Jest / TypeScript | Docs tests and `tsc --noEmit` checks | ## Testing Philosophy @@ -85,22 +85,27 @@ tflint --config=.tflint.hcl tflint main.tf variables.tf ``` -#### Testing Framework +#### Native Terraform Tests -Use Terratest for integration testing: +Use Terraform's native test framework for component tests. Place `.tftest.hcl` +files under the component's `terraform/tests/` directory and use `command = plan` +for fast validation that does not deploy Azure resources. ```bash -# Navigate to test directory -cd src/000-cloud/010-security-identity/tests +# Navigate to a component Terraform directory +cd src/000-cloud/000-resource-group/terraform -# Run Go tests -go test -v -timeout 30m +# Initialize providers and modules +terraform init + +# Run the component's .tftest.hcl files +terraform test -# Run specific test -go test -v -run TestTerraformSecurityIdentity +# Run through the repository npm helper +npm run tf-test -- src/000-cloud/000-resource-group/terraform -# Run tests with verbose output -go test -v -timeout 30m ./... +# Run all Terraform tests +npm run tf-test-all ``` ### Bicep Testing @@ -138,6 +143,41 @@ az bicep lint --file main.bicep az bicep lint --file main.bicep --level Error ``` +### Rust Testing + +Rust services in this repository are tested from their service directories. The +root `Cargo.toml` intentionally has no workspace members, so run `cargo test` +from the crate that changed. + +```bash +# Navigate to the changed Rust service +cd src/500-application/512-avro-to-json/operators/avro-to-json + +# Run unit and integration tests for that crate +cargo test + +# Optional static checks for Rust changes +cargo fmt --check +cargo clippy --all-targets --all-features +``` + +### JavaScript and Docs Testing + +The Docusaurus docs package uses Jest for docs tests and TypeScript for static +type validation. + +```bash +# Run docs tests through the root helper +npm run docs:test + +# Run tests and type checks directly from the docs package +npm run --prefix docs/docusaurus test +npm run --prefix docs/docusaurus typecheck + +# Build the docs site before publishing larger docs changes +npm run docs:build +``` + ## Validation Tools ### Security Scanning with Checkov @@ -192,10 +232,10 @@ Maintain documentation quality: npm run cspell # Check specific file -npx cspell docs/contributor/testing-validation.md +npx cspell docs/contributing/testing-validation.md # Add words to project dictionary -echo "terratest" >> .cspell-dictionary.txt +echo "Docusaurus" >> .cspell-dictionary.txt ``` ## Pre-Commit Validation @@ -251,13 +291,12 @@ src/000-cloud/010-security-identity/ ├── terraform/ │ ├── main.tf │ ├── variables.tf -│ └── outputs.tf -├── tests/ -│ ├── go.mod -│ ├── go.sum -│ ├── terraform_test.go -│ └── fixtures/ -│ └── test-parameters.tfvars +│ ├── outputs.tf +│ └── tests/ +│ ├── naming-convention.tftest.hcl +│ ├── output-validation.tftest.hcl +│ └── setup/ +│ └── main.tf └── ci/ └── terraform/ ├── main.tf @@ -266,56 +305,55 @@ src/000-cloud/010-security-identity/ ### Writing Component Tests -Create comprehensive test coverage: - -```go -// Example: tests/terraform_test.go -package test - -import ( - "testing" - "github.com/gruntwork-io/terratest/modules/terraform" - "github.com/stretchr/testify/assert" -) - -func TestTerraformSecurityIdentity(t *testing.T) { - t.Parallel() +Create comprehensive test coverage with `.tftest.hcl` files: - terraformOptions := terraform.WithDefaultRetryableErrors(t, &terraform.Options{ - TerraformDir: "../terraform", - VarFiles: []string{"fixtures/test-parameters.tfvars"}, - }) - - defer terraform.Destroy(t, terraformOptions) +```hcl +# Example: terraform/tests/naming-convention.tftest.hcl +provider "azurerm" { + storage_use_azuread = true + features {} +} - // Apply the Terraform configuration - terraform.InitAndApply(t, terraformOptions) +run "resource_group_uses_expected_name" { + command = plan - // Validate outputs - keyVaultName := terraform.Output(t, terraformOptions, "key_vault_name") - assert.NotEmpty(t, keyVaultName) + variables { + resource_prefix = "edge-ai" + environment = "dev" + location = "eastus2" + instance = "001" + } - // Additional validations - resourceGroupName := terraform.Output(t, terraformOptions, "resource_group_name") - assert.Contains(t, resourceGroupName, "test") + assert { + condition = azurerm_resource_group.new[0].name == "rg-edge-ai-dev-001" + error_message = "Resource group name should follow the expected naming convention" + } } ``` ### Test Data Management -Use fixture files for test parameters: +Use test setup modules or inline `variables` blocks for test data. The resource +group component provides a setup module for generated prefixes: ```hcl -# tests/fixtures/test-parameters.tfvars -prefix = "test" -environment = "dev" -location = "East US" -enable_monitoring = true +# terraform/tests/setup/main.tf +resource "random_string" "resource_prefix" { + length = 8 + special = false + upper = false +} + +output "resource_prefix" { + value = random_string.resource_prefix.result +} ``` ## Blueprint Testing -Some blueprints include comprehensive test suites using Go and the Terratest framework. The testing infrastructure validates both IaC declarations and actual deployments. +Blueprint testing should follow the same layered approach as component testing: +fast static validation first, then optional deployment validation when Azure +credentials and cost controls are available. ### Blueprint Test Architecture @@ -327,14 +365,13 @@ Provides reusable testing functions for all blueprints including: - Deployment and cleanup utilities - Output normalization across frameworks -**Reference Implementation:** [blueprints/full-single-node-cluster/tests/](https://github.com/microsoft/edge-ai/tree/main/blueprints/full-single-node-cluster/tests/) +**Reference Implementation:** [src/000-cloud/000-resource-group/terraform/tests/](https://github.com/microsoft/edge-ai/tree/main/src/000-cloud/000-resource-group/terraform/tests/) Complete test suite demonstrating: -- Contract tests for both Terraform and Bicep -- End-to-end deployment validation -- Helper scripts for test execution -- Output contract definitions +- Native Terraform test files using `command = plan` +- Naming, tag, boundary, and output assertions +- Shared setup module usage for generated test values ### Contract Testing @@ -350,17 +387,12 @@ Complete test suite demonstrating: **Running Contract Tests:** ```bash -cd blueprints/full-single-node-cluster/tests - -# Test both frameworks -./run-contract-tests.sh both - -# Test specific framework -./run-contract-tests.sh terraform -./run-contract-tests.sh bicep +cd src/000-cloud/000-resource-group/terraform +terraform init +terraform test -# Direct Go execution -go test -v -run Contract +# Or run through the repository helper +npm run tf-test -- src/000-cloud/000-resource-group/terraform ``` ### Deployment Testing @@ -377,43 +409,39 @@ go test -v -run Contract **Running Deployment Tests:** ```bash -cd blueprints/full-single-node-cluster/tests - -# Enable automatic cleanup -export CLEANUP_RESOURCES=true +cd blueprints/full-single-node-cluster/terraform -# Test specific framework -./run-deployment-tests.sh terraform -./run-deployment-tests.sh bicep +# Terraform deployment check +terraform init +terraform plan -var-file="test.tfvars" +terraform apply -var-file="test.tfvars" -auto-approve -# Direct Go execution -go test -v -run TestTerraformFullSingleNodeClusterDeploy -timeout 2h -go test -v -run TestBicepFullSingleNodeClusterDeploy -timeout 2h +# Clean up billable resources after validation +terraform destroy -var-file="test.tfvars" -auto-approve ``` **Environment Variables:** -- `CLEANUP_RESOURCES` - Auto-delete resources after test (default: `false`) +- `ARM_CLIENT_ID`, `ARM_CLIENT_SECRET`, `ARM_TENANT_ID`, `ARM_SUBSCRIPTION_ID` - Azure credentials for non-interactive Terraform runs - `TEST_ENVIRONMENT` - Environment name (default: `dev`) - `TEST_LOCATION` - Azure region (default: `eastus2`) - `TEST_RESOURCE_PREFIX` - Resource naming prefix (default: `t6`) -- `SKIP_BICEP_DEPLOYMENT` - Use existing deployment (default: `false`) ### Blueprint Test Organization Each blueprint test suite includes: ```text -blueprints/{blueprint-name}/tests/ -├── outputs.go # Output contract definition -├── contract_terraform_test.go # Terraform contract validation -├── contract_bicep_test.go # Bicep contract validation -├── deploy_terraform_test.go # Terraform deployment test -├── deploy_bicep_test.go # Bicep deployment test -├── validation.go # Shared validation functions -├── setup.go # Post-deployment setup -├── run-contract-tests.sh # Contract test runner -└── run-deployment-tests.sh # Deployment test runner +blueprints/{blueprint-name}/terraform/ +├── main.tf +├── variables.tf +├── outputs.tf +├── tests/ +│ ├── contract.tftest.hcl # Fast plan-time contract validation +│ ├── output-validation.tftest.hcl +│ └── setup/ +│ └── main.tf # Optional shared setup values +└── test.tfvars # Optional deployment validation values ``` ### Blueprint Integration Testing @@ -440,11 +468,11 @@ terraform destroy -var-file="test.tfvars" -auto-approve When creating a new blueprint, add comprehensive test coverage: -1. **Define output contract** in `tests/outputs.go` with struct tags for both frameworks -2. **Create contract tests** for static validation -3. **Create deployment tests** for end-to-end validation -4. **Add helper scripts** for simplified test execution -5. **Document test requirements** in blueprint README +1. **Define output assertions** in `.tftest.hcl` files +2. **Create plan-time contract tests** with `command = plan` +3. **Create deployment validation steps** for scenarios that require Azure resources +4. **Use setup modules** for generated test values that are shared across runs +5. **Document test requirements** in the blueprint README **See:** [Blueprint Developer Guide](../getting-started/blueprint-developer.md#testing-and-validation) for detailed instructions @@ -505,17 +533,8 @@ The CI/CD pipeline includes comprehensive testing: #### Test Stage ```yaml -- task: GoTool@0 - displayName: 'Use Go 1.19' - inputs: - version: '1.19' - -- task: Go@0 - displayName: 'Run Infrastructure Tests' - inputs: - command: 'test' - arguments: '-v -timeout 30m ./tests/...' - workingDirectory: '$(System.DefaultWorkingDirectory)' +- name: Run Terraform tests + run: npm run tf-test-all ``` ### Quality Gates @@ -567,11 +586,11 @@ az account set --subscription "your-subscription-id" #### Test Timeout Issues ```bash -# Increase timeout for long-running tests -go test -v -timeout 60m ./tests/... +# Narrow the run to one Terraform test file while debugging +terraform test -filter=tests/naming-convention.tftest.hcl -# Run tests in parallel with limited concurrency -go test -v -parallel 2 ./tests/... +# Run one component at a time through the helper +npm run tf-test -- src/000-cloud/000-resource-group/terraform ``` ### Debugging Test Failures @@ -601,11 +620,10 @@ az deployment group create \ #### Test Data Investigation ```bash -# Preserve test resources for investigation -export SKIP_teardown=true -go test -v -run TestSpecificCase +# Inspect a plan before applying it +terraform plan -var-file="test.tfvars" -out=tfplan -# Manual cleanup after investigation +# Clean up after investigation terraform destroy -auto-approve ``` @@ -631,11 +649,11 @@ az deployment group show \ Optimize test execution time: ```bash -# Run tests in parallel -go test -v -parallel 4 ./tests/... +# Run all Terraform tests through the parallel repository helper +npm run tf-test-all -# Profile test performance -go test -v -cpuprofile=cpu.prof -memprofile=mem.prof ./tests/... +# Run a single component while narrowing a failure +npm run tf-test -- src/000-cloud/000-resource-group/terraform ``` ## Best Practices @@ -652,7 +670,7 @@ go test -v -cpuprofile=cpu.prof -memprofile=mem.prof ./tests/... ### Best Practices for Test Data - **Use parameterized tests** for multiple scenarios -- **Clean up test resources** automatically (set `CLEANUP_RESOURCES=true`) +- **Clean up test resources** with `terraform destroy` after deployment validation - **Isolate test environments** to prevent interference - **Use realistic test data** that represents production scenarios - **Run contract tests first** to catch errors before expensive deployments From 09c844e52a49a0ec528d878184eeaa5a62bb8839 Mon Sep 17 00:00:00 2001 From: Omri SirComp Date: Sat, 16 May 2026 00:15:43 +0300 Subject: [PATCH 2/3] docs: address testing guide review feedback --- docs/contributing/testing-validation.md | 47 ++++++++++++++++++------- 1 file changed, 35 insertions(+), 12 deletions(-) diff --git a/docs/contributing/testing-validation.md b/docs/contributing/testing-validation.md index 86a19ed63..ef6b5a714 100644 --- a/docs/contributing/testing-validation.md +++ b/docs/contributing/testing-validation.md @@ -32,7 +32,7 @@ To maintain code quality and the OSSF Best Practices Badge, we enforce the follo | Technology | Framework | Minimum Requirement | |:---------------|:------------------------|:------------------------------------------------------| | **Terraform** | native `terraform test` | One `.tftest.hcl` per component with `command = plan` | -| **Rust** | `cargo test` | Run from each service crate with a local `Cargo.toml` | +| **Rust** | `cargo test` | `#[cfg(test)]` module covering core logic | | **.NET** | xUnit / NUnit | Test project covering business logic | | **JavaScript** | Jest / TypeScript | Docs tests and `tsc --noEmit` checks | @@ -314,19 +314,26 @@ provider "azurerm" { features {} } -run "resource_group_uses_expected_name" { +# Call the setup module to create a random resource prefix +run "setup_tests" { + module { + source = "./tests/setup" + } +} + +run "naming_with_different_environments" { command = plan variables { - resource_prefix = "edge-ai" - environment = "dev" + resource_prefix = run.setup_tests.resource_prefix + environment = "prod" location = "eastus2" instance = "001" } assert { - condition = azurerm_resource_group.new[0].name == "rg-edge-ai-dev-001" - error_message = "Resource group name should follow the expected naming convention" + condition = azurerm_resource_group.new[0].name == "rg-${run.setup_tests.resource_prefix}-prod-001" + error_message = "Resource group name should follow convention with environment = prod" } } ``` @@ -338,14 +345,24 @@ group component provides a setup module for generated prefixes: ```hcl # terraform/tests/setup/main.tf -resource "random_string" "resource_prefix" { - length = 8 +terraform { + required_providers { + random = { + source = "hashicorp/random" + version = ">= 3.5.1" + } + } + required_version = ">= 1.12.0, < 2.0" +} + +resource "random_string" "prefix" { + length = 4 special = false upper = false } output "resource_prefix" { - value = random_string.resource_prefix.result + value = "a${random_string.prefix.id}" } ``` @@ -355,17 +372,22 @@ Blueprint testing should follow the same layered approach as component testing: fast static validation first, then optional deployment validation when Azure credentials and cost controls are available. -### Blueprint Test Architecture +### Terraform Test Reference + +The resource group component is the current concrete Terraform test reference. +Blueprint tests should follow the same layered approach: fast static validation +first, then optional deployment validation when Azure credentials and cost +controls are available. **Shared Test Utilities:** [src/900-tools-utilities/904-test-utilities/](https://github.com/microsoft/edge-ai/tree/main/src/900-tools-utilities/904-test-utilities/) -Provides reusable testing functions for all blueprints including: +Provides reusable testing functions for blueprints including: - Contract validation functions for Terraform and Bicep - Deployment and cleanup utilities - Output normalization across frameworks -**Reference Implementation:** [src/000-cloud/000-resource-group/terraform/tests/](https://github.com/microsoft/edge-ai/tree/main/src/000-cloud/000-resource-group/terraform/tests/) +**Terraform Reference Implementation:** [src/000-cloud/000-resource-group/terraform/tests/](https://github.com/microsoft/edge-ai/tree/main/src/000-cloud/000-resource-group/terraform/tests/) Complete test suite demonstrating: @@ -414,6 +436,7 @@ cd blueprints/full-single-node-cluster/terraform # Terraform deployment check terraform init terraform plan -var-file="test.tfvars" +# Requires ARM_* credentials; provisions billable Azure resources. terraform apply -var-file="test.tfvars" -auto-approve # Clean up billable resources after validation From 8eb7c259b5e31d7e010cf28dcc50122f046c3619 Mon Sep 17 00:00:00 2001 From: Omri SirComp Date: Sat, 13 Jun 2026 22:16:36 +0300 Subject: [PATCH 3/3] docs: preserve blueprint Terratest guidance --- docs/contributing/testing-validation.md | 98 +++++++++++++------------ 1 file changed, 51 insertions(+), 47 deletions(-) diff --git a/docs/contributing/testing-validation.md b/docs/contributing/testing-validation.md index ef6b5a714..a3532a74f 100644 --- a/docs/contributing/testing-validation.md +++ b/docs/contributing/testing-validation.md @@ -14,6 +14,7 @@ keywords: - bicep testing - pester - terraform test + - terratest - checkov - security testing --- @@ -31,10 +32,11 @@ To maintain code quality and the OSSF Best Practices Badge, we enforce the follo | Technology | Framework | Minimum Requirement | |:---------------|:------------------------|:------------------------------------------------------| -| **Terraform** | native `terraform test` | One `.tftest.hcl` per component with `command = plan` | -| **Rust** | `cargo test` | `#[cfg(test)]` module covering core logic | -| **.NET** | xUnit / NUnit | Test project covering business logic | -| **JavaScript** | Jest / TypeScript | Docs tests and `tsc --noEmit` checks | +| **Terraform components** | native `terraform test` | One `.tftest.hcl` per component with `command = plan` | +| **Blueprint IaC** | Go / Terratest | Contract and deployment tests under blueprint `tests/` | +| **Rust** | `cargo test` | `#[cfg(test)]` module covering core logic | +| **.NET** | xUnit / NUnit | Test project covering business logic | +| **JavaScript** | Jest / TypeScript | Docs tests and `tsc --noEmit` checks | ## Testing Philosophy @@ -368,32 +370,26 @@ output "resource_prefix" { ## Blueprint Testing -Blueprint testing should follow the same layered approach as component testing: -fast static validation first, then optional deployment validation when Azure -credentials and cost controls are available. +Some blueprints include comprehensive test suites using Go and the Terratest framework. The testing infrastructure validates both IaC declarations and actual deployments. -### Terraform Test Reference - -The resource group component is the current concrete Terraform test reference. -Blueprint tests should follow the same layered approach: fast static validation -first, then optional deployment validation when Azure credentials and cost -controls are available. +### Blueprint Test Architecture **Shared Test Utilities:** [src/900-tools-utilities/904-test-utilities/](https://github.com/microsoft/edge-ai/tree/main/src/900-tools-utilities/904-test-utilities/) -Provides reusable testing functions for blueprints including: +Provides reusable testing functions for all blueprints including: - Contract validation functions for Terraform and Bicep - Deployment and cleanup utilities - Output normalization across frameworks -**Terraform Reference Implementation:** [src/000-cloud/000-resource-group/terraform/tests/](https://github.com/microsoft/edge-ai/tree/main/src/000-cloud/000-resource-group/terraform/tests/) +**Reference Implementation:** [blueprints/full-single-node-cluster/tests/](https://github.com/microsoft/edge-ai/tree/main/blueprints/full-single-node-cluster/tests/) Complete test suite demonstrating: -- Native Terraform test files using `command = plan` -- Naming, tag, boundary, and output assertions -- Shared setup module usage for generated test values +- Contract tests for both Terraform and Bicep +- End-to-end deployment validation +- Helper scripts for test execution +- Output contract definitions ### Contract Testing @@ -409,12 +405,17 @@ Complete test suite demonstrating: **Running Contract Tests:** ```bash -cd src/000-cloud/000-resource-group/terraform -terraform init -terraform test +cd blueprints/full-single-node-cluster/tests -# Or run through the repository helper -npm run tf-test -- src/000-cloud/000-resource-group/terraform +# Test both frameworks +./run-contract-tests.sh both + +# Test specific framework +./run-contract-tests.sh terraform +./run-contract-tests.sh bicep + +# Direct Go execution +go test -v -run Contract ``` ### Deployment Testing @@ -431,40 +432,43 @@ npm run tf-test -- src/000-cloud/000-resource-group/terraform **Running Deployment Tests:** ```bash -cd blueprints/full-single-node-cluster/terraform +cd blueprints/full-single-node-cluster/tests -# Terraform deployment check -terraform init -terraform plan -var-file="test.tfvars" -# Requires ARM_* credentials; provisions billable Azure resources. -terraform apply -var-file="test.tfvars" -auto-approve +# Enable automatic cleanup +export CLEANUP_RESOURCES=true -# Clean up billable resources after validation -terraform destroy -var-file="test.tfvars" -auto-approve +# Test specific framework +./run-deployment-tests.sh terraform +./run-deployment-tests.sh bicep + +# Direct Go execution +go test -v -run TestTerraformFullSingleNodeClusterDeploy -timeout 2h +go test -v -run TestBicepFullSingleNodeClusterDeploy -timeout 2h ``` **Environment Variables:** -- `ARM_CLIENT_ID`, `ARM_CLIENT_SECRET`, `ARM_TENANT_ID`, `ARM_SUBSCRIPTION_ID` - Azure credentials for non-interactive Terraform runs +- `CLEANUP_RESOURCES` - Auto-delete resources after test (default: `false`) - `TEST_ENVIRONMENT` - Environment name (default: `dev`) - `TEST_LOCATION` - Azure region (default: `eastus2`) - `TEST_RESOURCE_PREFIX` - Resource naming prefix (default: `t6`) +- `SKIP_BICEP_DEPLOYMENT` - Use existing deployment (default: `false`) ### Blueprint Test Organization Each blueprint test suite includes: ```text -blueprints/{blueprint-name}/terraform/ -├── main.tf -├── variables.tf -├── outputs.tf -├── tests/ -│ ├── contract.tftest.hcl # Fast plan-time contract validation -│ ├── output-validation.tftest.hcl -│ └── setup/ -│ └── main.tf # Optional shared setup values -└── test.tfvars # Optional deployment validation values +blueprints/{blueprint-name}/tests/ +├── outputs.go # Output contract definition +├── contract_terraform_test.go # Terraform contract validation +├── contract_bicep_test.go # Bicep contract validation +├── deploy_terraform_test.go # Terraform deployment test +├── deploy_bicep_test.go # Bicep deployment test +├── validation.go # Shared validation functions +├── setup.go # Post-deployment setup +├── run-contract-tests.sh # Contract test runner +└── run-deployment-tests.sh # Deployment test runner ``` ### Blueprint Integration Testing @@ -491,11 +495,11 @@ terraform destroy -var-file="test.tfvars" -auto-approve When creating a new blueprint, add comprehensive test coverage: -1. **Define output assertions** in `.tftest.hcl` files -2. **Create plan-time contract tests** with `command = plan` -3. **Create deployment validation steps** for scenarios that require Azure resources -4. **Use setup modules** for generated test values that are shared across runs -5. **Document test requirements** in the blueprint README +1. **Define output contract** in `tests/outputs.go` with struct tags for both frameworks +2. **Create contract tests** for static validation +3. **Create deployment tests** for end-to-end validation +4. **Add helper scripts** for simplified test execution +5. **Document test requirements** in blueprint README **See:** [Blueprint Developer Guide](../getting-started/blueprint-developer.md#testing-and-validation) for detailed instructions