Standardize repo file/folder structure & naming (canonical service names, consistent files, per-feature subdirectories)

[thzgajendra]

Summary

The repo's file/folder layout and naming have drifted service-by-service, so the same logical service is named differently in every layer, files for the same feature use different conventions, and a single "feature" is smeared across three unrelated directories with no home. This tracks an audit of the scatter and a standard, generic convention so that every existing service can be aligned and every future service/feature drops into a predictable place.

Decision (chosen): keep the existing 3-layer architecture (shared abstraction → per-cloud mock → per-cloud wire) — it's sound — but enforce one canonical name per service across all layers, one file-naming rule, a fixed per-directory file layout, and same-named subdirectories for services that span multiple sub-surfaces.

---

The problem (audit)

1. The same service has a different name in every layer — there is no canonical name to grep:

Service	providers/<cloud>/	server/<cloud>/	services/
IAM	awsiam / azureiam / gcpiam	iam / iam / iam	iam
Storage	s3 / blobstorage / gcs	s3 / blob / gcs	storage
Compute	ec2(+vpc) / virtualmachines / gce	ec2 / virtualmachines / compute	compute
Load balancer	elb / azurelb / gcplb	elbv2 / loadbalancer / loadbalancer	loadbalancer
Database (Cosmos)	cosmosdb	cosmos	database

2. Cloud-prefixing on provider dirs is arbitrary (half-and-half within each cloud): AWS awsiam vs ec2/s3; Azure azureai/azurecache/azuredns vs acr/aks/blobstorage; GCP gcpiam/gcplb/gcpvpc vs gce/gcs/gke.

3. Two file-naming conventions for the same feature + abbreviation drift — providers use flat-lowercase abbreviations, servers use snake_case full words:

Feature	provider file	server file
NAT gateway	natgw.go	nat_gateway.go
Internet gateway	igw.go	internet_gateway.go
Elastic IP	eip.go	address.go
Network interface	eni.go	network_interface.go
Traffic mirroring	trafficmirror.go	traffic_mirror.go

4. A feature has no home — e.g. Traffic Mirroring is an interface fragment in services/networking/driver/aws_capabilities.go + a mock in providers/aws/vpc/trafficmirror.go + a wire handler in server/aws/ec2/traffic_mirror.go. Nothing is named for the feature; reading or adding one means editing three trees.

5. Inconsistent granularity — within one service, some sub-features are flat files and some are whole subdirectories (server/azure/databricks/ has arm_*.go/dataplane_*.go files and dbfs/, pipelines/, scim/, unitycatalog/, … subdirs) with no rule.

6. Layers are not 1:1 — Azure virtualmachines provider fans out to server dirs virtualmachines+disks+images+snapshots+sshpublickeys; AWS ec2+vpc providers collapse into one server/aws/ec2. Provider↔server can't be mapped by name.

7. The services/ abstraction layer has no single philosophy — mixes generic capabilities (compute, storage, networking) with product names (bedrock, vertexai, databricks, bigtable); some have a driver/ subdir, some (cost, kubernetes, scope) don't.

---

Proposed convention

1. Canonical name (one per service, all layers)

Every service has exactly one canonical name, used verbatim in all three layers, with no cloud prefix (the parent aws/, azure/, gcp/ already encodes the cloud):

services/<canonical>/            # shared abstraction + portable API + driver interface
providers/<cloud>/<canonical>/   # in-memory mock
server/<cloud>/<canonical>/      # wire handler

• Prefer the generic capability name where a shared abstraction exists (compute, storage, networking, loadbalancer, iam, database, dns, monitoring, logging, secrets, messagequeue, eventbus, serverless).
• Product-specific services with no cross-cloud abstraction keep their product name consistently in all layers (bedrock, vertexai, databricks, bigtable).
• Kill the cloud prefixes: awsiam→iam, azurecache→cache, gcpvpc→networking, blobstorage→storage, gce→compute, elb/elbv2→loadbalancer, etc.

2. File naming (one rule, everywhere)

• snake_case, full words, no abbreviations, in all layers. natgw.go/nat_gateway.go → nat_gateway.go everywhere; trafficmirror.go/traffic_mirror.go → traffic_mirror.go everywhere; eip.go→elastic_ip.go; eni.go→network_interface.go.
• A feature is the same filename across the triple, so one grep finds all three layers of it.

3. Fixed per-directory layout (template)

services/<canonical>/
  driver/driver.go     # the interface (always a driver/ subdir — normalize the 4 that lack it)
  <canonical>.go       # portable API + do() pipeline
  <canonical>_test.go

providers/<cloud>/<canonical>/
  <canonical>.go       # store + core CRUD
  clone.go             # copy-on-write helpers
  <feature>.go         # one file per feature
  <canonical>_test.go

server/<cloud>/<canonical>/
  handler.go           # routing/dispatch
  types.go             # wire DTOs + mapping
  <feature>.go         # one wire file per feature, same name as the provider's

4. Subdirectories for multi-surface services

When a service spans multiple sub-surfaces / a large feature set (e.g. networking → traffic mirroring, network insights, VPC block-public-access, transit gateway, VPN; or compute → instances, disks, snapshots, images), group each sub-surface into a same-named subdirectory in each of the three layers instead of dozens of flat files:

services/networking/traffic_mirror/     (interface fragment)
providers/aws/networking/traffic_mirror/ (mock)
server/aws/networking/traffic_mirror/    (wire)

Rule of thumb: > ~5 files or a self-contained sub-API ⇒ promote it to a same-named subdirectory across all layers. Small services stay flat. This gives a feature a real home (one subdir per layer) while keeping the layer separation — adding a new feature = adding one same-named subdirectory in each layer with the fixed file set above.

---

Rollout (incremental, not a big bang)

1. Land this convention as a short doc (e.g. docs/STRUCTURE.md / a CONTRIBUTING section) — the source of truth.
2. Add a CI parity check that fails when the three layers of a service don't share the canonical name, or when a file breaks the snake_case rule — so drift can't return.
3. Convert service-by-service (one PR per service, mechanical git mv + import fixups + wiring rename), starting with the worst offenders (IAM, storage, compute/EC2+VPC, load balancer, the networking capability sprawl). No behavior change per PR.
4. Split the ec2+vpc provider and the virtualmachines+disks+… server fan-out onto the canonical compute/networking names with subdirectories.

Acceptance criteria

• Convention documented as the single source of truth.
• One canonical name per service across services/, providers/<cloud>/, server/<cloud>/; no cloud-prefixed dirs.
• One file-naming rule (snake_case, full words) enforced repo-wide; a feature shares its filename across all three layers.
• Every services/<x>/ has a driver/ subdir.
• Multi-surface services use same-named subdirectories per layer; a template exists for both flat and subdir services.
• CI check prevents future naming/structure drift.
• All existing services migrated (tracked as sub-tasks), each a no-behavior-change PR.