Terraform builds
the theatre.

Fix your provider versions. A provider without a fixed version is a time bomb that goes off on a Friday. The azurerm provider is on version 4.x, and several setting names changed from 3.x, which is the most common reason a copied snippet does not apply.

terraform {
  required_version = "~> 1.9"

  required_providers {
    azurerm = { source = "hashicorp/azurerm", version = "~> 4.30" }
    azuread = { source = "hashicorp/azuread", version = "~> 3.0" }
  }

  # remote state: shared, locked, versioned
  backend "azurerm" {
    resource_group_name  = "rg-tfstate"
    storage_account_name = "stcontosotfstate"
    container_name       = "tfstate"
    key                  = "prod/aks.tfstate"
    use_azuread_auth     = true   # no storage keys anywhere
  }
}

provider "azurerm" {
  features {}
  use_oidc = true   # federated auth from CI — no client secret
}

resource "azurerm_virtual_network" "main" {
  name                = "vnet-contoso-prod"
  address_space       = ["10.40.0.0/16"]
  location            = azurerm_resource_group.main.location
  resource_group_name = azurerm_resource_group.main.name
}

resource "azurerm_subnet" "aks" {
  name                 = "snet-aks-nodes"
  resource_group_name  = azurerm_resource_group.main.name
  # ↓ THIS reference is what creates the dependency edge.
  #   No depends_on needed — Terraform reads the graph from your code.
  virtual_network_name = azurerm_virtual_network.main.name
  address_prefixes     = ["10.40.0.0/20"]
}

resource "azurerm_kubernetes_cluster" "main" {
  name                = "aks-contoso-prod"
  location            = azurerm_resource_group.main.location
  resource_group_name = azurerm_resource_group.main.name
  dns_prefix          = "contoso-prod"

  # Standard tier buys a financially-backed control-plane SLA.
  # Free tier gives you an SLO and no money back. Use Standard in prod.
  sku_tier            = "Standard"

  kubernetes_version        = "1.35"   # AKS keeps 3 GA minors in support
  automatic_upgrade_channel = "patch"  # CVEs patch themselves
  node_os_upgrade_channel   = "NodeImage"

  # passwordless everything
  oidc_issuer_enabled       = true
  workload_identity_enabled = true
  local_account_disabled    = true   # no static kubeconfig admin creds

  identity { type = "SystemAssigned" }

  azure_active_directory_role_based_access_control {
    azure_rbac_enabled     = true
    admin_group_object_ids = [var.platform_admins_group_id]
  }

  default_node_pool {
    name                 = "system"
    vm_size              = "Standard_D4as_v5"
    vnet_subnet_id       = azurerm_subnet.aks.id
    auto_scaling_enabled = true     # was enable_auto_scaling in azurerm 3.x
    min_count            = 2
    max_count            = 4
    only_critical_addons_enabled = true  # keep apps off the system pool
    zones                = ["1", "2", "3"]
    upgrade_settings { max_surge = "33%" }
  }

  network_profile {
    # Overlay: pods get IPs from a private space, not your VNet.
    # You stop running out of subnet at 400 pods. Do this.
    network_plugin      = "azure"
    network_plugin_mode = "overlay"
    network_policy      = "cilium"
    network_data_plane  = "cilium"
    pod_cidr            = "192.168.0.0/16"
    load_balancer_sku   = "standard"
  }

  web_app_routing {                     # managed NGINX ingress
    dns_zone_ids = [azurerm_dns_zone.shop.id]
  }

  oms_agent {
    log_analytics_workspace_id      = azurerm_log_analytics_workspace.main.id
    msi_auth_for_monitoring_enabled = true
  }

  lifecycle {
    # the cluster autoscaler owns node_count at runtime — don't fight it
    ignore_changes = [default_node_pool[0].node_count]
  }
}

# app workloads live in their own pool, scaled independently
resource "azurerm_kubernetes_cluster_node_pool" "apps" {
  name                  = "apps"
  kubernetes_cluster_id = azurerm_kubernetes_cluster.main.id
  vm_size               = "Standard_D4as_v5"
  vnet_subnet_id        = azurerm_subnet.aks.id
  auto_scaling_enabled  = true
  min_count             = 3
  max_count             = 30
  zones                 = ["1", "2", "3"]
}

# let the cluster pull images without a single password
resource "azurerm_role_assignment" "acr_pull" {
  scope                = azurerm_container_registry.main.id
  role_definition_name = "AcrPull"
  principal_id         = azurerm_kubernetes_cluster.main.kubelet_identity[0].object_id
}

Three lines worth discussing in code review: network_plugin_mode = "overlay" (stops you running out of VNet IP addresses), local_account_disabled = true (removes the shared admin kubeconfig that ends up pasted in Teams), and ignore_changes on node_count. Without that last one, Terraform and the cluster autoscaler will fight forever, each one "fixing" the other's work.

The old way: a connection string in a Kubernetes Secret, base64-encoded (which is encoding, not encryption), and sooner or later someone commits it. The new way: the pod proves who it is with a short-lived token, and Azure gives it access. Nothing to leak, nothing to rotate, nothing to find in Git history two years from now.

# 1. the pod's Azure identity
resource "azurerm_user_assigned_identity" "catalog" {
  name                = "id-catalog-api"
  location            = azurerm_resource_group.main.location
  resource_group_name = azurerm_resource_group.main.name
}

# 2. the trust: "the catalog-sa account in namespace 'shop' IS this identity"
resource "azurerm_federated_identity_credential" "catalog" {
  name                = "fic-catalog-api"
  resource_group_name = azurerm_resource_group.main.name
  parent_id           = azurerm_user_assigned_identity.catalog.id
  audience            = ["api://AzureADTokenExchange"]
  issuer              = azurerm_kubernetes_cluster.main.oidc_issuer_url
  subject             = "system:serviceaccount:shop:catalog-sa"
}

# 3. and what it's allowed to touch — nothing more
resource "azurerm_role_assignment" "catalog_kv" {
  scope                = azurerm_key_vault.main.id
  role_definition_name = "Key Vault Secrets User"
  principal_id         = azurerm_user_assigned_identity.catalog.principal_id
}

On the .NET side, that is the whole change: one line, no secret.

builder.Configuration.AddAzureKeyVault(
    new Uri("https://kv-contoso.vault.azure.net/"),
    new DefaultAzureCredential());   // picks up workload identity automatically

One folder per environment, one shared module. Dev and prod differ only by a variables file, never by a separate copy of the code. The moment they drift apart, "it works in dev" stops meaning anything.

infra/
├── modules/
│   └── aks-platform/        # cluster + network + acr + identities
│       ├── main.tf
│       ├── variables.tf
│       └── outputs.tf
└── envs/
    ├── dev/
    │   ├── main.tf          # module "platform" { source = "../../modules/aks-platform" }
    │   └── dev.tfvars       # node_min=1  node_max=3  sku_tier="Free"
    └── prod/
        ├── main.tf
        └── prod.tfvars      # node_min=3  node_max=30 sku_tier="Standard"

# envs/prod/main.tf
module "platform" {
  source = "../../modules/aks-platform"

  environment   = "prod"
  location      = "westeurope"
  node_min      = 3
  node_max      = 30
  sku_tier      = "Standard"
}

This is the declaration. replicas: 6 is the "what you want". Everything else tells the scheduler how to keep that promise safely.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: catalog-api
spec:
  replicas: 6
  selector:
    matchLabels: { app: catalog-api }
  template:
    metadata:
      labels:
        app: catalog-api
        azure.workload.identity/use: "true"
    spec:
      serviceAccountName: catalog-sa      # ← matches the federated credential
      containers:
      - name: api
        image: contosoacr.azurecr.io/catalog-api:1.14.0
        ports: [{ containerPort: 8080 }]
        # requests = what the scheduler reserves. limits = the ceiling.
        resources:
          requests: { cpu: "150m", memory: "192Mi" }
          limits:   { cpu: "800m", memory: "512Mi" }
        # "am I alive?" — fail and I get restarted
        livenessProbe:
          httpGet: { path: /healthz, port: 8080 }
          initialDelaySeconds: 10
        # "am I ready for traffic?" — fail and I'm pulled from the load balancer
        readinessProbe:
          httpGet: { path: /readyz, port: 8080 }
          periodSeconds: 5

The two probes are the most under-used feature in Kubernetes. Liveness answers "should I be killed?" Readiness answers "should I get traffic?" Get readiness wrong and every deploy drops requests, because traffic arrives before your .NET app has finished setting up its DI container and EF Core model.

In ASP.NET Core, the endpoints are three lines:

// Program.cs
builder.Services.AddHealthChecks()
    .AddDbContextCheck<CatalogContext>("db", tags: ["ready"]);

app.MapHealthChecks("/healthz", new() { Predicate = _ => false });      // process is up
app.MapHealthChecks("/readyz",  new() { Predicate = r => r.Tags.Contains("ready") });

See that there is no password anywhere. OIDC federation lets the workflow prove it is this repo, this branch and get a short-lived Azure token.

name: catalog-api
on:
  push:
    branches: [main]
    paths: ['src/Catalog/**']     # only build what changed

permissions:
  id-token: write            # ← required for OIDC login
  contents: read

env:
  ACR: contosoacr.azurecr.io
  IMAGE: catalog-api

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Test
        run: dotnet test ./src/Catalog --logger trx

      - uses: azure/login@v2
        with:
          client-id:       ${{ secrets.AZURE_CLIENT_ID }}
          tenant-id:       ${{ secrets.AZURE_TENANT_ID }}
          subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}

      - name: Build & push image
        run: |
          az acr login --name contosoacr
          # tag with the commit SHA — never 'latest' in production
          TAG=${GITHUB_SHA::7}
          docker build -f src/Catalog/Dockerfile \
            -t $ACR/$IMAGE:$TAG .
          docker push $ACR/$IMAGE:$TAG

      - name: Scan image
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: ${{ env.ACR }}/${{ env.IMAGE }}:${{ github.sha }}
          severity: CRITICAL,HIGH
          exit-code: '1'          # fail the build, don't just warn

And the Dockerfile: multi-stage, not running as root, with a chiselled runtime. Chiselled images have no shell and no package manager, which removes most of what a security scanner would complain about:

FROM mcr.microsoft.com/dotnet/sdk:9.0 AS build
WORKDIR /src
COPY *.sln .
COPY src/Catalog/*.csproj src/Catalog/
RUN dotnet restore src/Catalog          # cached layer: restore before copying code
COPY . .
RUN dotnet publish src/Catalog -c Release -o /app

FROM mcr.microsoft.com/dotnet/aspnet:9.0-noble-chiseled AS final
WORKDIR /app
COPY --from=build /app .
USER $APP_UID                            # non-root by default
ENTRYPOINT ["dotnet", "Contoso.Catalog.dll"]

In GitHub Actions, the approval is an environment with protection rules. The job names the environment, and GitHub handles the pause, the notification, the audit trail and the "you cannot approve your own deploy" rule. Azure DevOps environments have the same idea as Approvals and Checks.

  deploy-canary:
    needs: build
    runs-on: ubuntu-latest
    environment: production-canary      # no reviewers — automatic
    steps:
      - run: az aks get-credentials -g rg-contoso-prod -n aks-contoso-prod
      - run: |
          helm upgrade --install orders-canary ./charts/orders \
            --namespace shop --set image.tag=${GITHUB_SHA::7} \
            --set replicaCount=1 --set canary=true --atomic

      - name: Analyse canary for 10 minutes
        run: ./scripts/canary-analysis.sh --window 10m \
                 --max-error-rate 0.5 --max-p95-regression 20

  deploy-prod:
    needs: deploy-canary
    runs-on: ubuntu-latest
    # ↓ THIS is the human gate. Configured in repo settings:
    #   Required reviewers: @contoso/payments-oncall (2)
    #   Wait timer: 0 · Prevent self-review: on
    #   Deployment branches: main only
    environment:
      name: production
      url: https://shop.contoso.com
    steps:
      - run: |
          helm upgrade --install orders ./charts/orders \
            --namespace shop --set image.tag=${GITHUB_SHA::7} \
            --atomic --timeout 5m
      - run: kubectl rollout status deploy/orders-api -n shop --timeout=180s

Two details that make the approval real, not just for show. --atomic means a rollout that does not become healthy within the timeout is rolled back automatically, so the approver is allowing an attempt, not a one-way door. And prevent self-review means the person who wrote the change is not the person who approves it, which is the whole point of having an approval.

The infrastructure pipeline is deliberately slower

Run plan on the pull request so reviewers see the difference. Apply only after a person clicks approve.

jobs:
  plan:
    runs-on: ubuntu-latest
    steps:
      - run: terraform init
      - run: terraform plan -out=tfplan -lock-timeout=5m
      - run: terraform show -no-color tfplan > plan.txt
      - uses: actions/github-script@v7      # post the plan as a PR comment
        with: { script: '...comment plan.txt on the PR...' }
      - uses: actions/upload-artifact@v4
        with: { name: tfplan, path: tfplan }

  apply:
    needs: plan
    if: github.ref == 'refs/heads/main'
    environment: prod-infra          # ← required reviewers live here too
    steps:
      - uses: actions/download-artifact@v4
        with: { name: tfplan }
      # apply the SAVED plan, not a fresh one — what was reviewed is what runs
      - run: terraform apply -auto-approve tfplan

Applying the saved plan file matters more than it looks. If you plan again at apply time, what runs is not what anyone reviewed. Drift, a teammate's portal click or a new provider version can all change it in between, and then your approval was for a document that no longer exists.

spec:
  replicas: 8
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 2          # may temporarily run 10
      maxUnavailable: 0    # never dip below 8 ready
  minReadySeconds: 10       # "ready" must hold for 10s before we trust it
  template:
    spec:
      terminationGracePeriodSeconds: 45   # let in-flight requests finish
      topologySpreadConstraints:              # don't put all 8 in one zone
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: ScheduleAnyway
        labelSelector:
          matchLabels: { app: catalog-api }

Scale on the thing that predicts load

CPU tells you about load only after it has arrived. For orders-api, the queue length warns you before it arrives. KEDA scales on the queue length directly, and can even scale down to zero when the queue is empty:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata: { name: orders-worker }
spec:
  scaleTargetRef: { name: orders-worker }
  minReplicaCount: 1
  maxReplicaCount: 40
  triggers:
  - type: azure-servicebus
    metadata:
      queueName: orders-to-fulfil
      messageCount: "20"       # one pod per 20 queued messages
    authenticationRef: { name: azure-workload-identity }

Protect yourself from your own maintenance

A PodDisruptionBudget stops planned disruptions, like node upgrades and autoscaler clean-up, from taking down more pods than you can afford. Without it, an AKS node image upgrade can drain three nodes and leave you with just one pod for a while:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: catalog-pdb }
spec:
  minAvailable: "70%"
  selector:
    matchLabels: { app: catalog-api }

And graceful shutdown on the .NET side

Kubernetes sends SIGTERM and waits. If your app exits immediately, you lose whatever requests were in progress. The .NET Generic Host handles this correctly if you let it:

builder.Services.Configure<HostOptions>(o =>
    o.ShutdownTimeout = TimeSpan.FromSeconds(30));   // < terminationGracePeriod

Alert on problems your customers feel, not on their causes. High CPU is not an incident. A 12% error rate is.

That last one is where the two heroes talk to each other: the alert fires because Kubernetes has used up the space Terraform gave it. The fix is a pull request that changes max_count, reviewed, planned and applied, not a click in the portal.