Four services, six repositories, and one configuration that holds them all together. An architect's walk through the launch sequence, and an honest discussion about where configuration should live.
Four names in a chat thread. The flight director asks the only question that matters, one desk at a time.
PILLAR — go. GATEHOUSE — go. FACADE — go. KEYSTONE — go.
Nobody is nervous. Nobody has a rollback script open in another window. Nobody has said "as long as nothing else changed". At 15:00 the release goes out, and at 15:04 four services are running a version that did not exist this morning.
A team that can say go without fear does not have better people. It is a team where someone, months earlier, took a series of boring but important decisions about repositories, build outputs, approval gates and configuration. Every one of those decisions was taken so that on release day there is less to be afraid of.
This article builds that kind of team setup. We are designing Project Keystone: four services on AKS, delivered by Azure DevOps, created with Terraform, with Key Vault holding the secrets and Application Insights keeping watch. We will design it the way an architect works: the brief first, then the decisions the brief did not mention, then the one decision everybody gets wrong.
| Service | Responsibility | Why it's separate |
|---|---|---|
| Keystone | Configuration for the whole ecosystem | The stone at the top of an arch. Every other stone leans on it; remove it and the arch falls. |
| Pillar | Domain service — the business logic | Carries the weight. Owns the model, the rules and the database. Changes carefully. |
| Gatehouse | Integration gateway — third parties | The guarded entrance. Everything from outside comes in here and is made safe before it goes further. |
| Facade | Backend-for-frontend | The face users see. Built entirely around what the UI needs, and changes as often as the UI does. |
Four services, four repositories. That is the brief as given, and it is a sensible brief. But, as we will see in Act III, it is two repositories short, and those two missing ones decide whether this system is still easy to maintain eighteen months from now.
The main text is the design discussion, in plain English, with no background needed. Every Under the hood panel opens the real YAML, HCL and C#. The Decision blocks are architecture decision records: what we chose, what else we considered, and what it costs. Read only the decisions and you still get the design.
Before you design a pipeline, you have to be honest about what it delivers. Four services is not one system deployed four times. It is four systems that change at four different speeds, cause four different sizes of damage when they break, and have four different answers to "what happens if this is wrong at 3 a.m."
Those differences are the whole design. A pipeline that treats Facade and Pillar the same way is either too slow for Facade or too risky for Pillar. So we start by writing down how they differ.
| Service | Deploys | If it's wrong | Gate |
|---|---|---|---|
| Facade | Several times a day | A screen looks odd. Can be undone in minutes. | Automated only |
| Gatehouse | Weekly | A partner integration breaks. Can be fixed, but embarrassing. | Automated + contract tests |
| Pillar | Weekly, carefully | Wrong data gets saved. Maybe impossible to undo. | Human approval |
| Keystone | Hourly, in theory | All three at once. | Depends completely on Act II |
Look at that last row. Configuration changes the most often and causes the widest damage in the whole system. That combination is rare and dangerous, and it is why configuration, not the pipeline and not the cluster, is the real design problem here.
Click through the system. Every box is something you will create in Terraform, deploy from a pipeline, or stare at during an incident.
A keystone is the wedge at the top of an arch. It carries almost no weight itself, but every other stone passes its load through it, and until it is in place the arch cannot stand. It is the smallest piece and also the one you cannot remove. That is exactly what configuration is: a few kilobytes of text that the whole structure leans on.
The brief says Keystone "serves configuration to the other three". That one sentence hides a lot of work, and how you read it will decide more about this system's reliability than any other choice you make. So let us not assume. Let us question it.
There are three honest answers, and big companies run all three at scale. None of them is stupid. Each one gives up something different, and that trade-off is what you are really choosing.
| Model | What it means | Config change takes |
|---|---|---|
| A · Runtime service | Keystone runs as a pod. The other three call it when they start and check again from time to time. | Seconds |
| B · Baked at deploy | The Keystone repo publishes versioned files; pipelines turn them into ConfigMaps and deploy them along with the image. | A full deploy |
| C · Managed store | The Keystone repo is the source of truth, copied into Azure App Configuration. Services read from there. | Seconds |
Everyone compares these when everything is working, and then they look almost the same. The only useful comparison is on a bad night. Pick a model, then break it in four different ways.
Work through the "whole cluster restarts" scenario on model A and you find the argument that decides it. It is a normal Tuesday on AKS. A node pool gets upgraded. A zone has a short outage. Someone drains a node. Suddenly a large number of your pods restart at the same time.
Every Pillar, Gatehouse and Facade pod comes up at the same moment and immediately asks Keystone for its configuration.
Keystone is also restarting. It has one or two copies. Now it is getting every start-up request in the cluster at once, from services that cannot continue without an answer.
You have made every service in the system depend on one service, exactly at the moment when everything is already struggling.
Be fair to model A, because the careless version of this argument is wrong. A configuration call that blocks start-up should be controlled by a readiness probe, not a liveness probe, and a startupProbe exists exactly so that a slow-starting pod is not killed and restarted again and again. Set those up properly and the rush becomes "everything starts slowly" instead of "nothing starts at all". Anyone who says a runtime config service will always collapse is really describing badly configured probes.
The real argument is simpler and harder to escape: you now own and run one more service. Save the last good configuration on disk. Spread out start-up times. Size the copies for the rush at cold start, not for normal reads. Add a PodDisruptionBudget. Make reads non-blocking with sensible defaults. Keep all of this working as the system grows. Each of these is real, permanent engineering work, spent copying what a managed configuration store gives you on day one, for a service whose only job is to return a few kilobytes of text.
keystone-config is the Git source of truth. Its pipeline checks the configuration and publishes it to two places: versioned files that are built into deployments, and Azure App Configuration snapshots for values read while running.Here is the rule that makes the two-mechanism design work. It is the single most useful idea in this article:
Do not ask "where does config live?". Ask two questions about each setting: how quickly might I need to change this, and what happens if it is wrong?
Anything you might need to change during an incident must not need a deploy. Anything that must be repeatable must be fixed and versioned with the code it configures. Anything that is a secret is neither: it is referenced, never stored.
That gives three classes, and every setting belongs to exactly one. Try it; the lines are less obvious than they look.
The two that trip up almost everyone are the connection string and the log level. A connection string looks structural: it points at infrastructure Terraform created and changes only when that infrastructure changes. But the moment a password appears anywhere in the value, the whole value becomes a secret, because a string is only as safe as its most sensitive part. The server name is structural; the string with the password is not. The log level goes the other way. It feels small and permanent, but the whole reason it exists is so you can turn it up at 02:00 while looking at a problem. That makes it operational, and building it into the deployment makes it useless exactly when you need it.
Secrets go in Key Vault and are referenced, never copied. Not into a ConfigMap, not into App Configuration as plain text, not into a pipeline variable marked secret and then printed into a manifest. App Configuration stores a reference to a Key Vault secret, and the pod reads it at runtime using its own workload identity. If a secret value ever shows up in a Git diff, a pipeline log or kubectl get configmap, the design has failed, however well everything else was built.
The Keystone repo holds plain YAML for each service in each environment. The deploy pipeline fills it in, takes a hash of it, and names the ConfigMap after that hash. So a config change creates a new ConfigMap and forces a real rollout, instead of changing the old one in place with no rollout at all:
# keystone-config/services/pillar/prod.yaml
database:
commandTimeoutSeconds: 30
maxPoolSize: 200
endpoints:
gatehouse: "http://gatehouse.keystone-prod.svc.cluster.local"
telemetry:
cloudRoleName: "pillar"
samplingPercentage: 25# rendered by the pipeline — note the content hash in the name
apiVersion: v1
kind: ConfigMap
metadata:
name: pillar-config-7f3a91c # changes when content changes
namespace: keystone-prod
immutable: true # Kubernetes will refuse edits
data:
appsettings.Production.json: |
{ "Database": { "CommandTimeoutSeconds": 30, ... } }immutable: true makes Kubernetes refuse any in-place change to data. So "someone edited the ConfigMap at 3 a.m." now needs a delete and re-create, which shows up in the audit log, instead of a silent kubectl edit. It is not impossible, just very visible, and that is most of the value.
The hash-naming trick needs two more things to work. The Deployment's pod spec must be updated to point to the new ConfigMap name; a new ConfigMap that nothing points to changes nothing, and this is the step teams forget. And since old ConfigMaps pile up forever, you need a clean-up job that skips any ConfigMap still used by a kept ReplicaSet, or one day you will delete exactly what a rollback needed.
Read it using a sentinel key, so the app makes one cheap check instead of watching every setting, and reloads only when the sentinel changes:
// Program.cs — Pillar, Gatehouse and Facade all do this identically
var env = builder.Environment.EnvironmentName;
builder.Configuration.AddAzureAppConfiguration(options =>
{
options.Connect(new Uri(appConfigEndpoint), new DefaultAzureCredential())
.Select("Pillar:*", labelFilter: env)
// Key Vault references resolve with the pod's own identity
.ConfigureKeyVault(kv => kv.SetCredential(new DefaultAzureCredential()))
.ConfigureRefresh(refresh =>
{
// ← the label MUST match Select's. Register without it and you
// watch a null-label sentinel nobody ever bumps: refresh
// silently never fires. This is the classic mistake.
refresh.Register("Pillar:Sentinel", env, refreshAll: true)
.SetRefreshInterval(TimeSpan.FromSeconds(30));
});
},
// ← without optional:true an unreachable store throws at Build()
optional: true);
// the middleware needs this registration or UseAzureAppConfiguration throws
builder.Services.AddAzureAppConfiguration();
var app = builder.Build();
app.UseAzureAppConfiguration(); // drives the refresh check per requestChange ten related values, then update Pillar:Sentinel once. Every pod picks up all ten together within the refresh interval. No half-applied changes, no deploy.
The pod signs in as itself using workload identity; no secret is ever written into the cluster. In App Configuration, the value is a reference, not the actual secret:
# stored in App Configuration as a Key Vault reference
key : Pillar:ConnectionStrings:Sql
label : Production
value : {"uri":"https://kv-keystone-prod.vault.azure.net/secrets/pillar-sql-connection"}
type : application/vnd.microsoft.appconfig.keyvaultref+jsonThe SDK sees the reference, calls Key Vault using the pod's managed identity, and gives the application the actual value without saving it anywhere. Change the secret in Key Vault and every user of it gets the new value, with no deploy and no config change at all.
Model C still puts a network call in your start-up path, so it deserves the same questions we asked about model A. Three things make its risk very different:
The honest warning: a pod still needs the store at start-up unless you say otherwise. Stack the providers so structural values come from the built-in ConfigMap first and App Configuration adds to them when it is reachable, and mark it optional. That one setting turns "the store is down, so nothing starts" into "the store is down, so we run on the last deployed values":
builder.Configuration
.AddJsonFile("appsettings.json", optional: false)
.AddJsonFile($"/config/appsettings.{env}.json", optional: true) // baked ConfigMap
.AddAzureAppConfiguration(o => { ... }, optional: true); // wins if reachableLater providers win, so the cluster can always start. Note that WebApplicationBuilder has already added appsettings.json and environment variables before your code runs. Adding them again moves them to the end of the list and changes which value wins, which is worth knowing before you copy this into an app that depends on the defaults.
One thing not to claim: an environment variable is a poor emergency switch here. Changing one means editing the Deployment, which is a rollout — the exact deploy we just criticised model B for needing.
The brief said four repositories. The brief was describing the product, not the whole system, and there are two more things here that change on their own schedule, can cause their own damage, and need their own reviewers. Giving them their own home is the difference between a platform and a pile of stuff.
Terraform needs a repository. The cluster, the container registry, the Key Vault, the configuration store and the Application Insights workspace do not belong to any one service. They change weekly, not hourly. A mistake there takes down all four services, not one. And the person who should approve "resize the node pool" is not the person who should approve "change a button label". If infrastructure sits inside a service repo, every normal deploy carries a hidden risk of re-creating a node pool.
Pipeline templates need a repository. Four services means four pipelines, and without a shared template that means four copies of the build steps, the security scan, the image naming rules and the deployment logic. Fix a security issue in the scan step and you have to fix it four times, badly, at four different times. Worse, a service team can remove the scan step from its own pipeline and nobody will notice.
keystone-config, pillar-domain, gatehouse-integration, facade-bff, plus keystone-platform (Terraform) and keystone-pipelines (shared YAML templates, tagged and versioned).This is where the folder layout starts to matter. Inside keystone-config, configuration is layered, and the folder layout decides how far a change can spread. A change under services/facade/ can only ever affect Facade. A change under global/ affects everything, so it should be rare, obvious in a diff and harder to approve.
Change a file and see what runs.
Two things to notice. First, a change to global/ starts three pipelines and so three deploys, which is exactly why it needs an approval that a single-service change does not. Second, editing the README starts nothing at all. That sounds minor, but on a busy repo, path filters are the difference between a CI system people trust and one they have learnt to ignore.
Every trigger in this design runs after a merge, which is the moment a mistake becomes production's problem. The cheapest check in the whole system is the one that runs before the merge: a build validation policy on each repo, so pull requests must build and pass tests; JSON schema checks on every changed config file; and terraform plan posted as a comment on the platform repo's pull requests. For a change under global/ that would otherwise go to three production deploys, catching it during review costs minutes and stops most of the problems the later checks exist for.
If you use a template repo without fixing a version, every service rebuilds on whatever main happens to be at that moment, without asking. One bad commit to the template repo breaks all four pipelines at once, including the pipeline you would use to fix it. Pin to a tag, ref: refs/tags/v3.2.0, and upgrade it on purpose, one service at a time.
config/
├── global/ # affects all three services
│ ├── telemetry.yaml
│ └── resilience.yaml
├── environments/
│ ├── dev.yaml
│ ├── test.yaml
│ └── prod.yaml # stricter gate on this path
└── services/
├── pillar/{dev,test,prod}.yaml
├── gatehouse/{dev,test,prod}.yaml
└── facade/{dev,test,prod}.yaml
schema/
└── config.schema.json # every file validated against thisA service pipeline is kept very small on purpose. It says what it is and leaves everything else to the template. Note that it uses extends, not template:
trigger:
branches: { include: [main] }
# note the globs: 'docs/*' and '*.md' match one level only
paths: { exclude: ['docs/**', '**/*.md'] }
resources:
repositories:
- repository: templates
type: git
name: Keystone/keystone-pipelines
ref: refs/tags/v3.2.0 # pinned, never a moving branch
# REDEPLOY (not rebuild) when shared configuration changes. A config-only
# change must not produce a new image: rebuilding from unchanged source
# yields a different digest, which is the exact thing Act IV forbids.
pipelines:
- pipeline: config
source: keystone-config-publish
trigger: { branches: { include: [main] } }
# extends, not template: the service CANNOT add arbitrary steps
extends:
template: service-pipeline.yml@templates
parameters:
serviceName: pillar
projectPath: src/Pillar.Api/Pillar.Api.csproj
# selects WHICH pre-configured environment this lands on.
# The checks on that environment are owned by the platform team.
riskTier: 1
runContractTests: trueAnd here is the template that defines what a pipeline is allowed to be:
parameters:
- { name: serviceName, type: string }
- { name: projectPath, type: string }
- { name: riskTier, type: number, default: 3 }
- { name: runContractTests, type: boolean, default: false }
stages:
# skip the build entirely when the run was started by the config pipeline —
# we redeploy the digest already in production, with new config beside it
- stage: build
condition: ne(variables['Build.Reason'], 'ResourceTrigger')
jobs:
- job: build_scan_push
steps:
- template: steps/dotnet-build.yml
- template: steps/unit-test.yml
- ${{ if eq(parameters.runContractTests, true) }}:
- template: steps/contract-test.yml
- template: steps/container-scan.yml # not optional, by construction
- template: steps/push-acr.yml
# one build, three deployments of the SAME digest
- template: stages/deploy.yml
parameters: { env: dev, serviceName: ${{ parameters.serviceName }} }
- template: stages/deploy.yml
parameters: { env: test, serviceName: ${{ parameters.serviceName }} }
- template: stages/deploy.yml
parameters: { env: prod, serviceName: ${{ parameters.serviceName }} }Why extends and not a normal template include. An include is just a convenience: the calling pipeline can still do anything else it wants alongside it. An extends template defines the structure of the pipeline, and the service repo can only fill in the parameters the template accepts. Add a required template check on the production environment, and a pipeline that does not extend this template simply cannot deploy to production. The security scan stops being a step people are trusted to include, and becomes a rule of the system.
The platform repo creates everything shared. The detail worth copying is the service connection: workload identity federation, so Azure DevOps signs in to Azure with a short-lived token instead of a client secret someone has to change regularly.
# keystone-platform — the pieces the pipelines depend on
resource "azurerm_container_registry" "main" {
name = "acrkeystone"
sku = "Premium"
resource_group_name = azurerm_resource_group.platform.name
location = azurerm_resource_group.platform.location
}
# App Configuration data-plane RBAC has NO key or label scoping: Data Reader
# on the store grants read on EVERY key-value in it. Labels are a client-side
# convenience, not a security boundary — so one store PER ENVIRONMENT, or dev
# pods can read production values and production Key Vault reference URIs.
resource "azurerm_app_configuration" "main" {
for_each = toset(["dev", "test", "prod"])
name = "appcs-keystone-${each.key}"
sku = "standard" # snapshots + 30-day revision history
resource_group_name = azurerm_resource_group.platform.name
location = azurerm_resource_group.platform.location
}
# the cluster must opt in to workload identity — both are off by default
resource "azurerm_kubernetes_cluster" "main" {
# ... sku_tier, node pools, network profile ...
oidc_issuer_enabled = true
workload_identity_enabled = true
}
# one identity per (service, environment) — dev and prod must not share one
locals {
svc_env = {
for pair in setproduct(["pillar", "gatehouse", "facade"], ["dev", "test", "prod"]) :
"${pair[0]}-${pair[1]}" => { svc = pair[0], env = pair[1] }
}
}
resource "azurerm_user_assigned_identity" "service" {
for_each = local.svc_env
name = "id-${each.key}"
location = azurerm_resource_group.platform.location
resource_group_name = azurerm_resource_group.platform.name
}
# a federated credential carries exactly ONE subject — no wildcards.
# One FIC per namespace, or dev pods fail with AADSTS70021.
resource "azurerm_federated_identity_credential" "service" {
for_each = local.svc_env
name = "fic-${each.key}"
resource_group_name = azurerm_resource_group.platform.name
parent_id = azurerm_user_assigned_identity.service[each.key].id
audience = ["api://AzureADTokenExchange"]
issuer = azurerm_kubernetes_cluster.main.oidc_issuer_url
subject = "system:serviceaccount:keystone-${each.value.env}:${each.value.svc}"
}
# read its own environment's store — the isolation is the store, not the label
resource "azurerm_role_assignment" "appconfig_read" {
for_each = local.svc_env
scope = azurerm_app_configuration.main[each.value.env].id
role_definition_name = "App Configuration Data Reader"
principal_id = azurerm_user_assigned_identity.service[each.key].principal_id
}
# without this the Key Vault references resolve to nothing at runtime
resource "azurerm_role_assignment" "kv_read" {
for_each = local.svc_env
scope = azurerm_key_vault.main[each.value.env].id
role_definition_name = "Key Vault Secrets User"
principal_id = azurerm_user_assigned_identity.service[each.key].principal_id
}
# and the cluster needs to be able to pull the images it is asked to run
resource "azurerm_role_assignment" "acr_pull" {
scope = azurerm_container_registry.main.id
role_definition_name = "AcrPull"
principal_id = azurerm_kubernetes_cluster.main.kubelet_identity[0].object_id
}The Kubernetes part is easy to forget, and without it things fail with no clear error: the ServiceAccount must be annotated with the client ID, and the pod must be labelled to use it:
apiVersion: v1
kind: ServiceAccount
metadata:
name: pillar
namespace: keystone-prod
annotations:
azure.workload.identity/client-id: "<id-pillar-prod client id>"
---
# and on the pod template
metadata:
labels:
azure.workload.identity/use: "true"Every credential in this system is federated or managed. There is no client secret in a variable group, no connection string in a pipeline, and nothing to change at 2 a.m. because someone left the company. This is more than a nice security touch; it removes a whole category of incidents.
This snippet still leaves two things open, because they are decisions, not mistakes. Key Vault and App Configuration are shown with default public network access; in a regulated environment you would put private endpoints in front of both and accept the extra DNS work. And the platform pipeline itself needs the discipline the Terraform article talks about: remote state with locking, plan on the pull request, and apply the saved plan, not a fresh one.
Here is the rule that separates a real delivery pipeline from a build script, and it is broken all the time: what reaches production must be exactly what you tested. Not something similar. Not something rebuilt from the same commit with the same Dockerfile. The same bytes.
Rebuilding for each environment feels harmless: same code, same steps, so surely the same result. It is not. Base images change. Indirect dependencies resolve differently an hour later. A scan that passed at 14:00 passed an image that no longer exists at 15:00. What you approved and what you shipped are two different things that just look alike.
The build creates one image that never changes, identified by its digest. Every environment after that is a deployment, not a build. The digest is like a flight number, and it does not change between the launch pad and orbit.
Run the sequence. Watch the digest, and notice what stays the same.
Two checks in that sequence need a closer look, because they are what make the go/no-go call at the start possible.
The approval is not in the YAML. It is set on the environment by the platform team, and the pipeline author cannot change it — which is the whole point. A check that the person being checked can edit is not a check. The same feature gives you branch control, business-hours windows, and an exclusive lock so two releases cannot deploy to production at the same time.
The Azure Monitor check is the one most teams never set up, and it turns Application Insights from a dashboard into an active part of the release. It looks at your alert rules and will not let a stage start while that service has alerts firing.
There is a detail here that catches almost everyone, including many published examples. Every environment check in Azure Pipelines runs before deployment. YAML pipelines have no after-deployment check; that existed in classic release pipelines and did not survive the move. So an Azure Monitor check on production does not watch the deployment you just did. It stops you from deploying onto a service that is already unhealthy, which is useful, but different.
To check health after a rollout, add a second stage that points to its own environment and depends on the deploy stage. That stage's before-deployment check then runs after the rollout has finished and had some time to settle. That is after-deploy checking, built from the only tool the platform gives you. It is a small trick, and it is the difference between "the pods started" and "the service is still healthy".
(image@sha256:…, appconfig-snapshot-id). The digest is what runs; the snapshot is a permanent record of the configuration that ran with it. Rolling back means restoring both: the digest by redeploying it, and the configuration by restoring the store to that snapshot's point in time.Everything above assumes rollback is possible. For Pillar it often is not, and any design for a service that owns its database has to say this clearly.
Once a database migration has run, "restore the previous digest" brings back code that may not understand the new schema. The pair from Decision 03 becomes a triple: image, config and schema version. And the third one is the only one that will not go backwards. A dropped column does not come back just because you redeployed yesterday's container.
Expand, migrate, contract, and never in one release. Add the new column as nullable and deploy code that writes to both. Fill in the old data. Deploy code that reads the new column. Only in a later release, once you are sure you will not roll back before that point, drop the old column.
Each step can be undone on its own. The release that removes your ability to roll back happens on a calm afternoon, on purpose, and not on the same day as the feature.
In practice: migrations run as a separate step before the rollout, they only move forward and only add things within a release, and the pipeline records the schema version along with the digest and the snapshot. And Pillar's approval should be honest about what it approves. It is not "ship this code" but "accept that from now on, rollback means restoring from backup, not redeploying".
# keystone-pipelines/stages/deploy.yml
parameters:
- { name: env, type: string }
- { name: serviceName, type: string }
# passed in explicitly — variables do NOT flow across stages implicitly.
# $(imageDigest) set in the build stage renders as empty string here.
- { name: imageDigest, type: string }
stages:
- stage: deploy_${{ parameters.env }}
# sequential: two releases never touch the same environment at once
lockBehavior: sequential
jobs:
- deployment: deploy
# ↓ PER SERVICE, not one shared environment. Environment checks apply to
# every pipeline targeting that environment — one shared keystone-prod
# would force Facade's daily deploys through Pillar's 2-approver gate.
# The ".keystone-${env}" suffix binds the Kubernetes resource.
environment: keystone-${{ parameters.env }}-${{ parameters.serviceName }}.keystone-${{ parameters.env }}
strategy:
runOnce:
deploy:
steps:
# the rendered ConfigMap comes from the CONFIG pipeline, not this one
- download: config
artifact: rendered-config
- task: KubernetesManifest@1
inputs:
action: deploy
namespace: keystone-${{ parameters.env }}
manifests: |
$(Pipeline.Workspace)/config/rendered-config/${{ parameters.serviceName }}/*.yaml
$(Pipeline.Workspace)/manifests/*.yaml
# digest, never a tag — a tag can be moved, a digest cannot
containers: acrkeystone.azurecr.io/${{ parameters.serviceName }}@${{ parameters.imageDigest }}
- task: AzureCLI@2
displayName: Annotate the release in App Insights
inputs:
azureSubscription: sc-keystone-wif # workload identity federation
scriptType: bash
inlineScript: |
# puts a marker on every latency chart at this exact moment
az monitor app-insights component update ...
# a SEPARATE stage on the same environment, so its pre-deployment
# Azure Monitor check evaluates AFTER the rollout above has landed.
# This is the only way to get post-deploy verification from a check.
- stage: verify_${{ parameters.env }}
dependsOn: deploy_${{ parameters.env }}
jobs:
- deployment: verify
environment: keystone-${{ parameters.env }}-${{ parameters.serviceName }}-verify
strategy:
runOnce:
deploy:
steps:
- script: echo "Alert rules were quiet for the soak window."Checks set on the keystone-prod environment, in the UI, by the platform team:
| Check | Configuration | Stops |
|---|---|---|
| Approval | 2 reviewers from Platform; the person who asked cannot approve | Shipping unreviewed |
| Branch control | refs/heads/main only, protection required | Shipping from a feature branch |
| Business hours | 09:00–16:00 Mon–Thu | Friday-evening heroics |
| Required template | Must extend service-pipeline.yml | Skipping the security scan |
| Exclusive lock | On; ordering set by lockBehavior in YAML | Two releases colliding |
| Azure Monitor | No firing alerts on the service's App Insights rules | Deploying onto an already-unhealthy service |
Notice that none of these appear in any YAML file a service developer can edit. That separation — pipeline logic in code, deployment permission in the platform — is what turns automation into governance. (The one exception is the exclusive lock, where the check turns locking on but the runLatest or sequential behaviour is chosen in YAML.)
Because checks are tied to environments, not pipelines, one shared keystone-prod would apply Pillar's two-approver check to Facade's daily deploys. So each service gets its own environment, keystone-prod-facade and keystone-prod-pillar, set up in advance by the platform team with the checks that level of risk needs. The riskTier parameter in a service pipeline chooses which environment it goes to; it does not decide what that environment demands. A team can change its tier and land in a stricter room, but it cannot weaken the room it is standing in.
A business-hours check from 09:00 to 16:00 is perfect for planned work and useless at 02:14, so the design needs a written-down way through. Give the on-call team a separate emergency environment with a single-approver check and no business-hours limit, which can only be reached by a pipeline run with a clear reason entered, and send an alert every time it is used. A set of checks with no emergency path does not get respected. It gets bypassed by someone with permissions nobody remembered they had.
A configuration value is wrong in production. Not disastrously wrong — a retry count set to 0 instead of 3 — but Gatehouse now fails every short partner timeout instead of retrying it, and the error rate is climbing past the alert limit.
Everything in the last four acts was practice for the next four minutes. And now the only question that matters is the one we designed for in Act II: how quickly can this be put right?
Same incident, three designs. Start the clock.
The interesting result is that the two self-managed options end up at about the same time. The built-in model takes eleven minutes, because configuration has to go through a build and a rolling deployment to reach a pod: safe, fully audited, and slow. The runtime service could in theory be instant, yet takes thirteen, because most of the time goes into finding the problem, not delivering the fix. Without a revision history, "put it back to how it was at 20:00" is not something you can do. You have to work out which values changed and reverse them by hand, correctly, at 02:14.
This is the part worth remembering. Everyone compares how fast a system can apply a configuration change. What matters at 02:14 is how fast it can undo one, and the two have nothing to do with each other.
Not the technology, but the point-in-time restore. App Configuration keeps a history of every value (30 days on Standard and Premium), and az appconfig kv restore --datetime puts the store back to how it looked at a chosen moment. The fix was not "find which value is wrong and edit it under pressure at 02:14". It was "put it back to 20:00 yesterday": one command, no investigation. Being able to go back to a known good state before understanding what broke is the most valuable thing an operational system can offer.
Note what this is not. A snapshot cannot be restored over the store. Snapshots support create, archive and recover, where "recover" only means un-archiving the snapshot itself. Snapshots are the permanent record you link a release to and show an auditor; revision history is the lever you pull at 02:14. Mixing up the two is a common and costly mistake.
That emergency restore was done directly on the store, so now the store and keystone-config disagree. The next time anyone merges to that repo, the publish pipeline will overwrite the fix and bring the incident back, at some random later time, with nobody watching.
This is the standard failure of every "Git is the source of truth, copied to a managed store" design, and it is better to build for it than to rely on memory. Two cheap mechanisms: a CI job that compares the store with the repo on a schedule and alerts if they differ, and a strict rule that an emergency change is not finished until the pull request copying it back into the repo is merged. The incident is closed when the repo matches production, not when the graph looks fine again.
All three designs assume someone knew within a minute that something was wrong. That did not come from the configuration design. It came from an alert rule on an Application Insights metric, connected to an on-call team, plus a release marker on the response-time chart that made the cause obvious the moment someone looked.
Build the fastest rollback in the world and it is worth nothing if it takes forty minutes for anyone to notice the problem. Set up monitoring first. The pipeline can only be as good as the signal that tells it something went wrong.
One workspace, four service names, and the trace ID travelling across every hop. Distributed tracing is the difference between "the system is slow" and "Gatehouse's call to the payments partner is slow, and here is the exact request":
// identical in all four services; only the role name differs
builder.Services.AddApplicationInsightsTelemetry();
builder.Services.AddSingleton<ITelemetryInitializer>(
new CloudRoleNameInitializer(builder.Configuration["Telemetry:CloudRoleName"]));
// the release annotation: makes "which deploy caused this?" a one-glance question
builder.Services.Configure<TelemetryConfiguration>(c =>
c.TelemetryInitializers.Add(new VersionInitializer(
Environment.GetEnvironmentVariable("IMAGE_DIGEST"))));The alert rules that block deployments, and that woke someone up at 02:14:
| Signal | Threshold | Wired to |
|---|---|---|
| Failed request rate | > 2% over 5 min, per role | Pager + the Azure Monitor deployment check |
| Response time of the slowest 5% of requests | > 2× the usual value over the last 7 days | Pager |
| Dependency failures | > 5% to any single partner | Gatehouse team channel |
| Config refresh failures | Any, sustained 10 min | Platform team — the store is unreachable |
| Config drift | Store differs from keystone-config main | Platform team — an emergency edit was never back-ported |
That last row is the one people forget. If pods stop refreshing configuration, nothing breaks straight away. They just fall behind, until the day someone changes a value and half the pods never get it.
Six repositories, because things that change at different speeds and break at different scales should not share a release. One shared pipeline template that services extend instead of copy, enforced by a required-template check so the security scan cannot be skipped. One build per commit, promoted by digest through three environments, controlled by approvals the pipeline author cannot edit. And configuration split three ways by two questions — how quickly might I need to change this, and what happens if it is wrong — plus one rule above both: if it contains a credential, it is a secret, wherever else it might have belonged.
Every decision in this design is really the same decision in different clothes: keep apart the things that change at different speeds.
Infrastructure from application. Structural config from operational config. Secrets from both. Pipeline logic from deployment permission. Build from deploy. Get that separation right and the go/no-go call becomes boring, which is the best compliment a release process can get.
| # | Decision | The reason in one line |
|---|---|---|
| 01 | Keystone is a repo and a pipeline, not a pod | Do not make every service's uptime depend on one service at cold start |
| 02 | Six repositories, not four | Terraform and shared templates have their own schedule, their own risk and their own reviewers |
| 03 | Deployable unit is (digest, snapshot) | Rolling back half a release creates a combination nobody tested |
| 04 | Config classified by how quickly it may need to change | Changes during an incident must not need a deploy; things that must be repeatable must never change |
| 05 | Gates live on environments, not in YAML | A check that the person being checked can edit is not a check |
| 06 | Federated identity everywhere | Removes a whole category of incidents instead of managing it |
| # | Do this | Not this |
|---|---|---|
| 1 | Write the classification rule down before the first setting exists | Deciding per-setting, forever, by argument |
| 2 | One shared template repo, tagged, from day one | Copy-paste the first pipeline three times |
| 3 | Build once; promote by digest | A build stage per environment |
| 4 | Workload identity federation before the first secret exists | A client secret in a variable group "for now" |
| 5 | Path filters and layered config folders on the config repo | One flat folder that rebuilds everything |
| 6 | Alerts and release annotations before the first production deploy | Discovering you have no signal during the first incident |
| 7 | Practise a rollback deliberately, before you need one | Finding out the snapshot restore was never tested |
This design is sized for four services, three environments and one team that owns the platform. At twelve services, think hard about whether one pipeline per service still works, or whether you are ready for a pull-based GitOps controller inside the cluster. For one service, most of this is unnecessary ceremony; a single repo and a single pipeline will serve you better, and you should not feel bad about that. Architecture is an answer to a specific scale, and applying a design to a problem of a different size is how most architecture goes wrong.
Every diagram on this page is a live simulation, not a screenshot. The timings in Diagram 6 are examples, based on realistic build, rollout, refresh and investigation times, not measured from a real system. What is described here is Azure DevOps and Azure App Configuration as of August 2026: snapshots cannot be changed and support only create, archive and recover (recover un-archives the snapshot; it does not put the store back; that is what the 30-day revision history and az appconfig kv restore are for); App Configuration access control works on the whole store, not on individual keys or labels; all environment checks run before deployment and are set by resource owners, not in YAML; and workload identity federation for Azure service connections is generally available. This article was checked against those docs after writing, and a dozen statements were corrected, including, embarrassingly, its own first description of snapshot rollback. The service names Keystone, Pillar, Gatehouse and Facade belong to this article, not to any product.
Discussion