One recurring problem in infrastructure documentation is diagram drift.

The architecture diagram in documentation says one thing. Terraform state says another. The deployed environment says something else entirely.

Over time diagrams stop being trusted.

Recently I started experimenting with an idea to reduce this drift:

Generate architecture diagrams automatically from Terraform module outputs.

This is still a work in progress, but the early prototype is promising and worth sharing.

The Core Idea

Terraform modules already know what infrastructure they create.

So instead of manually drawing architecture diagrams, we can generate them directly from Terraform outputs.

The flow looks like this:

Terraform Apply
      │
      ▼
terraform output -json
      │
      ▼
Renderer Script
      │
      ▼
Generated draw.io diagrams

Result:

generated/
 ├─ overview.drawio
 └─ modules/
      ├─ network.drawio
      └─ aks.drawio

Instead of manually updating diagrams, they become derived artifacts of infrastructure code.

Design Goals

The approach is built around a few principles.

1. Eliminate Diagram Drift

The diagram reflects actual deployed infrastructure, because the values come directly from Terraform outputs.

No manual updates required.

2. Module Ownership

Each Terraform module owns its own diagram template.

Example module structure:

modules/
  network/
    main.tf
    outputs.tf
    diagram/
      overview-layer.xml.tpl
      standalone.xml.tpl

This keeps documentation close to the infrastructure definition.

3. Composable Architecture Views

Each module produces two diagram artifacts:

1. Standalone module diagram

network.drawio
aks.drawio

1. Layer for a shared platform overview

overview.drawio

The overview diagram composes all modules together into a system view.

4. Reusable Modules

Because diagram templates live inside modules, they travel with the module.

If another team reuses a module, they automatically inherit its documentation.

Terraform Module Diagram Contract

Each module exports structured metadata describing what should appear in the diagram.

Example:

output "diagram" {
  value = {
    module_name = "network"
    title       = "Network"

    layout = {
      x = 80
      y = 80
    }

    resources = {
      vnet_name   = azurerm_virtual_network.this.name
      subnet_app  = azurerm_subnet.app.name
      subnet_data = azurerm_subnet.data.name
    }
  }
}

This becomes the contract between Terraform and the diagram renderer.

The renderer does not inspect Terraform state directly. It only consumes these outputs.

Template-Based Diagrams

Each module contains a draw.io template fragment.

Example file:

modules/network/diagram/overview-layer.xml.tpl

Example template snippet:

<mxCell id="_vnet"
        value="VNet: "
        style="rounded=1;whiteSpace=wrap;html=1;"
        vertex="1"
        parent="_group">
  <mxGeometry x="20" y="50" width="260" height="60" as="geometry"/>
</mxCell>

Tokens like this:

are replaced with Terraform output values during rendering.

Rendering the Diagrams

After Terraform runs:

terraform output -json > outputs.json
python render_drawio.py outputs.json

The renderer performs three simple steps:

1. Load Terraform outputs
2. Replace tokens inside templates
3. Assemble the final draw.io document

Resulting Architecture View

The generated diagram groups modules visually.

Example conceptual layout:

+--------------------+      +---------------------+
| Network            |      | AKS                 |
|                    |      |                     |
| VNet               |─────▶| Kubernetes Cluster  |
| Subnets            |      | Node Resource Group |
+--------------------+      +---------------------+

Each module becomes a self-contained block that can be reused or moved independently.

Why draw.io?

draw.io (diagrams.net) works well for generated diagrams because:

• widely used
• simple XML format
• easy to generate programmatically
• still editable visually when needed

This makes it a good compromise between automation and flexibility.

Current Limitations

This approach is still experimental and intentionally simple.

XML Templates Are Verbose

draw.io XML is not particularly pleasant to maintain.

A future version may replace XML templates with a simpler YAML-based specification.

Layout Is Static

Modules currently define fixed coordinates:

layout = {
  x = 80
  y = 80
}

Eventually layout could be automated.

Cross-Module Relationships

Modules do not create edges to other modules directly.

Relationships are currently defined separately to avoid tight coupling between modules.

Future Directions

If the approach proves useful, several improvements are possible.

Shared Style Library

Standard icons and styles for common infrastructure components.

YAML Diagram Specification

Instead of raw XML templates:

nodes:
  - id: vnet
    label: "VNet: "
    x: 20
    y: 50

Automatic Layout

Modules could declare relationships rather than absolute coordinates.

CI/CD Integration

Pipelines could automatically publish diagrams after infrastructure deployment.

Final Thoughts

Infrastructure has increasingly become fully code-driven:

• infrastructure
• networking
• security policies
• deployment pipelines

But diagrams are often still maintained manually.

Generating diagrams directly from Terraform modules could help keep architecture documentation closer to reality.

This is still an early prototype, but the idea of diagrams-as-code owned by Terraform modules looks promising.

More updates as the experiment evolves.

Terraform is excellent at provisioning infrastructure, but large Terraform codebases tend to accumulate problems:

• Copy‑pasted backend configuration\
• Repeated provider configuration\
• Environment drift across dev/stage/prod\
• Dependency ordering between stacks\
• Painful module orchestration

Terragrunt solves these operational problems without replacing Terraform. It wraps Terraform and adds features for composition, DRY configuration, and orchestration of infrastructure stacks.

This guide explains how experienced engineers actually structure Terragrunt in production environments.

What Terragrunt Really Does

Terragrunt is a thin wrapper around Terraform that adds operational capabilities Terraform intentionally avoids.

Capability Why It Matters ———————————– ———————————– DRY Terraform configuration Avoid copy‑pasting backend and providers

Environment hierarchy Clean dev/stage/prod structure

Dependency management Apply stacks in correct order

Remote state automation Automatically configure state

Module orchestration Run apply across multiple modules ———————————————————————–

Terraform itself remains responsible for infrastructure provisioning. Terragrunt focuses on codebase management and orchestration.

Typical Terraform Scaling Problem

A real infrastructure repository often starts simple and gradually becomes messy.

terraform/
 ├── dev
 │   ├── vpc
 │   ├── eks
 │   └── rds
 ├── stage
 │   ├── vpc
 │   ├── eks
 │   └── rds
 └── prod
     ├── vpc
     ├── eks
     └── rds

Each directory typically contains:

• backend configuration
• provider configuration
• repeated variables
• identical module references

As the infrastructure grows, copy‑paste drift becomes inevitable.

Terragrunt Repository Layout

Terragrunt separates infrastructure modules from environment configuration.

infrastructure-live/
 ├── dev
 │   ├── vpc
 │   │   └── terragrunt.hcl
 │   ├── eks
 │   │   └── terragrunt.hcl
 │   └── rds
 │       └── terragrunt.hcl
 ├── stage
 └── prod

Terraform modules live in a separate repository:

infrastructure-modules/
 ├── vpc
 ├── eks
 └── rds

Terragrunt orchestrates module usage across environments.

Installing Terragrunt

brew install terragrunt

Or download the binary:

https://terragrunt.gruntwork.io/docs/getting-started/install/

Verify installation:

terragrunt --version

Minimal Terragrunt Configuration

Example terragrunt.hcl:

terraform {
  source = "git::ssh://git@github.com/company/infrastructure-modules.git//vpc"
}

inputs = {
  cidr_block = "10.0.0.0/16"
}

Execution:

terragrunt apply

Terragrunt downloads the module and executes Terraform internally.

The Most Important Feature: include

Large infrastructures require shared configuration. Terragrunt solves this with inheritance via include.

Root configuration:

infrastructure-live/terragrunt.hcl

Example:

remote_state {
  backend = "s3"

  config = {
    bucket         = "company-terraform-state"
    key            = "${path_relative_to_include()}/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "terraform-locks"
  }
}

Child module:

dev/vpc/terragrunt.hcl

include {
  path = find_in_parent_folders()
}

terraform {
  source = "git::ssh://git@github.com/company/infrastructure-modules.git//vpc"
}

inputs = {
  cidr_block = "10.0.0.0/16"
}

All modules inherit shared configuration automatically.

Remote State Without Duplication

Terraform normally requires each module to define its backend.

Terragrunt can generate it automatically.

remote_state {
  backend = "s3"

  generate = {
    path      = "backend.tf"
    if_exists = "overwrite"
  }

  config = {
    bucket = "company-terraform-state"
    key    = "${path_relative_to_include()}/terraform.tfstate"
    region = "us-east-1"
  }
}

When Terragrunt runs, backend.tf is generated dynamically.

Environment Configuration

Terragrunt supports environment‑level variables.

live/
 ├── terragrunt.hcl
 ├── dev
 │   └── env.hcl
 ├── stage
 │   └── env.hcl
 └── prod
     └── env.hcl

Example env.hcl:

locals {
  env = "dev"
}

Load it in a module:

locals {
  env_config = read_terragrunt_config(find_in_parent_folders("env.hcl"))
}

inputs = {
  environment = local.env_config.locals.env
}

This allows environment‑aware configuration without repeating variables.

Managing Dependencies

Infrastructure stacks often depend on one another.

Example:

VPC -> EKS -> Application

Terragrunt provides a dependency block.

dependency "vpc" {
  config_path = "../vpc"
}

inputs = {
  vpc_id = dependency.vpc.outputs.vpc_id
}

Terragrunt reads outputs from Terraform state automatically.

Running Multiple Modules

Instead of applying each module individually:

cd vpc
terraform apply

cd eks
terraform apply

Use Terragrunt orchestration:

terragrunt run-all apply

Terragrunt:

1. Builds a dependency graph
2. Applies modules in correct order
3. Parallelizes independent stacks

Module Version Pinning

Modules are typically referenced from Git.

terraform {
  source = "git::ssh://git@github.com/company/infrastructure-modules.git//eks?ref=v1.4.0"
}

Benefits:

• controlled upgrades
• reproducible infrastructure
• easy rollback

Provider Generation

Provider configuration can also be generated.

generate "provider" {
  path      = "provider.tf"
  if_exists = "overwrite"

  contents = <<EOF
provider "aws" {
  region = "us-east-1"
}
EOF
}

This prevents provider duplication across modules.

Terragrunt Repository Design (Production Layout)

Most teams eventually converge on a layered layout.

repo-root
 ├── infrastructure-modules
 │    ├── vpc
 │    ├── eks
 │    └── rds
 │
 └── infrastructure-live
      ├── terragrunt.hcl
      │
      ├── dev
      │   ├── env.hcl
      │   ├── vpc
      │   ├── eks
      │   └── rds
      │
      ├── stage
      │   └── ...
      │
      └── prod
          └── ...

Responsibilities:

Modules repository:

• reusable Terraform modules
• versioned releases
• no environment values

Live repository:

• environment configuration
• Terragrunt orchestration
• module version pinning

Multi‑Account / Multi‑Region Pattern

Terragrunt scales well to multi‑account architectures.

Example:

live/
 ├── prod
 │   ├── account.hcl
 │   ├── us-east-1
 │   │   ├── vpc
 │   │   └── eks
 │   └── us-west-2
 │       └── vpc

Example account.hcl:

locals {
  account_id = "123456789012"
}

Provider configuration can dynamically assume roles based on this configuration.

CI/CD Integration

Terragrunt works well inside pipelines.

Example CI stage:

terragrunt run-all plan --terragrunt-non-interactive

Typical automation tools:

• Atlantis
• Spacelift
• GitHub Actions
• GitLab CI
• Azure DevOps

When Terragrunt Is Worth Using

Terragrunt becomes valuable when:

• Terraform repositories exceed ~10 modules
• multiple environments exist
• backend duplication becomes painful
• stack dependencies grow complex

For very small infrastructures, plain Terraform is simpler.

Key Takeaways

Terragrunt does not replace Terraform.

It solves the operational scaling problems Terraform intentionally avoids.

The most valuable capabilities are:

1. DRY configuration
2. dependency orchestration
3. environment hierarchy
4. remote state automation
5. multi‑module execution

These features allow Terraform infrastructures to scale without turning into copy‑paste chaos.

When engineers open a Helm chart for the first time, the immediate reaction is often confusion. The repository contains multiple YAML templates, a large values.yaml, helper files, and sometimes nested subcharts. Reading every template line-by-line is inefficient.

Instead, experienced engineers reconstruct the deployment model by scanning a few key signals in a specific order.

The goal is to quickly answer:

• What workloads does this chart deploy?
• What parts of the deployment are configurable?
• What dependencies does it include?
• What infrastructure assumptions does it make?

Once those answers are clear, the rest of the chart becomes much easier to reason about.

Step 1 — Start With Chart.yaml

This file defines the identity and dependencies of the chart.

Look for:

name
version
appVersion
dependencies

Example:

name: payments-api
version: 0.3.2
appVersion: 1.12.0

Signals to extract quickly:

• Is this an application chart or platform component?
• Does the chart depend on other charts (databases, ingress controllers, monitoring)?
• Does the chart bundle infrastructure components or assume they already exist?

The dependencies section is particularly important. It reveals whether the chart deploys additional systems like Redis, PostgreSQL, or Prometheus.

This tells you how self‑contained the deployment really is.

Step 2 — Scan values.yaml (Not the Templates)

Most Helm charts are driven almost entirely by values.yaml.

The templates simply interpolate those values.

Engineers should scan values.yaml first because it reveals:

• configurable components
• optional features
• scaling behavior
• external integrations

Look for sections like:

image
resources
replicaCount
service
ingress
env

These usually map directly to Kubernetes constructs.

Example mental model:

values.yaml
      ↓
templates/*.yaml
      ↓
Rendered Kubernetes manifests

If you understand the values structure, you already understand how the deployment behaves.

Step 3 — Identify the Workload Type

Next locate the core workload in templates/.

Typical files include:

deployment.yaml
statefulset.yaml
daemonset.yaml
job.yaml
cronjob.yaml

The workload type reveals the runtime model of the application.

Example signals:

Deployment
→ stateless service

StatefulSet
→ database or stateful workload

DaemonSet
→ node‑level agent (monitoring, logging, networking)

Understanding this immediately tells you how the system behaves operationally.

Step 4 — Look for External Exposure

Next determine how the application is exposed.

Search for templates containing:

service.yaml
ingress.yaml
gateway.yaml
route.yaml

Signals:

Service type

ClusterIP
NodePort
LoadBalancer

Ingress or Gateway configuration indicates the application expects external traffic.

This reveals how traffic enters the system.

Step 5 — Check Resource Configuration

One of the most important operational signals is how the chart defines resource limits.

Look for:

resources:
  limits:
  requests:

These values affect:

• pod scheduling
• cluster capacity
• performance characteristics

Charts without proper resource definitions often cause production issues.

Experienced engineers always scan this section early.

Step 6 — Look for Environment Injection

Next identify how runtime configuration is injected.

Common patterns:

env
envFrom
configMapRef
secretRef

This reveals where application configuration originates.

Example signals:

ConfigMap
→ non‑sensitive configuration

Secret
→ credentials or tokens

External secret systems may also appear through integrations with secret operators.

Understanding this tells you where configuration lives outside the chart.

Step 7 — Check Helpers and Template Logic

Most mature charts include helper functions in:

templates/_helpers.tpl

These contain reusable template logic such as:

• naming conventions
• label generation
• chart metadata

You typically do not need to read every helper function, but scanning them reveals:

• naming patterns
• resource label structure
• how multiple components are grouped

This helps interpret rendered manifests later.

Step 8 — Look for Subcharts

Some charts embed other charts inside:

charts/

or reference them through dependencies.

This often indicates the chart deploys a complete stack, not just an application.

Example:

application
↓
redis
↓
database
↓
metrics stack

Subcharts increase operational complexity, so identifying them early is important.

Reconstructing the Deployment Model

After scanning these areas, you should be able to reconstruct the architecture mentally.

Example:

Helm Chart
   ↓
Deployment (API)
   ↓
Service (ClusterIP)
   ↓
Ingress (public access)
   ↓
ConfigMaps + Secrets
   ↓
Optional Redis subchart

You now understand the core topology without reading every template.

Signals That a Helm Chart Is Complex

Experienced engineers also watch for these indicators:

Large values.yaml (hundreds of lines)

Heavy template logic in _helpers.tpl

Multiple workload types in templates/

Embedded subcharts

Conditional blocks such as:

These patterns usually indicate the chart supports multiple deployment modes.

Key Takeaway

When reading an unfamiliar Helm chart, scan in this order:

Chart.yaml
values.yaml
templates/ workload
service / ingress
resources
environment configuration
subcharts

This sequence allows you to reconstruct the deployment model quickly without reading every file.

When a Deployment becomes part of a production incident, reading it top to bottom is usually too slow. By the time you finish scanning every field, the real question has already shifted: what part of this object actually controls rollout behavior, runtime behavior, or recovery behavior?

Seasoned engineers usually do not read a Deployment as YAML. They read it as an operational contract between the application, the scheduler, and the rollout controller.

The fastest way to understand a Deployment is to answer a small set of questions:

• What pods is this object trying to keep alive?
• What image is actually being deployed?
• How does rollout happen?
• What makes a pod healthy or unhealthy?
• What scheduling or runtime constraints exist?
• What other objects does this Deployment depend on?

Once those answers are clear, most of the remaining YAML becomes supporting detail.

Step 1 — Start With Metadata Only Long Enough to Establish Context

Do not get stuck in labels immediately. Start by identifying the basic context:

metadata:
  name:
  namespace:

That tells you where this object lives and usually what system or bounded context it belongs to.

Then glance at labels and annotations only for high-signal clues such as:

• release ownership
• GitOps ownership
• team or service identity
• sidecar injection hints
• restart or checksum annotations

Examples of useful signals:

• app.kubernetes.io/name
• app.kubernetes.io/part-of
• argocd.argoproj.io/instance
• sidecar.istio.io/inject
• checksum annotations tied to ConfigMaps or Secrets

This step is not about detail. It is about understanding what broader system is managing the Deployment.

Step 2 — Find the Pod Template Immediately

The most important part of a Deployment is not the Deployment object itself. It is the pod template under:

spec:
  template:

This is the future state the controller keeps trying to realize.

If you understand the pod template, you understand the real workload.

At minimum, scan for:

• container images
• ports
• environment injection
• volume mounts
• service account
• resource requests and limits

A good mental shortcut is:

Deployment = rollout logic + pod template

If the pod template changes, Kubernetes creates a new ReplicaSet and begins rollout behavior.

That is why most operational questions eventually come back to the template.

Step 3 — Check replicas Before Anything Fancy

Look at:

spec:
  replicas:

This tells you the intended steady-state pod count.

It sounds obvious, but in practice this answers several important questions immediately:

• Is this workload expected to be highly available?
• Is it intentionally single replica?
• Are we dealing with a horizontally scaled service or a singleton process?

For example:

• replicas: 1 means update strategy and readiness become much more sensitive
• replicas: 2 or more suggests some availability expectations
• missing replicas may indicate HPA-managed behavior or default assumptions

For incident response, this single field often explains why a rollout created downtime or why there is no failover behavior.

Step 4 — Read the Selector Carefully

Look at:

spec:
  selector:
    matchLabels:

This is one of the highest-risk parts of the object because it defines which pods belong to this Deployment.

Experienced engineers treat the selector as identity, not decoration.

Why it matters:

• it determines which ReplicaSets the Deployment manages
• it must align with pod template labels
• bad selector design creates dangerous ownership confusion

Then compare it with:

spec:
  template:
    metadata:
      labels:

Those labels must match the selector correctly.

When debugging unexpected rollouts or pod ownership issues, this is one of the first places worth checking.

Step 5 — Read the Container Image Like a Supply-Chain Signal

Inside the pod template, go straight to:

spec:
  template:
    spec:
      containers:
      - name:
        image:

This is not just “what image runs.” It tells you:

• what artifact is being deployed
• whether the deployment is pinned or floating
• whether the image naming aligns with environment and registry conventions

High-signal things to notice:

• specific immutable tag vs generic tag
• internal registry vs public registry
• image naming patterns tied to platform conventions

Examples:

• myregistry.azurecr.io/payments/api:1.4.7
• repo/service:latest

Seasoned engineers get nervous when they see mutable tags like latest, because rollout behavior becomes harder to reason about and recovery becomes less deterministic.

Step 6 — Check Rollout Strategy Before You Check Probes

Look at:

spec:
  strategy:
    type:
    rollingUpdate:
      maxSurge:
      maxUnavailable:

This tells you how Kubernetes replaces old pods with new ones.

This is where you determine whether the Deployment is optimized for:

• availability
• speed
• conservative rollout
• aggressive replacement

Examples:

• maxUnavailable: 0 favors continuity
• maxSurge: 0 may create tighter capacity behavior
• default RollingUpdate behavior may be acceptable for stateless services but fragile for constrained clusters

For experienced engineers, rollout strategy often explains production pain faster than probes do. Many “application issues” are really rollout math issues under limited capacity.

Step 7 — Then Read Probes as Recovery Policy

Now inspect:

livenessProbe
readinessProbe
startupProbe

Do not read probes as health checks only. Read them as traffic control and restart policy signals.

What each really means operationally:

• readinessProbe controls when the pod is eligible for traffic
• livenessProbe controls when Kubernetes kills and restarts the container
• startupProbe protects slow-starting applications from premature restart loops

This is where you ask:

• Can the app start slowly?
• Can it accept traffic before dependencies are ready?
• Can a bad liveness probe create artificial restarts?
• Can readiness failures explain why rollout stalls?

In production, many “deployment problems” are actually probe problems.

Step 8 — Read Resources as Scheduling Intent

Check:

resources:
  requests:
  limits:

This is one of the most important sections for platform engineers because it expresses how the workload negotiates with the scheduler and node capacity.

Read it as:

• what minimum capacity the pod requires
• what maximum runtime envelope it may consume
• whether the values seem realistic for the application type

Signals to look for:

• missing requests
• equal requests and limits
• suspiciously small CPU or memory values
• very high limits relative to requests

These values influence:

• placement
• eviction pressure
• autoscaling behavior
• noisy-neighbor effects

A Deployment without sensible resource settings is often a future incident waiting to happen.

Step 9 — Check Environment and Configuration Injection

Next inspect:

env:
envFrom:
configMapRef:
secretRef:
volumes:
volumeMounts:

This reveals where runtime configuration comes from and what external dependencies the workload assumes.

Important questions:

• Does the app require ConfigMaps or Secrets to start?
• Is configuration mounted as files or injected as environment variables?
• Are there external certificates, tokens, or identity bindings involved?
• Is the pod coupled to storage or projected volumes?

This step often explains why a Deployment looks correct but pods still fail at runtime.

The Deployment may be syntactically fine while its dependencies are missing, stale, or out of sync.

Step 10 — Scan Scheduling and Identity Constraints

Then inspect high-signal pod spec fields such as:

• serviceAccountName
• nodeSelector
• tolerations
• affinity
• topologySpreadConstraints
• security context fields

These fields reveal where the pod is allowed to run and under what identity.

This is operationally important because many production issues come from scheduling constraints rather than application logic.

Examples:

• wrong service account → cloud identity failures
• strict node selectors → unschedulable pods
• missing tolerations → pods never land on intended node pools
• topology constraints → rollout stalls in small clusters

For seasoned engineers, this section often explains “why pods are Pending” faster than events do.

Step 11 — Understand What the Deployment Does Not Tell You

A Deployment alone does not fully explain a running service.

It usually depends on surrounding objects:

• Service
• Ingress / Gateway
• ConfigMaps
• Secrets
• HPA
• PDB
• NetworkPolicy
• ServiceAccount and RBAC
• external secret or identity systems

One of the fastest ways to avoid misdiagnosis is to treat a Deployment as one part of a workload bundle, not the full application definition.

A Deployment may be valid while the real failure lives in one of those adjacent objects.

Reconstruct the Operational Model

After scanning those sections, you should be able to build a mental model quickly.

Example:

Deployment
  ↓
3 replicas of an API pod
  ↓
Rolling update with no downtime target
  ↓
Traffic gated by readiness probe
  ↓
Restart policy driven by liveness probe
  ↓
Config from Secret + ConfigMap
  ↓
Scheduled only on workload nodes
  ↓
Uses cloud identity via service account

That is the point of the exercise. You are not memorizing YAML. You are reconstructing the workload’s operational behavior.

Signals That a Deployment Deserves Extra Attention

Experienced engineers usually slow down when they see patterns like these:

• mutable image tags
• no resource requests
• liveness probe without startup probe on slow apps
• strict affinity combined with small clusters
• single replica plus aggressive rollout settings
• heavy use of annotations from multiple controllers
• environment injection spread across many sources
• checksum annotations implying config-driven restarts

These are not always wrong, but they usually indicate higher operational sensitivity.

Key Takeaway

To understand a Kubernetes Deployment quickly, scan in this order:

metadata context
pod template
replicas
selector and pod labels
image
rollout strategy
probes
resources
configuration injection
scheduling and identity constraints
adjacent dependencies

That sequence helps you reconstruct how the workload behaves in production, which is far more useful than simply knowing what the YAML syntax means.

Dockerfiles are often opened during incidents, security reviews, or performance investigations. When that happens, reading them line‑by‑line is rarely the fastest way to understand what is going on.

Experienced engineers read Dockerfiles as image construction pipelines. The goal is to reconstruct how the runtime environment is built and identify signals that affect reproducibility, security posture, build speed, and runtime behavior.

Instead of parsing every instruction, focus on a small number of signals that reveal the entire container lifecycle.

The fastest way to understand a Dockerfile is to answer these questions:

• What base image defines the runtime environment?
• Is this a multi‑stage build?
• What dependencies are installed?
• What application artifacts are copied into the image?
• What process actually runs in the container?

Once those answers are clear, the rest of the file usually becomes predictable.

🧱 Step 1 — Identify the Base Image (The Supply Chain Root)

Start with the first FROM instruction.

Example:

FROM node:20-alpine

or

FROM mcr.microsoft.com/dotnet/aspnet:8.0

This line defines:

• the operating system layer
• the runtime environment
• the security patch source
• the expected base image size

Seasoned engineers immediately look for these signals:

• pinned version vs floating tag
• alpine, slim, or minimal variants
• internal registry vs public registry

Examples:

FROM node:20-alpine
FROM python:3.11-slim
FROM mycompany.azurecr.io/platform/base-runtime:2.4

This step answers a fundamental question:

What environment does every container instance ultimately inherit from?

🧩 Step 2 — Detect Multi‑Stage Builds

Next scan for multiple FROM statements.

Example:

FROM node:20 AS build
FROM nginx:alpine

Multiple stages usually mean:

build image
↓
compile or bundle artifacts
↓
copy only runtime artifacts
↓
create smaller final image

Mental model:

Build Stage
   ↓
Compile / package application
   ↓
Runtime Stage
   ↓
Minimal production container

This pattern reduces:

• final image size
• attack surface
• unnecessary toolchains in runtime containers

When investigating performance or security issues, multi‑stage builds are a strong signal of image optimization maturity.

📦 Step 3 — Locate Dependency Installation

Next scan RUN instructions that install dependencies.

Common patterns:

RUN apt-get install
RUN apk add
RUN pip install
RUN npm install
RUN dotnet restore

These instructions reveal:

• language ecosystem
• system library dependencies
• build-time toolchains
• potential security exposure surface

Example:

RUN apt-get update && apt-get install -y     curl     ca-certificates     libpq-dev

Large dependency blocks often explain:

• slow container builds
• large image sizes
• expanded vulnerability surface

Experienced engineers quickly check whether build dependencies accidentally remain in the runtime image.

📁 Step 4 — Understand File Copy Strategy

Next inspect how the application code enters the container.

Typical instructions:

COPY
ADD

Example:

COPY package.json ./
COPY package-lock.json ./
RUN npm install

COPY src/ ./src

This ordering is intentional.

Experienced engineers look for caching patterns:

Copy dependency manifests
↓
Install dependencies
↓
Copy application source code

Why this matters:

Docker layer caching allows dependency installation to be reused when only source files change.

Poor ordering causes dependency layers to rebuild on every commit, slowing CI pipelines dramatically.

⚙️ Step 5 — Inspect Environment and Build Arguments

Next check for:

ENV
ARG

Examples:

ENV NODE_ENV=production
ARG BUILD_VERSION

Key differences:

ARG → build-time variables
ENV → runtime environment variables

Signals revealed here:

• environment assumptions
• runtime configuration defaults
• version injection patterns

Example:

ARG VERSION
ENV APP_VERSION=$VERSION

These patterns often connect the Docker build process with CI/CD pipelines.

🚀 Step 6 — Identify the Runtime Process

Now find the container’s startup command.

Look for:

CMD
ENTRYPOINT

Example:

CMD ["node", "server.js"]

or

ENTRYPOINT ["dotnet", "payments-api.dll"]

This line reveals the actual workload process.

Everything earlier in the Dockerfile simply prepares the environment required to run this command.

When debugging runtime behavior, this is one of the most important lines in the file.

🔐 Step 7 — Look for Security Signals

Experienced engineers also quickly scan for security posture signals.

Things worth checking:

• containers running as root
• absence of a USER directive
• leftover package managers
• unnecessary build toolchains in runtime images

Example improvement pattern:

RUN adduser --system appuser
USER appuser

Running containers as non‑root is a common baseline in hardened Kubernetes platforms.

Another signal:

FROM node:latest

Mutable tags like latest make image reproducibility harder and complicate incident debugging.

🧠 Reconstruct the Image Build Pipeline

After scanning these sections, you should be able to mentally reconstruct the container build process.

Example:

Base runtime image (node:20-alpine)
        ↓
Install system dependencies
        ↓
Install Node dependencies
        ↓
Copy application code
        ↓
Set environment configuration
        ↓
Start application process

At this point you understand how the container is assembled and what environment the application runs inside.

⚠️ Signals That a Dockerfile Deserves Extra Attention

Experienced engineers slow down when they see patterns like:

• mutable base image tags (latest)
• large dependency installs in runtime images
• missing .dockerignore
• application code copied before dependency manifests
• unnecessary package managers left installed
• lack of explicit runtime user

These signals often correlate with:

• slow builds
• oversized images
• increased vulnerability surface
• inconsistent deployments

🧭 Key Takeaway

To understand a Dockerfile quickly, scan in this order:

base image
multi-stage structure
dependency installation
file copy strategy
environment configuration
runtime command
security signals

This sequence allows you to reconstruct how the container image is built and how the workload will behave in production — without reading every line of the Dockerfile.

Opening a Terraform module for the first time can be disorienting. Mature infrastructure repositories often contain dozens of resources, nested modules, dynamic expressions, and environment‑specific behavior. Reading the code from top to bottom rarely helps.

Experienced engineers instead reconstruct the infrastructure topology by scanning a few structural signals. The goal is not to understand every line immediately — it is to answer a small set of questions quickly:

• What infrastructure does this module create?
• What external systems does it depend on?
• What inputs control its behavior?
• What outputs does it expose to the rest of the platform?

Once those answers are clear, the rest of the module becomes predictable.

Step 1 — Identify the Provider and Platform

Start by locating the provider configuration.

Typical signals appear in:

providers.tf
main.tf
terraform blocks

Example:

provider “azurerm” { features {} }

or

required_providers { aws = { source = “hashicorp/aws” } }

This immediately tells you which infrastructure domain the module controls.

Examples:

azurerm → Azure infrastructure
aws → AWS infrastructure
google → GCP infrastructure
kubernetes → cluster resources
helm → application deployments

This step narrows the scope of what the module can possibly create.

Step 2 — Identify the Primary Resource Type

Next search for the dominant resource blocks.

Example:

resource “azurerm_kubernetes_cluster” “this” {}

resource “aws_vpc” “main” {}

resource “azurerm_storage_account” “logs” {}

Most modules revolve around one primary resource type. Everything else usually supports that resource.

For example:

AKS module → network, identities, monitoring attached to cluster
VPC module → subnets, routing tables, gateways
Storage module → networking, encryption, access policies

Identifying the primary resource reveals the module’s architectural purpose.

Step 3 — Scan for Nested Modules

Many production Terraform modules are composed of smaller modules.

Look for:

module “network” {} module “monitoring” {} module “identity” {}

These often represent platform building blocks.

Example architecture reconstruction:

module.cluster ↓ module.network module.identity module.monitoring

This step shows whether the module represents:

• a small reusable component
• a platform layer
• a full environment stack

Understanding this hierarchy prevents you from misreading supporting infrastructure as the main purpose of the module.

Step 4 — Check Data Sources (External Dependencies)

Data sources reveal infrastructure that already exists.

Search for:

data “…”

Examples:

data “azurerm_subnet” data “aws_ami” data “azurerm_resource_group”

This tells you the module depends on external infrastructure.

Typical signals:

existing network topology
existing identity systems
shared platform resources

This step helps reconstruct the boundary between this module and the wider platform.

Step 5 — Identify Input Variables

Variables define how the module is controlled.

Look for:

variable “…”

Typical examples:

variable “environment” {} variable “location” {} variable “cluster_name” {} variable “subnet_id” {}

Experienced engineers read variables not to understand syntax but to identify:

• required inputs
• optional configuration paths
• environment-specific behavior

Large variable surfaces often indicate the module is designed to support multiple deployment patterns.

Step 6 — Scan Locals for Derived Architecture

Locals often encode important design decisions.

Example:

locals { cluster_name = “${var.environment}-aks” tags = merge(var.tags, { platform = “core” }) }

These blocks frequently reveal:

naming conventions
tagging strategy
environment isolation
derived resource structure

Scanning locals can quickly expose organizational infrastructure patterns.

Step 7 — Identify Outputs

Outputs reveal how other modules depend on this module.

Search for:

output “…”

Example:

output “cluster_id” {} output “subnet_ids” {} output “vnet_id” {}

Outputs usually represent integration points with the rest of the infrastructure.

Example mental model:

Network Module ↓ AKS Module ↓ Application Platform

Understanding outputs tells you where this module fits in the larger architecture.

Step 8 — Watch for Conditional Infrastructure

Terraform modules often support multiple deployment modes using conditionals.

Examples:

count = var.enable_monitoring ? 1 : 0

or

for_each = var.subnets

These patterns indicate:

feature flags
environment-specific behavior
optional platform components

When you see many conditional expressions, expect the module to support several operational configurations.

Reconstructing the Architecture

After scanning these areas, you should be able to reconstruct the infrastructure model quickly.

Example mental model:

Terraform Module ↓ Primary Resource (AKS Cluster) ↓ Supporting Infrastructure • networking • managed identity • monitoring ↓ Outputs exposed to platform modules

This allows you to reason about the module without reading every line of Terraform.

Signals That a Terraform Module Is Complex

Experienced engineers watch for these indicators:

large variable surfaces
many nested modules
heavy conditional logic
dynamic blocks
cross‑module outputs

These signals often indicate the module supports multiple environments or platform layers.

Key Takeaway

To understand a Terraform module quickly, scan in this order:

provider / terraform block
primary resource types
nested modules
data sources
variables
locals
outputs

This sequence reveals the module’s role in the infrastructure architecture without reading the entire codebase.

If you inherit a Packer template in an existing infrastructure repository, reading it line-by-line is usually the wrong approach.

Senior engineers typically reconstruct the architecture first, then fill in details only if needed.

With some pattern recognition you can usually understand a Packer build in under a minute by answering four questions:

• What platform is the image built on?
• What base operating system does it start from?
• What software is installed during provisioning?
• Where is the final image stored?

Once those are clear, you already understand most of the pipeline.

Step 1 — Identify the Target Platform

Start by locating the builder / source block.

Examples:

source “azure-arm” “image” {} source “amazon-ebs” “image” {} source “googlecompute” “image” {} source “vmware-iso” “image” {}

This tells you where the image is being built.

Common builders:

Once you see the builder you can already visualize the architecture:

Packer → Temporary VM → Provision → Capture Image

Step 2 — Identify the Base Image

Next determine what operating system the image starts from.

Look for fields like:

image_publisher
image_offer
image_sku

or on AWS:

source_ami

Example:

image_publisher = “Canonical” image_offer = “UbuntuServer” image_sku = “20_04-lts”

Meaning the build starts from Ubuntu 20.04.

The base image defines the entire starting state of the system. If it changes, everything downstream changes as well.

Many enterprise pipelines fail simply because the upstream image was updated or deprecated.

Step 3 — Locate Provisioners

Provisioners describe what happens inside the temporary VM.

Search for:

provisioner

Common types include:

provisioner “shell”
provisioner “powershell”
provisioner “ansible”
provisioner “file”

These blocks reveal the purpose of the image.

Example:

provisioner “shell” { inline = [ “apt install docker”, “apt install nginx” ] }

You can immediately infer that the image likely prepares a web server environment.

Another example:

provisioner “ansible” { playbook_file = “hardening.yml” }

This typically indicates a security-hardened base image.

Provisioners usually reveal why the image exists.

Step 4 — Check the Communicator

Packer needs a way to connect to the temporary machine.

Look for:

communicator

Common values:

Example:

communicator = “ssh”

This indicates a Linux build environment.

Step 5 — Identify Image Output

Next determine where the final image is stored.

Look for fields such as:

shared_image_gallery
managed_image_name
ami_name

Modern Azure builds typically publish to:

Azure Shared Image Gallery

This matters because other infrastructure systems depend on this image, such as:

Terraform deployments
VM scale sets
AKS node pools

Understanding the output tells you how the image participates in the wider infrastructure pipeline.

Step 6 — Scan Variables and Locals

Variables reveal how the template integrates with CI/CD pipelines.

Example:

variable “client_id” { default = env(“pipeline-client-id”) }

This indicates that credentials are injected by the pipeline environment.

Locals are helper values used during the build.

Example:

locals { imagetag = formatdate(“MMDDYYYY-HHmm”, timestamp()) }

This pattern typically creates unique image versions for each build.

Step 7 — Check Post‑Processors (Optional)

Some templates include post‑processors such as:

docker-tag
manifest
compress
vagrant

These steps usually publish metadata or push artifacts after the image is built.

Not every template uses them.

Reconstructing the Architecture Quickly

After scanning those sections you should be able to mentally reconstruct the pipeline.

Example mental model:

Base Image (Ubuntu 20.04) ↓ Temporary Azure VM ↓ Provisioners install Docker, monitoring agents, and security tools ↓ Image captured ↓ Stored in Shared Image Gallery ↓ Terraform deploys infrastructure from that image

You now understand the system without reading every line.

The Typical Enterprise Image Pipeline

Most organizations structure Packer pipelines roughly like this:

Git Repository
↓
CI/CD Pipeline
↓
Packer Build
↓
Shared Image Gallery
↓
Terraform Deployment
↓
VM Scale Sets or Platform Nodes

This pattern is commonly referred to as a golden image pipeline.

Common Causes of Packer Pipeline Failures

Base Image Changes

Using a dynamic reference like “latest” can cause builds to break when the upstream image changes.

Provisioner Timing Issues

Examples include:

apt lock conflicts
services not ready
reboots not handled correctly

These often create flaky builds.

Missing Credentials

Many templates rely on pipeline‑injected secrets through environment variables.

If those variables are not injected correctly, authentication errors occur.

Where Packer Fits in Modern Infrastructure

In many modern platforms Packer is no longer used for traditional application servers.

Instead it typically produces:

• hardened base operating systems
• platform VM images
• AKS node base images
• enterprise baseline machine images

It becomes the first stage of the infrastructure pipeline, producing standardized images that downstream systems deploy.

Key Takeaway

To understand most Packer templates quickly, scan these sections:

Builder / Source
Base Image
Provisioners
Communicator
Image Destination
Variables

With those pieces you can usually reconstruct the entire pipeline in under a minute.

Large CloudFormation templates can easily reach hundreds or thousands of lines. Reading them sequentially rarely helps when the real goal is to understand what infrastructure is actually being created.

Experienced engineers treat a CloudFormation template as a dependency graph, not a text document. The objective is to quickly reconstruct:

• what infrastructure exists
• how resources depend on each other
• which parts are configurable
• which outputs other systems consume

Once that mental model is clear, the rest of the template becomes easier to navigate.

Step 1 — Identify the Stack Purpose

Start by scanning the template description and top-level structure.

CloudFormation templates usually follow this structure:

Parameters
Mappings
Conditions
Resources
Outputs

Immediately jump to Resources to understand what the stack actually builds. Everything else mostly controls behavior around those resources.

Step 2 — Scan the Resource Types First

Look for:

Type: AWS::

Examples:

AWS::EC2::Instance
AWS::EC2::VPC
AWS::RDS::DBInstance
AWS::Lambda::Function
AWS::EKS::Cluster

Experienced engineers scan resource types before reading properties because resource types quickly reveal the architectural layer of the stack.

Examples:

VPC, Subnet, RouteTable
→ networking layer

ECS Service, Lambda, EKS
→ compute layer

S3, DynamoDB, RDS
→ data layer

IAM Role, Policy
→ identity layer

Within a few seconds you can determine whether the template defines:

• foundational infrastructure
• application platform resources
• a single service deployment

Step 3 — Identify the Primary Resource

Most templates revolve around one main resource and supporting infrastructure.

Examples:

EKS cluster template
→ main resource: AWS::EKS::Cluster

Lambda stack
→ main resource: AWS::Lambda::Function

VPC stack
→ main resource: AWS::EC2::VPC

Supporting resources often include:

security groups
roles
logging resources
network attachments

Identifying the primary resource tells you why the stack exists.

Step 4 — Look for Implicit Dependency Signals

CloudFormation builds a dependency graph automatically.

Key signals include:

Ref
Fn::GetAtt
DependsOn

Examples:

Ref: MySecurityGroup

Fn::GetAtt:

• MyLoadBalancer
• DNSName

These references tell you how resources connect.

Example mental model:

Load Balancer ↓ Target Group ↓ Auto Scaling Group ↓ EC2 instances

You are reconstructing the resource graph, not reading YAML.

Step 5 — Check Parameters for External Control

Parameters define how the template is controlled by users, pipelines, or higher-level stacks.

Example:

Parameters: Environment: Type: String

InstanceType: Type: String

Signals to extract quickly:

• environment configuration
• instance sizing
• networking inputs
• external resource identifiers

Large parameter sets usually indicate the template supports multiple environments or deployment modes.

Step 6 — Check Conditions for Deployment Variants

Conditions allow templates to deploy different infrastructure depending on environment.

Example:

Conditions: IsProduction: !Equals [ !Ref Environment, prod ]

These often control:

optional logging systems
high-availability resources
multi-AZ behavior
monitoring stacks

Conditional resources usually signal environment-aware infrastructure.

Step 7 — Scan Outputs to Understand Stack Integration

Outputs show what the stack exposes to other stacks.

Example:

Outputs: VpcId: Value: !Ref VPC

ClusterEndpoint: Value: !GetAtt EKSCluster.Endpoint

Outputs usually reveal:

network identifiers
service endpoints
resource ARNs

These values often feed into:

Terraform deployments
CloudFormation nested stacks
CI/CD pipelines

Outputs tell you how this stack fits into the larger system architecture.

Step 8 — Watch for Nested Stacks

Some templates include:

Type: AWS::CloudFormation::Stack

This indicates a nested stack architecture.

Example:

network stack
↓ security stack
↓ application stack

Nested stacks usually mean the infrastructure is split into logical layers.

Reconstruct the Architecture

After scanning these signals, you should be able to build a mental model quickly.

Example:

VPC stack ↓ subnets and route tables ↓ security groups ↓ EKS cluster ↓ node groups ↓ application workloads

You now understand the core architecture without reading every property.

Signals That a CloudFormation Template Is Complex

Experienced engineers slow down when they see:

very large parameter sets
many conditions controlling resources
heavy use of intrinsic functions
deep nested stacks
large IAM policy blocks

These signals indicate the template supports multiple deployment patterns or complex environments.

Key Takeaway

To understand a CloudFormation template quickly, scan in this order:

resource types
primary resource
dependency references
parameters
conditions
outputs
nested stacks

This approach reconstructs the infrastructure graph quickly without reading the entire template.

• How to slice the cluster into node pools
• How to slice workloads via namespaces, tenants, and QoS
• How to use taints/tolerations, priority classes, PDBs, and topology to control behavior
• When to make more clusters vs fewer clusters

I’ll keep each pattern fairly tight so you can remix them.

1. Node Pool Segmentation Patterns

1.1 General vs Specialized Pools

Pattern:

• general-pool for 80–90% of workloads

• One or more specialized pools:

  • perf (CPUManager, TopologyManager)
  • gpu
  • batch
  • db or stateful

Mechanics:

• Labels:

  kubectl label node node-1 node-pool=general
  kubectl label node node-2 node-pool=perf
  

• Taints on special pools:

  kubectl taint node node-2 perf-only=true:NoSchedule
  

• Workload spec:

  nodeSelector:
    node-pool: perf
  tolerations:
    - key: "perf-only"
      operator: "Exists"
      effect: "NoSchedule"
  

When to use: almost always. This is the baseline pattern.

1.2 Horizontal Isolation by “Noisy Class”

Separate node pools for:

• system (CNI, CSI, metrics, logging)
• user-apps
• noisy-batch (Spark, ETL, big cronjobs)

Idea: Keep noisy, spiky workloads from contaminating general services.

Mechanics:

• System DaemonSets:

  nodeSelector:
    node-role.kubernetes.io/system: "true"
  

• Batch node pool tainted:

  kubectl taint node batch-pool batch-only=true:NoSchedule
  

1.3 Cost/Hardware Pools

Pools by machine type:

• spot or preemptible
• standard
• high-mem
• ssd-local

Use them like:

• Non-critical workers → spot
• Latency-critical → standard
• Memory-heavy → high-mem
• Spark/Redis → ssd-local

Key: Every pool has labels & taints; workloads choose via nodeSelector / nodeAffinity + tolerations.

2. Namespace & Tenant Patterns

2.1 Namespace-per-team / namespace-per-product

Pattern:

• team-a-dev, team-a-prod
• product-x-dev, product-x-prod

Controls per namespace:

• ResourceQuota
• LimitRange
• NetworkPolicy
• RBAC

Example:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-a-quota
  namespace: team-a-prod
spec:
  hard:
    requests.cpu: "40"
    requests.memory: "80Gi"
    limits.cpu: "80"
    limits.memory: "160Gi"
    pods: "200"

When to use: Multi-team clusters, platform teams serving app teams.

2.2 Soft Multi-Tenancy vs Hard Multi-Tenancy

• Soft: Same cluster, tenants isolated via namespaces, quotas, network policies, RBAC. Most enterprises.
• Hard: Separate clusters per tenant or per BU, sometimes separate accounts/subscriptions.

Rules of thumb:

• If tenants can be semi-trusted & share infra → soft.
• If you need strong isolation / different compliance regimes / noisy security boundaries → multiple clusters.

3. Workload Admission & QoS Patterns

3.1 Enforce Requests & Limits via Policy

Use an admission policy (OPA/Gatekeeper, Kyverno, or built-in ValidatingAdmissionPolicy) to:

• Reject Pods without resources.requests & resources.limits
• Forbid BestEffort except for debug namespaces
• Enforce max/min resource sizes per namespace

Pattern:

• Default: require at least requests and limits.memory.
• Exception: special allow-bursty namespace.

3.2 Priority Classes for SLO Layers

Define PriorityClasses like:

• system-critical (CNI, kube-dns)
• platform-critical (ingress, logging, metrics)
• business-critical (user-facing prod services)
• batch (ETL, reports)
• best-effort (preemptible stuff)

Example:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: business-critical
value: 900
globalDefault: false

Use in Pod spec:

priorityClassName: business-critical

Behavior:

• On resource pressure, lower-priority Pods get evicted first.
• Scheduler gives high-priority workloads first dibs on resources.

3.3 PodDisruptionBudget (PDB) + Autoscaling

Pattern:

• For every stateful or important stateless workload, define PDB:

apiVersion: policy/v1
kind: PodDisruptionBudget
spec:
  minAvailable: 2

Combine with:

• HPA for scale-out
• Cluster Autoscaler / Karpenter for node scale-out

This gives:

• Safe rollouts
• Safe node drain / spot preemption
• Enough replicas for resilience

4. Topology & Failure-Domain Patterns

4.1 Spread Across Zones / Nodes

Use topology spread constraints or anti-affinity:

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app: my-api

Or simpler:

affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          topologyKey: kubernetes.io/hostname
          labelSelector:
            matchLabels:
              app: my-api

Goal: Avoid all replicas landing on same node or same AZ.

4.2 Zone-aware Node Pools

Per cloud:

• Separate node pools per AZ
• Label nodes with zone
• Use topologySpreadConstraints to distribute workloads evenly

This prevents:

• All traffic going through a single zone
• Single-AZ outages taking entire app down

5. Security & Network Isolation Patterns

5.1 Zero-Trust-by-default NetworkPolicy

Base policy in each namespace:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

Then explicit “allow” policies for:

• namespace-local communication
• calls to specific backends (DBs, APIs)
• calls to observability stack

Pattern: No ingress/egress allowed by default → everything opt-in.

5.2 Security Boundary Namespaces

For particularly sensitive apps, combine:

• Dedicated namespace
• Dedicated node pool (taints)
• Strict NetworkPolicy
• Stricter PodSecurity / PSP replacement (restricted baseline)
• Separate secrets store (external KMS, Vault, AKV, etc.)

This is a cluster-within-a-cluster pattern.

6. Multi-Cluster Patterns

6.1 Env-tier Clusters

One of the most common:

• prod cluster(s)
• nonprod cluster(s) (dev/uat/stage)

Sometimes:

• prod-us, prod-eu (data residency)

Pros:

• Strong blast-radius isolation
• Simple mental model: “prod is sacred”

Cons:

• More control-plane overhead
• You need a GitOps story that understands multiple clusters (ArgoCD, Flux).

6.2 Function-based Clusters

Patterns like:

• core-platform cluster (ingress, observability, shared platform services)
• app-tenant clusters for main product lines
• data cluster for Kafka/Spark/Cassandra

This is helpful if:

• Data-plane loads are wildly different than API-plane loads
• Observability stack is heavy and you want to isolate it

7. Putting It Together – Example Design

Here’s a concrete cluster design pattern you can adapt:

Clusters

• corp-nonprod
• corp-prod

Node Pools in each cluster

• system (small, stable, for CNI/CSI/monitoring)
• general (default microservice nodes, D/E/m6i/n2)
• perf (CPUManager+TopologyManager, latency/cpu-critical)
• batch (cheaper, spot, larger nodes)
• db (memory-heavy, local SSD, tainted)

Namespaces

• platform-system (CNI, CSI, logging, metrics, ingress)
• platform-observability (Prometheus, Loki, Tempo, etc.)
• team-a-dev, team-a-prod
• team-b-dev, team-b-prod
• shared-services (auth, messaging, etc.)

Controls

• ResourceQuota + LimitRange per team namespace
• NetworkPolicy default-deny per namespace

• PriorityClasses:

  • system-critical
  • platform-critical
  • business-critical
  • batch-low

Scheduling hints

• Platform & observability → system & general pools
• Latency-critical apps → perf pool (Guaranteed, pinned CPUs)
• Spark jobs → batch pool (spot, large nodes, local SSD)
• Redis/DB → db pool (memory-heavy, local SSD)

8. Quick design checklist

When you design or refactor a cluster, ask:

1. Do I have at least two node pools? (general + something else)
2. Are system components isolated or competing with apps?
3. Do teams have clear namespace boundaries, quotas, and limits?
4. Are BestEffort workloads controlled or confined?
5. Do I have PriorityClasses & PDBs for production services?
6. Are workloads spread across zones and nodes?
7. Do sensitive workloads have network & node isolation?
8. Do I need multiple clusters for prod vs nonprod or for legal isolation?

If the answer to most of these is “yes”, you’re in serious platform-engineering territory already.

Choosing the wrong node size leads to:

• Constant evictions
• Memory pressure
• CPU throttling
• NUMA imbalance
• Poor inference latency
• Overpaying for unused cores
• Underpowered control-plane components (CNIs, CSI, monitoring agents)

This guide will help you select the best node types for:

• Microservices
• JVM workloads
• High-throughput services
• Dataplanes (Cilium, Envoy)
• Redis, Postgres
• AI/ML
• Spark
• GPU workloads

Let’s go deep.

SEGMENT 12 — Ultimate Node Sizing Guide for AKS, EKS, and GKE

We will cover:

1. The principles for choosing node sizes
2. CPU-to-memory ratios that actually work
3. Understanding NUMA (critical!)
4. Choosing VM families in each cloud (AKS/EKS/GKE)
5. Node sizes for different workload types
6. When to use large nodes vs many small nodes
7. When to use local SSD
8. Cost optimization rules

PART 1 — Principles of Good Node Sizing

These are universal across AKS/EKS/GKE.

1. Memory pressure kills nodes — not CPU

Always design node capacity with memory as primary constraint.

Nodes rarely fail from high CPU usage. Nodes frequently fail from memory exhaustion → eviction → OOM → kubelet death → NotReady.

2. NUMA topology heavily affects performance

Nodes with ≥ 2 sockets or ≥ 2 NUMA nodes require careful placement.

• JVM
• Redis
• AI inference
• network dataplanes cannot randomly bounce across NUMA nodes.

Prefer:

• single-NUMA nodes for latency-sensitive workloads

3. Avoid nodes with > 64 vCPUs unless you use pinned CPU workloads

Large nodes → more NUMA topology → more cgroup fragmentation → lower efficiency.

4. Prefer more medium nodes over fewer huge nodes

• reduces blast radius
• avoids multi-Pod NUMA fragmentation
• improves bin packing
• reduces eviction chain reactions

5. Always leave space for system daemons

Rule of thumb:

• Reserve 6–12% of node memory
• Reserve 0.5–1.5 vCPU for system/kube daemons

PART 2 — Recommended CPU : Memory Ratios

Use these ratios as starting points:

PART 3 — NUMA Topology Explained (Critical Selection Factor)

How to think about NUMA:

1. Single NUMA node = predictable, consistent latency

2. Multiple NUMA nodes =

   • Remote memory access
   • 20–80% slowdown for AI/Redis/Envoy
   • Complex scheduling

Cloud providers rarely document NUMA, but here’s the real mapping:

AWS (EKS) NUMA

• m5 / c5 / r5 → 1 NUMA node up to 24–32 vCPUs
• m6i / c6i / r6i → 1 NUMA until ~32–48 vCPUs
• m5.24xlarge / c5.24xlarge → 2 NUMA nodes

Azure (AKS) NUMA

Azure uses “CPU groups”, but effectively:

• D-series, E-series → 1 NUMA up to ~32 vCPUs
• F-series → 1 NUMA up to ~16 vCPUs
• Lsv2 → 2+ NUMA nodes (local SSD optimized)

GCP (GKE) NUMA

• n2-standard, e2-standard → single NUMA up to 32 vCPUs
• n2-highmem/highcpu → single NUMA up to 48 vCPUs
• a2 / g2 GPU nodes → big NUMA topology

PART 4 — Recommended VM Families Per Cloud

AKS (Azure)

⭐ Best General Purpose Workload Nodes:

• D4s_v5, D8s_v5, D16s_v5 Balance of:
• memory
• CPU
• no NUMA surprises

⭐ Best Compute Nodes:

• F4s_v2, F8s_v2 Best for:
• Cilium agents
• API gateways
• small services Avoid > F16 (NUMA segmentation)

⭐ Best Memory-Optimized:

• E8ds_v5, E16ds_v5, E20 Ideal for:
• Java
• Elasticsearch
• Redis

⭐ Best for NVMe-heavy workloads:

• L8s_v3, L16s_v3 For:
• Spark
• batch
• caching
• databases with high random IO

⭐ Best CPU-optimized for AI/DPDK:

• D8as_v5, F8as_v4 (start with 8 cores to keep single NUMA)

EKS (AWS)

⭐ Best general workloads:

• m6i.large / xlarge / 2xlarge / 4xlarge

⭐ Best for high-throughput:

• c6i.xlarge / 2xlarge

⭐ Best memory-heavy:

• r6i.xlarge / 2xlarge / 4xlarge

⭐ Best AI CPU-side pre/post processing:

• c7g (Graviton3) — extreme performance/price
• m7g — best balance

⭐ Best with local SSD:

• i3.xlarge / 2xlarge (best throughput in AWS)

Avoid:

• m5.24xlarge
• c5.18xlarge (NUM A splitting → inconsistent performance)

GKE (Google Cloud)

⭐ Best general workloads:

• n2-standard-4 / 8 / 16

⭐ Best memory workloads:

• n2-highmem-4 / 8 / 16

⭐ Best CPU-heavy:

• c2-standard-4 / 8

⭐ Best local SSD:

• n2-standard-8 w/ Local SSD

Avoid:

• n1 or older instance types
• Very large machine types (> 64 vCPUs)

PART 5 — Node Sizes Per Workload Type

1. Microservices (Go, Node, Python)

Best sizes:

• 4 vCPU / 16 GiB
• 8 vCPU / 32 GiB

Why:

• Good bin packing
• No NUMA pressure
• Fits 10–25 Pods safely

Avoid:

• Very small nodes (inefficient)
• Very large nodes (blast radius)

2. JVM Apps (Spring Boot, Pega, Kafka clients)

Needs:

• high memory per Pod
• JVM heap + direct buffers

Best sizes:

• 8 vCPU / 64 GiB
• 16 vCPU / 128 GiB

If each POD needs 4Gi:

• Node with 64Gi can fit 10–12 properly
• With headroom for system daemons

3. Redis / Memcached

Needs:

• single NUMA node
• predictable CPU
• local SSD optional

Best sizes:

• 8 vCPU / 64 GiB
• 16 vCPU / 128 GiB

Never deploy Redis on:

• multi-NUMA 32–64 core nodes (unless CPU pinned)

4. Envoy Proxy / API Gateway

Needs:

• stable CPU
• no throttling
• low jitter

Best sizes:

• 4 vCPU / 8 GiB
• 8 vCPU / 16 GiB

Run fewer Pods per node for isolation.

5. AI/ML Inference (CPU-bound)

Needs:

• NUMA alignment
• large memory for models
• predictable batching latency

Best sizes:

• 8 vCPU / 32 GiB
• 16 vCPU / 64 GiB

With CPUManager:

• Pin 4–8 CPUs exclusively for inference worker

6. AI/ML with GPU

CPU sizing is secondary.

Good rule:

• 4–6 vCPUs per GPU
• 16–32 GiB memory per GPU

Node example:

• A10 GPU node → 8 vCPU / 32 GiB
• A100 GPU node → 32 vCPU / 128–256 GiB

7. Databases (Postgres, MySQL, Elasticsearch)

Needs:

• huge page cache
• high memory
• stable IO

Best sizes:

• 8 vCPU / 64 GiB
• 16 vCPU / 128 GiB

With local SSD:

• Lsv2 (AKS)
• i3/i4i (EKS)
• n2-standard w/ local SSD (GKE)

Avoid:

• memory-poor compute nodes

8. Spark / Flink / Ray

Executors need:

• memory
• local SSD
• CPU bursts

Best sizes:

• 16 vCPU / 64 GiB
• 32 vCPU / 128 GiB
• with local SSD

Avoid:

• small nodes (executor fragmentation)
• massive nodes (NUMA issues)

PART 6 — When to Use Large Nodes vs Small Nodes

Use small/medium nodes (<16 vCPU) for:

• microservices
• latency-sensitive workloads
• Cilium/Envoy
• Redis
• AI inference
• clusters with high Pod churn

Benefits:

• low blast radius
• easier bin packing
• fast autoscaling

Use large nodes (32–64 vCPU) for:

• Spark executors
• Flink task managers
• ETL workloads
• AI training (multi-GPU nodes)

Avoid very large nodes (>64 vCPU) unless:

• you’re doing ML training
• pods are pinned to cores
• you fully understand NUMA management

PART 7 — Local SSD Guidance

Use nodes with local SSD when:

• Redis
• Postgres WAL/logs
• Spark shuffle
• ML preprocessing
• High local IO workloads

Avoid local SSD for:

• general microservices (no benefit)
• workloads using remote storage (EBS/EFS/Azure Disk/Premium)

PART 8 — Cost Optimization Rules

1. Use medium nodes for better bin packing

   • 8 vCPU / 32 GiB is the global sweet spot

2. Avoid high-memory SKUs unless necessary

   • r-series / E-series cost premium

3. Graviton (AWS) or Ampere (GKE/Oracle) > x86

   • 20–40% cheaper
   • better perf

4. GPU nodes: choose smallest CPU SKU that meets throughput

   • oversizing CPU around GPUs is the #1 cost waste in AI clusters

5. Use autoscaling with Pod Disruption Budgets

   • avoids evacuation storms

SEGMENT 12 SUMMARY

You now have a cloud-agnostic, workload-driven node sizing strategy:

Core Principles

• memory > CPU
• avoid NUMA fragmentation
• prefer several medium nodes
• leave room for system daemons

Best VM Families

• Azure: D-series, E-series, F-series, Lsv2 for SSD
• AWS: m6i, c6i, r6i, c7g (Graviton), i3/i4i
• GCP: n2-standard, n2-highmem, c2-standard

Per-Workload Node Size Playbooks

• Microservices → 4–8 vCPU
• JVM → 8–16 vCPU, high-memory
• Redis → 8 vCPU single-NUMA
• AI inference → 8–16 vCPU
• AI GPU → 4–6 CPUs per GPU
• Spark → 16–32 vCPU, local SSD

Cost Optimization

• medium nodes pack best
• avoid big NUMA nodes
• Graviton/Ampere highly efficient
• GPU nodes should minimize CPU