Managing Terraform Phantom Diffs: A Practical Guide

What Are Phantom Diffs?

Phantom diffs — also called perpetual diffs or ghost changes — are changes that appear in your terraform plan output every single run, even when you haven't modified any code or infrastructure. They create noise, erode trust in your plans, and can mask real changes hiding among the false positives.

# azurerm_resource_group.example will be updated in-place
~ resource "azurerm_resource_group" "example" {
      id       = "/subscriptions/.../resourceGroups/my-rg"
      name     = "my-rg"
    ~ tags     = {
        - "created_by" = "terraform" -> null
      }
  }

You stare at the diff. You didn't change anything. You run plan again — same diff. Welcome to the world of phantom diffs.

---

Why Do Phantom Diffs Happen?

There are several root causes, and understanding them is the key to fixing them.

1. Provider Defaults and API Normalization

Cloud APIs often return values in a different format than what you send. The provider compares your config to the API response and sees a "difference" every time.

Example: Case sensitivity

resource "azurerm_resource_group" "example" {
  name     = "My-Resource-Group"
  location = "East US"
}

Azure might normalize location to eastus internally. Every plan, Terraform sees "East US" in your config vs "eastus" in state, and reports a diff.

Fix: Match the API's canonical form:

resource "azurerm_resource_group" "example" {
  name     = "My-Resource-Group"
  location = "eastus"
}

2. Computed Attributes Conflicting with Config

Some attributes are both configurable and computed by the provider. If you set a value that the API overrides or augments, you'll get a perpetual diff.

Example: Tags merged by Azure Policy

resource "azurerm_resource_group" "example" {
  name     = "my-rg"
  location = "eastus"

  tags = {
    environment = "dev"
  }
}

If an Azure Policy automatically adds "created_by" = "policy", every plan shows:

~ tags = {
    + "created_by" = "policy"    # Terraform wants to remove this
      "environment" = "dev"
  }

Terraform wants to enforce your declared state, which doesn't include that tag.

Fix: Use ignore_changes in a lifecycle block:

resource "azurerm_resource_group" "example" {
  name     = "my-rg"
  location = "eastus"

  tags = {
    environment = "dev"
  }

  lifecycle {
    ignore_changes = [tags["created_by"]]
  }
}

Or, if the external system adds many unpredictable tags:

  lifecycle {
    ignore_changes = [tags]
  }

3. Attribute Ordering and Serialization

Some resources contain list or set attributes where order is semantically irrelevant but syntactically significant to Terraform.

Example: Security group rules

resource "aws_security_group" "example" {
  name = "my-sg"

  ingress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["10.0.0.0/8", "172.16.0.0/12"]
  }
}

If the AWS API returns cidr_blocks as ["172.16.0.0/12", "10.0.0.0/8"], you get a phantom diff every run.

Fix: Sort your values to match API return order, or use ignore_changes:

  cidr_blocks = ["172.16.0.0/12", "10.0.0.0/8"]  # Match API order

4. Default Values Applied Server-Side

Resources often have optional attributes that the API fills in with defaults. If the provider reads these back but you didn't set them, Terraform may show an unwanted diff.

Example: AWS Launch Template

resource "aws_launch_template" "example" {
  name_prefix   = "my-lt-"
  image_id      = "ami-0abcdef1234567890"
  instance_type = "t3.micro"
}

The API might populate metadata_options with defaults. Next plan:

~ metadata_options {
    ~ http_endpoint               = "enabled" -> "enabled"
    ~ http_put_response_hop_limit = 1 -> 1
    ~ http_tokens                 = "optional" -> "optional"
  }

Fix: Explicitly declare the defaults in your config:

resource "aws_launch_template" "example" {
  name_prefix   = "my-lt-"
  image_id      = "ami-0abcdef1234567890"
  instance_type = "t3.micro"

  metadata_options {
    http_endpoint               = "enabled"
    http_put_response_hop_limit = 1
    http_tokens                 = "optional"
  }
}

5. Sensitive or Computed-Only Values in State

Some providers mark attributes as sensitive or don't store them in state properly. This causes Terraform to believe the value is missing or changed.

Example: Passwords and secrets

resource "azuredevops_serviceendpoint_generic" "example" {
  project_id            = azuredevops_project.example.id
  service_endpoint_name = "my-endpoint"
  server_url            = "https://example.com"
  password              = "my-secret"
}

If the provider can't read the password back from the API (it's write-only), every plan shows:

~ password = (sensitive value)

Fix: Use ignore_changes for write-only secrets:

  lifecycle {
    ignore_changes = [password]
  }

Note: Terraform v1.7+ introduced the ephemeral attribute concept and write-only arguments in v1.11+ to handle this pattern more elegantly.

6. Timestamp and Auto-Generated Fields

Resources that include timestamps (last_modified, updated_at) or auto-generated IDs that change on read will always show diffs.

Example:

~ last_modified = "2026-03-19T10:00:00Z" -> "2026-03-20T08:30:00Z"

Fix:

  lifecycle {
    ignore_changes = [last_modified]
  }

7. Inconsistent jsonencode / JSON Formatting

When you pass JSON as a string (e.g., IAM policies, API definitions), whitespace or key-ordering differences between your config and the API cause phantom diffs.

Example:

resource "aws_iam_role" "example" {
  name = "my-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action    = "sts:AssumeRole"
      Effect    = "Allow"
      Principal = { Service = "ec2.amazonaws.com" }
    }]
  })
}

The API may return the policy with different key ordering or whitespace, causing:

~ assume_role_policy = jsonencode(
    ~ {
        - Statement = [...]
        + Statement = [...]  # Same content, different formatting
      }
  )

Fix: Use the dedicated policy resources or data sources:

data "aws_iam_policy_document" "assume_role" {
  statement {
    actions = ["sts:AssumeRole"]
    effect  = "Allow"
    principals {
      type        = "Service"
      identifiers = ["ec2.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "example" {
  name               = "my-role"
  assume_role_policy = data.aws_iam_policy_document.assume_role.json
}

---

Strategies for Handling Phantom Diffs

Strategy 1: lifecycle { ignore_changes = [...] }

The most common and direct fix. Use it when an external system modifies attributes outside Terraform's control.

resource "kubernetes_deployment" "example" {
  metadata {
    name = "my-app"
  }

  lifecycle {
    ignore_changes = [
      metadata[0].annotations["kubectl.kubernetes.io/last-applied-configuration"],
      spec[0].template[0].metadata[0].annotations,
    ]
  }
}

When to use: External systems (policies, operators, other tools) modify resources after Terraform applies them.

When NOT to use: You're hiding a real configuration problem. ignore_changes is a scalpel, not a sledgehammer.

Strategy 2: Match the API's Canonical Form

Before reaching for ignore_changes, check if you can simply adjust your config to match what the API returns.

# Check what the API actually stores:
terraform show -json | jq '.values.root_module.resources[] | select(.address == "aws_s3_bucket.example")'

Then update your config to match. This is the cleanest solution.

Strategy 3: Use terraform state show to Inspect

terraform state show 'module.my_module.azuredevops_serviceendpoint_nuget.this'

Compare the state values with your config to identify exactly which attribute drifts.

Strategy 4: Pin Provider Versions

Provider updates can introduce or fix phantom diffs. Always pin your versions:

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 3.85.0"
    }
  }
}

Check provider changelogs — phantom diff fixes are common in patch releases.

Strategy 5: Use replace_triggered_by Instead of Fighting Diffs

Sometimes a resource will always show a diff because a dependency changed. Rather than suppress it, embrace it:

resource "null_resource" "config_reload" {
  lifecycle {
    replace_triggered_by = [
      azuredevops_variable_group.secrets
    ]
  }
}

Strategy 6: Automate Phantom Diff Detection in CI/CD

Add a check to your pipeline that flags plans with only known phantom diffs:

#!/bin/bash
terraform plan -detailed-exitcode -out=plan.tfplan 2>&1
EXIT_CODE=$?

if [ $EXIT_CODE -eq 2 ]; then
  # Changes detected — check if they're all known phantoms
  terraform show -json plan.tfplan | jq -r '
    .resource_changes[]
    | select(.change.actions != ["no-op"])
    | .address
  ' > changed_resources.txt

  KNOWN_PHANTOMS="azuredevops_serviceendpoint_generic.password_reset"

  while read -r resource; do
    if ! echo "\(KNOWN_PHANTOMS" | grep -q "\)resource"; then
      echo "REAL CHANGE DETECTED: $resource"
      exit 2
    fi
  done < changed_resources.txt

  echo "Only phantom diffs detected — safe to skip apply."
  exit 0
fi

---

Decision Framework

Symptom	Root Cause	Fix
Case/format differences	API normalization	Match canonical form
External tags/annotations	External system modification	ignore_changes on specific keys
Sensitive values always changing	Write-only API fields	ignore_changes on secret fields
JSON/policy reformatting	Serialization differences	Use dedicated data sources
Ordering changes in lists	Non-deterministic API responses	Sort values or use ignore_changes
New attributes appearing after upgrade	Provider version change	Pin provider, set explicit defaults

---

Key Takeaways

1. Diagnose before suppressing. Use terraform state show and terraform show -json to understand why a diff appears before reaching for ignore_changes.
2. ignore_changes is a trade-off. It solves the noise problem but creates a blind spot — Terraform will never manage that attribute again until you remove the lifecycle rule.
3. Match the API, not the docs. The API's canonical representation is the source of truth. Adjust your config to match it.
4. Pin provider versions. Phantom diff behavior can change between provider versions. Pin versions and upgrade deliberately.
5. Automate detection. In CI/CD, distinguish phantom diffs from real changes to prevent pipeline fatigue and rubber-stamp approvals.
6. Report bugs upstream. Many phantom diffs are provider bugs. File issues — the maintainers fix them regularly.

Phantom diffs are an inevitable part of working with Terraform at scale, but they don't have to be a source of constant frustration. A methodical approach to diagnosing and resolving them keeps your plans clean and trustworthy.