# Managing Terraform Phantom Diffs: A Practical Guide

## What Are Phantom Diffs?

Phantom diffs — also called **perpetual diffs** or **ghost changes** — are changes that appear in your `terraform plan` output every single run, even when you haven't modified any code or infrastructure. They create noise, erode trust in your plans, and can mask real changes hiding among the false positives.

```
# azurerm_resource_group.example will be updated in-place
~ resource "azurerm_resource_group" "example" {
      id       = "/subscriptions/.../resourceGroups/my-rg"
      name     = "my-rg"
    ~ tags     = {
        - "created_by" = "terraform" -> null
      }
  }
```

You stare at the diff. You didn't change anything. You run `plan` again — same diff. Welcome to the world of phantom diffs.

---

## Why Do Phantom Diffs Happen?

There are several root causes, and understanding them is the key to fixing them.

### 1. Provider Defaults and API Normalization

Cloud APIs often return values in a different format than what you send. The provider compares your config to the API response and sees a "difference" every time.

**Example: Case sensitivity**

```hcl
resource "azurerm_resource_group" "example" {
  name     = "My-Resource-Group"
  location = "East US"
}
```

Azure might normalize `location` to `eastus` internally. Every plan, Terraform sees `"East US"` in your config vs `"eastus"` in state, and reports a diff.

**Fix:** Match the API's canonical form:

```hcl
resource "azurerm_resource_group" "example" {
  name     = "My-Resource-Group"
  location = "eastus"
}
```

### 2. Computed Attributes Conflicting with Config

Some attributes are both configurable *and* computed by the provider. If you set a value that the API overrides or augments, you'll get a perpetual diff.

**Example: Tags merged by Azure Policy**

```hcl
resource "azurerm_resource_group" "example" {
  name     = "my-rg"
  location = "eastus"

  tags = {
    environment = "dev"
  }
}
```

If an Azure Policy automatically adds `"created_by" = "policy"`, every plan shows:

```
~ tags = {
    + "created_by" = "policy"    # Terraform wants to remove this
      "environment" = "dev"
  }
```

Terraform wants to enforce *your* declared state, which doesn't include that tag.

**Fix:** Use `ignore_changes` in a lifecycle block:

```hcl
resource "azurerm_resource_group" "example" {
  name     = "my-rg"
  location = "eastus"

  tags = {
    environment = "dev"
  }

  lifecycle {
    ignore_changes = [tags["created_by"]]
  }
}
```

Or, if the external system adds many unpredictable tags:

```hcl
  lifecycle {
    ignore_changes = [tags]
  }
```

### 3. Attribute Ordering and Serialization

Some resources contain list or set attributes where order is semantically irrelevant but syntactically significant to Terraform.

**Example: Security group rules**

```hcl
resource "aws_security_group" "example" {
  name = "my-sg"

  ingress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["10.0.0.0/8", "172.16.0.0/12"]
  }
}
```

If the AWS API returns `cidr_blocks` as `["172.16.0.0/12", "10.0.0.0/8"]`, you get a phantom diff every run.

**Fix:** Sort your values to match API return order, or use `ignore_changes`:

```hcl
  cidr_blocks = ["172.16.0.0/12", "10.0.0.0/8"]  # Match API order
```

### 4. Default Values Applied Server-Side

Resources often have optional attributes that the API fills in with defaults. If the provider reads these back but you didn't set them, Terraform may show an unwanted diff.

**Example: AWS Launch Template**

```hcl
resource "aws_launch_template" "example" {
  name_prefix   = "my-lt-"
  image_id      = "ami-0abcdef1234567890"
  instance_type = "t3.micro"
}
```

The API might populate `metadata_options` with defaults. Next plan:

```
~ metadata_options {
    ~ http_endpoint               = "enabled" -> "enabled"
    ~ http_put_response_hop_limit = 1 -> 1
    ~ http_tokens                 = "optional" -> "optional"
  }
```

**Fix:** Explicitly declare the defaults in your config:

```hcl
resource "aws_launch_template" "example" {
  name_prefix   = "my-lt-"
  image_id      = "ami-0abcdef1234567890"
  instance_type = "t3.micro"

  metadata_options {
    http_endpoint               = "enabled"
    http_put_response_hop_limit = 1
    http_tokens                 = "optional"
  }
}
```

### 5. Sensitive or Computed-Only Values in State

Some providers mark attributes as sensitive or don't store them in state properly. This causes Terraform to believe the value is missing or changed.

**Example: Passwords and secrets**

```hcl
resource "azuredevops_serviceendpoint_generic" "example" {
  project_id            = azuredevops_project.example.id
  service_endpoint_name = "my-endpoint"
  server_url            = "https://example.com"
  password              = "my-secret"
}
```

If the provider can't read the password back from the API (it's write-only), every plan shows:

```
~ password = (sensitive value)
```

**Fix:** Use `ignore_changes` for write-only secrets:

```hcl
  lifecycle {
    ignore_changes = [password]
  }
```

> **Note:** Terraform v1.7+ introduced the `ephemeral` attribute concept and write-only arguments in v1.11+ to handle this pattern more elegantly.

### 6. Timestamp and Auto-Generated Fields

Resources that include timestamps (`last_modified`, `updated_at`) or auto-generated IDs that change on read will always show diffs.

**Example:**

```
~ last_modified = "2026-03-19T10:00:00Z" -> "2026-03-20T08:30:00Z"
```

**Fix:**

```hcl
  lifecycle {
    ignore_changes = [last_modified]
  }
```

### 7. Inconsistent `jsonencode` / JSON Formatting

When you pass JSON as a string (e.g., IAM policies, API definitions), whitespace or key-ordering differences between your config and the API cause phantom diffs.

**Example:**

```hcl
resource "aws_iam_role" "example" {
  name = "my-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action    = "sts:AssumeRole"
      Effect    = "Allow"
      Principal = { Service = "ec2.amazonaws.com" }
    }]
  })
}
```

The API may return the policy with different key ordering or whitespace, causing:

```
~ assume_role_policy = jsonencode(
    ~ {
        - Statement = [...]
        + Statement = [...]  # Same content, different formatting
      }
  )
```

**Fix:** Use the dedicated policy resources or data sources:

```hcl
data "aws_iam_policy_document" "assume_role" {
  statement {
    actions = ["sts:AssumeRole"]
    effect  = "Allow"
    principals {
      type        = "Service"
      identifiers = ["ec2.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "example" {
  name               = "my-role"
  assume_role_policy = data.aws_iam_policy_document.assume_role.json
}
```

---

## Strategies for Handling Phantom Diffs

### Strategy 1: `lifecycle { ignore_changes = [...] }`

The most common and direct fix. Use it when an external system modifies attributes outside Terraform's control.

```hcl
resource "kubernetes_deployment" "example" {
  metadata {
    name = "my-app"
  }

  lifecycle {
    ignore_changes = [
      metadata[0].annotations["kubectl.kubernetes.io/last-applied-configuration"],
      spec[0].template[0].metadata[0].annotations,
    ]
  }
}
```

**When to use:** External systems (policies, operators, other tools) modify resources after Terraform applies them.

**When NOT to use:** You're hiding a real configuration problem. `ignore_changes` is a scalpel, not a sledgehammer.

### Strategy 2: Match the API's Canonical Form

Before reaching for `ignore_changes`, check if you can simply adjust your config to match what the API returns.

```bash
# Check what the API actually stores:
terraform show -json | jq '.values.root_module.resources[] | select(.address == "aws_s3_bucket.example")'
```

Then update your config to match. This is the cleanest solution.

### Strategy 3: Use `terraform state show` to Inspect

```bash
terraform state show 'module.my_module.azuredevops_serviceendpoint_nuget.this'
```

Compare the state values with your config to identify exactly which attribute drifts.

### Strategy 4: Pin Provider Versions

Provider updates can introduce or fix phantom diffs. Always pin your versions:

```hcl
terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 3.85.0"
    }
  }
}
```

Check provider changelogs — phantom diff fixes are common in patch releases.

### Strategy 5: Use `replace_triggered_by` Instead of Fighting Diffs

Sometimes a resource will always show a diff because a dependency changed. Rather than suppress it, embrace it:

```hcl
resource "null_resource" "config_reload" {
  lifecycle {
    replace_triggered_by = [
      azuredevops_variable_group.secrets
    ]
  }
}
```

### Strategy 6: Automate Phantom Diff Detection in CI/CD

Add a check to your pipeline that flags plans with *only* known phantom diffs:

```bash
#!/bin/bash
terraform plan -detailed-exitcode -out=plan.tfplan 2>&1
EXIT_CODE=$?

if [ $EXIT_CODE -eq 2 ]; then
  # Changes detected — check if they're all known phantoms
  terraform show -json plan.tfplan | jq -r '
    .resource_changes[]
    | select(.change.actions != ["no-op"])
    | .address
  ' > changed_resources.txt

  KNOWN_PHANTOMS="azuredevops_serviceendpoint_generic.password_reset"

  while read -r resource; do
    if ! echo "$KNOWN_PHANTOMS" | grep -q "$resource"; then
      echo "REAL CHANGE DETECTED: $resource"
      exit 2
    fi
  done < changed_resources.txt

  echo "Only phantom diffs detected — safe to skip apply."
  exit 0
fi
```

---

## Decision Framework

| Symptom | Root Cause | Fix |
|---|---|---|
| Case/format differences | API normalization | Match canonical form |
| External tags/annotations | External system modification | `ignore_changes` on specific keys |
| Sensitive values always changing | Write-only API fields | `ignore_changes` on secret fields |
| JSON/policy reformatting | Serialization differences | Use dedicated data sources |
| Ordering changes in lists | Non-deterministic API responses | Sort values or use `ignore_changes` |
| New attributes appearing after upgrade | Provider version change | Pin provider, set explicit defaults |

---

## Key Takeaways

1. **Diagnose before suppressing.** Use `terraform state show` and `terraform show -json` to understand *why* a diff appears before reaching for `ignore_changes`.
2. **`ignore_changes` is a trade-off.** It solves the noise problem but creates a blind spot — Terraform will never manage that attribute again until you remove the lifecycle rule.
3. **Match the API, not the docs.** The API's canonical representation is the source of truth. Adjust your config to match it.
4. **Pin provider versions.** Phantom diff behavior can change between provider versions. Pin versions and upgrade deliberately.
5. **Automate detection.** In CI/CD, distinguish phantom diffs from real changes to prevent pipeline fatigue and rubber-stamp approvals.
6. **Report bugs upstream.** Many phantom diffs are provider bugs. File issues — the maintainers fix them regularly.

Phantom diffs are an inevitable part of working with Terraform at scale, but they don't have to be a source of constant frustration. A methodical approach to diagnosing and resolving them keeps your plans clean and trustworthy.