HowToFixMe
AWS CloudFormation

CloudFormation drift detection IN_SYNC but resource changed

By Sai Kiran Pandrala · Last verified: 2026-05-31 · Source: AWS re:Post, community Q&A, AWS docs

At a glance
ServiceAWS CloudFormation
CloudAmazon Web Services (AWS)
Guide typeProcedure
Skill levelIntermediate to advanced
Time15 - 60 minutes depending on account size

CloudFormation drift detection IN_SYNC but resource changed on AWS CloudFormation sits in the most-reported issues list across r/aws, AWS re:Post, and StackOverflow. The recovery path is mostly known, the AWS docs just bury it under three layers of conceptual material.

What cloudformation drift detection in_sync but resource changed actually involves on AWS CloudFormation

Real-world context. Cost envelope: ~Rs 0 INR for the fix itself, support plan adds Rs 2,500 to Rs 1,00,000 INR per month (around $30 to $1,200 USD/month). Time at the keyboard: ~15 to 45 minutes. Time end-to-end including verification: ~1 to 4 hours including IAM review and post-fix validation. Have an admin IAM role, the AWS CLI v2, and a CloudTrail filter pointed at the affected resource staged before the first command so you do not stall on missing inputs.

This task on AWS CloudFormation is one of the more searched operational topics on AWS in the last 12 months. The procedure below is the path that works in a current AWS account with default IAM and standard VPC config.

The rest of this page is the structured fix path. Start with diagnose, then remediation, then the automation options so you do not have to do this by hand the next time it surfaces. Verify and safety sections at the end are the discipline that keeps the fix from regressing in production.

Identify

Look at the CloudTrail event for the failed call, even if you are not enrolled in CloudTrail Lake. The basic 90-day event history works for most diagnostic purposes and lives in the console under CloudTrail > Event history. Filter by event name (the API action) and time range; the event JSON shows the exact user identity, source IP, request parameters, and error code.

Diff against last known good. The last config change you made is the cause about three quarters of the time, even when the change should not have mattered. Use AWS Config history (or your Terraform / CloudFormation drift report) to see the actual delta between the resource state when it worked and when it broke. The change you remember is often not the only change that happened.

Run aws sts get-caller-identity first. About one in five 'why does this not work' tickets are actually 'I am in the wrong account' or 'my session expired and the SDK is using stale creds'. The 5-second sanity check costs nothing and saves real time when the answer is that simple.

Solution-focused remediation path

If you cannot reproduce the failure consistently, the cause is probably a race condition or a session-cache issue. Run the call with --profile set to a fresh STS session, in a different region you control, with a single concurrent request. If it works there but fails in your normal setup, the difference is the bug.

For IAM and STS issues, the timing matters. STS sessions can take up to 60 seconds to propagate after creation. The first call right after assume-role can fail with a permission error even when the policy is correct. Add a small retry with backoff before treating the first failure as definitive.

When the failure happens in production but not in dev, do not just compare the IAM policy. Compare the SCP / RCP at the OU level, the permission boundary on the role, and the resource-based policy on the target. One of those is almost always different between accounts. AWS Config conformance packs make this comparison routine.

Automate this fix so you do not do it twice

Add a CloudWatch alarm so you know next time

The cheapest way to never see the same incident twice is a CloudWatch alarm on the metric that would have warned you. For AWS CloudFormation, the relevant metrics live under AWS/cloudformation namespace or under custom metrics published by your Lambda or ECS task. Set thresholds based on observed normal range plus one or two standard deviations, not on round-number guesses. CloudWatch anomaly-detection alarms remove the threshold-guessing problem entirely for metrics with regular seasonality.

Add a Systems Manager Automation runbook

For multi-step fixes that include a manual approval, use SSM Automation. Document the fix as a runbook with aws:approve steps where a human signs off and aws:executeAwsApi steps where the runbook calls the AWS API. Approvers are notified by SNS; the runbook execution shows up in CloudTrail with the approver's identity attached. This makes audit trails easy and stops production fixes from being one-person operations.

Automate the fix with the AWS CLI

The CLI one-liner pattern for AWS CloudFormation operations is roughly: aws cloudformation describe-... --query ... to read state, aws cloudformation modify-... --no-dry-run to apply the change, and aws cloudformation describe-... --query ... again to verify. Wrap it in a shell script that sets a region variable at the top and exits on first error with set -euo pipefail so a partial run does not leave the account in a half-fixed state.

# Template - replace placeholders with your account specifics
export AWS_REGION=us-east-1
export AWS_PROFILE=prod
aws cloudformation describe-... --query 'Resources[?Status==`FAILED`].[Id,Reason]' --output table
aws cloudformation modify-... --resource-id RESOURCE_ID --no-dry-run
aws cloudformation describe-... --resource-id RESOURCE_ID --query 'Status'

Pitfalls to dodge

The most common pitfall when fixing this on AWS CloudFormation is treating it as a one-off rather than as a recurring class of incident. The same misconfiguration tends to happen again after a deployment, a role rotation, or a region migration unless the fix is codified. Add a CloudFormation hook, Service Control Policy condition, or AWS Config rule that prevents the same misconfig from being introduced again. Documentation alone does not survive turnover.

Another common trap: confirming the fix on a single resource and assuming the fleet is healthy. Loop your check across every account, region, and IAM principal that could exhibit the same symptom. If you cannot enumerate the affected scope without a script, you do not yet understand the scope.

Resolve

Safety, rollback, blast radius

FAQ

How long does cloudformation drift detection in_sync but resource changed typically take on AWS?
For most AWS CloudFormation environments, 15 to 60 minutes including verification. Large multi-account setups, anything touching SCPs at the Organizations level, or cross-region replication can stretch to half a day because AWS has to wait for replication and IAM session caches.
Is there a rollback path?
Yes for most AWS CloudFormation changes. Export the existing config to JSON via aws cloudformation describe-... first, then commit it before you change anything. A few operations are one-way (KMS key deletion past the pending window, region migration, account closure). Check the AWS doc for the specific API before you commit.
Will this affect dependent AWS services?
Often yes. AWS CloudFormation resources are usually referenced by other workloads (Lambda, ECS tasks, IAM-bound apps, CloudFront origins, downstream pipelines). Use IAM Access Analyzer + CloudTrail to enumerate consumers before changing a shared resource.
What if my AWS Console layout does not match these steps?
AWS Console UI moves quarterly. The Console layout in this page is current as of 2026-05-31 but the underlying CLI / SDK calls do not change as fast. If the Console version differs, fall back to aws CLI or SDK calls - those almost always still work.
Where do I get AWS Support help if I am still stuck?
Open a case via the AWS Support Center with: the request ID + correlation ID, the exact error string, CloudTrail event, and your reproduction steps. AWS re:Post is the no-cost public alternative - search there first; 80% of common AWS CloudFormation issues already have an answer with an AWS-staff-verified flag.

References

Related guides worth a look while you sort this one out: