CodePipeline V2 pipeline type variable resolution
| Service | AWS CodePipeline |
|---|---|
| Cloud | Amazon Web Services (AWS) |
| Guide type | Procedure |
| Skill level | Intermediate to advanced |
| Time | 15 - 60 minutes depending on account size |
When CodePipeline V2 pipeline type variable resolution bites you on AWS CodePipeline, the first instinct is to open a ticket. Most of the time you do not have to. The steps below are the ones AWS Support would walk you through on the call.
What codepipeline v2 pipeline type variable resolution actually involves on AWS CodePipeline
This task on AWS CodePipeline is one of the more searched operational topics on AWS in the last 12 months. The procedure below is the path that works in a current AWS account with default IAM and standard VPC config.
The rest of this page is the structured fix path. Start with diagnose, then remediation, then the automation options so you do not have to do this by hand the next time it surfaces. Verify and safety sections at the end are the discipline that keeps the fix from regressing in production.
Spot the symptom
Look at the CloudTrail event for the failed call, even if you are not enrolled in CloudTrail Lake. The basic 90-day event history works for most diagnostic purposes and lives in the console under CloudTrail > Event history. Filter by event name (the API action) and time range; the event JSON shows the exact user identity, source IP, request parameters, and error code.
Pull the AWS request ID from the response headers: x-amz-request-id for most services, x-amzn-RequestId for API Gateway, both x-amz-request-id and x-amz-id-2 for S3. AWS Support needs these IDs to look up your call in their internal logs - without them, the first reply on a ticket will ask you to reproduce the call and capture them. Save them with a timestamp; AWS Support cannot retrieve calls older than 90 days for most services.
Reproduce the failure with the AWS CLI in --debug mode. The full SigV4 request payload it emits, plus the exact endpoint URL it resolved to, is what AWS Support uses to verify policy, region, or parameter issues without you having to share IAM credentials. Save the debug output to a file with aws ... --debug 2> debug.log and you can search it for the failed aws.request entry.
Solution-focused remediation path
When the failure happens in production but not in dev, do not just compare the IAM policy. Compare the SCP / RCP at the OU level, the permission boundary on the role, and the resource-based policy on the target. One of those is almost always different between accounts. AWS Config conformance packs make this comparison routine.
For IAM and STS issues, the timing matters. STS sessions can take up to 60 seconds to propagate after creation. The first call right after assume-role can fail with a permission error even when the policy is correct. Add a small retry with backoff before treating the first failure as definitive.
If the issue points at IAM, do not start by adding * to a policy. Use IAM Access Analyzer (Policy Generator) against the failed action to see the minimum scope. Adding * is the fastest way to fail your next AWS Well-Architected security review, and it usually does not even fix the issue because the explicit deny is often coming from a higher level (SCP, RCP, or permission boundary), not a missing allow.
Automate this fix so you do not do it twice
Automate the fix with Python and boto3
For anything you do more than twice, write a small Python script. The boto3 pattern below uses paginators (so it does not blow up on accounts with thousands of resources), explicit region binding, and a dry-run flag that defaults to True. Keep the script under 100 lines; if it grows beyond that, you are building a tool and should put it behind a Lambda with proper logging.
import boto3, sys
DRY_RUN = '--apply' not in sys.argv
client = boto3.client('codepipeline', region_name='us-east-1')
paginator = client.get_paginator('describe_...')
for page in paginator.paginate(): for item in page.get('Items', []): if item.get('Status') == 'FAILED': if DRY_RUN: print(f'[dry-run] would fix {item["Id"]}') else: client.modify_...(ResourceId=item['Id']) print(f'fixed {item["Id"]}')Add a Systems Manager Automation runbook
For multi-step fixes that include a manual approval, use SSM Automation. Document the fix as a runbook with aws:approve steps where a human signs off and aws:executeAwsApi steps where the runbook calls the AWS API. Approvers are notified by SNS; the runbook execution shows up in CloudTrail with the approver's identity attached. This makes audit trails easy and stops production fixes from being one-person operations.
Codify the fix in Terraform or CloudFormation
When you reach for the console to fix the same issue twice, the third occurrence should be solved in IaC, not in the console. Terraform's terraform import and CloudFormation's resource importer let you adopt the existing resource into state without recreating it. Lock the corrected attribute behind a variable so the next operator does not have to rediscover the value. Add a moved {} block or CloudFormation resource refactor to keep the diff clean.
Pitfalls
The pitfall most teams hit on AWS CodePipeline is moving too fast and skipping the read-only validation step. Before any write, list the current state and save it. AWS APIs are eventually consistent for many resource types, so the validation snapshot is your only reliable reference if you need to undo. Save the output of the describe call to S3, not to your laptop.
Second pitfall: confusing IAM permission errors with networking errors. AccessDenied can be IAM (policy missing), networking (VPC endpoint policy blocking the call), or KMS (key policy missing). The error string looks identical for all three. Distinguish by looking at the CloudTrail event's errorCode and the encoded authorization message; do not assume IAM is the culprit just because the message says AccessDenied.
Full fix path
- Reproduce the original symptom path. If it still surfaces in any account or region or IAM role, you have not fixed it.
- Watch for 24 to 48 hours. AWS metrics and policy systems can mask issues with cached health for 6 to 12 hours, especially CloudFront and Route 53.
- Run a smoke test under realistic load. Happy-path tests miss race conditions and IAM session-cache issues.
- Capture the new state in a runbook so the next person on call does not have to rediscover this. Push it to Confluence or your team wiki, not into Slack.
- If the fix involved a permission change, run IAM Access Analyzer one more time to confirm you did not open a separate hole while closing this one.
Safety, rollback, blast radius
- Test in a non-production account if your environment has Control Tower or AWS Organizations. The cost of one sandbox account is cheaper than one rollback meeting.
- Export the existing config before changing it. Most AWS CodePipeline resources support describe + export to JSON via CLI - capture that to source control before you start.
- Know your rollback path. Some AWS CodePipeline operations are one-way (region migration, account-level feature opt-in, KMS key deletion past pending window). Confirm reversibility on the AWS doc before you commit.
- Be aware of cross-service impact. IAM role changes ripple to every service trusting that role. KMS key changes break every workload depending on that key. VPC endpoint changes affect every VPC consumer of that endpoint.
- Maintenance window discipline: if the change touches DNS, certificate rotation, or anything that emits TLS handshakes, line up a window with stakeholder notification, not a heroic mid-day swap.
FAQ
aws codepipeline describe-... first, then commit it before you change anything. A few operations are one-way (KMS key deletion past the pending window, region migration, account closure). Check the AWS doc for the specific API before you commit.aws CLI or SDK calls - those almost always still work.References
- docs.aws.amazon.com - official documentation for AWS CodePipeline
- AWS re:Post (formerly forums) - community Q&A with AWS-staff-verified answers
- AWS Health Dashboard at health.aws.amazon.com
- AWS Service Quotas console and AWS Well-Architected Tool
Related fixes
Related guides worth a look while you sort this one out:
- OAC for Lambda function URL AWS_IAM auth-type and SigV4 signing config
- AppSync 200 OK Unauthorized type Not Authorized to access
- AppSync pipeline resolver function early return
- CDK context lookup failed pipeline read-only
- CodeBuild buildspec Commands to be of string type subkeys
- CodePipeline Bitbucket connection state ERROR