How to migrate App Engine Standard from Python 2 to Python 3
| Service | App Engine |
|---|---|
| Cloud | Google Cloud (GCP) |
| Guide type | Procedure |
| Skill level | Intermediate to advanced |
| Time | 15 - 60 minutes depending on account size |
When How to migrate App Engine Standard from Python 2 to Python 3 bites you on App Engine, the first instinct is to open a ticket. Most of the time you do not have to. The steps below are the ones Google Cloud Support would walk you through on the call.
What how to migrate app engine standard from python 2 to python 3 actually involves on App Engine
This task on App Engine is one of the more searched operational topics on AWS in the last 12 months. The procedure below is the path that works in a current AWS account with default IAM and standard VPC config.
The rest of this page is the structured fix path. Start with diagnose, then remediation, then the automation options so you do not have to do this by hand the next time it surfaces. Verify and safety sections at the end are the discipline that keeps the fix from regressing in production.
Diagnose first, fix second
Look at the Cloud Audit Log event for the failed call, even if you are not enrolled in Cloud Logging Log Router. The basic 90-day event history works for most diagnostic purposes and lives in the console under Cloud Audit Logs > Event history. Filter by event name (the API action) and time range; the event JSON shows the exact user identity, source IP, request parameters, and error code.
Diff against last known good. The last config change you made is the cause about three quarters of the time, even when the change should not have mattered. Use Asset Inventory snapshot history (or your Terraform / Deployment Manager or Terraform drift report) to see the actual delta between the resource state when it worked and when it broke. The change you remember is often not the only change that happened.
Reproduce the failure with the gcloud CLI in --debug mode. The full SigV4 request payload it emits, plus the exact endpoint URL it resolved to, is what Google Cloud Support uses to verify policy, region, or parameter issues without you having to share IAM credentials. Save the debug output to a file with gcloud ... --debug 2> debug.log and you can search it for the failed aws.request entry.
Solution-focused remediation path
When the failure happens in production but not in dev, do not just compare the IAM policy. Compare the Org Policy / RCP at the OU level, the permission boundary on the role, and the resource-based policy on the target. One of those is almost always different between accounts. Policy Intelligence recommendations bundles make this comparison routine.
If networking is suspect, use Network Intelligence Connectivity Tests. It is the only tool that simulates the full ENI-to-ENI path including firewall rules, hierarchical firewall policies, routes, and VPC Service Controls perimeters in one call. Manual trace is slower and misses transitive issues. The analyzer charges $0.10 per analysis - cheaper than a 30-minute call with your network team.
If quotas are suspect, the Quotas page in Cloud Console (IAM & Admin > Quotas) console shows current usage and the active limit side by side. Request increases through Quotas page in Cloud Console (IAM & Admin > Quotas), not through Support tickets - quota dashboard requests usually approve faster (often within minutes for soft limits) and they are auditable in Cloud Audit Logs. Set up Quotas page in Cloud Console (IAM & Admin > Quotas) + Cloud Monitoring alert policys at 80 percent usage so you get notified before you hit the wall.
Automate this fix so you do not do it twice
Add a Cloud Monitoring alert policy so you know next time
The cheapest way to never see the same incident twice is a Cloud Monitoring alert policy on the metric that would have warned you. For App Engine, the relevant metrics live under compute.googleapis.com/app namespace or under custom metrics published by your Cloud Run service or GKE pod. Set thresholds based on observed normal range plus one or two standard deviations, not on round-number guesses. Cloud Monitoring anomaly-based alert policies remove the threshold-guessing problem entirely for metrics with regular seasonality.
Codify the fix in Terraform or Deployment Manager
When you reach for the console to fix the same issue twice, the third occurrence should be solved in IaC, not in the console. Terraform's terraform import and Deployment Manager or Terraform's resource importer let you adopt the existing resource into state without recreating it. Lock the corrected attribute behind a variable so the next operator does not have to rediscover the value. Add a moved {} block or Deployment Manager or Terraform resource refactor to keep the diff clean.
Wire the fix into Eventarc for self-healing
If the failure mode is recurring, automate the remediation instead of the diagnosis. Eventarc Scheduler or rules that watch Cloud Logging events for the specific error code can invoke a Lambda that runs the same fix you would run by hand. The Lambda must be idempotent (re-running it on already-healthy resources must be a no-op) and must emit a Cloud Monitoring metric so you can track how often the auto-fix fires. A spike in auto-fix invocations is itself a signal worth alerting on.
# Eventarc rule pattern (JSON)
{ "source": ["aws.app"], "detail-type": ["Google Cloud API Call via Cloud Audit Logs"], "detail": { "errorCode": ["AccessDenied", "ThrottlingException"] }
}
Common pitfalls and what to watch for
The pitfall most teams hit on App Engine is moving too fast and skipping the read-only validation step. Before any write, list the current state and save it. Google Cloud APIs are eventually consistent for many resource types, so the validation snapshot is your only reliable reference if you need to undo. Save the output of the describe call to S3, not to your laptop.
Second pitfall: confusing IAM permission errors with networking errors. AccessDenied can be IAM (policy missing), networking (VPC endpoint policy blocking the call), or KMS (key policy missing). The error string looks identical for all three. Distinguish by looking at the Cloud Audit Log event's errorCode and the encoded authorization message; do not assume IAM is the culprit just because the message says AccessDenied.
Verify the fix worked
- Reproduce the original symptom path. If it still surfaces in any account or region or IAM role or service account, you have not fixed it.
- Watch for 24 to 48 hours. Cloud Monitoring metrics and Cloud Asset Inventory can mask issues with cached health for 6 to 12 hours, especially Cloud CDN and Cloud DNS.
- Run a smoke test under realistic load. Happy-path tests miss race conditions and IAM session-cache issues.
- Capture the new state in a runbook so the next person on call does not have to rediscover this. Push it to Confluence or your team wiki, not into Slack.
- If the fix involved a permission change, run IAM Access Analyzer one more time to confirm you did not open a separate hole while closing this one.
Safety, rollback, blast radius
- Test in a non-production account if your environment has Resource Manager and Organization Policy or Cloud Resource Manager (organizations, folders, projects). The cost of one sandbox account is cheaper than one rollback meeting.
- Export the existing config before changing it. Most App Engine resources support describe + export to JSON via CLI - capture that to source control before you start.
- Know your rollback path. Some App Engine operations are one-way (region migration, account-level feature opt-in, Cloud KMS key deletion past pending window). Confirm reversibility on the Google Cloud doc before you commit.
- Be aware of cross-service impact. IAM role or service account changes ripple to every service trusting that role. Cloud KMS key changes break every workload depending on that key. VPC endpoint changes affect every VPC consumer of that endpoint.
- Maintenance window discipline: if the change touches DNS, certificate rotation, or anything that emits TLS handshakes, line up a window with stakeholder notification, not a heroic mid-day swap.
FAQ
gcloud app describe-... first, then commit it before you change anything. A few operations are one-way (Cloud KMS key deletion past the pending window, region migration, account closure). Check the Google Cloud doc for the specific API before you commit.aws CLI or SDK calls - those almost always still work.References
- docs.cloud.google.com - official documentation for App Engine
- Google Cloud Community - community Q&A with Google-staff-verified answers
- Cloud Service Health Dashboard at health.cloud.google.com
- Quotas page in Cloud Console (IAM & Admin > Quotas) and Architecture Framework checklists
Related fixes
Related guides worth a look while you sort this one out:
- How to migrate App Engine Standard to Cloud Run
- How to access Cloud SQL from App Engine Standard via Unix socket
- How to enable IAP on App Engine Standard
- How to fix DeadlineExceededError on App Engine Standard
- How to set up cron.yaml on App Engine Standard
- How to set up queue.yaml on App Engine Standard