Connectivity on Database Migration Service, what causes it and how to fix
| Service | Google Database Migration Service |
|---|---|
| Cloud | Google Cloud (GCP) |
| Guide type | Procedure |
| Skill level | Intermediate to advanced |
| Time | 15 - 60 minutes depending on account size |
Running into Connectivity on Database Migration Service, what causes it and how to fix on Google Database Migration Service is one of the more searched issues on Google Cloud Community and StackOverflow in the last 12 months. Here is what actually moves the needle when the Google Cloud docs are too generic.
What connectivity on database migration service, what causes it and how to fix actually involves on Google Database Migration Service
The Connectivity error from AWS typically surfaces with the message "test failed routing misconfigured in Private Service Connect". The error code itself is what you grep for in AWS re:Post or in AWS Support cases, not the human-readable line.
On Database Migration Service, this most often comes from one of three causes: a missing or restrictive IAM permission, a service-level limit you have hit, or a transient AWS-side capacity issue. The fix path differs by which.
The rest of this page is the structured fix path. Start with diagnose, then remediation, then the automation options so you do not have to do this by hand the next time it surfaces. Verify and safety sections at the end are the discipline that keeps the fix from regressing in production.
Diagnose first, fix second
Diff against last known good. The last config change you made is the cause about three quarters of the time, even when the change should not have mattered. Use Asset Inventory snapshot history (or your Terraform / Deployment Manager or Terraform drift report) to see the actual delta between the resource state when it worked and when it broke. The change you remember is often not the only change that happened.
Check Cloud Monitoring Logs for the calling service. Lambda, ECS, EKS, Step Functions, API Gateway, and most managed services write detailed traces to Cloud Monitoring Logs under predictable log group names. Use Cloud Monitoring Logs Insights with fields @timestamp, @message | filter @message like /ERROR/ | sort @timestamp desc | limit 50 to surface the most recent failures.
Run gcloud auth list and gcloud config list first. About one in five 'why does this not work' tickets are actually 'I am in the wrong account' or 'my session expired and the SDK is using stale credentials or ADC pointed at the wrong project'. The 5-second sanity check costs nothing and saves real time when the answer is that simple.
Solution-focused remediation path
If quotas are suspect, the Quotas page in Cloud Console (IAM & Admin > Quotas) console shows current usage and the active limit side by side. Request increases through Quotas page in Cloud Console (IAM & Admin > Quotas), not through Support tickets - quota dashboard requests usually approve faster (often within minutes for soft limits) and they are auditable in Cloud Audit Logs. Set up Quotas page in Cloud Console (IAM & Admin > Quotas) + Cloud Monitoring alert policys at 80 percent usage so you get notified before you hit the wall.
When the failure happens in production but not in dev, do not just compare the IAM policy. Compare the Org Policy / RCP at the OU level, the permission boundary on the role, and the resource-based policy on the target. One of those is almost always different between accounts. Policy Intelligence recommendations bundles make this comparison routine.
If the issue points at IAM, do not start by adding * to a policy. Use IAM Policy Troubleshooter and IAM Recommender against the failed action to see the minimum scope. Adding * is the fastest way to fail your next Google Cloud Architecture Framework security review, and it usually does not even fix the issue because the explicit deny is often coming from a higher level (Org Policy, RCP, or permission boundary), not a missing allow.
Automate this fix so you do not do it twice
Codify the fix in Terraform or Deployment Manager
When you reach for the console to fix the same issue twice, the third occurrence should be solved in IaC, not in the console. Terraform's terraform import and Deployment Manager or Terraform's resource importer let you adopt the existing resource into state without recreating it. Lock the corrected attribute behind a variable so the next operator does not have to rediscover the value. Add a moved {} block or Deployment Manager or Terraform resource refactor to keep the diff clean.
Add a Cloud Monitoring alert policy so you know next time
The cheapest way to never see the same incident twice is a Cloud Monitoring alert policy on the metric that would have warned you. For Google Database Migration Service, the relevant metrics live under compute.googleapis.com/google namespace or under custom metrics published by your Cloud Run service or GKE pod. Set thresholds based on observed normal range plus one or two standard deviations, not on round-number guesses. Cloud Monitoring anomaly-based alert policies remove the threshold-guessing problem entirely for metrics with regular seasonality.
Automate the fix with the gcloud CLI
The CLI one-liner pattern for Google Database Migration Service operations is roughly: gcloud google describe RESOURCE --format=json --filter ... to read state, gcloud google update RESOURCE --quiet to apply the change, and gcloud google describe RESOURCE --format=json --filter ... again to verify. Wrap it in a shell script that sets a region variable at the top and exits on first error with set -euo pipefail so a partial run does not leave the account in a half-fixed state.
# Template - replace placeholders with your account specifics
export GOOGLE_CLOUD_REGION=us-central1
export GOOGLE_CLOUD_PROJECT=prod-project
gcloud google describe RESOURCE --format=json --filter 'Resources[?Status==`FAILED`].[Id,Reason]' --output table
gcloud google modify-... --resource-id RESOURCE_ID --no-dry-run
gcloud google describe RESOURCE_ID --query 'Status'
Common pitfalls and what to watch for
A subtle pitfall on Google Database Migration Service is that the Cloud Console and the SDK can disagree about resource state during a configuration change. Console UI is cached for performance and may show the old config for up to 10 minutes after you change it via API or Deployment Manager or Terraform. Always confirm with describe-* CLI calls during a change window, not with screenshots from the Console.
The other pitfall: assuming that an automated remediation is correct because it succeeded. A Lambda that fires on a Cloud Monitoring alert policy and runs a remediation step should also publish a metric for every remediation; sudden surges in auto-fix invocations are themselves an outage signal. Otherwise you can hide a slow-burn regression behind a quiet remediation loop for weeks.
Verify the fix worked
- Reproduce the original symptom path. If it still surfaces in any account or region or IAM role or service account, you have not fixed it.
- Watch for 24 to 48 hours. Cloud Monitoring metrics and Cloud Asset Inventory can mask issues with cached health for 6 to 12 hours, especially Cloud CDN and Cloud DNS.
- Run a smoke test under realistic load. Happy-path tests miss race conditions and IAM session-cache issues.
- Capture the new state in a runbook so the next person on call does not have to rediscover this. Push it to Confluence or your team wiki, not into Slack.
- If the fix involved a permission change, run IAM Access Analyzer one more time to confirm you did not open a separate hole while closing this one.
Safety, rollback, blast radius
- Test in a non-production account if your environment has Resource Manager and Organization Policy or Cloud Resource Manager (organizations, folders, projects). The cost of one sandbox account is cheaper than one rollback meeting.
- Export the existing config before changing it. Most Google Database Migration Service resources support describe + export to JSON via CLI - capture that to source control before you start.
- Know your rollback path. Some Google Database Migration Service operations are one-way (region migration, account-level feature opt-in, Cloud KMS key deletion past pending window). Confirm reversibility on the Google Cloud doc before you commit.
- Be aware of cross-service impact. IAM role or service account changes ripple to every service trusting that role. Cloud KMS key changes break every workload depending on that key. VPC endpoint changes affect every VPC consumer of that endpoint.
- Maintenance window discipline: if the change touches DNS, certificate rotation, or anything that emits TLS handshakes, line up a window with stakeholder notification, not a heroic mid-day swap.
FAQ
gcloud google describe-... first, then commit it before you change anything. A few operations are one-way (Cloud KMS key deletion past the pending window, region migration, account closure). Check the Google Cloud doc for the specific API before you commit.aws CLI or SDK calls - those almost always still work.References
- docs.cloud.google.com - official documentation for Google Database Migration Service
- Google Cloud Community - community Q&A with Google-staff-verified answers
- Cloud Service Health Dashboard at health.cloud.google.com
- Quotas page in Cloud Console (IAM & Admin > Quotas) and Architecture Framework checklists
Related fixes
Related guides worth a look while you sort this one out:
- Error on Database Migration Service: what causes it and how to fix
- FAILED_PRECONDITION on Database Migration Service, what causes it and how to fix
- Logical on Database Migration Service, what causes it and how to fix
- Migration on Database Migration Service, what causes it and how to fix
- Replication on Database Migration Service, what causes it and how to fix
- Service on Database Migration Service, what causes it and how to fix