OpenAI-compatible Chat Completions endpoint on Vertex Gemini

By Sai Kiran Pandrala · Last verified: 2026-05-31 · Source: Google Cloud docs, Google Cloud Community, community Q&A

Running into OpenAI-compatible Chat Completions endpoint on Vertex Gemini on Vertex AI Model Garden is one of the more searched issues on Google Cloud Community and StackOverflow in the last 12 months. Here is what actually moves the needle when the Google Cloud docs are too generic.

What openai-compatible chat completions endpoint on vertex gemini actually involves on Vertex AI Model Garden

This task on Vertex AI Model Garden is one of the more searched operational topics on AWS in the last 12 months. The procedure below is the path that works in a current AWS account with default IAM and standard VPC config.

The rest of this page is the structured fix path. Start with diagnose, then remediation, then the automation options so you do not have to do this by hand the next time it surfaces. Verify and safety sections at the end are the discipline that keeps the fix from regressing in production.

Diagnose first, fix second

Check the Google Cloud Service Health at status.cloud.google.com and the per-product status board for ongoing service events in your region. About one in ten user-reported outages turn out to be region-scoped Google Cloud service degradation already being tracked. Cloud Service Health also exposes an API and Eventarc events, so you can wire a Lambda hook that pages on-call only when the failure correlates with an active Cloud Service Health event in the same region and service.

Run gcloud auth list and gcloud config list first. About one in five 'why does this not work' tickets are actually 'I am in the wrong account' or 'my session expired and the SDK is using stale credentials or ADC pointed at the wrong project'. The 5-second sanity check costs nothing and saves real time when the answer is that simple.

Reproduce the failure with the gcloud CLI in --debug mode. The full SigV4 request payload it emits, plus the exact endpoint URL it resolved to, is what Google Cloud Support uses to verify policy, region, or parameter issues without you having to share IAM credentials. Save the debug output to a file with gcloud ... --debug 2> debug.log and you can search it for the failed aws.request entry.

Solution-focused remediation path

If you cannot reproduce the failure consistently, the cause is probably a race condition or a session-cache issue. Run the call with --profile set to a fresh STS session, in a different region you control, with a single concurrent request. If it works there but fails in your normal setup, the difference is the bug.

When the fix involves a destructive operation (delete VPC endpoint, swap Cloud KMS key, rotate root credential), do it during a maintenance window with at least one teammate watching. Several Vertex AI Model Garden operations have implicit dependencies that only show up when traffic starts flowing again. Document the rollback path before you start, not during the incident.

If networking is suspect, use Network Intelligence Connectivity Tests. It is the only tool that simulates the full ENI-to-ENI path including firewall rules, hierarchical firewall policies, routes, and VPC Service Controls perimeters in one call. Manual trace is slower and misses transitive issues. The analyzer charges $0.10 per analysis - cheaper than a 30-minute call with your network team.

Automate this fix so you do not do it twice

Automate the fix with Python and boto3

For anything you do more than twice, write a small Python script. The boto3 pattern below uses paginators (so it does not blow up on accounts with thousands of resources), explicit region binding, and a dry-run flag that defaults to True. Keep the script under 100 lines; if it grows beyond that, you are building a tool and should put it behind a Lambda with proper logging.

import boto3, sys
DRY_RUN = '--apply' not in sys.argv
client = boto3.client('vertex', region_name='us-east-1')
paginator = client.get_paginator('describe_...')
for page in paginator.paginate(): for item in page.get('Items', []): if item.get('Status') == 'FAILED': if DRY_RUN: print(f'[dry-run] would fix {item["Id"]}') else: client.modify_...(ResourceId=item['Id']) print(f'fixed {item["Id"]}')

Wire the fix into Eventarc for self-healing

If the failure mode is recurring, automate the remediation instead of the diagnosis. Eventarc Scheduler or rules that watch Cloud Logging events for the specific error code can invoke a Lambda that runs the same fix you would run by hand. The Lambda must be idempotent (re-running it on already-healthy resources must be a no-op) and must emit a Cloud Monitoring metric so you can track how often the auto-fix fires. A spike in auto-fix invocations is itself a signal worth alerting on.

# Eventarc rule pattern (JSON)
{ "source": ["aws.vertex"], "detail-type": ["Google Cloud API Call via Cloud Audit Logs"], "detail": { "errorCode": ["AccessDenied", "ThrottlingException"] }
}

Automate the fix with the gcloud CLI

The CLI one-liner pattern for Vertex AI Model Garden operations is roughly: gcloud vertex describe RESOURCE --format=json --filter ... to read state, gcloud vertex update RESOURCE --quiet to apply the change, and gcloud vertex describe RESOURCE --format=json --filter ... again to verify. Wrap it in a shell script that sets a region variable at the top and exits on first error with set -euo pipefail so a partial run does not leave the account in a half-fixed state.

# Template - replace placeholders with your account specifics
export GOOGLE_CLOUD_REGION=us-central1
export GOOGLE_CLOUD_PROJECT=prod-project
gcloud vertex describe RESOURCE --format=json --filter 'Resources[?Status==`FAILED`].[Id,Reason]' --output table
gcloud vertex modify-... --resource-id RESOURCE_ID --no-dry-run
gcloud vertex describe RESOURCE_ID --query 'Status'

Common pitfalls and what to watch for

The pitfall most teams hit on Vertex AI Model Garden is moving too fast and skipping the read-only validation step. Before any write, list the current state and save it. Google Cloud APIs are eventually consistent for many resource types, so the validation snapshot is your only reliable reference if you need to undo. Save the output of the describe call to S3, not to your laptop.

Second pitfall: confusing IAM permission errors with networking errors. AccessDenied can be IAM (policy missing), networking (VPC endpoint policy blocking the call), or KMS (key policy missing). The error string looks identical for all three. Distinguish by looking at the Cloud Audit Log event's errorCode and the encoded authorization message; do not assume IAM is the culprit just because the message says AccessDenied.

Verify the fix worked

• Reproduce the original symptom path. If it still surfaces in any account or region or IAM role or service account, you have not fixed it.
• Watch for 24 to 48 hours. Cloud Monitoring metrics and Cloud Asset Inventory can mask issues with cached health for 6 to 12 hours, especially Cloud CDN and Cloud DNS.
• Run a smoke test under realistic load. Happy-path tests miss race conditions and IAM session-cache issues.
• Capture the new state in a runbook so the next person on call does not have to rediscover this. Push it to Confluence or your team wiki, not into Slack.
• If the fix involved a permission change, run IAM Access Analyzer one more time to confirm you did not open a separate hole while closing this one.

Safety, rollback, blast radius

• Test in a non-production account if your environment has Resource Manager and Organization Policy or Cloud Resource Manager (organizations, folders, projects). The cost of one sandbox account is cheaper than one rollback meeting.
• Export the existing config before changing it. Most Vertex AI Model Garden resources support describe + export to JSON via CLI - capture that to source control before you start.
• Know your rollback path. Some Vertex AI Model Garden operations are one-way (region migration, account-level feature opt-in, Cloud KMS key deletion past pending window). Confirm reversibility on the Google Cloud doc before you commit.
• Be aware of cross-service impact. IAM role or service account changes ripple to every service trusting that role. Cloud KMS key changes break every workload depending on that key. VPC endpoint changes affect every VPC consumer of that endpoint.
• Maintenance window discipline: if the change touches DNS, certificate rotation, or anything that emits TLS handshakes, line up a window with stakeholder notification, not a heroic mid-day swap.

FAQ

Related fixes

Related guides worth a look while you sort this one out:

• Model Garden private deployment to private endpoint networking
• Multimodal Gemini request fails on video over 1 minute
• Tune Gemini with supervised fine-tuning dataset format errors
• Deploy fine-tuned Gemini model to endpoint IAM and quota path
• Audit usage with Cloud Logging logName aiplatform googleapis com
• Blocked on Vertex AI Model Garden, what causes it and how to fix