DynamoDB ACID transactions and the 100-item-per-transaction limit
| Service | Amazon DynamoDB |
|---|---|
| Cloud | Amazon Web Services (AWS) |
| Guide type | Procedure |
| Skill level | Intermediate to advanced |
| Time | 15 - 60 minutes depending on account size |
DynamoDB ACID transactions and the 100-item-per-transaction limit on Amazon DynamoDB sits in the most-reported issues list across r/aws, AWS re:Post, and StackOverflow. The recovery path is mostly known, the AWS docs just bury it under three layers of conceptual material.
What dynamodb acid transactions and the 100-item-per-transaction limit actually involves on Amazon DynamoDB
This task on Amazon DynamoDB is one of the more searched operational topics on AWS in the last 12 months. The procedure below is the path that works in a current AWS account with default IAM and standard VPC config.
The rest of this page is the structured fix path. Start with diagnose, then remediation, then the automation options so you do not have to do this by hand the next time it surfaces. Verify and safety sections at the end are the discipline that keeps the fix from regressing in production.
What you'll see
Run aws sts get-caller-identity first. About one in five 'why does this not work' tickets are actually 'I am in the wrong account' or 'my session expired and the SDK is using stale creds'. The 5-second sanity check costs nothing and saves real time when the answer is that simple.
Look at the CloudTrail event for the failed call, even if you are not enrolled in CloudTrail Lake. The basic 90-day event history works for most diagnostic purposes and lives in the console under CloudTrail > Event history. Filter by event name (the API action) and time range; the event JSON shows the exact user identity, source IP, request parameters, and error code.
Diff against last known good. The last config change you made is the cause about three quarters of the time, even when the change should not have mattered. Use AWS Config history (or your Terraform / CloudFormation drift report) to see the actual delta between the resource state when it worked and when it broke. The change you remember is often not the only change that happened.
Solution-focused remediation path
If quotas are suspect, the Service Quotas console shows current usage and the active limit side by side. Request increases through Service Quotas, not through Support tickets - quota dashboard requests usually approve faster (often within minutes for soft limits) and they are auditable in CloudTrail. Set up Service Quotas + CloudWatch alarms at 80 percent usage so you get notified before you hit the wall.
If you cannot reproduce the failure consistently, the cause is probably a race condition or a session-cache issue. Run the call with --profile set to a fresh STS session, in a different region you control, with a single concurrent request. If it works there but fails in your normal setup, the difference is the bug.
When the failure happens in production but not in dev, do not just compare the IAM policy. Compare the SCP / RCP at the OU level, the permission boundary on the role, and the resource-based policy on the target. One of those is almost always different between accounts. AWS Config conformance packs make this comparison routine.
Automate this fix so you do not do it twice
Wire the fix into EventBridge for self-healing
If the failure mode is recurring, automate the remediation instead of the diagnosis. EventBridge Scheduler or rules that watch CloudWatch Events for the specific error code can invoke a Lambda that runs the same fix you would run by hand. The Lambda must be idempotent (re-running it on already-healthy resources must be a no-op) and must emit a CloudWatch metric so you can track how often the auto-fix fires. A spike in auto-fix invocations is itself a signal worth alerting on.
# EventBridge rule pattern (JSON)
{ "source": ["aws.dynamodb"], "detail-type": ["AWS API Call via CloudTrail"], "detail": { "errorCode": ["AccessDenied", "ThrottlingException"] }
}Codify the fix in Terraform or CloudFormation
When you reach for the console to fix the same issue twice, the third occurrence should be solved in IaC, not in the console. Terraform's terraform import and CloudFormation's resource importer let you adopt the existing resource into state without recreating it. Lock the corrected attribute behind a variable so the next operator does not have to rediscover the value. Add a moved {} block or CloudFormation resource refactor to keep the diff clean.
Automate the fix with Python and boto3
For anything you do more than twice, write a small Python script. The boto3 pattern below uses paginators (so it does not blow up on accounts with thousands of resources), explicit region binding, and a dry-run flag that defaults to True. Keep the script under 100 lines; if it grows beyond that, you are building a tool and should put it behind a Lambda with proper logging.
import boto3, sys
DRY_RUN = '--apply' not in sys.argv
client = boto3.client('dynamodb', region_name='us-east-1')
paginator = client.get_paginator('describe_...')
for page in paginator.paginate(): for item in page.get('Items', []): if item.get('Status') == 'FAILED': if DRY_RUN: print(f'[dry-run] would fix {item["Id"]}') else: client.modify_...(ResourceId=item['Id']) print(f'fixed {item["Id"]}')
Common traps
A subtle pitfall on Amazon DynamoDB is that the AWS Console and the SDK can disagree about resource state during a configuration change. Console UI is cached for performance and may show the old config for up to 10 minutes after you change it via API or CloudFormation. Always confirm with describe-* CLI calls during a change window, not with screenshots from the Console.
The other pitfall: assuming that an automated remediation is correct because it succeeded. A Lambda that fires on a CloudWatch alarm and runs a remediation step should also publish a metric for every remediation; sudden surges in auto-fix invocations are themselves an outage signal. Otherwise you can hide a slow-burn regression behind a quiet remediation loop for weeks.
The repair
- Reproduce the original symptom path. If it still surfaces in any account or region or IAM role, you have not fixed it.
- Watch for 24 to 48 hours. AWS metrics and policy systems can mask issues with cached health for 6 to 12 hours, especially CloudFront and Route 53.
- Run a smoke test under realistic load. Happy-path tests miss race conditions and IAM session-cache issues.
- Capture the new state in a runbook so the next person on call does not have to rediscover this. Push it to Confluence or your team wiki, not into Slack.
- If the fix involved a permission change, run IAM Access Analyzer one more time to confirm you did not open a separate hole while closing this one.
Safety, rollback, blast radius
- Test in a non-production account if your environment has Control Tower or AWS Organizations. The cost of one sandbox account is cheaper than one rollback meeting.
- Export the existing config before changing it. Most Amazon DynamoDB resources support describe + export to JSON via CLI - capture that to source control before you start.
- Know your rollback path. Some Amazon DynamoDB operations are one-way (region migration, account-level feature opt-in, KMS key deletion past pending window). Confirm reversibility on the AWS doc before you commit.
- Be aware of cross-service impact. IAM role changes ripple to every service trusting that role. KMS key changes break every workload depending on that key. VPC endpoint changes affect every VPC consumer of that endpoint.
- Maintenance window discipline: if the change touches DNS, certificate rotation, or anything that emits TLS handshakes, line up a window with stakeholder notification, not a heroic mid-day swap.
FAQ
aws dynamodb describe-... first, then commit it before you change anything. A few operations are one-way (KMS key deletion past the pending window, region migration, account closure). Check the AWS doc for the specific API before you commit.aws CLI or SDK calls - those almost always still work.References
- docs.aws.amazon.com - official documentation for Amazon DynamoDB
- AWS re:Post (formerly forums) - community Q&A with AWS-staff-verified answers
- AWS Health Dashboard at health.aws.amazon.com
- AWS Service Quotas console and AWS Well-Architected Tool
Related fixes
Related guides worth a look while you sort this one out:
- Aurora zero-ETL integration with Redshift and DynamoDB
- Bedrock Converse API token count exceeded model limit
- CloudFront cache invalidation cost 0.005 per path after first 1000 per month
- CloudFront KeyValueStore for CloudFront Functions when to pick over DynamoDB
- Comprehend BatchDetectEntities ThrottlingException TPS limit
- Comprehend syntax detection token limit exceeded