How to Fix Azure Container Storage Issues on AKS

Why Azure Container Storage Issues Happen

Here's a scenario I see constantly: you've just set up Azure Container Storage on your AKS cluster, you apply your workload YAML, and the pod just sits there , Pending, no movement, no helpful error message. Or maybe your persistent volume claim (PVC) never binds. Or the storage class simply doesn't show up in kubectl get storageclass. You go looking for answers and hit walls of vague documentation.

I've personally worked through Azure Container Storage setup problems on dozens of AKS clusters, and the root causes almost always fall into one of four buckets:

• Driver pods not running. The acstor CSI driver components in the kube-system namespace failed to start , maybe because the HelmRelease didn't apply cleanly, maybe because the node pool VM SKU doesn't support the storage type you selected.
• Wrong annotation on PVCs. This is one of the most common silent failures. Azure Container Storage version 2.x.x changed the required annotation for persistent volume claims that use local NVMe. If you're copying YAML from older tutorials you'll be using the outdated acstor.azure.com/accept-ephemeral-storage: "true" annotation, and it simply won't work. The correct one is localdisk.csi.acstor.io/accept-ephemeral-storage: "true". That small difference causes PVCs to hang in a Pending state indefinitely.
• VM SKU mismatch. Local NVMe disks are only available on specific Azure VM families, think storage-optimized SKUs like the Lsv3 series. If your node pool is running on a general-purpose VM like Standard_D4ds_v5, local NVMe simply doesn't exist on that hardware and the driver will fail to find any backing devices.
• Volume binding mode confusion. The local storage class uses WaitForFirstConsumer binding mode, which means the PVC won't bind until a pod is actually scheduled onto a node. This is intentional, the volume must be co-located with the workload, but if you're watching PVC status expecting immediate binding, you'll think something is broken when it's actually working correctly.

Microsoft's error messages in this space are notoriously unhelpful. A PVC stuck in Pending doesn't tell you whether the driver isn't running, whether the node lacks NVMe hardware, or whether you used a deprecated annotation. That's exactly what this guide is for. Browse all Microsoft fix guides →

The good news: every one of these problems is diagnosable with a handful of kubectl commands, and in most cases you'll have a working setup within 20 minutes. Let's get into it.

The Quick Fix, Try This First

Before you dive into multi-step debugging, run this one command. It tells you immediately whether the acstor driver components are actually up and running in your cluster:

kubectl get pod -n kube-system | grep acstor

You're looking for output that shows the CSI driver pods, the cluster manager, the Geneva telemetry pods, node agents, and the OpenTelemetry collector, all in a Running state. Specifically, a healthy Azure Container Storage installation shows entries like acstor-azuresan-csi-driver-* with 7/7 Running, the acstor-cluster-manager-* pods showing 2/2 Running, and the node agents each showing 1/1 Running.

If you see zero output, the installer never ran or the HelmRelease failed. If you see pods in CrashLoopBackOff or Error, you have a component-level failure that needs targeted attention. If everything shows Running but workloads still fail, the problem is almost certainly in your PVC spec or storage class configuration, and we'll fix that in the steps below.

Next, confirm your storage classes are actually present:

kubectl get storageclass

For a cluster with both local NVMe and Azure Elastic SAN configured, you should see a local storage class using the localdisk.csi.acstor.io provisioner and an azuresan storage class using san.csi.azure.com. If either of those is missing, the corresponding storage type was either not installed or the installation failed partway through.

Advanced Troubleshooting for Azure Container Storage

Restricting Local CSI Drivers to Specific Node Pools

In a multi-node-pool AKS cluster, you typically don't want the local NVMe CSI driver running on every node, only on the pools that actually have NVMe hardware. Deploying it everywhere wastes resources and can cause confusing scheduling behavior. Azure Container Storage handles this through node affinity on the StorageClass itself, using the storageoperator.acstor.io/nodeAffinity annotation.

Here's how to create a storage class that only targets specific node pools, in this example, pools named mygpu and mygpu2:

cat <<EOF | kubectl apply -f -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-nvme
  annotations:
    storageoperator.acstor.io/nodeAffinity: |
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.azure.com/agentpool
            operator: In
            values: [mygpu,mygpu2]
provisioner: localdisk.csi.acstor.io
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
EOF

One thing that bites people constantly here: the match expressions for node pool names are case-sensitive. The label value mygpu is different from MyGPU or MYGPU. Before writing this annotation, always verify the exact label values on your actual nodes:

kubectl get nodes -o custom-columns="NAME:.metadata.name,AGENTPOOL:.metadata.labels.kubernetes\.azure\.com/agentpool"

Copy the exact strings from that output into your node affinity configuration. A single character case difference will cause the affinity to silently not match any nodes, and your PVCs will hang in Pending forever with no obvious error message explaining why.

Checking Node Ephemeral Disk Capacity

Before provisioning volumes, especially on tightly packed clusters, check how much ephemeral disk capacity each node actually has available. Ephemeral volumes are allocated on a single node, your requested volume size must fit within that node's available NVMe capacity. If you request 500Gi on a node that only has 200Gi free, the volume will fail to provision. Use kubectl describe node <node-name> and look at the allocatable storage resources to understand available headroom before sizing your PVCs.

Interpreting Event Viewer Equivalent, Kubernetes Events

In the Windows world, you'd reach for Event Viewer when something mysterious happens. On AKS with Azure Container Storage, your equivalent is the Kubernetes events stream. Watch it during a problematic deployment:

kubectl get events -n kube-system --watch

Warning events with reasons like FailedMount, FailedAttach, or ProvisioningFailed tell you exactly where the chain is breaking. A FailedMount on a pod that references a PVC usually means the PVC bound successfully but the volume couldn't be mounted onto the node, often a node-level NVMe driver issue. A ProvisioningFailed on the PVC itself means the CSI driver couldn't create the backing volume at all.

Prevention & Best Practices for Azure Container Storage

Getting Azure Container Storage working is one thing. Keeping it working, especially in a production AKS cluster where node pools scale, workloads get updated, and team members deploy new manifests, takes some discipline up front. Here's what actually prevents the problems we've covered.

Always verify the driver after cluster changes. Whenever you add a node pool, upgrade the AKS cluster version, or modify the Azure Container Storage extension, immediately re-run kubectl get pod -n kube-system | grep acstor to confirm all driver components are still healthy. Node pool additions don't automatically extend the CSI driver to new nodes unless the node pool uses a compatible VM SKU and the extension is reconfigured to include it.

Understand what "ephemeral" really means before designing your architecture. Local NVMe through Azure Container Storage is ephemeral storage, the data lives on the physical VM hosting your AKS node. Stop or deallocate that VM, and your data is gone. This is intentional and documented, but I've seen teams design stateful workloads on this storage without accounting for it. Use local NVMe for workloads that can reconstruct data, distributed database replicas, cache layers, intermediate ML training checkpoints. Don't use it as your primary persistence layer for anything you can't afford to lose.

Choose VM SKUs deliberately. Since Azure Container Storage automatically stripes data across all NVMe drives on a VM, the size of your VM directly determines your maximum I/O performance. Plan your VM SKU selection before deploying workloads, not after you hit a performance wall. The Lsv3 series scales linearly, moving from L8s_v3 to L16s_v3 roughly doubles your throughput because you go from one NVMe drive to two.

Pin your annotation version. When you find a working YAML manifest, document which version of Azure Container Storage it targets. The annotation change from acstor.azure.com to localdisk.csi.acstor.io between version 1.x.x and 2.x.x is a breaking change that manifests silently. Keep a comment in your manifests noting the minimum required version.

Frequently Asked Questions