Azure Data Explorer: Fix Common Setup & Config Errors
Why This Is Happening
I've seen this exact situation play out dozens of times: an engineering team spins up an Azure Data Explorer cluster for the first time, points their IoT pipeline or log stream at it , and then something breaks. Maybe the cluster shows as "Running" in the portal but queries return nothing. Maybe ingestion silently fails. Maybe KQL errors surface that make zero sense given that the data is clearly there. The frustration is real, and Azure's default error messages rarely tell you which of the five possible root causes you're actually dealing with.
Here's the core thing to understand about Azure Data Explorer: it is a fully managed, high-performance big data analytics platform built specifically for near real-time analysis of massive data volumes. It is not a general-purpose relational database. It does not behave like SQL Server or Cosmos DB, and troubleshooting it requires a different mental model. The platform ingests structured, semi-structured, and unstructured data , logs, metrics, telemetry, time series, and organizes it into tables with strongly-typed schemas inside databases, which live inside clusters. One cluster can hold up to 10,000 databases, and each database can hold up to 10,000 tables.
Most of the problems I see fall into a handful of buckets. First, there are Azure Data Explorer cluster creation and provisioning errors, the cluster either doesn't finish deploying or the networking configuration blocks inbound traffic. Second, there are data ingestion failures in Azure Data Explorer, where events disappear into the pipeline without any obvious failure signal. Third, there are Kusto Query Language (KQL) permission errors and query timeouts that hit users hard the first time they try to run a complex aggregation over a large dataset. Fourth, there are Azure Data Explorer cluster scaling problems, teams that hit the wall because they didn't configure horizontal or vertical scaling ahead of time.
The reason Microsoft's error messages don't help much here is that Azure Data Explorer was built by the team that invented KQL, and the product's internal design assumes you know what a "data shard" (called an extent) is, what "queued ingestion" means versus "streaming ingestion", and why a schema mismatch during mapping can silently drop rows. If you don't have that context yet, you're flying blind. This guide gives you that context and shows you the exact fixes.
I know this can feel like a black box, especially when it's blocking a production pipeline. Browse all Microsoft fix guides →
The Quick Fix, Try This First
If your Azure Data Explorer cluster is running but queries return no data, the most common cause is an ingestion mapping failure. Before you do anything else, run this diagnostic query directly in the Azure Data Explorer web UI at dataexplorer.azure.com:
.show ingestion failures
| where Table == "YourTableName"
| sort by FailedOn desc
| take 20That command pulls the last 20 ingestion failure records for your table. Look at the Details column. The two most frequent errors I see are:
- "Schema mismatch", your source data has a field that doesn't match the column type in the table definition.
- "Mapping reference not found", you're referencing an ingestion mapping by name that doesn't exist or was deleted.
If you see a schema mismatch, go to the Azure Data Explorer web UI, open your database, click Query, and run:
.show table YourTableName schema as jsonCompare that output against the actual shape of your incoming data. A single field coming in as a string when the column expects a datetime will silently drop every row in that batch during queued ingestion, no alert, no retry, just gone.
If you see "Mapping reference not found", recreate the mapping with:
.create table YourTableName ingestion json mapping "YourMappingName"
'[{"column":"Timestamp","path":"$.time","datatype":"datetime"},
{"column":"EventId","path":"$.id","datatype":"string"}]'Adjust the column names, JSON paths, and data types to match your actual schema. After recreating the mapping, re-trigger your ingestion pipeline. Within a couple of minutes, run .show ingestion failures again, if the failure list stays empty and row counts increase on .show table YourTableName details, you're fixed.
getschema on your actual data before wiring up a production Event Hub or Event Grid connection, it saves you from silent row drops at scale.
Before troubleshooting ingestion or queries, confirm the cluster itself is healthy. Go to the Azure portal at portal.azure.com, search for Azure Data Explorer Clusters, and click your cluster name. On the Overview blade, confirm the Status field shows Running, not "Starting", "Stopping", or "Failed".
If the status shows Failed, scroll down to the Activity log blade on the left sidebar and look for any red-flagged provisioning operations in the last 24 hours. Click the failed operation and expand the JSON payload under the Properties tab. The statusMessage field will usually contain the actual error, such as a quota limit on the subscription or a virtual network configuration conflict.
Once the cluster is confirmed running, verify your database exists:
.show databasesRun that in the Azure Data Explorer web UI after connecting to your cluster endpoint (format: https://<clustername>.<region>.kusto.windows.net). If your database is missing from the output, you need to create it. In the Azure portal, click your cluster, then Databases on the left nav, then + Add database. Give it a name and set the hot cache and retention periods. The default hot cache is 31 days, data in hot cache is served from SSD and returns in milliseconds; cold cache data goes to Azure Storage and is slower.
If the cluster endpoint isn't responding at all, you get a connection timeout or a DNS resolution failure, check the Networking blade on your cluster in the portal. If the cluster was deployed into a virtual network, confirm the subnet has outbound connectivity and that no Network Security Group (NSG) rule is blocking port 443 inbound to the cluster nodes. You should see a 200 response when you open the cluster URI in a browser (it redirects to the web UI).
Azure Data Explorer supports two ingestion modes: queued ingestion (the default, batch-based, higher throughput) and streaming ingestion (low latency, under 10 seconds, but lower throughput). Knowing which one you're using matters because they fail differently and have different diagnostic paths.
For queued ingestion failures, the command you already saw works well: .show ingestion failures. But also run this to check the ingestion operations table:
.show operations
| where Operation == "DataIngestPull" or Operation == "TableSetOrAppend"
| where State == "Failed"
| sort by StartedOn desc
| take 10For streaming ingestion issues, first confirm streaming ingestion is enabled on the cluster. In the Azure portal, go to your cluster, click Configurations on the left nav, and toggle Streaming ingestion to On. Then confirm it's enabled at the database level too:
.alter database YourDatabase policy streamingingestion enableIf you're ingesting from an Event Hub and seeing dropped events, check the Event Hub's Metrics blade for Incoming Messages vs. Outgoing Messages, a gap there means Azure Data Explorer's consumer group isn't keeping up or is misconfigured. Verify the consumer group name in your Data connections blade inside the Azure Data Explorer cluster matches exactly what's configured on the Event Hub side. Consumer group name mismatches are one of the top causes of Event Hub ingestion silently stalling.
After resolving the issue, validate data arrived with:
YourTableName
| countRow count should increase within 5 minutes for queued ingestion.
If you're seeing "Principal 'aaduser=...' is not authorized to read database" errors, the fix is straightforward, you need to grant the user or service principal the appropriate role on the database. There are four main roles to know: Admin, User, Viewer, and Monitor.
To grant a user the Viewer role on a database (read-only query access), run this in the web UI as an Admin:
.add database YourDatabase viewers ('aaduser=user@yourtenant.com')
'Granting viewer access for data analyst team'For service principals (used by pipelines, applications, or automated scripts), the syntax is slightly different:
.add database YourDatabase viewers
('aadapp=<app-client-id>;<tenant-id>')
'Service principal for pipeline reads'If the error is query timeout, you see Request is throttled or the query just hangs past 30 seconds, the issue is usually one of three things. First, your query is doing a full table scan without a time filter. Azure Data Explorer stores data partitioned by ingestion time, so always anchor your queries with a time range:
YourTableName
| where ingestion_time() > ago(1h)
| summarize count() by bin(Timestamp, 5m)Second, you may be hitting the default query timeout of 4 minutes for the User scope. You can extend this at the request level for admin-tier queries:
set query_timeout = time(10m);
YourTableName
| summarize ...Third, if multiple users are hammering the cluster simultaneously and you're seeing widespread throttling, that's a capacity issue, jump to the Advanced Troubleshooting section on cluster scaling. If the query returns results without errors, you're good.
One of the most common performance problems I see is teams who spun up a Dev/Test cluster SKU and then tried to run production-scale analytics against it. Azure Data Explorer supports two types of scaling: horizontal scaling (adding more instances/nodes) and vertical scaling (upgrading the compute SKU per node). You can manage both from the portal.
For horizontal scaling, go to your cluster in the portal and click Scale out on the left nav. You can configure a minimum and maximum instance count, and enable Optimized Autoscale, which automatically adjusts node count based on CPU and query load. I strongly recommend enabling Optimized Autoscale for any workload that's not perfectly flat. Set minimum instances to 2 (so you're not at zero headroom) and maximum based on your expected peak load.
For vertical scaling, click Scale up on the left nav. This is where you change the compute SKU, for example, from Standard_D11_v2 to Standard_D14_v2. Note that vertical scaling requires a cluster restart and will cause a brief downtime window, so plan it during low-traffic hours. Horizontal scaling is live with no downtime.
To check current cluster utilization before deciding which direction to scale, run:
.show cluster
| project ClusterName, State, NodeCount, TotalCpuPercent, TotalDiskPercentIf TotalCpuPercent is consistently above 80% during query peaks, you need more compute. If TotalDiskPercent is high, consider reducing the hot cache period on tables where historical data doesn't need sub-second response times. Once autoscale is enabled and the cluster stabilizes, query latency should drop noticeably.
Flying blind is how small problems turn into production incidents. Azure Data Explorer has first-class monitoring support through Azure Monitor metrics and diagnostic logs, and most teams don't enable it until something has already broken. Don't be that team.
To enable diagnostic logs, go to your cluster in the portal, click Diagnostic settings under Monitoring on the left nav, then click + Add diagnostic setting. Give it a name, check the following log categories: SucceededIngestion, FailedIngestion, IngestionBatching, Command, and Query. Send them to a Log Analytics workspace, if you don't have one, create one in the same resource group.
Once diagnostic logs are flowing (allow 5–10 minutes), you can query ingestion history directly from Log Analytics:
ADXIngestionBatching
| where TimeGenerated > ago(1h)
| summarize AvgLatency=avg(BatchTimeSeconds) by bin(TimeGenerated, 5m)
| render timechartFor built-in Azure Monitor metrics, the most useful ones to pin to a dashboard are: CPU, Ingestion latency in seconds, Ingestion result (split by result), Keep alive (cluster health heartbeat), and Query duration. Navigate to your cluster → Metrics → Add metric to pin these. Set up an Alert rule on Ingestion result where result = Failed with a count threshold above zero, this will page you the moment ingestion breaks, instead of you finding out hours later from an empty dashboard.
After this is configured, go to Workbooks under Monitoring on your cluster, Microsoft ships a pre-built Azure Data Explorer monitoring workbook that gives you a one-page view of cluster health, ingestion health, and query performance. If all your metrics show green and no ingestion failures are firing, your monitoring baseline is solid.
Advanced Troubleshooting
Once you've handled the basics, there's a class of harder problems that show up in enterprise deployments and domain-joined environments. Here's what I see most often and how to work through each one.
Azure Active Directory Authentication Failures
If users are authenticated to Azure but still can't connect to the Azure Data Explorer web UI or the Kusto client libraries, check whether Conditional Access policies are blocking the KustoService app registration. In Entra ID (formerly Azure Active Directory), go to Security → Conditional Access → Sign-in logs and filter by Application = "Azure Data Explorer". Failed sign-ins here will show you exactly which CA policy is blocking access. The fix is usually adding the user's group to an exclusion on the offending policy, or adding the Kusto service principal to the allowed apps list.
Follower Database Configuration Problems
Azure Data Explorer supports a "follower database" architecture where a read-only copy of a database in one cluster is attached to a second cluster for query isolation. If your follower database is showing stale data or isn't updating, check the leader cluster's Databases blade for the database in question. The follower attachment status will show either Online, Degraded, or Error. A Degraded state usually means the follower cluster's managed identity doesn't have AllDatabasesViewer or AllDatabasesAdmin permission on the leader. Fix it with:
.add cluster follower databases all viewers
('aadapp=<follower-cluster-managed-identity-client-id>;<tenant-id>')Materialized Views Not Refreshing
Materialized views, precomputed scheduled aggregates, are one of Azure Data Explorer's most powerful features for making dashboards fast. If yours has stopped updating, run:
.show materialized-view YourViewName
| project Name, SourceTable, Query, IsEnabled, LastRunResult, LastRunTimeIf LastRunResult is Failed, the most common cause is that the source table schema changed after the view was created. Drop and recreate the view with the updated column references. If IsEnabled is false, re-enable it with .alter materialized-view YourViewName autoUpdateSchema = true.
Private Endpoint and VNet Injection Issues
Enterprise clusters deployed inside a virtual network with private endpoints need DNS configuration that's easy to get wrong. The cluster's private IP must be resolvable from within the VNet. If your application can reach the cluster by public hostname but not by private endpoint, check whether the private DNS zone privatelink.<region>.kusto.windows.net is linked to your VNet in the Azure portal under Private DNS zones. A missing VNet link is almost always the root cause here.
Some situations genuinely require Microsoft's involvement: cluster provisioning failures that persist after re-deploying from ARM template, cluster stuck in a "Stopping" state for more than 30 minutes, or data loss confirmed via ingestion logs where rows are missing with no corresponding failure record. Before opening a support ticket, export your diagnostic logs, note your cluster resource ID (from the Overview blade → Properties), and document the exact time range of the issue, this cuts support resolution time significantly. You can open a ticket directly at Microsoft Support. For Azure billing issues or quota increases, use the "New support request" flow inside the Azure portal itself.
Prevention & Best Practices
Most Azure Data Explorer incidents are preventable. Here's what the teams that run it smoothly do differently from the ones who are constantly firefighting.
Design your schema before you ingest. Azure Data Explorer uses strongly-typed schemas, changing a column's data type after ingestion requires a table alteration that won't backfill existing data. Before you wire up your first real data source, push a sample payload through the ingestion wizard in the web UI. Let the wizard auto-suggest the schema, review it carefully, and lock it down. Changing string to datetime three weeks after millions of rows are in the table is painful.
Set retention and caching policies per table, not per database. Azure Data Explorer's hot cache (SSD-backed, fast) and cold cache (Azure Storage-backed, slower) can be tuned independently per table. High-frequency query targets, your last 7 days of telemetry, your active dashboards, should have a hot cache period matching your query window. Historical tables that only get queried monthly don't need 31 days of hot cache. Tuning this correctly cuts your compute costs significantly on large clusters.
Always use time filters in production KQL queries. Every query that hits a table without a time filter does a full extent scan. At petabyte scale, this is expensive and slow. Establish a team coding standard that every production query must include | where ingestion_time() > ago(Xh) or a timestamp filter. You can enforce this with .alter database policy querythrottling to set default timeouts that catch unbounded queries early.
Use update policies for real-time transformation instead of pre-processing pipelines. If you're running a separate service to clean or reshape your raw telemetry before it lands in Azure Data Explorer, consider replacing it with an update policy. Update policies run a KQL function every time data lands in a source table and write the transformed output to a target table, on the server side, at ingestion time, with no extra infrastructure.
- Enable Optimized Autoscale from day one, don't wait until a traffic spike forces you to manually scale at 2 AM
- Create a dedicated consumer group for each Azure Data Explorer data connection on your Event Hub, sharing consumer groups causes event loss
- Tag every managed identity and service principal with a description comment in your
.addcommands so you know what they're for six months later - Schedule monthly reviews of
.show ingestion failures | summarize count() by Table, FailureKindto catch low-level drip failures before they become data quality incidents
Frequently Asked Questions
When should I use Azure Data Explorer instead of Azure SQL or Cosmos DB?
Azure Data Explorer is the right choice when you need interactive analysis over high-velocity, high-volume raw data, think IoT telemetry, application logs, security event streams, or time-series metrics where you're ingesting millions of events per second and need query results in seconds or milliseconds. Azure SQL is better for transactional workloads with row-level updates and deletes. Cosmos DB is better for globally distributed document or key-value access patterns. If your main use case involves aggregation, anomaly detection, time series forecasting, or exploring data that isn't fully structured into a star schema, Azure Data Explorer will outperform both alternatives dramatically at scale.
How much data can Azure Data Explorer actually handle, what are the real limits?
The platform can ingest terabytes of data in minutes via queued ingestion and sustain millions of events per second. Query performance scales to petabytes with results returning within milliseconds to seconds depending on data volume and query complexity. A single cluster can manage up to 10,000 databases, and each database can hold up to 10,000 tables. In practice, the limits you'll hit first are subscription-level compute quotas (request a quota increase through the Azure portal if needed) and the hot cache size on your chosen SKU. The platform's linear scale architecture means adding more nodes gives you proportional throughput, there's no hidden ceiling that bites you at a specific data size.
Is KQL (Kusto Query Language) hard to learn if I already know SQL?
KQL has a genuinely different syntax from SQL, it's pipe-based, meaning you chain operators left to right rather than writing nested subqueries. Most SQL developers pick up the basics within a day or two. Microsoft even provides a SQL-to-KQL cheat sheet in the official documentation specifically for this transition. The language is open source and was designed to be simple to understand and learn while being highly productive for analytics. Start with | where, | summarize, | extend, and | render, those four operators cover the majority of real-world analytics queries. Azure Data Explorer also supports T-SQL for teams that need it during migration, so you're not forced to rewrite everything on day one.
Can multiple users or applications query Azure Data Explorer at the same time without slowing each other down?
Yes, concurrent query access is one of the things Azure Data Explorer handles well. The platform is explicitly designed for high query concurrency, with multiple users and processes querying simultaneously without the kind of lock contention you'd see in a traditional RDBMS. That said, unbounded queries (no time filter, full table scans) from multiple users at once will still saturate cluster CPU if you haven't sized the cluster appropriately. The practical fix is combining proper KQL query discipline (always use time ranges) with Optimized Autoscale configured on the cluster. You can also use follower databases to physically separate read-heavy dashboard traffic from write-heavy ingestion workloads on separate clusters.
Why is my Azure Data Explorer ingestion taking hours instead of minutes?
Queued ingestion is batch-based and typically completes within 5 minutes under normal conditions, but a few things can delay it significantly. The most common culprit is the ingestion batching policy, by default, Azure Data Explorer batches incoming data until it hits 1 GB, 1000 files, or a 5-minute time window (whichever comes first). If you're ingesting small files infrequently, you may wait close to 5 minutes per batch. You can tune this with .alter table YourTable policy ingestionbatching @'{"MaximumBatchingTimeSpan":"00:01:00"}' to reduce the batching window. If ingestion is taking hours, check .show ingestion failures, it's likely failing and retrying, not just slow.
Should I start with a free Azure Data Explorer cluster or go straight to a production SKU?
Microsoft offers a genuinely free Azure Data Explorer cluster (no Azure subscription required, up to 100 GB storage, available at dataexplorer.azure.com/freecluster) that's great for learning KQL, testing ingestion pipelines, and evaluating whether the platform fits your use case. I'd also recommend trying the Kusto Detective Agency, an interactive puzzle-based tutorial Microsoft built specifically to teach KQL in a hands-on way. For anything heading toward production with real data volumes, move to a paid cluster SKU early: the free cluster has throttled compute and no SLA. Starting on Standard_D11_v2 with 2 nodes and Optimized Autoscale enabled is a sensible baseline for most initial production deployments.