Azure Data Factory Not Working: Fix Pipelines, Errors & Setup
Why This Is Happening
I've seen this exact situation play out dozens of times: you set up an Azure Data Factory pipeline, everything looks fine in the authoring canvas, and then you hit that green Trigger Now button , and the pipeline fails with a cryptic error that doesn't tell you much of anything useful. Or worse, it just hangs in a "Queued" state and never moves. I know this is frustrating, especially when it's blocking a data migration deadline or a daily ETL job that the business depends on.
Azure Data Factory is Microsoft's cloud-scale ETL and data integration service. It's designed to handle complex hybrid extract-transform-load (ETL), extract-load-transform (ELT), and data integration scenarios , think pulling petabytes of raw data from on-premises SQL servers, combining it with cloud storage data in Azure Blob, transforming it through mapping data flows, and landing it in Azure Synapse Analytics. That's a lot of moving parts, which means a lot of places things can go wrong.
The root causes I see most often fall into a handful of buckets:
- Integration Runtime misconfiguration, the most common culprit. Either the self-hosted IR (SHIR) is offline, the Azure IR region is mismatched, or the SHIR can't reach the target data source due to firewall rules.
- Linked Service connection failures, credentials expired, Key Vault access revoked, or the connection string points to a resource that's been moved or renamed.
- Copy Activity failures, schema drift, column mapping errors, or data type mismatches between source and sink.
- Trigger misconfiguration, tumbling window triggers that overlap incorrectly, schedule triggers hitting the wrong time zone, or storage event triggers pointing at a container that no longer exists.
- Role-Based Access Control (RBAC) gaps, ADF's managed identity wasn't granted the right permissions on the target resource, or Entra ID (formerly Azure Active Directory) tokens expired mid-pipeline run.
The maddening thing about Azure Data Factory error messages is that they're often one level of abstraction above where the real problem lives. You'll see ErrorCode=UserErrorInvalidColumnMappingColumnNotFound without any hint about which column in which dataset is the offender. Or you'll get a generic Failure happened on 'Source' side message that could mean six different things. Microsoft's monitoring pane gives you activity run details, but you often need to dig two or three levels deep to find the actual stack trace.
This guide walks through the most common Azure Data Factory issues I've helped teams fix, from pipeline failures and Integration Runtime problems to Copy Activity errors and data flow debugging, with exact steps that match what you'll see in the Azure portal today. Browse all Microsoft fix guides →
The Quick Fix, Try This First
Before diving into deep troubleshooting, do this. It resolves about 40% of Azure Data Factory pipeline failures and takes under five minutes.
Step 1: In the Azure portal, navigate to your Data Factory resource. Click Launch Studio to open the ADF authoring experience.
Step 2: Click the Monitor icon (the bar chart icon on the left sidebar). Under Pipeline runs, find your failed run and click the pipeline name to expand it.
Step 3: Click the failed activity (it will show a red X icon). In the right panel, click Details, the small glasses icon. This opens the full activity run output JSON. Look for the errorCode and message fields. Copy that error code.
Step 4: Now go to Manage (the toolbox icon) → Integration runtimes. Check the status column. If your Integration Runtime shows Unavailable or a red status, that's your problem right there, stop here and jump to Step 2 in the detailed section below.
Step 5: If the IR looks healthy, go to Manage → Linked services. Find the linked service your failed pipeline uses and click Test connection. If it fails, you've found your issue, the connection credentials or network path is broken, not the pipeline itself.
Most of the time, one of these two checks, Integration Runtime status or Linked Service test, will immediately surface the real problem. Everything else in this guide builds on fixing what those checks uncover.
Generic pipeline failure messages in Azure Data Factory are almost never the full story. The real diagnostic information is buried in the activity run output JSON, and getting to it properly is the first skill you need.
In ADF Studio, go to Monitor → Pipeline runs. Select the time range that covers your failed run using the date picker at the top. Click the pipeline name, not the status icon, the actual name, to drill into activity-level runs.
Find the failed activity (red X) and click the Output icon (square with arrow). This gives you the raw JSON output. The fields you care about most:
{
"errorCode": "2200",
"message": "ErrorCode=UserErrorInvalidConnectionString",
"failureType": "UserError",
"target": "Copy_from_SQL_to_Blob"
}The failureType field is particularly useful. A value of UserError means the problem is in your configuration, credentials, schema mismatch, missing permissions. A value of SystemError means ADF's infrastructure had a transient issue and a simple re-run might succeed. Don't spend an hour debugging configuration if it's a SystemError, just rerun the pipeline first.
For Copy Activities specifically, click the Details icon (glasses) rather than Output. This gives you the data read/written stats, throughput numbers (in MBps), and the specific phase where the failure occurred, whether that's at the Source side, the Sink side, or during the transfer itself. That distinction saves significant debugging time.
If you see the failure is on the Source side with error code 2200 or 2108, you're almost certainly dealing with a connectivity or authentication issue on the source data store. Move to Step 2.
The Self-Hosted Integration Runtime (SHIR) is the bridge between Azure Data Factory and your on-premises or private network data sources. When it goes down, every pipeline that depends on it fails. I've seen this happen after Windows Updates restart the host machine without restarting the SHIR service, after network policy changes, and after credential rotations that weren't propagated to the IR configuration.
First, check the status in ADF Studio: Manage → Integration runtimes. If the SHIR shows Unavailable, go to the Windows machine hosting the SHIR and check the service status:
# Run in PowerShell as Administrator on the SHIR host machine
Get-Service -Name "DIAHostService" | Select-Object Name, Status, StartTypeIf the service is stopped, start it:
Start-Service -Name "DIAHostService"If the service is running but the portal still shows Unavailable, check the SHIR logs at C:\Users\[username]\AppData\Local\Microsoft\Integration Runtime\[version]\Shared\, look for the most recent log file. Connectivity errors to Azure Service Bus endpoints (*.servicebus.windows.net on port 443) are a very common cause. Your network team may have blocked those endpoints.
You can also open the Microsoft Integration Runtime Configuration Manager from the Start menu on the host machine. Click the Diagnostics tab and run the connectivity test. It will explicitly tell you which endpoints are unreachable.
After fixing the service or network issue, wait about 60 seconds and refresh the Integration Runtimes page in ADF Studio. A healthy SHIR shows a green dot and a Running status. Once you see that, rerun your pipeline.
Linked Services in Azure Data Factory are your connection definitions, they hold the connection string, authentication method, and Integration Runtime assignment for each data source. They break more often than people expect, usually because of credential expiration, Key Vault access policy changes, or because someone renamed or moved the underlying resource.
In ADF Studio, go to Manage → Linked services. Click the linked service that your failed pipeline uses, then click Edit (pencil icon). Scroll to the bottom and click Test connection.
If the test fails with a message about authentication, and you're using a Service Principal or Managed Identity for authentication, verify the role assignment in the Azure portal. Go to the target resource (say, your Azure SQL Database or Storage Account), click Access control (IAM), then Role assignments. Confirm that ADF's managed identity (it will be named the same as your Data Factory) has an appropriate role, at minimum, Storage Blob Data Contributor for Blob Storage, or db_datareader/db_datawriter for SQL.
If you're using Azure Key Vault to store connection strings or secrets, a very common issue is that the Key Vault access policy no longer includes ADF's managed identity. In the Key Vault resource, go to Access policies and confirm ADF has at least Get and List permissions on Secrets.
After correcting the permissions, go back to ADF Studio and click Test connection again. A successful test shows a green checkmark with Connection successful. Publish the updated Linked Service by clicking Publish all in the top toolbar, unpublished changes won't be picked up by pipeline runs.
Copy Activity errors related to schema mapping are extremely common when your source data changes, a new column gets added, a data type gets altered, or a column gets renamed. ADF's error codes for these are specific enough to work with once you know what to look for.
The error code UserErrorInvalidColumnMappingColumnNotFound means a column in your explicit column mapping no longer exists in the source or sink dataset. The error code UserErrorDataTypeNotSupported means ADF can't auto-convert between the source and sink data types for a specific column.
To fix schema mapping issues, open your Copy Activity in the pipeline canvas (double-click to edit). Click the Mapping tab. If you see any rows with a warning icon or a column name in red, those are the broken mappings. You have two options:
Option A, Auto-map (quickest for simple fixes): Delete all existing mappings and click Import schemas. ADF will re-detect the source and sink schemas and auto-map by column name. This works if the column names match between source and sink.
Option B, Manual correction: Edit the specific broken mapping. Click the source column dropdown and re-select the correct column name from the refreshed schema. For data type mismatches, you may need to add a type conversion, click the + next to the mapping row and select Add conversion.
After fixing mappings, use the Data preview button in the Source dataset settings to confirm you can actually read rows from the source before triggering another full pipeline run. Catching schema issues at preview time is far faster than waiting for a pipeline run to fail.
Triggers in Azure Data Factory are what make pipelines run automatically, on a schedule, when a file lands in storage, or when a custom event fires. Trigger problems are sneaky because the pipeline itself might be perfectly healthy; the issue is purely in how and when it gets kicked off.
In ADF Studio, go to Monitor → Trigger runs (in the left sub-menu under Monitor). If you see failed trigger runs, click the trigger name to see the error. A common one for Storage Event Triggers is TriggerNotSubscribedToStorageAccount, this means the event subscription between Azure Event Grid and your storage account was broken, often because the storage account was recreated or the subscription was manually deleted.
To fix a broken Storage Event Trigger: go to Manage → Triggers, click your trigger, hit Edit, and then click Continue through the wizard. On the final step, click Finish, this re-creates the Event Grid subscription. Then deactivate and reactivate the trigger using the toggle switch.
For Schedule Triggers that are running at the wrong time: the most common culprit is time zone misconfiguration. When you edit a Schedule Trigger, the Start time and Recurrence settings have a Time zone dropdown. If this is set to UTC but your team set the recurrence time thinking in EST or IST, your pipeline is running at the wrong wall-clock time. Fix the time zone to match the intended local time zone and save.
For Tumbling Window Triggers that show failures or concurrency errors, check the Max concurrency setting in the trigger definition. If downstream pipeline runs take longer than the window interval, ADF queues up backfill runs. Increasing max concurrency or reducing window size usually clears this.
After fixing any trigger, always click Publish all, trigger changes are not live until published.
Advanced Troubleshooting
If the steps above haven't resolved your Azure Data Factory issue, you're likely dealing with something at the network, identity, or enterprise configuration layer. These require a bit more digging but are very fixable.
Mapping Data Flow Debug Failures
Mapping Data Flows in Azure Data Factory use a Spark-based execution cluster, and they have their own failure modes. When you enable Data flow debug mode in ADF Studio, it spins up a debug cluster (typically takes 5–7 minutes). If debug mode fails to start, check whether your ADF instance has outbound network access to the Spark cluster's subnet. In VNet-injected scenarios, verify that the NSG (Network Security Group) rules on the Integration Runtime subnet allow outbound traffic on ports 443 and 8080.
For data flow transformations that fail with DF-SYS-01 or DF-EXECUTOR-SOURCEINVALIDPAYLOAD, run the data flow in debug mode with a sample dataset first. Use the Data preview button on each transformation step to identify exactly which transformation is producing unexpected nulls or type errors. The Spark execution log is available under Monitor → Data flow debug.
RBAC and Entra ID Token Issues
Enterprise scenarios often hit RBAC issues that don't appear in test environments. If pipelines work when triggered manually but fail when triggered by a schedule or automation, check whether the managed identity's token is expiring. Azure Data Factory managed identity tokens are valid for one hour; if a pipeline run spans longer than that and attempts a new resource access call, the token may have expired mid-run.
Use Azure Monitor to check for Entra ID sign-in logs related to your ADF managed identity. In the Azure portal, go to Entra ID → Monitoring → Sign-in logs, filter by the Application name (your ADF instance name), and look for failed sign-in events with error code 700016 (app not found in directory) or AADSTS65001 (consent required).
VNet Integration and Private Endpoint Issues
If your ADF is configured with a Managed Virtual Network and you're using private endpoints to connect to data sources, connectivity failures often show as UserErrorForbidden or timeouts with no error code. In the Azure portal, navigate to your Data Factory → Managed private endpoints under the Manage tab in ADF Studio. Verify that each private endpoint shows Approved status, a pending or rejected approval from the target resource owner will silently block all connections. Contact the resource owner to approve the private endpoint connection in their resource's Private endpoint connections blade.
CI/CD Pipeline Failures After ARM Template Deployment
Teams using ADF with Azure DevOps or GitHub Actions for CI/CD sometimes see pipelines fail after a deployment because ARM templates don't carry over trigger states. Triggers are disabled by default after an ARM template deployment. Use a post-deployment script to re-enable them:
# PowerShell, re-enable all triggers after ADF ARM deployment
$triggers = Get-AzDataFactoryV2Trigger -ResourceGroupName "myRG" -DataFactoryName "myADF"
foreach ($trigger in $triggers) {
Start-AzDataFactoryV2Trigger -ResourceGroupName "myRG" `
-DataFactoryName "myADF" `
-Name $trigger.Name -Force
}SystemError failures (not UserError) that persist across multiple retries, if your SHIR registers successfully but shows intermittent connectivity drops that you can't correlate with any local event, or if your ADF instance shows pipeline run history gaps (runs that never appear in Monitor), these are platform-side issues that need a support ticket. Open a case through Microsoft Support and include your correlation IDs from the activity run output JSON, those IDs allow Microsoft engineers to pull the server-side logs directly.
Prevention & Best Practices
Once you've fixed the immediate problem, here's how to stop it from coming back. I've watched teams go from weekly pipeline incidents to months of clean runs by putting these practices in place.
Set up ADF alerts proactively. In the Azure portal, go to your Data Factory resource → Monitoring → Alerts → New alert rule. Create alerts for the Failed pipeline runs metric with a threshold of greater than 0. Route these to an email action group or a Teams/Slack webhook. You want to know about failures before your data consumers do.
Use parameterized Linked Services and Datasets. Hard-coded connection strings in Linked Services are brittle. When you parameterize them, pulling values from Azure Key Vault or pipeline parameters, you get a single place to update credentials when they rotate. ADF's Linked Service editor has a Parameters tab specifically for this pattern. It's a bit more setup upfront, but it eliminates the most common cause of Linked Service failures.
Pin your Integration Runtime to a specific Azure region. The default "Auto-resolve" Integration Runtime picks the nearest Azure region dynamically, which occasionally causes unexpected routing when Azure region capacity shifts. For production pipelines that connect to specific regional resources, create a dedicated Azure IR pinned to the same region as your data sources. This is a one-minute change in Manage → Integration runtimes → New.
Monitor Copy Activity throughput trends. ADF's monitoring pane shows throughput (MBps) for every Copy Activity run. Baseline your normal throughput numbers after initial setup. A sudden drop in throughput, even if the activity doesn't fail, often signals an emerging network or source-side issue before it becomes an outright failure. The official ADF documentation specifically calls out Copy Activity performance troubleshooting as a distinct capability, and the throughput metrics are your early warning system.
Version control your ADF artifacts from day one. ADF Studio has native Git integration, connect your factory to an Azure DevOps repo or GitHub repo under Manage → Git configuration. With Git integration enabled, every pipeline, dataset, and Linked Service change is tracked. You can always roll back to a working state without recreating anything from scratch, and you get a clear audit trail of what changed before a pipeline started failing.
- Enable ADF alert rules for failed pipeline runs with email notification, takes 3 minutes to set up and saves hours of reactive firefighting
- Store all credentials in Azure Key Vault and reference them via Linked Service parameters, never hardcode connection strings
- Set retry policies on Copy Activities (2–3 retries with 30-second intervals) to automatically recover from transient network hiccups
- Connect ADF Studio to a Git repo before making any production changes, rollback is only possible if you have version history
Frequently Asked Questions
What exactly is Azure Data Factory and what does it do?
Azure Data Factory is Microsoft's cloud-based managed ETL (extract-transform-load) and data integration service. You use it to build data pipelines that pull raw data from different sources, SQL databases, cloud storage, REST APIs, on-premises systems, transform that data, and load it into destinations like Azure Synapse Analytics, Azure Data Lake Storage, or Cosmos DB. It's built for large-scale, complex data movement scenarios where you need scheduling, monitoring, and transformation logic all in one place, without managing the underlying infrastructure yourself.
What data sources can I connect to with Azure Data Factory?
ADF supports a very wide range of connectors, Azure services like Blob Storage, SQL Database, Synapse, Cosmos DB, and Data Lake Storage; on-premises databases like SQL Server, Oracle, SAP, and MySQL via the Self-Hosted Integration Runtime; cloud platforms like AWS S3 and Google BigQuery; REST and OData endpoints; file formats like CSV, Parquet, JSON, Avro, and Excel; and SaaS applications like Salesforce, ServiceNow, and SharePoint. The full connector list is maintained in the official ADF documentation and grows with each product update. If you're connecting to something on your internal network, you'll need to install the Self-Hosted Integration Runtime on a Windows machine that has network access to that resource.
What is a virtual network data gateway and when do I need one?
A virtual network data gateway is a Microsoft-managed gateway service that lets your Azure Data Factory (and other Azure services like Power BI) reach data sources inside an Azure Virtual Network without requiring you to set up and maintain your own Self-Hosted Integration Runtime on a VM. You'd use it when your data sources live in a VNet-isolated environment, like a private Azure SQL Database with a private endpoint, or resources behind a VNet firewall, but you don't want the overhead of managing a dedicated SHIR machine. It's configured in the Azure portal under the Data Gateway resource type, and you reference it from ADF Linked Services the same way you'd reference a SHIR. Keep in mind it has regional availability restrictions, so check that it's available in your target Azure region before designing your architecture around it.
My ADF pipeline says "Queued" but never starts running, what's wrong?
A pipeline stuck in "Queued" almost always means the Integration Runtime it depends on is unavailable or hasn't finished initializing. If you're using an Azure Integration Runtime, ADF may be waiting for the IR cluster to spin up, this is normal and typically takes 1–2 minutes for the first run. If it stays Queued beyond 5 minutes, check whether you hit your subscription's ADF pipeline concurrency limit (which varies by region and subscription tier). You can check active runs under Monitor → Pipeline runs and see if you have a large number of in-progress runs consuming the concurrency slots. For SHIR-dependent pipelines, verify the SHIR service is running on the host machine as described in Step 2 of this guide.
How is Azure Data Factory different from Data Factory in Microsoft Fabric?
Azure Data Factory (ADF) is the standalone Azure service, it lives in your Azure subscription as its own resource, integrates with Azure Synapse, Azure DevOps, and your existing Azure infrastructure, and is billed independently. Data Factory in Microsoft Fabric is a newer version built directly into the Microsoft Fabric unified analytics platform, sharing the same workspace, compute, and OneLake storage as Fabric's other workloads like Power BI, Synapse Data Engineering, and Real-Time Intelligence. Fabric Data Factory has a more modern authoring experience and tighter Fabric integration, but ADF has broader connector support and more mature CI/CD tooling as of now. Microsoft has published official migration guidance for moving ADF pipelines into Fabric, including a PowerShell upgrade tool, if that's a direction you're considering.
Can Azure Data Factory run my existing SSIS packages?
Yes, this is one of ADF's specific "lift and shift" features. You can run your existing SQL Server Integration Services (SSIS) packages in Azure using the SSIS Integration Runtime, which is a fully managed Azure compute cluster that hosts SSIS natively. You provision the SSIS IR through the Integration Runtimes section in ADF Studio, configure it with the right Azure-SSIS IR size and node count, and then execute your SSIS packages either from the SSIS catalog in Azure SQL Managed Instance or from a file share. According to Microsoft, SSIS packages run with full compatibility on the SSIS IR, you don't need to rewrite them. The main thing you'll need to update is any connection manager in your packages that pointed to on-premises servers; those need to be updated to point to Azure-accessible endpoints or routed through a SHIR.