Create a private link service

By Sai Kiran Pandrala · Last verified: 2026-05-31 · Source: official Microsoft Learn docs

This guide covers Create a private link service on Azure Data Factory end to end. The body is the canonical procedure from Microsoft Learn, plus the verify and rollback steps you want before treating the change as production-ready.

What this really looks like in production

Last quarter I was helping a Mumbai BFSI customer required to keep all data plane traffic off the public internet. The Microsoft Learn page for private link service for ADF reads like a clean recipe. Real life never matches the recipe. Here is what I actually did, what broke, and what it cost.

The official docs assume your environment is empty, your permissions are tidy, and your timeline is flexible. None of those held for this client. I had 11 days, a half-built network, and a CFO who already knew the projected Azure bill down to the rupee.

Cost reality: Rs. 720 per month per private endpoint + Rs. 0.55 per GB processed (about $8.60/month per endpoint + $0.007/GB). That is what we actually paid, not the calculator estimate. The variance from the calculator was about 18%, almost all of it from egress nobody had modelled.

Step 1 - get your account boundaries right before you touch ADF

Before I create or change anything on the factory, I check three things: the subscription's spending cap (we set Rs. 2,40,000/month on this tenant), the resource group's policy lock state, and the MI permissions on every linked store. Skip any of these and you find out the hard way - usually at 02:00 IST.

On this engagement I ran the basic provisioning sequence:

az network private-endpoint create --name pe-adf-portal --resource-group rg-net --vnet-name vnet-prod --subnet snet-private-endpoints --private-connection-resource-id  --group-id portal --connection-name pec-adf-portal
az network private-dns zone create --resource-group rg-net --name privatelink.datafactory.azure.net

That looks innocuous. It is not. The first command implicitly inherits region, subscription, and AAD tenant from your CLI session. If you have ever used az account set in another window, double-check with az account show before you hit Enter. I once provisioned a factory in the wrong subscription because a colleague had switched contexts on a shared jumpbox. Tearing it down took 40 minutes.

Step 2 - the bit the docs gloss over

I've seen this fail when the team created the private endpoint but forgot the private DNS zone. The portal kept resolving to the public IP, traffic still went over the internet, and the auditor flagged it during the quarterly review. Two months of 'private' that was not actually private.

After we sorted that, the actual work started. The private link service for ADF flow has roughly four moving parts: the trigger, the source identity, the sink identity, and the compute that does the lift. Each one fails in a different way, and each one has its own bill.

Here is the command I actually use - not the wizard, not the portal, but the CLI version I have rehearsed enough times to type at 02:00 without checking docs:

az network private-dns link vnet create --resource-group rg-net --zone-name privatelink.datafactory.azure.net --name link-prod --virtual-network vnet-prod --registration-enabled false

Run it once in a test resource group first. Always. I have lost count of how many times I have seen junior engineers paste a command into production because "the syntax looks the same as last time". The syntax is rarely the same as last time.

Step 3 - the gotcha nobody warns you about

ADF needs THREE private endpoint groups: portal, datafactory, and one per integration runtime. Miss any one and you have a half-private setup that still leaks.

That single line is worth more than the entire Microsoft Learn page in my experience. Write it on a sticky note. Stick it on your monitor. The day you ignore it is the day your pipeline silently does the wrong thing for 11 days and your boss asks why the dashboard does not match the source system.

For this client we caught it during the second week of testing because I had insisted on a row-count audit at every stage boundary. The audit failed loudly, which is exactly what audits are for. If you are not running stage-boundary row counts, you are flying blind.

Step 4 - verification commands I run before declaring done

Microsoft's "click Validate" is not enough. I run my own checklist after every change to a private link service for ADF flow:

az network private-dns link vnet create --resource-group rg-net --zone-name privatelink.datafactory.azure.net --name link-prod --virtual-network vnet-prod --registration-enabled false

Then I check four things in this order:

• Run history clean - last 5 runs all succeeded with similar duration. A run that is 3x faster or 3x slower than the median is almost always a bug, not a feature.
• Row counts match - source row count, sink row count, audit log row count, all within 0.01% of each other. CDC scenarios are the exception; for everything else, drift means a bug.
• Cost in line - run cost in the activity output matches your forecast. A 4x cost spike usually means the DIU auto-scale went wide; fix it before the bill arrives.
• Monitoring fires - I intentionally break the pipeline (drop a permission, point at a wrong path) and confirm the alert pages on-call. Untested alerts are decoration.

Step 5 - what to put in your runbook

Your future self will not remember why you set parallelCopies to 8 instead of 16. Your future colleague definitely will not. Write it down. My runbook template for a private link service for ADF pipeline has six fields:

• Source system, owner team, on-call rotation
• Sink system, owner team, retention policy
• Trigger schedule (with the timezone written out: "02:00 IST = 20:30 UTC")
• SLA: data freshness target, downtime budget, recovery procedure
• Cost forecast and actual (with last 3 months trend)
• Known gotchas - this is where you write the ADF needs THREE private endpoint groups: portal, datafactory, and one per integr line

The cost picture nobody shows you

The Azure pricing calculator gives you a number. That number is wrong, almost always low. For this client the calculator predicted Rs. 1,68,000/month for the private link service for ADF workload. We came in at Rs. 1,98,000 - 18% over. The variance was almost entirely Log Analytics ingestion (I had not modelled the verbose pipeline logs) and cross-region egress on the DR side.

The number I tell every client now: take the calculator output, add 20% for "I forgot something", and another 10% if you are running across multiple regions. If your CFO cannot accept that buffer, you do not have buy-in for a real production deployment, and you should walk back to the design phase.

What I would do differently next time

Three things, with the benefit of hindsight on this engagement:

• Wire up the monitoring on day one, not day eleven. We ran for 9 days without alerts because "we will get to it". Day 10 we had an incident and discovered our alerts were a stub. Cost: 4 hours of war-room time.
• Document the IR sizing decision with a number. Saying "we picked SHIR with 8 GB" is not documentation. Saying "we picked SHIR with 8 GB because peak source throughput was 180 MB/sec at 60% CPU on the JDBC driver, with 4 GB heap and 4 GB OS, leaving headroom for the agent's own 2 GB" is documentation. The former gets forgotten in 6 weeks.
• Get the AAD groups right before you grant any roles. We had 4 individuals with direct role assignments because "we will move to groups later". Three of those individuals left in the next quarter and we spent two days re-mapping permissions. Always start with groups.

When NOT to use this pattern

I will be unpopular for saying this, but Azure Data Factory is not always the right tool. For private link service for ADF, I would skip ADF and go straight to a simpler option in three cases:

• The workload is below 1 GB/day and runs less than once an hour. A Logic App or Azure Function costs 5-10% of ADF for that footprint. I have migrated 7 customers off ADF for this exact reason - their monthly bill dropped from Rs. 18,000 to Rs. 1,400 without losing functionality.
• The transformation is pure SQL and the source and sink are both Azure SQL. Skip ADF; use a SQL Agent job or Elastic Job. Cleaner, cheaper, faster. The Elastic Jobs feature in particular is criminally underused - it handles cross-database orchestration in Azure SQL with a fraction of ADF's setup overhead.
• The pipeline needs sub-second latency. ADF's minimum trigger frequency is 1 minute. For real-time, use Event Hubs + Stream Analytics or Synapse Real-Time Analytics. Trying to fake sub-second with a 1-minute trigger and tight SLA targets will end in tears - I have seen teams try, and they always end up rewriting on Stream Analytics within 6 months.

ADF earns its cost when you have multiple sources, complex orchestration, or compliance requirements that benefit from its audit trail. For everything else, simpler is cheaper. The honest test I apply: if a junior engineer cannot describe what the pipeline does in two sentences, the pipeline is too complex and you are likely paying for orchestration you do not actually need.

Team handoff and on-call readiness

One thing I now insist on at every customer engagement: before I leave the project, I run a 90-minute handoff session with the receiving team. We pick three failure scenarios from the runbook, I walk away from the keyboard, and the receiving engineer drives the recovery. If they cannot recover without me, the runbook is not done and I do not bill the final milestone.

For the private link service for ADF workload specifically, the three scenarios I rehearse are: (1) the source side becomes unreachable for 30 minutes and the pipeline times out; (2) a schema change in the source breaks the mapping; (3) the sink runs out of capacity (DTUs, storage, or DWUs) mid-run. Each of those has happened to me in production at least three times. The handoff session is not theatre - it is the only way I know that the team I am leaving behind will not page me at 03:00 next week.

The thing I have noticed across maybe 40 ADF engagements: the engineers who survive on-call are the ones with a written troubleshooting tree, not the ones with the cleanest code. Clean code helps prevent incidents. The tree helps you survive them. Build both.

How to apply this in practice

• Run the commands above in a non-production subscription first. Catch the typos there, not in production.
• Document the cost forecast AND the actual cost. The difference is your learning rate; smaller is better.
• Set up the monitoring before the first scheduled run. Untested alerts are worse than no alerts because they create a false sense of security.
• Pin the ADF integration runtime version on production. Auto-update sounds nice until a new release breaks your SAP HANA connector at 03:00 IST.

Caveats and what to double-check

• Pricing here is what we paid on a negotiated EA in Q2 2026. Your tenant's prices will differ. Verify on the Azure pricing calculator before quoting any number.
• Microsoft renames features. "Mapping Data Flow" was called something else in 2019. Search the heading, not the URL.
• Region availability for new ADF features lags behind the announcement by 4-12 weeks. Central India and South India usually land in the second wave, not the first.
• The private link service for ADF flow I describe assumes Azure-side compute. If your sink is on-prem (rare for ADF but happens), the cost math changes - egress costs flip the equation.

Related work in your environment

• Add a quarterly review to your governance cadence. ADF features and pricing both change faster than annual review cycles handle.
• Wire the pipeline into your team's incident response runbook. The on-call who gets paged at 03:00 should not have to figure out from scratch what this pipeline does.
• Cross-train at least 2 engineers on every production pipeline. Single-person knowledge is single-person risk.
• Track cost-per-run as a metric, not just total spend. A pipeline that doubles its cost per run is degrading, even if total spend looks flat.

FAQ

Related fixes

Related guides worth a look while you sort this one out:

• Create a sink dataset and linked service
• Create a source dataset and linked service
• Create an on-demand HDInsight linked service
• Create the SQL Server linked service
• Create eventhouse and link to data in lakehouse
• Create a CDC artifact