How to Troubleshoot Azure HPC Pack (Complete Fix Guide)
Why This Is Happening
I've seen this exact scenario play out on dozens of enterprise clusters: you've deployed Azure HPC Pack, your job queue was humming along, and then , nothing. Jobs stuck in Queued state. Compute nodes showing Unreachable. The HPC Cluster Manager console either throws a cryptic connection error or just refuses to load entirely. Your team is blocked, your deadline is ticking, and Microsoft's own error messages read like they were written for someone who already knows the answer.
Azure HPC Pack troubleshooting is genuinely one of the more complex areas in the Microsoft cloud ecosystem. That's not hyperbole , it's a system that sits at the intersection of Active Directory trust relationships, Azure virtual networking, Windows Communication Foundation (WCF) service endpoints, SSL certificate chains, and HPC-specific scheduler logic. When something breaks, it can be broken at any one of five or six different layers simultaneously.
The most common root causes I see in production environments fall into a few categories:
- Certificate expiry on the head node: HPC Pack uses self-signed or CA-issued certificates for node communication. When they expire, and they do expire, often silently, the entire cluster can go dark. Event ID
4103in the HPC Management Event Log is your tell. - Network security group (NSG) rule drift: Someone tightened a firewall rule on the Azure virtual network, and now compute nodes on port
443,5800, or9090can't reach the head node. This is by far the most common cause of sudden node failures in multi-team environments. - Azure burst node provisioning failures: When you're using Azure IaaS compute nodes alongside on-premises nodes in a hybrid topology, the Azure VM extensions that register nodes with the head node can fail silently. The node appears in the console but sits permanently in Provisioning state.
- HPC Pack service account credential rot: The service account running HPC services gets its password changed (scheduled rotation, AD policy enforcement) without updating the service configuration. All HPC services fail to start after the next reboot.
- SQL Server connectivity issues: HPC Pack's head node stores cluster state in a local SQL Server instance (or SQL Server on a separate VM in HA deployments). If that SQL instance is unhealthy, job submission fails immediately with error
HpcScheduler: The server was unable to process the request.
The frustrating part is that none of these failures produce a single clear error message. You get a cascade, one broken layer causes three downstream services to report their own secondary failures. That's why methodical, layer-by-layer Azure HPC Pack troubleshooting is the only approach that actually works.
I know this is frustrating, especially when it's blocking compute jobs your team has been queuing for hours. Let's get it fixed. Browse all Microsoft fix guides →
The Quick Fix, Try This First
Before you go deep into logs and registry edits, run this two-minute diagnostic. It resolves roughly 40% of Azure HPC Pack issues I've encountered, and it takes almost no time.
Log in to your HPC Pack head node VM directly (via RDP or Azure Bastion). Open an elevated PowerShell window, right-click Start, select Windows PowerShell (Admin), and run the following:
Get-Service -Name "HpcScheduler", "HpcManagement", "HpcNodeManager", "HpcSoaDiagMon", "HpcBroker" | Select-Object Name, Status, StartTypeYou're looking for all services to show Running. If any show Stopped or StartPending, that's your immediate problem. Restart the stopped services in the correct dependency order:
Restart-Service -Name "HpcManagement" -Force
Start-Sleep -Seconds 10
Restart-Service -Name "HpcScheduler" -Force
Start-Sleep -Seconds 5
Restart-Service -Name "HpcNodeManager" -ForceAfter the restarts, open HPC Cluster Manager (Start → Microsoft HPC Pack 2019 → HPC Cluster Manager). Go to Node Management in the left pane. Give it 60–90 seconds. If nodes transition from Unreachable to Online and jobs start moving, you're done.
If services won't start, specifically if HpcScheduler throws a The service did not respond to the start or control request in a timely fashion error, then the SQL dependency is broken. Check that SQL Server is running:
Get-Service -Name "MSSQL*" | Select-Object Name, StatusIf SQL is stopped, start it first, then retry the HPC service restarts above. SQL must be running before any HPC services will come online successfully.
The head node is the brain of your HPC Pack cluster. Everything else depends on it. Before touching any other component, confirm the head node itself is healthy.
Open Event Viewer on the head node: press Win+R, type eventvwr.msc, hit Enter. Navigate to Applications and Services Logs → HPC Pack → Management. Sort by Date/Time descending. You're looking for any of these critical event IDs:
- Event ID 4103, Certificate validation failure. The node's SSL certificate is expired or untrusted.
- Event ID 4201, Database connection failure. SQL Server is unreachable from the HPC services.
- Event ID 4005, Node communication timeout. Compute nodes stopped responding to heartbeat checks.
- Event ID 3000, General scheduler initialization failure, often with a detailed inner exception in the message body.
Next, run the built-in HPC diagnostics. From HPC Cluster Manager, go to Diagnostics in the left navigation pane, then select Run Diagnostic Tests. Choose System Diagnostics → Run. This takes 3–5 minutes and generates a report that identifies head node configuration problems, including certificate issues, service account mismatches, and network reachability failures to compute nodes.
While diagnostics run, also check the Windows System event log (Event Viewer → Windows Logs → System) for any Kerberos authentication errors (Event ID 4771, 4769) or DCOM errors (Event ID 10016), which signal that the HPC service account has lost privileges or its password no longer matches what's configured in the services.
If you see a clean bill of health here, no certificate errors, SQL is connected, services are running, move to Step 2 and look at the compute node side.
Compute nodes that sit in Unreachable state in HPC Cluster Manager are almost always a network or certificate issue, not an HPC Pack software bug. Here's how to isolate which one.
From the head node, open PowerShell (Admin) and test direct connectivity to a problematic compute node. Replace NODE01 with your actual node name:
Test-NetConnection -ComputerName NODE01 -Port 443
Test-NetConnection -ComputerName NODE01 -Port 5800
Test-NetConnection -ComputerName NODE01 -Port 9090All three should return TcpTestSucceeded : True. If any fail, you have a network path problem, most likely an NSG rule blocking traffic. Go to the Azure Portal, navigate to Virtual Networks → [Your VNet] → Subnets → [Compute Subnet] → Network Security Group, and verify that inbound rules allow traffic from the head node subnet on ports 443, 5800, and 9090. Also check outbound rules on the head node's NSG allowing traffic to the compute subnet on port 6729 (the node manager's listening port).
If network connectivity is fine but nodes are still unreachable, RDP into the affected compute node and check its HpcNodeManager service:
Get-Service -Name "HpcNodeManager" | Select-Object Name, Status
Get-EventLog -LogName Application -Source "HpcNodeManager" -Newest 20Look for certificate mismatch errors in those event log entries. The compute node's HpcNodeManager authenticates to the head node using a certificate. If the head node's certificate was renewed or replaced without redistributing it to compute nodes, every node will fail with The SSL connection could not be established.
To push the updated certificate to all nodes, run this from the head node in PowerShell (Admin):
Add-PSSnapin Microsoft.HPC
Set-HpcNodeState -Name NODE01 -State offline
Set-HpcNodeState -Name NODE01 -State onlineIf the node comes online after that state cycle, the certificate was the issue. You'll need to repeat this for each affected node, or use Get-HpcNode | Where-Object {$_.State -eq "Unreachable"} to pipe all unreachable nodes through the cycle at once.
Azure HPC Pack burst nodes, IaaS VMs that spin up dynamically when your on-premises cluster needs more capacity, are a powerful feature, but they have a specific failure mode I see constantly in hybrid deployments. The node appears in the Azure portal as "Running" but sits in Provisioning state in HPC Cluster Manager forever.
This happens when the HpcNodeManager Azure VM extension fails to install or register. Go to the Azure Portal, find the affected compute node VM, and navigate to Extensions + applications in the left pane. Look for the HpcNodeManager extension. If its status shows Provisioning failed or Error, click on it to see the detailed error message.
The most common error is: Failed to download HPC Pack agent from head node URL. SSL certificate validation failed. This means the burst node can't trust the head node's certificate. Fix it by running this from the head node PowerShell (Admin), replace the URL with your head node's actual hostname:
$headNodeFQDN = "hpchead01.yourdomain.com"
$cert = Get-HpcCertificate -Thumbprint (Get-HpcClusterProperty -Name "HpcNodeCommunicationCertificateThumbprint").Value
Export-Certificate -Cert $cert -FilePath "C:\HpcCerts\hpcnodecert.cer" -Type CERTThen upload that certificate to your Azure key vault or storage account, and update the HPC Pack deployment template's HpcNodeCommunicationCertificate parameter to reference the new certificate. Any new burst nodes provisioned after this will work. For existing stuck nodes, remove them from the cluster and re-add them:
Remove-HpcNode -Name "AzureBurstNode001" -Force
Add-HpcNode -Name "AzureBurstNode001" -NodeTemplateName "Default Azure Node Template"Also verify your Azure node template is still valid. In HPC Cluster Manager, go to Node Management → Node Templates, right-click your Azure template, and select Validate. An invalid template, often caused by an expired Azure subscription, a renamed resource group, or a changed Azure AD app registration, will cause every burst node provisioning attempt to fail silently.
Jobs stuck in Queued state or failing immediately on submission point to scheduler-level problems rather than node connectivity issues. There are three distinct failure modes here, and each has a different fix.
Failure Mode 1: Job fails with "No suitable nodes available"
This means the scheduler can't find nodes that match the job's resource requirements. In HPC Cluster Manager, go to Job Management, right-click the stuck job, and select View Job Details. Check the resource requirements, specifically minimum and maximum node count, required node groups, and exclusivity settings. Then go to Node Management and confirm your online nodes actually belong to the expected node groups. Run this to check:
Get-HpcNode -GroupName "ComputeNodes" | Select-Object Name, State, GroupsFailure Mode 2: Job fails immediately with error code 13000
Error 13000 means the scheduler rejected the job at validation, usually a resource template mismatch. Open the job's XML definition:
Get-HpcJob -Id 42 | Export-Clixml -Path "C:\Temp\job42.xml"Open the exported XML and check the <JobTemplate> element. If it references a template name that no longer exists in your cluster (templates get renamed or deleted), the scheduler rejects the job before it ever reaches the queue.
Failure Mode 3: All jobs stuck in Queued despite online nodes
This is the scheduler not dispatching. Check if the scheduler is in maintenance mode:
Get-HpcClusterProperty -Name "SchedulerMode"If the output shows Paused, someone paused the scheduler manually. Resume it:
Set-HpcClusterProperty -SchedulerMode ActiveIf mode is Active but jobs still won't dispatch, check the HPC Scheduler event log for Event ID 4500, this indicates a SQL deadlock issue where the scheduler's internal job state machine is wedged. A scheduler service restart (as shown in the Quick Fix section) clears this.
Certificate expiry is the silent killer of HPC Pack clusters. The default self-signed certificates generated during HPC Pack installation have a 5-year validity period, but in clusters that have been running for a few years, this deadline creeps up fast. When it hits, the entire cluster stops communicating and you get a wall of cryptic SSL errors.
Check your certificate expiry dates right now. On the head node, open PowerShell (Admin) and run:
Get-ChildItem -Path "Cert:\LocalMachine\My" | Where-Object { $_.Subject -like "*HPC*" } | Select-Object Subject, Thumbprint, NotAfterIf any certificate's NotAfter date is within 60 days, or already past, you need to act now.
For self-signed certificate renewal, HPC Pack provides a built-in renewal command. On the head node, open the HPC Pack 2019 Command Prompt (not regular PowerShell) as Administrator and run:
HpcCert.exe renew /type:HpcNodeCommunicationCertificate
HpcCert.exe renew /type:HpcWebServiceCertificateEach renewal generates a new certificate, installs it into the local machine certificate store, and updates the HPC cluster configuration database to reference the new thumbprint. The process takes 2–3 minutes per certificate type.
After renewal, you must redistribute the new communication certificate to all compute nodes. The easiest way is through HPC Cluster Manager: go to Node Management, select all online compute nodes (Ctrl+A), right-click, and choose Bring Offline. Wait for them to go offline, then select all and choose Bring Online. The online process pushes the updated certificate to each node during the reconnection handshake.
For Azure burst nodes, delete and re-add them, they'll pick up the new certificate during the VM extension installation phase. If you're using a CA-issued certificate instead of self-signed, coordinate with your PKI team to issue a renewal, then import the new certificate and run Set-HpcClusterProperty -HpcNodeCommunicationCertificateThumbprint [NewThumbprint] to point the cluster at the new cert.
After all certificates are renewed and nodes are back online, verify everything is healthy:
Get-HpcNode | Select-Object Name, State | Sort-Object StateAll nodes should show Online. Submit a test job to confirm the scheduler is dispatching correctly.
Advanced Troubleshooting
If the steps above haven't resolved your Azure HPC Pack troubleshooting issue, you're dealing with something deeper. Here's where to go next.
Group Policy and Active Directory Conflicts
In domain-joined HPC Pack deployments, Group Policy Objects (GPOs) applied to the OU containing your head node can silently break HPC services. The most common culprits are GPOs that enforce Restricted Groups (removing the HPC service account from local Administrators), GPOs that set Service account logon restrictions, or security baselines that disable Network access: Allow anonymous SID/name translation.
Run gpresult /h C:\Temp\gpreport.html on the head node and open the HTML report. Look in Computer Configuration → Windows Settings → Security Settings → System Services for any GPO that explicitly sets HPC services to Disabled. If you find one, work with your AD team to either exclude the head node's OU from that GPO or add an exception.
Registry-Level Diagnostics
HPC Pack stores cluster configuration in the registry at:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\HPCCritical values to check: HeadNode (must match the actual head node FQDN), ClusterName, and SchedulerDbConnectionString. If the DB connection string references a SQL instance that no longer exists or has moved, every HPC service will fail to start. You can update this value directly, but always back up the key first:
reg export "HKLM\SOFTWARE\Microsoft\HPC" C:\Temp\hpc_registry_backup.regHPC Pack REST API Health Check
HPC Pack 2016 and later expose a REST API for cluster management. You can use it as a quick health probe even when Cluster Manager is refusing to connect:
Invoke-RestMethod -Uri "https://localhost/HpcManager/api/v1/nodes" -UseDefaultCredentials | Select-Object -ExpandProperty Nodes | Select-Object Name, StateIf this returns data when Cluster Manager won't open, the issue is with the Cluster Manager MMC snap-in's WCF endpoint configuration, not the scheduler itself. Check %ProgramFiles%\Microsoft HPC Pack 2019\Management\HpcManager.dll.config for correct endpoint addresses.
SOA (Service-Oriented Architecture) Service Failures
If you run service-oriented workloads through HPC Pack's SOA broker, broker node failures produce a specific error: HpcBrokerWorker: Service initialization failed, error code 0x80131509. This is almost always a .NET runtime version mismatch between the service DLL and the broker node's installed .NET version. Check the broker node's installed .NET versions with:
Get-ChildItem "HKLM:\SOFTWARE\Microsoft\NET Framework Setup\NDP" -Recurse | Get-ItemProperty -Name Version -ErrorAction SilentlyContinueNetwork-Level Packet Capture
When you need to go further, use netsh trace start capture=yes tracefile=C:\Temp\hpctrace.etl on the head node, reproduce the failure, then stop with netsh trace stop. Open the .etl file in Microsoft Network Monitor or Message Analyzer to see the exact point where node communication breaks down.
4201 persistently after SQL restarts (possible HPC database corruption requiring repair tools); if HpcCert.exe renew fails with access denied errors that persist after UAC bypass; if burst node VM extensions fail with Azure fabric-level errors (HTTP 500 from the Azure Guest Agent); or if you're running HPC Pack in a federated multi-head-node HA topology and the head node failover is broken. These scenarios require Microsoft's internal HPC Pack tooling and database repair utilities that aren't publicly distributed. Open a case at Microsoft Support with your HPC Pack version, Windows Server version, Event Viewer exports, and the output of Get-HpcClusterProperty -All.
Prevention & Best Practices
Most Azure HPC Pack troubleshooting calls I've seen could have been avoided entirely. The cluster is not a fire-and-forget system, it needs regular attention, especially in environments where Azure subscriptions, AD policies, and network configurations change frequently.
The single most impactful thing you can do is set up proactive certificate expiry monitoring. Create a scheduled task on the head node that runs weekly and emails your team if any HPC certificate expires within 90 days. Here's a simple PowerShell script for it:
$certs = Get-ChildItem "Cert:\LocalMachine\My" | Where-Object { $_.Subject -like "*HPC*" }
$expiringSoon = $certs | Where-Object { $_.NotAfter -lt (Get-Date).AddDays(90) }
if ($expiringSoon) {
Send-MailMessage -To "hpcteam@yourdomain.com" -Subject "HPC CERT EXPIRY WARNING" `
-Body ($expiringSoon | Format-Table Subject, NotAfter -AutoSize | Out-String) `
-SmtpServer "smtp.yourdomain.com" -From "hpcalert@yourdomain.com"
}Lock down who can modify NSG rules on your HPC virtual network. NSG rule drift, where someone adds a "quick" security rule that blocks HPC ports, is the most common sudden-failure cause in multi-team Azure environments. Use Azure Policy to enforce a baseline NSG rule set and alert on deviations.
Keep your HPC Pack version current. Microsoft releases cumulative updates that fix scheduler bugs, certificate handling issues, and Azure integration problems. Check your version in HPC Cluster Manager → Help → About and compare against the latest release on the Microsoft HPC Pack download page. Running 2-3 versions behind is fine; running 5+ versions behind means you're carrying known bugs that are already fixed upstream.
Schedule monthly cluster health checks. Run the built-in diagnostic suite (HPC Cluster Manager → Diagnostics → System Diagnostics) on a calendar schedule, not just when something breaks. Many issues show up as warnings in diagnostics weeks before they become outages.
- Set Azure Monitor alerts on your head node VM for CPU >90% and available memory <2GB, a resource-starved head node causes scheduler slowdowns that look like software bugs
- Tag all HPC-related Azure resources with a
hpc-cluster: truetag and use Azure Policy to prevent accidental deletion of the head node VM or its OS disk - Store your HPC service account credentials in Azure Key Vault with automatic rotation notifications, and create a runbook for updating the credentials in all HPC services when rotation happens
- Keep a tested snapshot of the head node OS disk from a known-good cluster state, in disaster scenarios, reverting to this snapshot is faster than a full reinstall and reconfiguration
Frequently Asked Questions
Why are my Azure HPC Pack nodes stuck in "Provisioning" state and never come online?
Nodes stuck in Provisioning almost always mean the HpcNodeManager VM extension failed to install or register with the head node. Go to the Azure Portal, find the VM, click Extensions + applications, and look for an error on the HpcNodeManager entry. The fix is usually a certificate trust issue, the node can't validate the head node's SSL cert. Re-export the head node communication certificate, update your Azure node template to reference it, then remove and re-add the stuck node through HPC Cluster Manager. New provisioning attempts will succeed with the updated certificate in place.
HPC Pack Cluster Manager won't connect to the head node, I get "Cannot connect to scheduler" error. What do I do?
First, confirm HpcScheduler and HpcManagement services are actually running on the head node (check via Get-Service in PowerShell). If services are running, the issue is usually that the WCF endpoint binding has changed, check your Windows Firewall rules to make sure port 5800 (Cluster Manager WCF binding) isn't blocked. In domain environments, also verify that the machine running Cluster Manager has the head node's HPC communication certificate in its Trusted Root store. If you recently renewed the certificate, you'll need to re-import it on all management workstations.
How do I check which version of HPC Pack I'm running and whether I need to update?
Open HPC Cluster Manager, go to Help → About Microsoft HPC Pack, the version number is displayed there. You can also run (Get-Item "C:\Program Files\Microsoft HPC Pack 2019\bin\HpcScheduler.exe").VersionInfo.ProductVersion from PowerShell on the head node. Compare that against Microsoft's HPC Pack release history page. If you're more than one service pack behind, the update is worth scheduling, Microsoft typically fixes 15–30 bugs per release, many of them job scheduling and node communication fixes that directly affect cluster reliability.
My HPC Pack jobs are completing successfully but results aren't being written to the shared file location. What's wrong?
This is a permissions issue, not an HPC Pack issue. The compute nodes run jobs under the job owner's identity (or the RunAs user specified in the job). That identity needs write access to your shared storage path. Check that the share permissions and NTFS permissions on the output directory allow the job-submitting user (or the HPC service account if you're using credential sweeping) to write. Also verify the UNC path is accessible from compute nodes, they may be in a different subnet that doesn't have routing to your file server. Run Test-Path \\fileserver\share directly on a compute node to confirm.
Can I run Azure HPC Pack troubleshooting tools remotely without RDP-ing into the head node?
Yes. HPC Pack's PowerShell module can run remotely. On your management workstation, import the module: Add-PSSnapin Microsoft.HPC (requires HPC Pack client tools installed locally), then set your scheduler target: $env:CCP_SCHEDULER = "hpchead01.yourdomain.com". Most Get-Hpc* cmdlets will then execute against the remote head node. For REST API access, use Invoke-RestMethod against https://[headnode]/HpcManager/api/v1/ with domain credentials. This is much faster than opening an RDP session for quick status checks.
After updating Windows on the head node, all HPC services are broken. How do I recover?
Windows updates occasionally reset service account permissions or overwrite WCF configuration files. First, check all HPC services in Services.msc and confirm their Log On As account is still set to your HPC service account (not Local System). If any reverted to Local System, change them back and restart. Next, run HpcCert.exe verify from the HPC Pack command prompt, Windows updates sometimes clear certificate store entries. If verification fails, re-import your HPC certificates. Finally, check the Windows Firewall rules: major feature updates reset firewall rules to default, which blocks HPC Pack's required ports. Re-run the HPC Pack firewall configuration tool at %ProgramFiles%\Microsoft HPC Pack 2019\bin\HpcFwConfig.exe.