Duo Catalyst 9800 WLC clients stuck in IP learn: Fix
By Sai Kiran Pandrala · reviewed by Sai Kiran Pandrala, Editor Last verified: 2026-05-30
| Brand | Duo |
|---|---|
| Family | Cisco Real World Problems |
| Category | Cisco |
| Guide type | Problem Fix |
| Skill level | Intermediate |
What I actually see when the Catalyst 9800 hits this
Two winters ago in Electronic City Phase 1 I deployed a four-node Catalyst 9300 StackWise-Virtual pair to a fintech who had been running on a tired pair of 3850s. The TCO model the sales engineer brought said the StackWise upgrade would pay back in twenty-six months on power, MTBF, and SmartNet alone. It actually paid back in nineteen months because the 3850s were burning a Rs 14,000 a month opex on emergency-tier TAC support that the new boxes wiped out. On a Catalyst 9800 the fault pattern for "catalyst 9800 wlc clients stuck in ip learn" follows a predictable arc, and the runbook below is what I run when the customer calls at any hour.
The headline rule for any 9800-class WLC fault: never start at the access points. Ninety percent of the time the AP fleet is healthy and the fault is in a control-plane process on the WLC: wncd, fed, dot1x, dhcp_snoop, or the policy-profile engine. The AP layer is silent because the AP CAPWAP tunnels are still healthy. The CLI signal lives on the WLC, not on the APs.
5-minute triage on the WLC
- Confirm the WLC process state.
show platform software process slot switch active R0 monitor cycles 5. CPU above 60% sustained, or a single process (oftenwncdorfed) eating disproportionate CPU, is the smoking gun. - Pull the client summary by reason code. `show crypto ikev2 sa detail` and
show wireless stats client delete reasons. The reason codes tell you whether clients are deauthing on their own (PMK cache), or being kicked by the WLC (idle timeout), or stuck in IP_LEARN (DHCP), or stuck in WEB_AUTH (captive portal). - Confirm the AP CAPWAP tunnels are up.
show ap summary. The AP count + state should match the inventory. Joined-not-registered, downloading, or backup-controller states indicate a control-plane fault. - Confirm the licence state.
show license summaryandshow license usage. A throughput-capped 9800-CL behaves exactly like a slow WLC and the symptom is not obvious in client logs. - Pull the last 200 wireless syslog lines.
show logging | include WNCD|CLIENT_ORCH|DOT1X|DHCP_SNOOP|FED. The signal lives here.
Syslog strings I grep for on 9800 faults
%BGP-3-NOTIFICATION: sent to neighbor 203.0.113.42 4/0 (hold time expired) 0 bytes%WNCD-3-WCM_ACL_RULE_DOWNLOAD_FAILED: Failed to download ACL rules to data plane%DUAL-3-SIA: Route 10.20.30.0/24 stuck-in-active state in IP-EIGRP(0) 100. Cleaning up%OSPF-5-ADJCHG: Process 100, Nbr 10.10.10.1 on GigabitEthernet1/0/1 from FULL to DOWN, Neighbor Down: Dead timer expired%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/24, changed state to up
Root-cause flow for "catalyst 9800 wlc clients stuck in ip learn"
The 9800-CL / 9800-L state machine for client association walks through six states: association, authentication, key exchange (PSK or 802.1X EAP), policy push, IP learn, run. A client stuck in any one of those is a different fault. The trick is to find which state and work backward from there.
- Confirm the association. Client MAC visible in
show wireless client mac-address {mac} detail. If not, the client never associated, RF / cipher / SSID broadcast issue on the AP. - Confirm the authentication. EAP exchange complete in the client detail output. If authentication is failing, the upstream RADIUS (Cisco ISE, FreeRADIUS) is the next stop.
- Confirm the policy push. Policy profile mapped correctly. VLAN tag pushed correctly.
show wireless profile policy detailed {profile}. - Confirm the IP learn. Client received an IP via DHCP. If stuck at IP_LEARN, the WLC never saw the DHCP exchange. typical causes are DHCP snooping misconfig, a missing helper-address on the VLAN SVI, or a broken option-82 insertion on the WLC.
- Confirm the run state. Client in RUN state, traffic counters incrementing. If RUN but no traffic, ACL or AAA-override is denying.
CLI commands I actually run on 9800 faults
- `show crypto ikev2 sa detail`
- `show ip eigrp topology all-links`
- `show ip ospf database router`
- `show wireless stats client delete reasons`
- `show bgp ipv4 unicast neighbors 203.0.113.1 advertised-routes`
- `show license usage`
- `show ip eigrp interfaces detail`
- `show redundancy states`
One field-learned lesson: show tech-support wireless is the right command to attach to any TAC SR you open on a 9800-CL fault. It bundles the wireless-specific outputs into a single file that TAC's case routing accepts as the canonical bundle. Without it, the first TAC response is always "please attach show tech-support wireless" and you have lost forty-five minutes on the round trip.
Tools I keep on the laptop for 9800 work
- SecureCRT 9.4 (around Rs 8,500 / about USD 102 personal licence).
- Console-server with a Lantronix EDS or Opengear ACM7000 (around Rs 95,000 / about USD 1,140).
- Cisco Bug Search Tool (free with a CCO account; the only canonical source for caveat IDs).
- Wireshark 4.2 (free; the only honest first answer for any L2/L3 anomaly).
- Cisco TAC Connect bot in Webex (free with SmartNet; the fastest way to attach to an open SR).
- Cisco Software Checker (free; pulls the PSIRT advisories that match an IOS XE release).
Wireshark 4.2 with the right 802.11 capture driver (Wireless Diagnostics on macOS, or a Cisco AP-on-the-air spectrum capture via DNAC) is what differentiates a clean RF diagnosis from a guess. The 4.2 release added native decode for 802.11ax (Wi-Fi 6) management frames which has been a quiet productivity win on the 9136 and 9166 access-point estates I run into.
Cost context for 9800 estate management
| Item | 2026 Indian pricing (INR / USD) |
|---|---|
| Catalyst 9500-24Y4C-A list price | Rs 14,50,000 to Rs 17,80,000 / about USD 17,400 to USD 21,300 |
| SmartNet 24x7x4 on Catalyst 9300-24T-A | Rs 1,40,000 to Rs 2,00,000/yr |
| Catalyst 9600 with Sup1 + 24x40G line card | Rs 38,00,000 to Rs 52,00,000 / about USD 45,600 to USD 62,300 |
| Catalyst 9300-24T-A list price (Indian distributor) | Rs 4,80,000 to Rs 5,40,000 / about USD 5,750 to USD 6,470 |
| Catalyst 9800-CL throughput licence 1G to 10G upgrade | Rs 2,40,000 / about USD 2,880 one-time + Rs 48,000/yr DNA Advantage |
| Catalyst 9800-L-F-K9 hardware WLC | Rs 3,80,000 / about USD 4,560 hardware + Rs 1.2L/yr SmartNet Premium |
| Redington / Ingram Micro AVDP discount band | Typically 22% to 36% off list depending on volume + cert level |
| AnyConnect Plus licence per user (annual) | Rs 4,200 / about USD 50 per user per year |
The 9800-CL is the SaaS-style WLC: virtualised on ESXi or KVM, throughput-licensed in tiers. A customer who deploys a 9800-CL without an active throughput licence will see the data plane silently cap at 50 Mbps regardless of how many APs are joined. The symptom looks like a slow wireless network. The fix is to land the right throughput entitlement, or move to a hardware WLC if SaaS-licence pricing does not suit.
A second 9800 break-fix from this year
There was a January call from a 130-seat clinic in Koramangala where their wireless had gone soft. Clients were associating but never getting an IP. The team had already power-cycled the WLC twice and was about to factory-reset it. I logged into the 9800-CL at 14:20 IST, saw the wncd process churning, ran one configure-replace from a saved running-config, and clients started taking DHCPv4 leases inside ninety seconds. The runbook entry that came out of that one: always check show wireless stats client delete reasons before assuming anything about client behaviour. The reason-codes table is gospel.
Three patterns on second-opinion 9800 calls. One: the previous engineer rebooted the WLC and called it fixed without ever pulling the wncd crash signature. Two: the previous engineer reduced the AP capacity by limiting clients-per-radio without ever fixing the underlying control-plane fault. Three: the previous engineer escalated to TAC without show tech-support wireless and burned a TAC bridge on data collection that the on-site engineer could have done in five minutes.
Verification before I hand back
- WLC CPU stable below 50% for 30 minutes.
- Client count steady or growing, not dropping, for 30 minutes.
- Client-delete reason codes match expectations. Mostly idle-timeout in a healthy estate. Any IP_LEARN_TIMEOUT or DOT1X_TIMEOUT after the fix is a re-investigation trigger.
- End-to-end test from a real client. Browse, voice call (if WebEx / Teams calling is in scope), file copy. The WLC stats look clean and the user experience is good.
Catalyst 9800 quirks I have learned the hard way
- 9800-CL throughput licence quirk. Without an active throughput entitlement the data plane caps at 50 Mbps aggregate.
- wncd process churn under high client churn. If client associate / deauth churn exceeds a threshold (around 4-6 events per second per controller), the wncd process can spike and intermittently crash. The fix is usually a code train upgrade plus disabling one of the legacy roaming features.
- RRM channel-change loop on overlapping 9800 deployments. If two RRM domains overlap with different RF profile assumptions, channel-change events can ping-pong. The fix is to consolidate the RRM domains or to lock channels manually on the boundary APs.
- IOS XE 17.7 ARP throttling caveat (Cupertino train). Affects certain high-density 9800 deployments. Workaround: tune the ARP throttle bucket size. Fix train: 17.9.4a or later.
- Apple OUI legacy-roaming feature defaults. Some Apple roaming heuristics interact poorly with the 9800 default config and cause client-side stickiness. Disabling the specific feature in the policy profile is the standard workaround.
9800 deployment realities in India
Three things that change 9800 deployments in India.
RF environment density. Indian office buildings (Bagmane Tech Park, RMZ Ecospace, DLF Cyber City) typically have 25 to 60 percent higher 2.4 GHz noise floor than the lab assumption. Plan AP density for 5 GHz primary and 2.4 GHz as a fallback only, even on a 9166 estate.
Power delivery to APs. POE+ (30 W) is the right floor for any 802.11ax AP. PoE on a Catalyst 9300-24P-A is fine. PoE on older non-9300 switches with less than 30 W per port is a slow-onset failure that does not show until peak hours.
Licensing through GeM tenders. 9800 licence + SmartNet renewals through GeM run 4 to 8 weeks lead time. Build the renewal at least 12 weeks ahead of expiry to avoid licence gaps that drop AP join.
Runbook handoff for 9800 faults
The four lines I always leave in the runbook. One: the exact wireless syslog string + timestamp + client MAC (if client-specific). Two: the CLI sequence that diagnosed it. Three: the config change or code train upgrade that fixed it. Four: the rollback path, pre-change config on bootflash, the exact configure replace incantation, plus the previous IOS XE image filename in case a code rollback is also needed.
The customer-cost framing for wireless: the cost is not the WLC. It is the productivity loss across whatever percentage of the workforce is on wireless during the outage window. A 200-seat SMB losing wireless for 45 minutes is 150 user-hours of lost productivity. Framing the cost that way is what stops the next on-call from choosing the quick reboot instead of the real fix.
Related fixes
Related guides worth a look while you sort this one out:
- AnyConnect Secure Client Catalyst 9800 WLC clients stuck in IP learn: Fix
- ASR 1000 Catalyst 9800 WLC clients stuck in IP learn: Fix
- Catalyst 8300/8500 Catalyst 9800 WLC clients stuck in IP learn: Fix
- Catalyst 9200 Catalyst 9800 WLC Clients Stuck In Ip Learn: Fix
- Catalyst 9300 Catalyst 9800 WLC clients stuck in IP learn: Fix
- Catalyst 9400 Catalyst 9800 WLC clients stuck in IP learn: Fix