MikroTik RouterOS firewall (built-in on all routers): How to recover from a corrupted image during upgrade

Topology deep dive: where this bites a Tier-2 WISP

Most of the MikroTik gear I run sits in small-town ISP backhaul: a CCR or RB-series box in a roadside cabinet feeding a cluster of access points across a Tier-2 town. The uplink is usually a leased fibre from a regional carrier (sometimes a BSNL or Railtel pipe, sometimes a local last-mile reseller), and the MikroTik is the demarcation between my network and the subscriber pool. When something breaks here, it does not break for one customer. It breaks for the whole sector, and the WhatsApp group lights up before I have even logged in.

The thing people miss about RouterOS is that the firewall, the routing table, and the bridge all live in one box on cheap hardware. There is no separate supervisor and line card to blame. When I touch this on a CCR2004 at a tower site, I keep a serial cable in the bag because the Ethernet management can drop the moment a config goes sideways. The cabinet has one 4G failover SIM on an LtAP, and that out-of-band path is the only reason I have not driven 60 km at 2 a.m. more than once.

On the switching and routing side, the MikroTik usually wears two hats: a routed core for the subscriber subnets and a layer-2 bridge for the management trunk. When a port, a VLAN, or a route misbehaves, I always check which hat the traffic is wearing first, because the fix for a bridge problem and a RIB problem are not the same thing on RouterOS even when the symptom looks identical.

Configuration walkthrough I actually use

RouterOS upgrades are two packages stacked: the routeros system package and the separate RouterBOOT firmware. People upgrade the first and forget the second, then wonder why a feature is missing. My controlled-upgrade pattern always does both, with a verified download.

# Check what is installed and what bootloader is running
/system package print
/system routerboard print

# Stage the upgrade package, verify, then reboot
/system package update set channel=stable
/system package update check-for-updates
/system package update download
# After reboot, push the matching RouterBOOT
/system routerboard upgrade
/system reboot

I keep the previous npk file on a USB stick or on the box itself so a downgrade is one /system package downgrade away. On a 200-subscriber tower I never upgrade during business hours; the window is 2 a.m. to 4 a.m. with the failover SIM tested first.

Troubleshooting commands by platform

RouterOS is the platform here, but a backhaul link almost always has another vendor on the far end. When I am proving where a fault sits, I run the equivalent command on both sides of the link so the carrier cannot bounce the ticket back to me.

One field note: RouterOS /tool sniffer quick is gold for proving a problem to a carrier. I capture on the uplink, filter for the subscriber subnet, and screenshot the output for the ticket. A regional carrier NOC argues with a description; they do not argue with a packet trace timestamped from their own handoff.

India compliance and deployment notes

If you run a licensed ISP in India, a few rules touch this box directly. The DoT licence conditions and the CERT-In directions both expect time-synced, retained logs. RouterOS NTP plus remote syslog covers most of it, and I keep at least 180 days of logs off-box because that is the retention floor I work to. Set the clock to IST and lock NTP to a trusted source before you trust any timestamp in a dispute.

/system clock set time-zone-name=Asia/Kolkata
/system ntp client set enabled=yes
/system logging action set remote remote=10.20.0.5 remote-port=514
/system logging add topics=info,!debug action=remote

On the procurement side, this gear usually lands through a GeM tender or a distributor like Redington or a regional reseller. A CCR2004 runs roughly INR 55,000 to 70,000 depending on the USD-INR rate the week it ships; an RB5009 is closer to INR 18,000 to 22,000. There is no SmartNet equivalent on MikroTik, so my AMC budget goes into a shelf of cold spares rather than a support contract. For a 10-site WISP I keep two spare CCRs and a box of bidi optics; that is cheaper than downtime and far faster than an RMA. Under the DPDP framework, the subscriber data that transits this box (PPPoE usernames, session logs) is personal data, so I keep the syslog server itself access-controlled and inside the NOC, not on a cloud bucket with a guessable name.

A real deployment I did

One outage I will not forget: a whole access VLAN went dark across a Tier-2 town deployment, and the symptom looked exactly like this one. Subscribers could associate but not pass traffic. Running /interface bridge host print twice showed a MAC bouncing between two ports, a customer had looped a cheap unmanaged switch back into two of our access ports. I enabled loop protect on the access ports, killed the offending link, and the sector recovered in under a minute. Now every access port on every tower has loop protection on by default, because one careless subscriber should never take down a sector.

Extended FAQ for field operators

Can I do this remotely without a tower visit? Usually yes, if you have an out-of-band path. I always keep a 4G failover or a serial-over-IP console at unmanned sites, and I run risky changes behind RouterOS safe mode so a mistake reverts itself instead of stranding me.

How does this differ on a CCR versus a hAP or RB5009? The CLI is identical across RouterOS, but the bigger boxes have hardware offload and real SFP cages, so some commands show extra detail. The small boxes are CPU-bound, so the same fix can behave differently under load. Test on the actual model in your rack, not a different one on the bench.

What do I tell the carrier when I open a ticket? Give them a timestamp in IST, your interface counters, and a packet capture from the handoff. Regional carrier NOCs (BSNL, Railtel, or whoever owns your last mile) move faster when you hand them evidence rather than a story.

How long does this take in practice? A planned change inside a maintenance window is 15 to 30 minutes including the rollback safety net. A genuine hardware failure is bounded by how fast I can get a cold spare into the cabinet, which is why I budget for spares instead of a support contract that does not exist for this platform.