Upgrade Failure

Nvidia (Mellanox) SN3420: How to rollback to the previous image after a failed upgrade

Q: Where can I find the Nvidia (Mellanox) official documentation?

https://docs.nvidia.com/networking/ — search the product family + feature name.

By Sai Kiran Pandrala · reviewed by Sai Kiran Pandrala, Editor Last verified: 2026-05-30

⚡ At a glance

Vendor	Nvidia (Mellanox)
Operating system	Cumulus Linux / NVOS / SONiC
Category	Upgrade Failure
Skill level	Intermediate to advanced
DIY-able?	Yes with CLI access; some scenarios need Nvidia Enterprise Support + RMA.

I have run more Nvidia (Mellanox) upgrades than I can count and the only ones that hurt are the ones where I skipped image-integrity verification. Cumulus Linux / NVOS / SONiC either ships a `verify` step or expects you to checksum the file before onie-nos-install /home/cumulus/cumulus-linux-5.x.img.

On the SN3420 platform the activation phase is where you lose data-plane connectivity. Plan the change window around that window, not the full upgrade duration.

If something goes wrong, the rollback path on Cumulus Linux / NVOS / SONiC is well-trodden, but only if you saved running-config before starting. Do that now, before anything else.

What this guide covers

Rollback to the previous image after a failed upgrade on a Nvidia (Mellanox) SN3420 (Cumulus Linux / NVOS / SONiC).

Step-by-step

Confirm there's a previous image still on flash.
Set the boot variable to that previous image.
Reboot.
Verify the version is back to the prior release.
Investigate the upgrade failure separately. do not re-attempt without root cause.

CLI / commands

# Boot recovery prompt: ONIE:/ #

# Verify image
nv show system

# Upgrade
onie-nos-install /home/cumulus/cumulus-linux-5.x.img

# Save / commit
nv config save

# Rollback
nv config patch (revert to last apply)

Recovery options

Boot loader recovery (ONIE:/ #)
Rollback to the previous image with nv config patch (revert to last apply)
Force failover to a known-good standby (HA platforms)

Frequently asked questions

Will this work on my specific Cumulus Linux / NVOS / SONiC version?

The procedure reflects current Cumulus Linux / NVOS / SONiC behaviour. Older releases may need minor syntax adjustments, use the CLI help (? or tab-completion) to verify.

Should I open a Nvidia Enterprise Support case immediately?

Open one if you suspect hardware failure or the symptom persists after a maintenance-window reload. Make sure your support entitlement is active first.

Where can I find the Nvidia (Mellanox) official documentation?

https://docs.nvidia.com/networking/: search the product family + feature name.

Is this procedure safe in production?

Test in a lab or maintenance window first. Capture pre-change state so you can roll back.

All Nvidia (Mellanox) fix guides → /nvidia/
All vendor guides → /vendors/

Related guides worth a look while you sort this one out:

References

Nvidia (Mellanox) support portal: https://enterprise-support.nvidia.com/
Nvidia (Mellanox) knowledge base: https://docs.nvidia.com/networking/
Nvidia (Mellanox) security advisories: https://www.nvidia.com/en-us/security/
Open a case: https://enterprise-support.nvidia.com/s/createcase

Reference material, not professional advice. Validate against your specific Cumulus Linux / NVOS / SONiC version and test in a non-production environment before applying.

Common patterns we see

When this symptom shows up on a Nvidia device, three patterns repeat:

1. Recent firmware update changed behavior, the symptom started within a week of an OTA push. Rollback or wait for the hotfix. 2. Environmental trigger. temperature, humidity, line voltage, network changes. Look at what changed in the environment. 3. Cumulative wear, components like batteries, gaskets, fans degrade over time. Replace the consumable rather than chasing a software fix.

Knowing which pattern applies saves time on the wrong fix.

Safety + preconditions

Before any work on a Nvidia device:

Unplug from mains for any internal-access procedure.
Discharge stored energy (capacitors in PSUs, residual battery charge) per manufacturer guidance.
Use ESD-safe handling for boards and modules: no carpet, no wool sleeves.
Avoid moisture; never apply liquids near vents or connectors.
If you smell smoke, see scorch marks, or feel uneven heat, stop and escalate.

Quick verification

Before you walk away from a Nvidia device fix, run through:

1. Reproduce the original trigger, does the issue reappear? 2. Check the device's status / health screen for any new alerts. 3. Confirm paired devices (app, hub, controller) reconnected. 4. Save / commit any configuration changes per the device's normal workflow. 5. Note the change in your maintenance log with date + firmware version.

Escalation guide

For a Nvidia device, the right escalation depends on impact:

Cosmetic / minor: log a ticket via the Nvidia app or web portal. Response 1-3 business days.
Mid-impact: phone support. Have your serial number ready.
Critical (production down, safety issue): in-person dealer / TAC visit. Bring proof of purchase.
Out of warranty: third-party repair shop with manufacturer-certified technicians.