Upgrade Failure

Nvidia (Mellanox) SN2700: How to rollback to the previous image after a failed upgrade

Q: Where can I find the Nvidia (Mellanox) official documentation?

https://docs.nvidia.com/networking/ — search the product family + feature name.

By Sai Kiran Pandrala · reviewed by Sai Kiran Pandrala, Editor Last verified: 2026-05-30

⚡ At a glance

Vendor	Nvidia (Mellanox)
Operating system	Cumulus Linux / NVOS / SONiC
Category	Upgrade Failure
Skill level	Intermediate to advanced
DIY-able?	Yes with CLI access; some scenarios need Nvidia Enterprise Support + RMA.

Every Nvidia (Mellanox) upgrade I have shipped to production was paired with a written rollback. Cumulus Linux / NVOS / SONiC on the SN2700 family makes rollback cheap if you saved the previous image and config, and expensive if you did not.

The onie-nos-install /home/cumulus/cumulus-linux-5.x.img command on Cumulus Linux / NVOS / SONiC is straightforward once you have the right artifact staged. The trap is mismatched hardware-to-image: always cross-reference platform IDs from `nv show system` against the image name.

I file every upgrade run under a change number and attach the before/after `nv show system` and tech-support bundle. Nvidia Enterprise Support appreciates it; future me appreciates it even more.

What this guide covers

Rollback to the previous image after a failed upgrade on a Nvidia (Mellanox) SN2700 (Cumulus Linux / NVOS / SONiC).

Step-by-step

Confirm there's a previous image still on flash.
Set the boot variable to that previous image.
Reboot.
Verify the version is back to the prior release.
Investigate the upgrade failure separately, do not re-attempt without root cause.

CLI / commands

# Boot recovery prompt: ONIE:/ #

# Verify image
nv show system

# Upgrade
onie-nos-install /home/cumulus/cumulus-linux-5.x.img

# Save / commit
nv config save

# Rollback
nv config patch (revert to last apply)

Recovery options

Boot loader recovery (ONIE:/ #)
Rollback to the previous image with nv config patch (revert to last apply)
Force failover to a known-good standby (HA platforms)

Frequently asked questions

Will this work on my specific Cumulus Linux / NVOS / SONiC version?

The procedure reflects current Cumulus Linux / NVOS / SONiC behaviour. Older releases may need minor syntax adjustments. use the CLI help (? or tab-completion) to verify.

Should I open a Nvidia Enterprise Support case immediately?

Open one if you suspect hardware failure or the symptom persists after a maintenance-window reload. Make sure your support entitlement is active first.

Where can I find the Nvidia (Mellanox) official documentation?

https://docs.nvidia.com/networking/, search the product family + feature name.

Is this procedure safe in production?

Test in a lab or maintenance window first. Capture pre-change state so you can roll back.

All Nvidia (Mellanox) fix guides → /nvidia/
All vendor guides → /vendors/

Related guides worth a look while you sort this one out:

References

Nvidia (Mellanox) support portal: https://enterprise-support.nvidia.com/
Nvidia (Mellanox) knowledge base: https://docs.nvidia.com/networking/
Nvidia (Mellanox) security advisories: https://www.nvidia.com/en-us/security/
Open a case: https://enterprise-support.nvidia.com/s/createcase

Reference material, not professional advice. Validate against your specific Cumulus Linux / NVOS / SONiC version and test in a non-production environment before applying.

Why this matters for your day-to-day

A Nvidia device that's misbehaving costs more than the fix itself: lost productivity, missed calls, security risk, even safety risk in some categories. Treating the symptom quickly with a documented procedure is cheaper than letting it persist. The steps above are written to get you back to working in under an hour where possible, and to flag clearly when escalation is the right call.

Safety + preconditions

Before any work on a Nvidia device:

Unplug from mains for any internal-access procedure.
Discharge stored energy (capacitors in PSUs, residual battery charge) per manufacturer guidance.
Use ESD-safe handling for boards and modules: no carpet, no wool sleeves.
Avoid moisture; never apply liquids near vents or connectors.
If you smell smoke, see scorch marks, or feel uneven heat, stop and escalate.

How to confirm it's actually fixed

On a Nvidia device, the test is rarely "reboot and see". Use this list:

Active reproduction: trigger the original failure path on purpose.
Indirect reproduction: do an activity that would expose the same subsystem.
Status indicator review: every LED / display / app status should be green.
24-hour soak: leave the device under normal load overnight; check the next morning.
Telemetry check: review the device or app's diagnostic log for new error entries.

Escalation guide

For a Nvidia device, the right escalation depends on impact:

Cosmetic / minor: log a ticket via the Nvidia app or web portal. Response 1-3 business days.
Mid-impact: phone support. Have your serial number ready.
Critical (production down, safety issue): in-person dealer / TAC visit. Bring proof of purchase.
Out of warranty: third-party repair shop with manufacturer-certified technicians.