hash mismatch fixed-output on NixOS. what causes it and how to fix
| OS / Distro | NixOS |
|---|---|
| Category | Operating Systems |
| Guide type | Procedure |
| Skill level | Intermediate to advanced |
| Time | 15 - 60 minutes including verification |
Running into hash mismatch fixed-output on NixOS, what causes it and how to fix on NixOS is one of the more searched issues on distro forums and Unix StackExchange in the last 12 months. Here is what actually moves the needle when the official OS documentation is too generic.
What hash mismatch fixed-output on nixos, what causes it and how to fix actually involves on NixOS
The hash mismatch fixed-output error on NixOS typically surfaces with the message "error: hash mismatch in fixed-output derivation". The exact code or signature line is what you grep for in the distro forum, ServerFault, or Unix StackExchange, not the human-readable sentence next to it.
On NixOS this most often comes from one of three causes: a configuration file or unit override that drifted, a missing package or kernel module, or a resource limit (disk, RAM, file handles, inodes). The fix path differs by which.
The rest of this page is the structured fix path. Start with diagnose, then remediation, then the automation options so you do not have to do this by hand the next time it surfaces. Verify and safety sections at the end are the discipline that keeps the fix from regressing in production.
Diagnose first, fix second
Start by capturing the exact error string verbatim. Many UIs and installers truncate messages in popups, but the system log keeps the full record. Use journalctl -xe on systemd distros, logread on OpenWrt, dmesg for kernel messages, /var/log/messages on RHEL family, /var/log/syslog on Debian family. For ESXi use vmkernel.log in the DCUI. For Solaris use fmadm faulty. For AIX use errpt -a. The exact error code (for example NO_PUBKEY, 0x803fb112, NodeCreationFailure, NW-2-5) is the thing you grep for in forums and Stack Exchange, not the human-readable sentence next to it.
Pull the kernel ring buffer with dmesg --since '5 minutes ago' for hardware-level events, and journalctl --since '5 minutes ago' --no-pager for the systemd timeline of the same window. Cross-reference them. Most boot, network, and storage issues on {family} leave a signature in both at the same wall-clock timestamp.
Look at process state and resource pressure before blaming the application. top, htop, iotop, vmstat 1 5, and iostat -xz 1 answer the four questions every Linux incident needs: CPU saturated, memory exhausted, disk I/O bottlenecked, or context-switch storm. About a quarter of {family} 'service is broken' tickets turn out to be 'host is out of RAM and OOM killer fired'.
Solution-focused remediation path
When the fix involves a destructive operation (rm of a config file, dropping an LV, rewriting a partition table, replacing a kernel package), do it during a maintenance window with at least one teammate watching. Snapshot first if the filesystem supports it (Btrfs, ZFS, LVM thin). Document the rollback path before you start, not during the incident. Run script /tmp/incident.log first to capture the entire session.
For boot issues, the right primitive is the rescue console. UEFI dropdown to the firmware setup, boot from the install ISO, mount the root filesystem, and chroot into it. Once chrooted you can reinstall the bootloader (grub-install + update-grub on Debian family, grub2-install + grub2-mkconfig on RHEL family, bootctl install for systemd-boot), regenerate initramfs (update-initramfs -u -k all, dracut --force --regenerate-all, mkinitcpio -P), and reset the root password (passwd).
Most NixOS failures fall into one of three buckets: configuration drift (a setting changed and nobody documented it), dependency gap (a package, kernel module, or library is missing or wrong version), or resource exhaustion (disk, memory, file handles, or inodes). Triage in that order. It covers around 80 percent of real-world cases. If the failure does not fit any of the three, it is likely an upstream regression worth tracking against the distro bug tracker.
Automate this fix so you do not do it twice
Codify the fix as a systemd timer or cron job for unattended remediation
For workflows that need to run unattended (clear a stuck cache, rotate logs, fail over a service, rebuild an index) a systemd timer or a cron job is the right place. Timers can fire on boot, on schedule, or after a dependency unit reaches an active state. systemctl list-timers shows the next-fire time for every active timer. For interactive helper workflows, a wrapper shell script in /usr/local/bin/ documented in MOTD or the team wiki keeps the institutional knowledge accessible.
Automate the fix in shell with systemctl, journalctl, and the package manager
On most Linux and BSD systems the most reliable repair primitives are the built-in CLI tools. systemctl status reveals the current service state, journalctl -u exposes the structured log stream, and systemctl reload or restart applies config changes without a reboot. For package management use the distro tool: apt, dnf, zypper, pacman, pkg, opkg, apk. For hardware and inventory checks the canonical readers are lsblk, lspci, lscpu, dmidecode, and lsmod.
# Template - replace SERVICE with the failing unit name
systemctl status SERVICE --no-pager | head -40
journalctl -u SERVICE -n 100 --no-pager
ss -tlnp | grep -i SERVICE
ls -l /etc/SERVICE/ 2>/dev/null
cat /etc/os-releaseAdd a manual-approval gate with sudo and auditd for risky fixes
For multi-step fixes that include a destructive action (drop a database, delete a snapshot, fail over a cluster, wipe a partition) gate the script behind sudo with an auditd rule that logs every invocation. The audit trail lives in /var/log/audit/audit.log with the invoking UID and GID and the exact command. For change management requiring a second-person sign-off, wrap the destructive step in a configuration-management approval gate such as Ansible Tower or AWX, Puppet Enterprise, or Salt Master ACL.
Common pitfalls and what to watch for
The most common pitfall when fixing this on NixOS is treating it as a one-off rather than as a recurring class of incident. The same misconfiguration tends to happen again after a kernel upgrade, a major distro version bump, or a fleet rollout unless the fix is codified. Add an Ansible role, a Puppet manifest, a SaltStack state, or a Cloud-init drop-in that prevents the same misconfig from being reintroduced. Documentation alone does not survive team turnover.
Another common trap: confirming the fix on a single host and assuming the fleet is healthy. Loop your check across every node, container, and VM that could exhibit the same symptom. If you cannot enumerate the affected scope without a script, you do not yet understand the scope.
Verify the fix worked
- Reproduce the original symptom path. If it still surfaces on any host, container, or VM in the fleet, you have not fixed it.
- Watch for 24 to 48 hours.
journalctl --since '24 hours ago' -u <service> -p errand Prometheus query history can mask issues with cached health for 6 to 12 hours, especially for slow-burn memory leaks and disk-fill regressions. - Run a smoke test under realistic load. Happy-path tests miss race conditions, file-descriptor leaks, and cgroup limits.
- Capture the new state in a runbook so the next person on call does not have to rediscover this. Push it to Confluence or your team wiki, not into Slack.
- If the fix involved a permission or security change, run a CIS Benchmark or DISA STIG audit one more time to confirm you did not open a separate hole while closing this one.
Safety, rollback, blast radius
- Test in a non-production VM, container, or namespace if your environment supports it. The cost of one disposable VM is cheaper than one rollback meeting.
- Export the existing config before changing it. Most NixOS services support
--print-defaults,systemctl show, or a documented config-dump command. Capture that to source control before you start. - Know your rollback path. Some NixOS operations are one-way (irreversible filesystem upgrade like ext3 to ext4 inline, kernel ABI change, removal of an LVM physical volume). Confirm reversibility on the official OS documentation before you commit.
- Be aware of cross-service impact. A change to PAM ripples to every service using it. A change to /etc/resolv.conf affects every name lookup. A change to systemd default.target affects every reboot.
- Maintenance window discipline: if the change touches DNS, certificate rotation, kernel upgrade, or anything that emits TLS handshakes, line up a window with stakeholder notification, not a heroic mid-day swap.
FAQ
etckeeper commit, cp file file.bak.$(date +%F), or a Btrfs/ZFS snapshot), then commit it before you change anything. A few operations are one-way (in-place filesystem conversion, partition table rewrite, kernel ABI bump). Check the distro release notes for the specific operation before you commit.systemctl list-dependencies and lsof to enumerate consumers before changing a shared service or configuration file.man <command> on the host, or the upstream project documentation - those almost always still work.sosreport (RHEL family) or supportconfig (SUSE), and your reproduction steps. The distro forum is the no-cost public alternative - search there first; 80 percent of common NixOS issues already have a working answer marked as solved.References
- Official documentation for NixOS
- Distro forums and community Q&A (Ubuntu Discourse, Fedora Discussion, Arch BBS, openSUSE Forum, Reddit r/linux + distro subreddits, ServerFault, Unix StackExchange)
- Vendor status pages and release-notes feeds
- CIS Benchmarks and DISA STIG hardening guides for NixOS
Related fixes
Related guides worth a look while you sort this one out:
- APT_HASH_SUM_MISMATCH on Ubuntu. what causes it and how to fix
- USE flag mismatch on Gentoo Linux: what causes it and how to fix
- cached failure of attribute on NixOS: what causes it and how to fix
- sysupgrade returns no output dead on OpenWrt. what causes it and how to fix
- etcd cluster ID mismatch on Talos Linux, what causes it and how to fix
- EVC mode mismatch on VMware ESXi: what causes it and how to fix