Every server I ran for my business was a Ship of Theseus.

It started as a clean install. Then it got a package, a config tweak, a script from a forum post, a “temporary” fix, an upgrade that half worked, and another fix for the upgrade. After a year, every plank had been replaced, and nobody could tell you exactly what the boat was made of anymore. Including me.

I had my golden image. I set up my server the same way every single time. And it worked. Clients never complained, services ran, and business operations continued. Five nines be damned, I had a system that worked and was reproducible, in my mind. The issue arises when you go to update the system and something downstream that you didn’t anticipate causes everything to come to a halt. Now you have a dead system with no easy way to roll it back. All from a simple sudo apt upgrade.

The problem was that I couldn’t prove what it was, couldn’t rebuild it quickly, and couldn’t undo a change without a lot of hope. For a small business, “rebuild from scratch” isn’t a strategy. It’s a lost week.

So I tried NixOS. This is the technical version of why, how it works under the hood, and where it still hurts.

Recipes

Most of us manage Linux machines with recipes: a sequence of commands that, if everything goes right, leaves the system in the state we wanted.

Here is a perfectly normal recipe for setting up a shell:

#!/bin/bash
# "recipe" to set up my awesome shell.
set -e

echo "--- Installing dependencies ---"
sudo apt-get update
sudo apt-get install -y zsh fzf-IS-BROKEN

echo "--- Installing Oh My Zsh ---"
# This clones the master branch
sh -c "$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)" "" --unattended

echo "--- Cloning a custom plugin ---"
# This also clones the master branch
git clone https://github.com/zsh-users/zsh-autosuggestions.git ~/.oh-my-zsh/custom/plugins/zsh-autosuggestions

echo "--- Copying configuration ---"
# This is a one-time, destructive copy
cp .zshrc ~/

Count the problems:

  • It fails dirty. One bad package name and set -e stops the script halfway. apt-get update already ran. Whatever was installed before the failure stays installed. The system is now in a state nobody designed.
  • It isn’t reproducible. curl | sh and git clone both pull whatever is on master today. Run this next month and you get different code.
  • It has no guarantees. Nothing verifies that what was downloaded is what you expected. You are running a remote script as your user on trust.
  • It destroys history. cp .zshrc ~/ overwrites whatever was there. Running it twice isn’t the same as running it once if anything changed in between.

Scale that up to a web server, a firewall, a file share and a dozen services, and you get configuration drift. The state of the machine is an accident of history. Auditing it is forensics: you SSH in and start running dpkg -l, systemctl list-units and diff against whatever notes you hope are current.

Configuration management tools like Ansible help a lot. They make recipes repeatable and mostly idempotent. But they still converge a mutable system toward a desired state. Anything the playbook doesn’t mention, it doesn’t manage, and the drift lives in exactly those gaps.

Blueprints

NixOS flips the model. Instead of a list of steps, you write one description of the whole system:

{ config, pkgs, ... }:
{
  services.nginx = {
    enable = true;
    virtualHosts."intranet.example.com".root = "/srv/www";
  };

  networking.firewall.allowedTCPPorts = [ 80 ];
}

That isn’t a script. Nothing runs top to bottom. It’s an expression that Nix evaluates into a complete system: every package, every config file in /etc, every systemd unit, the firewall rules, the bootloader entry.

The difference sounds philosophical until you see what it buys you:

Recipe Blueprint
Focus How to get there What the end state is
State Accident of history Defined by the file
Auditing Forensics Reading
Failed step System may be half-changed System untouched

The last row is the one that matters at 2 a.m. To see why, you need to know what Nix does with that file.

How Nix Actually Builds a System

Everything Nix builds lands in /nix/store, under a path like this:

/nix/store/0c8kdd3xl5w7vjc4i4hvyg8j6s5f4ga9-nginx-1.28.0

That hash isn’t a version number. It’s derived from every input that went into the build: the source, the compiler, the build flags, the dependencies and their hashes, all the way down. Change any input and you get a different path.

Three properties fall out of that:

  1. Nothing gets overwritten. A new nginx doesn’t replace the old one. It gets its own path next to it. Store paths are read-only once built.
  2. Different versions coexist. Two services that need different OpenSSL versions each reference their own. Dependency hell mostly stops being a thing, because there is no single shared /usr/lib to fight over.
  3. The system is a closure. Your entire NixOS install is one store path (the system’s top-level derivation) that references everything it needs. Copy that closure to another machine and you have the same system.

When you run nixos-rebuild switch, Nix evaluates your configuration, builds (or downloads) whatever store paths are missing, and only then touches the running system.

The Atomic Switch

Here is the sequence nixos-rebuild switch goes through:

  1. Evaluate configuration.nix into a system derivation.
  2. Build it. Everything new goes into /nix/store. If anything fails here (a typo, a missing option, a package that doesn’t exist), the command stops and nothing on the running system has changed.
  3. Register a new generation. The system profile at /nix/var/nix/profiles/system is a symlink. Nix creates system-N-link pointing at the new build and moves the profile symlink to it. Swapping a symlink is a single atomic filesystem operation.
  4. Update the bootloader with an entry for the new generation.
  5. Activate: point /run/current-system at the new build, update /etc, and restart or reload any systemd units whose definitions changed.

That bad package name that broke the bash recipe halfway? In NixOS it fails at step 2, before anything is touched.

To be honest about the edges: steps 1–4 are atomic. Step 5 isn’t perfectly atomic, because services restart one after another and a service can still fail to start with its new config. But the definition of the system switched all at once, and you know exactly which definition is running.

nixos-rebuild has other modes worth knowing:

  • nixos-rebuild build builds the new system without activating it. Good for checking a change compiles before you commit to it.
  • nixos-rebuild test activates it now but doesn’t add a boot entry. Reboot and you’re back on the previous generation.
  • nixos-rebuild boot adds the boot entry but doesn’t activate until the next reboot.

The Undo Button

Because old generations are never overwritten, rolling back is just pointing the profile symlink at an older one:

# See every generation of the system
sudo nix-env --list-generations --profile /nix/var/nix/profiles/system

# Go back one
sudo nixos-rebuild switch --rollback

That takes seconds, because nothing gets rebuilt. The old generation is already sitting in the store.

The part that sold me is that the generations are also in the boot menu. While putting together a demo, I broke a NixOS VM on purpose: I changed the root filesystem’s UUID in hardware-configuration.nix to one that doesn’t exist and rebuilt. Nix happily built it, because it’s a valid configuration. It just describes a machine that can’t boot. On reboot, stage 1 sat waiting for the disk:

waiting for device /dev/disk/by-uuid/f51e21f2-... to appear.......
Timed out waiting for device /dev/disk/by-uuid/f51e21f2-..., trying to mount anyway.
mount: mounting /dev/disk/by-uuid/f51e21f2-... on /mnt-root/ failed: No such file or directory

An error occurred in stage 1 of the boot process, which must mount the
root filesystem on '/mnt-root' and then start stage 2.

On Debian, that’s a rescue USB, a chroot and an afternoon. On NixOS, I rebooted, picked the previous configuration from the GRUB menu, and was back at the desktop.

One detail that trips people up: picking an older generation in the boot menu boots it this time. The default is still the broken one. Once you’re back in, either fix the config and rebuild, or run nixos-rebuild switch --rollback to make the old generation the default again.

Pinning: Reproducible Means Reproducible

A configuration file alone doesn’t make a system reproducible. It also matters which nixpkgs you evaluated it against. With classic channels, that’s whatever the channel pointed to when you last ran nix-channel --update, which is its own kind of drift.

Flakes fix that with a lock file. A minimal flake.nix for one machine:

{
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-25.05";

  outputs = { nixpkgs, ... }: {
    nixosConfigurations.office-server = nixpkgs.lib.nixosSystem {
      system = "x86_64-linux";
      modules = [ ./configuration.nix ./hardware-configuration.nix ];
    };
  };
}

The first build writes flake.lock, which pins nixpkgs to an exact commit and a hash of its contents. Commit both files to git, and anyone (including you, on replacement hardware, after a disaster) gets the same system:

sudo nixos-rebuild switch --flake .#office-server

Updating becomes a deliberate act: nix flake update, rebuild, test, commit the new lock file. If the update breaks something, the old lock file is in git history and the old generation is in the boot menu.

Flakes are still officially “experimental,” so you have to turn them on:

nix.settings.experimental-features = [ "nix-command" "flakes" ];

In practice, most of the community and tooling already assume them.

The Shell, Done Right

Here’s the same shell setup from the recipe, as part of the system definition:

programs.zsh = {
  enable = true;
  # This adds autosuggestions and completions
  enableCompletion = true;
  autosuggestions.enable = true;

  # This replaces the insecure 'curl | sh' installer
  ohMyZsh = {
    enable = true;
    theme = "robbyrussell";
    plugins = [ "git" ];
  };

  # This replaces 'cp .zshrc ~/' and 'echo "alias ll=..."'
  shellAliases = {
    ll = "ls -la";
  };
};

Every package here comes from the pinned nixpkgs, either built from pinned sources or fetched prebuilt from the signature-checked binary cache. There’s no curl | sh, no unpinned master branch, and no overwritten dotfile. If I don’t like it, I delete the block, rebuild, and it’s gone. Not “uninstalled, plus whatever it left behind.” Gone.

What It Doesn’t Fix

This is where NixOS evangelism usually gets quiet. Rollbacks cover what Nix manages, which is a lot but not everything.

  • Your data doesn’t roll back. /home, ~/.config, /var/lib (where databases live) and /var/log are mutable state outside the store. If a new PostgreSQL version migrates your data on first start, rolling back the package won’t un-migrate the data. You still need real backups.
  • Secrets don’t belong in the store. Everything in /nix/store is world-readable. Put a password in configuration.nix and it ends up readable by every user on the box. Use something like sops-nix or agenix, which keep secrets encrypted in the repo and decrypt them at activation time outside the store.
  • Garbage collection deletes your undo history. Old generations take disk space, and nix-collect-garbage -d removes all of them. Prefer --delete-older-than 30d, and cap boot entries so /boot doesn’t fill up (boot.loader.grub.configurationLimit or boot.loader.systemd-boot.configurationLimit).
  • Firmware and hardware don’t roll back. A BIOS update, a dying disk or a changed network on the other end of the cable are still your problem.

Why This Is a Security Story

I’m presenting this at VetSecCon as a security talk, not a Linux talk, because most of these properties are security controls:

  • Auditing is reading. The configuration in git is the system. A change review is a git diff. You can see what a change will do to the system before activating it:

    nixos-rebuild build --flake .#office-server
    nix store diff-closures /run/current-system ./result
    

    That lists every package that would be added, removed or changed, with version numbers.

  • No undocumented changes survive. If someone hand-edits something Nix manages, the next rebuild puts it back. Drift has nowhere to hide in the parts Nix owns.

  • Recovery is fast. Ransomware, a bad update or a compromised box all end the same way: rebuild the system from the repo on clean hardware and restore data from backups. The system half of that takes minutes, not days.

  • The supply chain is pinned. Inputs are fixed by hash, and binaries from cache.nixos.org are signature-checked. You’re still trusting nixpkgs and its build infrastructure, but you trust a known, specific version of it.

Thorns on the Rose

It isn’t all upside.

  • The Nix language is a real language. It’s lazy and functional, and much more powerful than YAML. Error messages can be baffling. Expect a few evenings of confusion before it clicks.
  • The documentation is uneven. The manual covers the basics. A lot of practical knowledge lives in blog posts, GitHub issues and other people’s configs.
  • The community is brilliant and divided. Flakes vs. no flakes, which module system, which secrets tool. “Nix fixes this” is often true and almost never helpful on its own.

The configuration.nix on my demo VM is 137 lines. Getting a file like that right takes a while. It pays for itself the first time something breaks.

I can say with confidence that moving to NixOS was a good call for my business and what I was doing. Your mileage will absolutely vary. Is it worth a shot? Yes, test in dev, see how it feels.

The Bottom Line

I didn’t switch to NixOS because it’s clever. I switched because I was tired of being anxious every time I changed a production machine. I wanted a blueprint instead of a pile of recipes, a reset button that works even when the machine won’t boot, and the confidence that I could rebuild everything from a git repo.

A five-second fix instead of a five-day one.

Sources and Further Reading