Every server I ran for my business was a Ship of Theseus.
It started as a clean install. Then it got a package, a config tweak, a script from a forum post, a “temporary” fix, an upgrade that half worked, and another fix for the upgrade. After a year, every plank had been replaced, and nobody could tell you exactly what the boat was made of anymore. Including me.
I had my golden image. I set up my server the same way every single time. And it worked. Clients never complained, services ran, and business operations continued. Five nines be damned, I had a system that worked and was reproducible, in my mind. The issue arises when you go to update the system and something downstream that you didn’t anticipate causes everything to come to a halt. Now you have a dead system with no easy way to roll it back. All from a simple sudo apt upgrade.
The problem was that I couldn’t prove what it was, couldn’t rebuild it quickly, and couldn’t undo a change without a lot of hope. For a small business, “rebuild from scratch” isn’t a strategy. It’s a lost week.
So I tried NixOS. This is the technical version of why, how it works under the hood, and where it still hurts.
Recipes
Most of us manage Linux machines with recipes: a sequence of commands that, if everything goes right, leaves the system in the state we wanted.
Here is a perfectly normal recipe for setting up a shell:
#!/bin/bash
# "recipe" to set up my awesome shell.
set -e
echo "--- Installing dependencies ---"
sudo apt-get update
sudo apt-get install -y zsh fzf-IS-BROKEN
echo "--- Installing Oh My Zsh ---"
# This clones the master branch
sh -c "$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)" "" --unattended
echo "--- Cloning a custom plugin ---"
# This also clones the master branch
git clone https://github.com/zsh-users/zsh-autosuggestions.git ~/.oh-my-zsh/custom/plugins/zsh-autosuggestions
echo "--- Copying configuration ---"
# This is a one-time, destructive copy
cp .zshrc ~/
Count the problems:
- It fails dirty. One bad package name and
set -estops the script halfway.apt-get updatealready ran. Whatever was installed before the failure stays installed. The system is now in a state nobody designed. - It isn’t reproducible.
curl | shandgit cloneboth pull whatever is onmastertoday. Run this next month and you get different code. - It has no guarantees. Nothing verifies that what was downloaded is what you expected. You are running a remote script as your user on trust.
- It destroys history.
cp .zshrc ~/overwrites whatever was there. Running it twice isn’t the same as running it once if anything changed in between.
Scale that up to a web server, a firewall, a file share and a dozen services,
and you get configuration drift. The state of the machine is an accident of
history. Auditing it is forensics: you SSH in and start running dpkg -l,
systemctl list-units and diff against whatever notes you hope are current.
Configuration management tools like Ansible help a lot. They make recipes repeatable and mostly idempotent. But they still converge a mutable system toward a desired state. Anything the playbook doesn’t mention, it doesn’t manage, and the drift lives in exactly those gaps.
Blueprints
NixOS flips the model. Instead of a list of steps, you write one description of the whole system:
{ config, pkgs, ... }:
{
services.nginx = {
enable = true;
virtualHosts."intranet.example.com".root = "/srv/www";
};
networking.firewall.allowedTCPPorts = [ 80 ];
}
That isn’t a script. Nothing runs top to bottom. It’s an expression that Nix
evaluates into a complete system: every package, every config file in
/etc, every systemd unit, the firewall rules, the bootloader entry.
The difference sounds philosophical until you see what it buys you:
| Recipe | Blueprint | |
|---|---|---|
| Focus | How to get there | What the end state is |
| State | Accident of history | Defined by the file |
| Auditing | Forensics | Reading |
| Failed step | System may be half-changed | System untouched |
The last row is the one that matters at 2 a.m. To see why, you need to know what Nix does with that file.
How Nix Actually Builds a System
Everything Nix builds lands in /nix/store, under a path like this:
/nix/store/0c8kdd3xl5w7vjc4i4hvyg8j6s5f4ga9-nginx-1.28.0
That hash isn’t a version number. It’s derived from every input that went into the build: the source, the compiler, the build flags, the dependencies and their hashes, all the way down. Change any input and you get a different path.
Three properties fall out of that:
- Nothing gets overwritten. A new nginx doesn’t replace the old one. It gets its own path next to it. Store paths are read-only once built.
- Different versions coexist. Two services that need different OpenSSL
versions each reference their own. Dependency hell mostly stops being a
thing, because there is no single shared
/usr/libto fight over. - The system is a closure. Your entire NixOS install is one store path (the system’s top-level derivation) that references everything it needs. Copy that closure to another machine and you have the same system.
When you run nixos-rebuild switch, Nix evaluates your configuration, builds
(or downloads) whatever store paths are missing, and only then touches the
running system.
The Atomic Switch
Here is the sequence nixos-rebuild switch goes through:
- Evaluate
configuration.nixinto a system derivation. - Build it. Everything new goes into
/nix/store. If anything fails here (a typo, a missing option, a package that doesn’t exist), the command stops and nothing on the running system has changed. - Register a new generation. The system profile at
/nix/var/nix/profiles/systemis a symlink. Nix createssystem-N-linkpointing at the new build and moves the profile symlink to it. Swapping a symlink is a single atomic filesystem operation. - Update the bootloader with an entry for the new generation.
- Activate: point
/run/current-systemat the new build, update/etc, and restart or reload any systemd units whose definitions changed.
That bad package name that broke the bash recipe halfway? In NixOS it fails at step 2, before anything is touched.
To be honest about the edges: steps 1–4 are atomic. Step 5 isn’t perfectly atomic, because services restart one after another and a service can still fail to start with its new config. But the definition of the system switched all at once, and you know exactly which definition is running.
nixos-rebuild has other modes worth knowing:
nixos-rebuild buildbuilds the new system without activating it. Good for checking a change compiles before you commit to it.nixos-rebuild testactivates it now but doesn’t add a boot entry. Reboot and you’re back on the previous generation.nixos-rebuild bootadds the boot entry but doesn’t activate until the next reboot.
The Undo Button
Because old generations are never overwritten, rolling back is just pointing the profile symlink at an older one:
# See every generation of the system
sudo nix-env --list-generations --profile /nix/var/nix/profiles/system
# Go back one
sudo nixos-rebuild switch --rollback
That takes seconds, because nothing gets rebuilt. The old generation is already sitting in the store.
The part that sold me is that the generations are also in the boot menu. While
putting together a demo, I broke a NixOS VM on purpose: I changed the root filesystem’s UUID in
hardware-configuration.nix to one that doesn’t exist and rebuilt. Nix happily
built it, because it’s a valid configuration. It just describes a machine that
can’t boot. On reboot, stage 1 sat waiting for the disk:
waiting for device /dev/disk/by-uuid/f51e21f2-... to appear.......
Timed out waiting for device /dev/disk/by-uuid/f51e21f2-..., trying to mount anyway.
mount: mounting /dev/disk/by-uuid/f51e21f2-... on /mnt-root/ failed: No such file or directory
An error occurred in stage 1 of the boot process, which must mount the
root filesystem on '/mnt-root' and then start stage 2.
On Debian, that’s a rescue USB, a chroot and an afternoon. On NixOS, I rebooted, picked the previous configuration from the GRUB menu, and was back at the desktop.
One detail that trips people up: picking an older generation in the boot menu
boots it this time. The default is still the broken one. Once you’re back in,
either fix the config and rebuild, or run nixos-rebuild switch --rollback to
make the old generation the default again.
Pinning: Reproducible Means Reproducible
A configuration file alone doesn’t make a system reproducible. It also matters
which nixpkgs you evaluated it against. With classic channels, that’s
whatever the channel pointed to when you last ran nix-channel --update, which
is its own kind of drift.
Flakes fix that with a lock file. A minimal flake.nix for one machine:
{
inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-25.05";
outputs = { nixpkgs, ... }: {
nixosConfigurations.office-server = nixpkgs.lib.nixosSystem {
system = "x86_64-linux";
modules = [ ./configuration.nix ./hardware-configuration.nix ];
};
};
}
The first build writes flake.lock, which pins nixpkgs to an exact commit and
a hash of its contents. Commit both files to git, and anyone (including you, on
replacement hardware, after a disaster) gets the same system:
sudo nixos-rebuild switch --flake .#office-server
Updating becomes a deliberate act: nix flake update, rebuild, test, commit
the new lock file. If the update breaks something, the old lock file is in git
history and the old generation is in the boot menu.
Flakes are still officially “experimental,” so you have to turn them on:
nix.settings.experimental-features = [ "nix-command" "flakes" ];
In practice, most of the community and tooling already assume them.
The Shell, Done Right
Here’s the same shell setup from the recipe, as part of the system definition:
programs.zsh = {
enable = true;
# This adds autosuggestions and completions
enableCompletion = true;
autosuggestions.enable = true;
# This replaces the insecure 'curl | sh' installer
ohMyZsh = {
enable = true;
theme = "robbyrussell";
plugins = [ "git" ];
};
# This replaces 'cp .zshrc ~/' and 'echo "alias ll=..."'
shellAliases = {
ll = "ls -la";
};
};
Every package here comes from the pinned nixpkgs, either built from pinned
sources or fetched prebuilt from the signature-checked binary cache. There’s no curl | sh, no unpinned master branch, and no overwritten
dotfile. If I don’t like it, I delete the block, rebuild, and it’s gone. Not
“uninstalled, plus whatever it left behind.” Gone.
What It Doesn’t Fix
This is where NixOS evangelism usually gets quiet. Rollbacks cover what Nix manages, which is a lot but not everything.
- Your data doesn’t roll back.
/home,~/.config,/var/lib(where databases live) and/var/logare mutable state outside the store. If a new PostgreSQL version migrates your data on first start, rolling back the package won’t un-migrate the data. You still need real backups. - Secrets don’t belong in the store. Everything in
/nix/storeis world-readable. Put a password inconfiguration.nixand it ends up readable by every user on the box. Use something like sops-nix or agenix, which keep secrets encrypted in the repo and decrypt them at activation time outside the store. - Garbage collection deletes your undo history. Old generations take disk
space, and
nix-collect-garbage -dremoves all of them. Prefer--delete-older-than 30d, and cap boot entries so/bootdoesn’t fill up (boot.loader.grub.configurationLimitorboot.loader.systemd-boot.configurationLimit). - Firmware and hardware don’t roll back. A BIOS update, a dying disk or a changed network on the other end of the cable are still your problem.
Why This Is a Security Story
I’m presenting this at VetSecCon as a security talk, not a Linux talk, because most of these properties are security controls:
-
Auditing is reading. The configuration in git is the system. A change review is a
git diff. You can see what a change will do to the system before activating it:nixos-rebuild build --flake .#office-server nix store diff-closures /run/current-system ./resultThat lists every package that would be added, removed or changed, with version numbers.
-
No undocumented changes survive. If someone hand-edits something Nix manages, the next rebuild puts it back. Drift has nowhere to hide in the parts Nix owns.
-
Recovery is fast. Ransomware, a bad update or a compromised box all end the same way: rebuild the system from the repo on clean hardware and restore data from backups. The system half of that takes minutes, not days.
-
The supply chain is pinned. Inputs are fixed by hash, and binaries from
cache.nixos.orgare signature-checked. You’re still trusting nixpkgs and its build infrastructure, but you trust a known, specific version of it.
Thorns on the Rose
It isn’t all upside.
- The Nix language is a real language. It’s lazy and functional, and much more powerful than YAML. Error messages can be baffling. Expect a few evenings of confusion before it clicks.
- The documentation is uneven. The manual covers the basics. A lot of practical knowledge lives in blog posts, GitHub issues and other people’s configs.
- The community is brilliant and divided. Flakes vs. no flakes, which module system, which secrets tool. “Nix fixes this” is often true and almost never helpful on its own.
The configuration.nix on my demo VM is 137 lines. Getting a file like that
right takes a while. It pays for itself the first time something breaks.
I can say with confidence that moving to NixOS was a good call for my business and what I was doing. Your mileage will absolutely vary. Is it worth a shot? Yes, test in dev, see how it feels.
The Bottom Line
I didn’t switch to NixOS because it’s clever. I switched because I was tired of being anxious every time I changed a production machine. I wanted a blueprint instead of a pile of recipes, a reset button that works even when the machine won’t boot, and the confidence that I could rebuild everything from a git repo.
A five-second fix instead of a five-day one.
Sources and Further Reading
- NixOS Manual: configuration,
nixos-rebuild, rollbacks and the boot process - Nix Pills: how the store, derivations and profiles work under the hood
- NixOS Search: every option and package, with defaults
- Eelco Dolstra, The Purely Functional Software Deployment Model (2006): the thesis Nix came from
- sops-nix and agenix: secrets management outside the store
- Determinate Systems: Nix tooling and support for organizations