Supporting Multiple Pis

Explanation

Currently I am able to boot a completely diskless Raspberry Pi into a fully configured Ubuntu Server OS over the network, by utilizing an NFSv3 server as the root filesystem.

However, if I were to try and boot further machines off of the same root filesystem, I would no doubt be flooded with errors as each system attempts to modify the shared OS root and create handles on the files

Most guides or documentation online suggest working around this by provisioning a copy of the filesystem per machine, for example;

 - /mnt/nfsshare/ubuntu-server-24-04-node01
 - /mnt/nfsshare/ubuntu-server-24-04-node02

But I see this as wasteful and likely better suited to trying an alternative OS. In my humble opinion the OS root should be the same across all machines, immutable and immune to per machine drift.

So instead I offer this alternative:

1. Have the Pi mount the nfs share as read only root
2. Use OverlayFS to make any runtime changes in memory/tmpfs

Implementation

For multi‑Pi netboot the goal is simple: a single, read‑only OS image that every node boots, with any runtime changes kept off the shared root.

Overlayroot achieves this by placing a writable tmpfs layer over an immutable lower filesystem (our NFS export).

Each Pi sees a normal, writable root, but all writes land in RAM and vanish on reboot.

This gives a single shared golden image for all nodes (easy patching/rollbacks).

Kernel parameters

Point the kernel at the shared NFS root and enable the overlay layer:

# cmdline.txt
root=/dev/nfs rw \
nfsroot=192.168.1.66:/mnt/nfsshare/<image>,v3,tcp,ro \
modules-load=overlay overlayroot=tmpfs \
ip=dhcp

Notes:

• rw applies to the final overlay root; the lower NFS layer is made ro in nfsroot=....
• Short NFS options (v3,tcp) are required by early boot.

Enable OverlayFS

Overlayroot reads a simple config file during early boot:

/etc/overlayroot.conf

overlayroot="tmpfs"
overlayroot_overlayfs_opts="redirect_dir=off,index=off,metacopy=off"

The extra overlayfs_opts make OverlayFS happy when the lower filesystem is NFS (they disable features the kernel cannot guarantee on NFS lowers).

Keep logs light

To avoid noisy disk writes and journald watchdog timeouts on a volatile root, keep the system journal entirely in RAM with compression and strict size limits so it can’t exhaust memory on a diskless system.

/etc/systemd/journald.conf.d/volatile.conf

[Journal]
Storage=volatile
RuntimeMaxUse=64M
Compress=yes
SystemMaxFileSize=16M

Review

• Root is an overlay and writable:
  • findmnt -no FSTYPE,OPTIONS / → overlay rw,...
• Lower is NFS and read‑only:
  • findmnt -no FSTYPE,OPTIONS /media/root-ro → nfs ...,ro,vers=3,proto=tcp
• /home automount is armed after provisioning:
  • findmnt /home → systemd-1 autofs before first access; nfs after ls /home.
• A file in ~ubuntu appears on the NFS share.

With this setup you get an immutable, shared base OS for all Pis and clean, per‑node ephemeral state - while keeping user data safely on the NFS share.