Running a NAS, Part 2: Pools and Filesystems on a NAS

A NAS pools its drives in one of four main ways. That choice sets what it costs to add a drive, what it costs when one dies, and whether silent corruption gets repaired or only noticed. This is Part 2 of Running a NAS. Part 1, NAS Operating Systems Compared covered the systems, and every one of them brings a storage layer along. This part takes four of those layers apart: ZFS, Unraid’s parity array, SnapRAID with mergerfs, and Btrfs on Linux RAID the way Synology builds it.

Two terms come first. ParityParity is redundancy calculated from the data on several drives, so the contents of a lost drive can be rebuilt from the ones that remain. is redundancy calculated from the data on several drives, so the contents of a lost drive can be rebuilt from the others. Silent corruption is a drive returning wrong data without reporting an error. Both come up in every section below, and the redundant array of independent disks (RAID)RAID, a redundant array of independent disks, combines several drives into one array. Its parity levels let the array keep working when one drive fails. idea sits behind most of the designs. Redundancy keeps data available through a dead drive, but it doesn’t replace a backup. The Cloning and Backup guide explains how many copies to keep and where, and Part 7 applies that to a NAS. How drives that use shingled magnetic recording (SMR)Shingled magnetic recording overlaps each track on the one before to fit more data. Rewriting part of a band means rewriting the rest of it, which slows sustained writes. behave during a rebuild is in Part 9, and How a Hard Drive Works explains what shingling is.

ZFS builds pools out of vdevs

A ZFS pool is a set of virtual devices (vdevs), and each vdev carries its own redundancy. A mirror of N disks holds the capacity of one disk and survives N-1 failures. RAIDZ groups take single, double or triple parity, written raidz1, raidz2 and raidz3, and survive that many failed disks. Their capacity is about (N-P)*X bytes for N disks of size X with P of them parity. OpenZFS recommends 3 to 9 devices in one group.

OpenZFS says RAIDZ avoids the RAID 5 and 6 write hole, where data and parity disagree after a power loss. Redundancy still belongs to the vdev. Unraid’s ZFS documentation says if any one vdev fails, the whole pool fails, even when the others are healthy, so a pool of several vdevs is only as safe as its weakest one. It adds that ZFS treats every disk in a RAIDZ vdev as the size of the smallest one. Any bigger drive mixed in gives up its surplus.

Pools grow by whole vdevs. After zpool add, the new space is available to datasets at once, and the command refuses a vdev whose replication level conflicts with the pool unless forced. That stops a single disk from joining a pool of redundant vdevs by accident. Taking a vdev out is harder, because a pool that holds a top-level RAIDZ vdev can’t have any top-level vdev removed. That means a RAIDZ pool can’t shrink by dropping one. Mirrors grow another way, since attaching a disk to a single-disk vdev makes a two-way mirror, and attaching to a two-way mirror makes a three-way one, with a resilver after each.

Widening a RAIDZ vdev one disk at a time

RAIDZ expansion covers growing a RAIDZ vdev by one disk. OpenZFS 2.3.0 added it, so new devices can join an existing RAIDZ vdev without downtime. It reads all allocated data from the existing disks and rewrites it across the old and new disks. Redundancy stays in place throughout, and if a disk fails mid-expansion it pauses until the vdev is healthy again. Parity level doesn’t change, so a RAIDZ2 vdev stays RAIDZ2. Expansion can repeat, the new disk has to be at least as large as the smallest one already in it, and the pool needs the raidz_expansion feature. When the expansion ends, a pool-wide scrubA scrub reads stored data and checks it against its checksum or parity, so silent corruption is found and, where a good copy exists, repaired. starts to verify the copied blocks.

What expansion costs old data

Old blocks keep the ratio they were written with. One example in the manual is a 5-wide RAIDZ2, which has 3 data and 2 parity and keeps that split for old data after the vdev widens, while new blocks use the wider ratio. Old data therefore goes on spending two of every five blocks on parity. Expansion makes RAIDZ workable for a home pool that grows a drive at a time, and the old-data ratio is its price.

Checksums, rebuilds and snapshots in ZFS

ZFS checksums data by default, and a checksumA checksum is a short value calculated from a block of data and stored apart from it, so a later read can show whether the data has changed. error means a disk returned data that was expected to be correct but wasn’t. A redundantly stored block can be rebuilt from RAIDZ parity or a mirror copy. If ZFS can’t rebuild it, it lists the file as damaged in zpool status so the owner can restore it from a backup.

Scrubs go looking for that damage before a read trips over it. A scrub reads all data in the pool and verifies each block’s checksum, and on mirror and RAIDZ vdevs it repairs what it finds automatically. A resilver reads only the data known to be out of date, such as after a disk replacement, while a scrub reads everything to find silent errors. Part 8 covers how often to scrub on each system.

Swapping a failed disk

Replacing a disk goes through resilver too. zpool replace is the same as attaching the new disk, waiting for it to resilver, and detaching the old one, and the replacement has to be at least as large as the smallest disk in the vdev. The OpenZFS manuals give no rebuild time. Part 8 covers replacing a drive in practice and works out an estimate from a drive’s data sheet.

What a snapshot protects against

SnapshotsA snapshot is a point-in-time image of a file system or dataset, kept on the same drives, so changed or deleted files can be brought back from it. cover a different failure. A ZFS snapshot is a read-only copy of a dataset that can be created extremely quickly and uses no extra space at first. It takes more space as the live dataset changes away from it, and each snapshot is atomic, which makes it a consistent image of the dataset at one moment. Snapshots live in the same pool as the data, so they roll back a bad edit but won’t survive the loss of the pool. Part 7 covers copying snapshots to a second machine.

Synology puts Btrfs on top of Linux RAID

Classic RAID builds on the smallest drive, so with a 500GB smallest drive every other drive contributes only 500GB. Synology Hybrid RAID (SHR) is based on a Linux RAID management system and avoids that by dividing each drive into smaller chunks and adding redundancy across them. Synology’s own structure check explains that SHR may consist of several RAID arrays integrated into one storage pool through Logical Volume Manager (LVM).

Growth costs little under that layout. An SHR pool can expand as soon as two of its drives are upgraded and can form a redundant array. SHR-1 needs at least two drives and SHR-2 needs at least four. Adding a drive to a single-drive SHR pool gives fault tolerance and no extra capacity.

A degraded pool and drive swaps

After a drive fails, the repair help calls the pool Degraded: no data is lost and the pool is still accessible. A replacement has to be equal to or larger than the defective drive in an SHR-1 pool, and Part 8 covers the repair itself. Swapping drives for larger ones goes through Replace Drive, which replaces one drive at a time and keeps the pool protected by redundancy meanwhile.

Rot detection is Btrfs’s job

The Btrfs documentation lists checksums for data and metadata, with automatic detection of silent corruption, as a feature. Synology says Btrfs finds corrupted files by checksum and recovers the data from the RAID volume, whether RAID 1, RAID 5, RAID 6, RAID 10, F1 or SHR. Metadata mirroring and checksums are on by default for every Btrfs volume, while copy-on-write and checksums for file data have to be enabled separately.

That second switch decides whether repair works. Data scrubbing can’t check or repair a shared folder that lacks the data checksum option. It also runs only on Btrfs volumes, or on pools of SHR with three or more drives, RAID 5, RAID 6 or RAID F1. Synology doesn’t say why it pairs Btrfs with Linux RAID. This guide reads the pairing as a way to get checksums and repair while skipping the one part of Btrfs that its own documentation warns against, which the next section covers.

Btrfs RAID5/6: the part to leave alone

The Btrfs manual says RAID5/6 is unreliable in some corner cases and shouldn’t run in production, only in evaluation or testing. It adds that raid5 and raid6 should never hold metadata, and that raid1 or raid1c3 should. After an unclean shutdown RAID5/6 has a write hole because no write journal is implemented.

The mirrored profiles are in better standing. According to the status table, Btrfs repairs bad data automatically from a correct spare copy on DUP, RAID1, RAID10 and RAID56, when a good copy exists. That RAID56 entry does not make the profile safe, since the write hole and the corner cases above still apply. A Btrfs scrub is the validation pass that finds checksum errors and disk read errors across all data and metadata.

Unraid keeps parity apart from the data

Unraid stores data on individual drives rather than striping them into one volume. One parity drive protects all the others, and an array can use up to two. Drive sizes can mix, and drives go in one at a time without reorganizing what is already there. Because no data disk can be larger than the parity disk, the biggest drives go to parity. With dual parity both parity drives must be at least as big as the largest data disk, though they can differ from each other. By default a write reads the old data and parity, computes the new parity, and then writes the data.

My own server is a 4U, 24-bay box full of 20TB hard drives, and it runs Unraid with dual parity. That setup is the one that covers two failures at once, since Unraid says one parity device protects against one disk failure and two protect against two. The first parity drive uses XORXOR compares two bits and gives a one when they differ. Applying the same XOR twice undoes it, which is why ciphers use it to mix in key bits. calculations. The second uses Q-parity, a Reed-Solomon style code comparable to RAID 6, and that is what allows a rebuild from two simultaneous failures.

Adding a data disk to a parity-protected array requires clearing it first. Clearing runs in the background, so the array stays available. A failed drive is emulated from parity and the other drives while the array keeps running. Under single parity a second failure during the rebuild could lose data, which is the gap dual parity closes. Part 8 covers replacing a disk in practice.

Why parity checks don’t repair rot

Parity protects against a dead drive and does little for rot. A parity check reads every data disk and the parity drives and compares recomputed parity with what is stored. In correcting mode a mismatch updates the parity disk rather than the data. A mismatch caused by a flipped bit on a data disk therefore gets baked into parity. SnapRAID’s comparison page says Unraid lacks native checksums and ignores silent errors, and that is a rival project’s statement. Unraid’s own file systems page lists ZFS as the option with checksums to detect bit rot. The ZFS documentation adds that a ZFS disk inside the parity array is managed individually and gets none of the self-healing of a multi-disk pool.

ZFS pools inside Unraid

Version 6.12.0 added ZFS pools with raid0, mirror, raidz1, raidz2 and raidz3 profiles, and version 7.2.0 added RAIDZ expansion so single-vdev pools can grow one drive at a time. Other pool layouts need a whole new vdev, such as a new mirror pair or a RAIDZ vdev of the same width. Since 7.2.0 a two-device ZFS pool is a mirror by default, and it can be made RAIDZ1 up front if the plan is to expand it later. Part 4, SSDs in a NAS covers what SSDs do in ZFS pools and in Unraid’s cache pools.

SnapRAID and mergerfs: parity and pooling as separate jobs

SnapRAID stores parity for recovery from up to six disk failures, and each extra parity disk allows recovery from one more. Parity disks must be the largest in the array, because parity can grow to the size of the largest data disk. Mixed disk sizes are fine, disks can be added at any time, and disks already full of files need no reformatting.

It keeps a separate file system per disk and doesn’t stripe, so it gives no striping speed-up. Reading a file spins up only the disk that holds it. If more disks fail than there is parity for, the loss stays limited to the failed disks.

Parity that lags the data

SnapRAID’s parity is a snapshot taken at the last sync. After a disk fails, SnapRAID recovers data only up to that sync, and changes made since then on that disk are lost. The manual describes SnapRAID as suited to data that rarely changes. Without snapshot support in the data file system, deleting or changing files after a sync can prevent full recovery of other failed disks, although newly added files don’t. On Btrfs, ZFS, Bcachefs and NTFS, SnapRAID can keep a frozen reference copy to avoid that. Recovery also isn’t real time. With RAID you keep working when a disk fails, and with SnapRAID you don’t. The same snapshot-style parity lets it restore files that were deleted by accident.

Scrubs check about 8% at a time

SnapRAID hashes all data to detect silent corruption. Each scrub run checks about 8% of the array by default and skips data scrubbed in the previous 10 days. Permissions, ownership and extended attributes aren’t saved. Blocks with silent or I/O errors are marked bad during a scrub, and the fix command repairs them from parity.

mergerfs merges the paths

mergerfs handles the other job. It is a FUSE (Filesystem in Userspace) union file system that merges filesystem paths from many drives under one mount point, and it works on paths rather than block devices. Drives can be added or removed without touching the data on the others. One failing drive doesn’t affect the rest, since each keeps an ordinary file system. mergerfs offers no parity of its own. Its comparison page says SnapRAID can be combined with it for redundancy and notes that SnapRAID, unlike RAID5 or RAID6, works with drives of different sizes. A pool can’t hold a file larger than one drive, so on two 1TB drives no file can exceed 1TB.

Adding a drive, losing a drive, finding rot

The three questions from the start get different answers in each design. The table sets them side by side.

Source: the OpenZFS manuals, Synology’s DSM help and Btrfs page, the Unraid documentation, the SnapRAID manual and the mergerfs documentation, each cited in the sections above.

Only ZFS and Synology’s Btrfs both detect rot and repair it on live data, and SnapRAID repairs it back to the last sync. Unraid’s parity check works as availability protection for a dead drive. It hands rot nothing to repair from, because a mismatch makes parity follow the data disk.

Choosing a layout before the drives arrive

The choice mostly follows how the data behaves. Files that change all day need parity that updates with every write, which ZFS, Synology’s RAID layer and Unraid’s array provide. A media library that only grows can live with SnapRAID’s sync-based parity. Anyone who wants rot repaired on live data is left with ZFS or Btrfs on a redundant volume. None of the pool documentation gives a rebuild time, so that part of the decision waits for the drives themselves, and Part 8 estimates it from a data sheet. Drives for a pool are in the NAS tier of Best HDDs.

Part 3, Drive Pooling on Windows covers StableBit DrivePool duplication and what happens when a drive fails, Storage Spaces layouts and ReFS checksums, and SnapRAID beside a Windows pool.