I currently have a secondary pool (with raidz2) that I was originally going to use for my important documents, such as storage for Paperless-ngx, as raidz offers corruption detection and repair. The pool is encrypted.
However, I’m concerned about rebuild times (it’s a pool of 4 22TB drives). Is btrfs a better choice for this use case, or should I just go with raidz like I originally planned?
Edit: I should have mentioned that I already have 4-3-2 backups configured - I’m primarily interested in the “self-healing” aspect of ZFS so that I don’t have to recover from backups unless necessary, and to resolve corruption on the fly without me having to notice that a file is corrupt.
I have been enjoying using ZFS, although it’s a not a killer filesystem for every scenario, I think it would be the best solution for you. also, you can try the “free consulting” on the 2.5 Admins podcast (show@2.5admins.com). Jim and Allen are ZFS lovely weirdos, but they can explain better why would be ZFS a better solution for your case. Give it a try. I have done it before and really helped me.
Don’t use any btrfs raid levels besides mirroring. There are long-standing raid bugs that they wontfix in 5 and 6 that have led to data loss.
RAID is not a backup.
RAID is not for data safety.
RAID is for:
- Ensuring availability of data in the face of hardware failure. That means your files don’t disappear when a drive dies and you have some time to swap out for functional hardware and restore redundancy.
- Presenting multiple drives as one larger unit. This is what striping does, and to a lesser extent the parity-mode levels.
- Improving performance (sometimes). A RAID mirror is generally much faster to read from than any individual drive because reads can be interleaved across drive members. A stripe can be much faster because writes are distributed across drive members. This is less of a bonus today with solid state/nvme drives, but it’s still applicable to spinning rust.
If your concern is protecting your data, set up a 3-2-1 backup strategy.
RAID is not a backup.
True
RAID is not for data safety.
Not true.
- RAID helps prevent data loss in the event that a drive failure occurs before changes are replicated to backups. If you upload photos and then delete them from your camera, they will likely be stored in just one location for a period of time. If you have a drive failure without RAID, you could lose your only copy.
- With ZFS, RAID can be used to protect against things like bitrot. Even data at rest can become corrupt over time. ZFS stores checksums of the files to know if corruption occurs. And with one or more parity drives, ZFS can automatically repair the corruption when detected. Without this, detecting and fixing these kinds of issues can be much more difficult.
I’m in 100% agreement that RAID is not essential, and that backups are a much higher priority. In fact, without backups in place, I’m not generally in favor of RAID. RAID adds additional complexity. That complexity can result in data loss. Especially due to user error. But once backups are part of the equation, RAID can add additional layers of security for your data.
Yes, valid points that I didn’t really want to muddy the discussion with. In my experience, most people hear the Redundant in RAID and think that’s sufficient and that their data is safe. Maybe poor choice of words on my part, but that’s what I meant re: “data safety.”
You’re fully correct that a proper RAID setup can provide additional layers of availability atop a robust backup process, but I’d wager most of the people who are interested in that extra layer are already aware of the limitations of RAID.
I do run RAIDZ1 on my franken-nas due to limited drive sizes on that machine, the goal being to maximize usable space while providing a sufficient amount of time to address a drive failure. If drive prices ever become reasonable again, I’ll likely rebuild the system with 6-8 drives in a RAIDZ2 configuration just for a little more peace of mind, but as long as my off-site backups are running, I sleep at night just fine.
I’m very aware and have full 4-3-2 backups already. I’m also not interested in a standard raid. Thanks though! It’s always good to mention that raid is not a backup. I simply want to add more protection from disk corruption (not necessarily full failure) so that I don’t need to recover from backups unless I absolutely must. A benefit would also be resolving corruption before I even notice it.
I’ve been using a raid1 btrfs pool to store offline backups for around 10 years. It’s 4 rotating drives (2x4TB+2x12TB). I replaced / rebalanced 3 disks with larger / newer ones already (went fine). I identified a bad usb/sata controller, and lots of bitrots on one old disk (scrub was able to correct a few thousands errors).
I’m getting around 80MB/s read/write throughput (not great but OK for offline backup). I’m able to mount it on low-powered / low-memory devices (not the case for ZFS). Scrub takes around 2 days IIRC (for around 10TB of actual data), so I run it once a year.
I keep it simple and thus am not using advanced features (dedup / encryption / snapshots / subvolumes / raid5/6/10). So far its a good match for my needs.
I was originally going to use for my important documents
Not quite what you’re asking, but if your concern is avoiding data loss, if you haven’t already, I’d set up a backup before I started setting up a RAID or similar setup.
While redundancy in a drive setup helps, it’s not really a backup and thus not a “safe” way to store important information on it’s own.
That said, selecting the way you setup a raid system is based on risk and utility. I have a raid1 with a hotspare for important files. And use raid5 with 3&4 drives for less important stuff. You can also optimise for reading speed for example (as the same file can be drawn from multiple drives)
I’d stay on zfs, I simply don’t trust btrfs’ raid implementation. For very important documents I also set copies=2 (or 3) on that dataset, just in case.
And as others already said: 3-2-1 backups ;)
Yeah, I’m leaning towards ZFS for sure after reading about btrfs’ past…
Filesystem doesn’t really matter once you have a reliable, redundant off-site backup and recovery plan set up and tested.
Really, use what fs feels best for you. And do your backups.
Did I mention backups are important?
I prefer file systems that checksum data. Without this it is difficult to know when there has been corruption. I generally use brtfs for this reason.
I should have mentioned that I was (and still am) using XFS for my primary pool. XFS has been great, I was just hoping for a solution which would make it so I don’t have to hit my backups unless absolutely necessary (such as full array loss, for example). ZFS’ data corruption protection is very intriguing, it’s mostly the rebuild time and write speed that is making me think about doing something else.
If rebuild time is truly your main concern, you could configure the 4 drives as raid 1+0. On truenas I think it would be called a pool of 2 mirror vdevs. It would give you much faster rebuild time and still have 1-2 drive failure tolerance.
What about 2 mirrored pools of 2 drives each, then back up the main pool to the other with either ZFS snapshots or a tool like Restic.
Ideally you also need an offsite backup of important files too, but that gets you part way to a robust system that can handle corruption or accidental deletions.
Would this just be to help with the rebuild time? Raid10 in ZFS is an interesting idea, which would also require two mirrors and striping.
Yes mirrors are the fastest to rebuild I believe, it’s also to give you a backup, as any kind of raid or mirror is not a functional backup, it only provides redundancy.
I would not do raid 10 for the same reason of no backup that way.
For your situation I would be more likely to go with a single drive with btrfs and dup for metadata redundancy. Regular snapshots and scrubs.
Use a second drive in the same system with btrfs to store snapshots at wider scheduled intervals. These will be bigger since no CoW on the separate file system. Scheduled scrub here too.
Use a third drive with ext4 as a backup target using a separate backup mechanism.
Use the fourth drive as a spare, or in a separate location as a target to send the backups if you don’t already have an off-site solution.
Interesting idea.
If you’re storing, then offline, read only, checksummed and preferably encrypted. And above all, several tested backups. I’m partial to making erofs images from dirs, but any archive that fits the content type works.
For the random access cache on top, whatever.