I’ve gone from “no backups” to “raid is a backup” to “two zfs pools in one box”, and decided it’s finally time for a proper backup solution. So, I settled on Proxmox Backup Server! And today, I rebuild my HP Microserver Gen8 with 4x10T refurbished SAS drives, a new SAS controller card, and more! With this backup solution, I’m feeling a lot better about my data migration to Ceph.

Contents⌗

Video⌗

Thumbnail

SAS Drive Formatting⌗

Since these drives were refurbished they were formerly formatted for a hardware RAID controller and were giving me lots of protection errors in dmesg - specifically blk_update_request: protection error (and failing to read, but not write). Reading around I found the answer - format without hardware protection. The drives were likely previously used with a hardware RAID controller and had additional data integrity features enabled, something that ZFS doesn’t need. The command I used to do this is sg_format --format /dev/sdX and took about a full day per drive to complete (I ran them all in parallel, no I didn’t wait a week). This is different than a filesystem format, since it’s a SCSI command to request that the drive format itself with the sector size and layout requested (in this case, the manufacturer’s defaults).

I also ran SMART tests and they passed.

ZFS Pool Setup⌗

I wanted to use a ZFS Special Device, which stores filesystem metadata on a separate drive. This can let me use a smaller faster drive to speed up directory listings and large metadata operations, which is important in PBS since it can potentially store up to 65k files per directory. The chunk-based deduplication system splits data up into files named by their hash, so it relies on the filesystem to store a ton of files.

I created the zfs pool via the GUI, using RAIDZ1 with the 4x10T drives (30T usable). Then I added the special device from the shell: zpool add <poolname> -o ashift=12 special mirror /dev/sde /dev/sdf -f

Next Steps⌗

Going forward, I have a few plans for this system that I haven’t implemented yet:

  • Auto-start the server at backup time, and auto shut-down when backups and other tasks are done
  • LACP bond (2x1gig) when I have enough switch ports to do that
  • Add backup of CephFS directly on the PBS server (mount Ceph on PBS, run proxmox-backup-client to the CephFS mount, …)
  • TAPE BACKUP! I bought an LTO5 SAS drive to dip my toes in tape backup, to see if I like it before moving to an autoloader