Using ZFS in OS X
I recently decided to try out the OpenZFS implementation of ZFS on my mac desktop and so far, I’m impressed. In this post, I’ll cover some of the issues encountered along the way. Let me start by describing my configuration. My desktop is a hack(intosh) with Mavericks installed on two internal 1TB drives. The primary boot drive is partitioned into one big slice and I use CarbonCopy to clone this disk to the second disk. My goal is to migrate my home directories, /Users, to a ZFS mirror and keep OSX on it’s own partition. This is pretty simple with two disks but I thought I would cover the steps here and hopefully help others down the ZFS path.
You’ll first want to head over to OpenZFS and read all about it. Follow the link over to OpenZFS on OS X and download the disk image. Installation is as simple as double-clicking on the package. No reboots are required and you can now start creating zpools and ZFS file systems.
I would like to point out one issue I had with the OpenZFS_on_OS_X_1.2.0.dmg disk image. I had some leftovers from installing from source so I had drivers installed in two different directories and they were different versions. This was causing my zpool to fail importing at boot time. Check this with the following:
# cd / # ls -lR Library/Extensions/zfs.kext/ System/Library/Extensions/zfs.kext/
My install showed different months (January and March I think) so I fixed this with:
# cd /Library/Extensions # find ./zfs.kext -depth -print | cpio -pdmu /System/Library/Extensions/
Others have pointed out a missing symbolic link which prevents mounting at boot from working. This is resolved via:
# cd /System/Library/Extensions # ln -sf /Library/Extensions/zfs.kext . # ls -l zfs.kext lrwxr-xr-x 1 root wheel 28 Apr 13 11:39 zfs.kext -> /Library/Extensions/zfs.kext
Prepare Mirror Disk
Partition Mirror Disk
OSX doesn’t require a lot of disk space and depending on your additional software needs, you can probably resize the OSX boot partition to 250GB. Here’s a snapshot of my 1TB mirror disk after partitioning. I used Disk Utility to change the size then changed the TYPE for the ZFS partition #3 using the gdisk utility mentioned on the wiki.
/dev/disk1 #: TYPE NAME SIZE IDENTIFIER 0: GUID_partition_scheme *1.0 TB disk0 1: EFI EFI 209.7 MB disk0s1 2: Apple_HFS Clone 250.1 GB disk0s2 3: ZFS 749.7 GB disk0s3
Next, I clone my primary disk to the Clone partition using CarbonCopy except I exclude /Users which is where my home directory resides. When completed, you’ll see there’s still plenty of room for Applications.
# df -h /Volumes/Clone/ Filesystem Size Used Avail Capacity iused ifree %iused Mounted on /dev/disk1s2 233Gi 90Gi 143Gi 39% 23658814 37388846 39% /Volumes/Clone
Mount at Boot
We now need to add a few bits to the Clone disk so it will import our zpool at boot time. This is also documented on the wiki.
% cd ~/Developer/ % git clone https://gist.github.com/7549433.git autoimport-repo % sudo cp autoimport-repo/org.openzfsonosx.ilovezfs.zfs.zpool-import.plist /Volumes/Clone/Library/LaunchDaemons/ % cd /Volumes/Clone/Library/LaunchDaemons/ % sudo chown root:wheel org.openzfsonosx.ilovezfs.zfs.zpool-import.plist % sudo chmod 644 org.openzfsonosx.ilovezfs.zfs.zpool-import.plist
When creating the zpool, there are a few options you will want check and set appropriately:
I strongly suggest enabling lz4 compression for the entire pool. The CPU overhead is minimal and the compression is excellent. I have found I achieve better IO speeds with compression enabled as the actual data read or written to physical disk is less so fewer IO operations are needed
You’ll want to check your disk specifications and find the native block size for your drive. Common values are 9 for 512B block size and 12 for 4KB block size. Some drives are known to report 512B when in fact, internally they use 4KB block sizes and this can only be set when the pool is created.
- casesensitivity=[ sensitive | mixed | insensitive ]
OSX doesn’t really care if your home directory is case sensitive or not but many applications simply fail when installed to a case sensitive file system. I suggest setting this at the pool level for simplicity as it can always be changed when creating a new ZFS file system in the pool. This is discussed on the wiki.
# zpool create -o ashift=12 -O compression=lz4 -O casesensitivity=insensitive -O normalization=formD tank disk1s3
Check your work.
# zfs get all tank | egrep 'compres|sensit' tank compressratio 3.29x - tank compression lz4 local tank casesensitivity insensitive - tank refcompressratio 3.43x -
Create a ZFS file system for my home directory and for the Shared directory under /Users.
# zfs create tank/prophead # zfs create tank/Shared # zfs list NAME USED AVAIL REFER MOUNTPOINT tank 980K 685G 307K /tank tank/prophead 255K 685G 255K /tank/prophead tank/Shared 255K 685G 255K /tank/Shared # zfs get all tank/prophead | egrep 'compres|sensit' tank/prophead compressratio 3.21x - tank/prophead compression lz4 inherited from tank tank/prophead casesensitivity insensitive - tank/prophead refcompressratio 3.21x -
All child file systems will inherit the parent configuration settings. This can be changed when creating a new ZFS file system with -o casesensitivity=mixed or-o casesensitivity=sensitive.
Now for the incredibly boring task of migrating my data to it’s new home.
# cd /Users # find . -depth -print | cpio -pdmu /tank
when this is finished, we need to unmount the ZFS file systems and configure the pool to mount on /Users.
# zfs umount tank/prophead # zfs umount tank/Shared # zfs umount tank # zfs set mountpoint=/Users tank # zpool export tank
If we’ve done everything correctly, we should now be able to boot from the Clone and start using our new ZFS home directory. Before you reboot to theClone, make sure to run multibeast again and install the boot loader on Clone. I plan to run this way for a week or so to ensure all my Applications operate correctly. Oh, and don’t expect Time Machine to back up your zfs file systems because it will only back up HFS/HFS+ file systems.
You should also disable spotlight indexing of your zfs file systems after rebooting (at least until issue 116 is resolved):
# for f in `zfs list -H -o mountpoint`; do mdutil -i off $f; done
When you’re satisfied you want to make the switch, partition the original boot disk to match the Clone. Again, gdisk is your friend here. Attach the mirror disk to the pool:
# zpool attach tank disk1s3 disk0s3
Fire up CarbonCopy and clone from Clone to the original boot disk. Once the resilver is complete and the clone is finished, remember to apply the boot loader again with multibeast. The pool should now look something like this:
# zpool status -v pool: tank state: ONLINE scan: resilvered 409G in 6h6m with 0 errors on Sun Apr 13 04:11:54 2014 config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 disk0s3 ONLINE 0 0 0 disk1s3 ONLINE 0 0 0 errors: No known data errors
If you have problems getting the pool mounted at boot, try the following boot flags:
-f Ignore kernel caches
-s Boot single user
Once at single user and have performed the fsck and mount commands per the boot message, clear /Users if needed and import the pool.
# zpool import # zpool import -f tank # zfs list NAME USED AVAIL REFER MOUNTPOINT tank 408G 277G 828K /Users tank/Shared 140M 277G 140M /Users/Shared tank/prophead 408G 277G 408G /Users/prophead # exit
I’ve had to do this when booting from my alternate disk or when switching between disks for the boot. When performing a normal reboot, this step isn’t needed.
One of the powerful features of ZFS is file system snapshots. This allows me to keep a limited amount of history directly in the file system. OpenSolaris introduced this feature and it is still present in Solaris today. This doesn’t help us Mac (Hac) users though so I wrote a short perl script for automating snapshots. This script combined with a few plists placed in /Library/LaunchDaemons and we now have this functionality. I’ve zipped up this project and you can grab it here. I’ll provide a few tips to getting this working in your environment:
- Edit each of the plist files and change the path to the script depending on where you install this zipball
- The script reads the com.sun:auto-snapshot property from each zfs file system and will only snapshot those which have this set to true. You can change this on the fly using the zfs command:
# zfs set com.sun:auto-snapshot=true tank/prophead
- Copy the plist files into /Library/LaunchDaemons and reboot or load them using launchctl. I found a nice utility for managing launched called LaunchControl and highly recommend it.
The auto-snapshot environment is easily controlled simply by changing or adding launchd plist files. The snapshots are organized and controlled by the -land -k flags:
$ ./zfs-snapshot.pl -h usage: ./zfs-snapshot.pl -l LABEL -k xx ./zfs-snapshot.pl -l LABEL -k xx [ -n ] [ -r USER@HOST ] [ -f TARGET-ZFS-FS ] ./zfs-snapshot.pl -h -n : dry run; print what would be done but do nothing -v : more verbose output -l : label to use for snapshot -k : number of snaps to keep -r : name of remote host where snapshot will be sent -f : what zfs file system to use on remote host -h : print this help
I’ve not coded the -r or the -f features yet as I still have to covert my file server to zfs. I’ll update this later this month after this is done. You can see how simple it is to create a scheme for snapshots:
- hourly run every 60*60 seconds with the following flags: zfs-snapshot.pl -l hourly -k 24
- daily run every 60*60*24 seconds with the following flags: zfs-snapshot.pl -l daily -k 7
- weekly run every 60*60*24*7 seconds with the following flags: zfs-snapshot.pl -l weekly -k 4
You should create a spare admin account and set the home directory for this user outside of the zfs filesystem. This will allow you to log in and upgrade zfs or perform maintenance tasks when zfs is not available. (Thanks Alex)
It’s a good feeling knowing my data is no longer subject to bit rot and I now can take advantage of all the wonderful features of ZFS. Please let me know how your migration goes and do provide feedback to the developers using the forum.