I recently decided to try out the OpenZFS implementation of ZFS on my mac desktop and so far, I’m impressed. In this post, I’ll cover some of the issues encountered along the way. Let me start by describing my configuration. My desktop is a hack(intosh) with Mavericks installed on two internal 1TB drives. The primary boot drive is partitioned into one big slice and I use CarbonCopy to clone this disk to the second disk. My goal is to migrate my home directories, /Users, to a ZFS mirror and keep OSX on it’s own partition. This is pretty simple with two disks but I thought I would cover the steps here and hopefully help others down the ZFS path.

Install ZFS

You’ll first want to head over to OpenZFS and read all about it. Follow the link over to OpenZFS on OS X and download the disk image. Installation is as simple as double-clicking on the package. No reboots are required and you can now start creating zpools and ZFS file systems.

I would like to point out one issue I had with the OpenZFS_on_OS_X_1.2.0.dmg disk image. I had some leftovers from installing from source so I had drivers installed in two different directories and they were different versions. This was causing my zpool to fail importing at boot time. Check this with the following:

# cd /
# ls -lR Library/Extensions/zfs.kext/ System/Library/Extensions/zfs.kext/

My install showed different months (January and March I think) so I fixed this with:

# cd /Library/Extensions
# find ./zfs.kext -depth -print | cpio -pdmu /System/Library/Extensions/

Others have pointed out a missing symbolic link which prevents mounting at boot from working. This is resolved via:

# cd /System/Library/Extensions
# ln -sf /Library/Extensions/zfs.kext .
# ls -l zfs.kext
lrwxr-xr-x  1 root  wheel  28 Apr 13 11:39 zfs.kext -> /Library/Extensions/zfs.kext

Prepare Mirror Disk

Partition Mirror Disk

OSX doesn’t require a lot of disk space and depending on your additional software needs, you can probably resize the OSX boot partition to 250GB. Here’s a snapshot of my 1TB mirror disk after partitioning. I used Disk Utility to change the size then changed the TYPE for the ZFS partition #3 using the gdisk utility mentioned on the wiki.

/dev/disk1
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *1.0 TB     disk0
   1:                        EFI EFI                     209.7 MB   disk0s1
   2:                  Apple_HFS Clone                   250.1 GB   disk0s2
   3:                        ZFS                         749.7 GB   disk0s3

Next, I clone my primary disk to the Clone partition using CarbonCopy except I exclude /Users which is where my home directory resides. When completed, you’ll see there’s still plenty of room for Applications.

# df -h /Volumes/Clone/
Filesystem     Size   Used  Avail Capacity  iused    ifree %iused  Mounted on
/dev/disk1s2  233Gi   90Gi  143Gi    39% 23658814 37388846   39%   /Volumes/Clone

Mount at Boot

We now need to add a few bits to the Clone disk so it will import our zpool at boot time. This is also documented on the wiki.

% cd ~/Developer/
% git clone https://gist.github.com/7549433.git autoimport-repo
% sudo cp autoimport-repo/org.openzfsonosx.ilovezfs.zfs.zpool-import.plist /Volumes/Clone/Library/LaunchDaemons/
% cd /Volumes/Clone/Library/LaunchDaemons/
% sudo chown root:wheel org.openzfsonosx.ilovezfs.zfs.zpool-import.plist
% sudo chmod 644 org.openzfsonosx.ilovezfs.zfs.zpool-import.plist

Configure ZFS

When creating the zpool, there are a few options you will want check and set appropriately:

1. compression=lz4
   I strongly suggest enabling lz4 compression for the entire pool. The CPU overhead is minimal and the compression is excellent. I have found I achieve better IO speeds with compression enabled as the actual data read or written to physical disk is less so fewer IO operations are needed
2. ashift=??
   You’ll want to check your disk specifications and find the native block size for your drive. Common values are 9 for 512B block size and 12 for 4KB block size. Some drives are known to report 512B when in fact, internally they use 4KB block sizes and this can only be set when the pool is created.
3. casesensitivity=[ sensitive | mixed | insensitive ]
   OSX doesn’t really care if your home directory is case sensitive or not but many applications simply fail when installed to a case sensitive file system. I suggest setting this at the pool level for simplicity as it can always be changed when creating a new ZFS file system in the pool. This is discussed on the wiki.

# zpool create -o ashift=12 -O compression=lz4 -O casesensitivity=insensitive -O normalization=formD tank disk1s3

Check your work.

# zfs get all tank | egrep 'compres|sensit'
tank  compressratio          3.29x                  -
tank  compression            lz4                    local
tank  casesensitivity        insensitive            -
tank  refcompressratio       3.43x                  -

Create a ZFS file system for my home directory and for the Shared directory under /Users.

# zfs create tank/prophead
# zfs create tank/Shared
# zfs list
NAME            USED  AVAIL  REFER  MOUNTPOINT
tank            980K   685G   307K  /tank
tank/prophead   255K   685G   255K  /tank/prophead
tank/Shared     255K   685G   255K  /tank/Shared

# zfs get all tank/prophead | egrep 'compres|sensit'
tank/prophead  compressratio          3.21x                  -
tank/prophead  compression            lz4                    inherited from tank
tank/prophead  casesensitivity        insensitive            -
tank/prophead  refcompressratio       3.21x                  -

All child file systems will inherit the parent configuration settings. This can be changed when creating a new ZFS file system with -o casesensitivity=mixed or-o casesensitivity=sensitive.

Migrate Data

Now for the incredibly boring task of migrating my data to it’s new home.

# cd /Users
# find . -depth -print | cpio -pdmu /tank

when this is finished, we need to unmount the ZFS file systems and configure the pool to mount on /Users.

# zfs umount tank/prophead
# zfs umount tank/Shared
# zfs umount tank
# zfs set mountpoint=/Users tank
# zpool export tank

If we’ve done everything correctly, we should now be able to boot from the Clone and start using our new ZFS home directory. Before you reboot to theClone, make sure to run multibeast again and install the boot loader on Clone. I plan to run this way for a week or so to ensure all my Applications operate correctly. Oh, and don’t expect Time Machine to back up your zfs file systems because it will only back up HFS/HFS+ file systems.

You should also disable spotlight indexing of your zfs file systems after rebooting (at least until issue 116 is resolved):

# for f in `zfs list -H -o mountpoint`; do mdutil -i off $f; done

Configure Mirrors

When you’re satisfied you want to make the switch, partition the original boot disk to match the Clone. Again, gdisk is your friend here. Attach the mirror disk to the pool:

# zpool attach tank disk1s3 disk0s3

Fire up CarbonCopy and clone from Clone to the original boot disk. Once the resilver is complete and the clone is finished, remember to apply the boot loader again with multibeast. The pool should now look something like this:

# zpool status -v
  pool: tank
 state: ONLINE
  scan: resilvered 409G in 6h6m with 0 errors on Sun Apr 13 04:11:54 2014
config:

	NAME         STATE     READ WRITE CKSUM
	tank         ONLINE       0     0     0
	  mirror-0   ONLINE       0     0     0
	    disk0s3  ONLINE       0     0     0
	    disk1s3  ONLINE       0     0     0

errors: No known data errors

Troubleshooting

If you have problems getting the pool mounted at boot, try the following boot flags:

• -f Ignore kernel caches
• -s Boot single user

Once at single user and have performed the fsck and mount commands per the boot message, clear /Users if needed and import the pool.

# zpool import
# zpool import -f tank
# zfs list
NAME           USED  AVAIL  REFER  MOUNTPOINT
tank           408G   277G   828K  /Users
tank/Shared    140M   277G   140M  /Users/Shared
tank/prophead  408G   277G   408G  /Users/prophead
# exit

I’ve had to do this when booting from my alternate disk or when switching between disks for the boot. When performing a normal reboot, this step isn’t needed.

Automatic Snapshots

One of the powerful features of ZFS is file system snapshots. This allows me to keep a limited amount of history directly in the file system. OpenSolaris introduced this feature and it is still present in Solaris today. This doesn’t help us Mac (Hac) users though so I wrote a short perl script for automating snapshots. This script combined with a few plists placed in /Library/LaunchDaemons and we now have this functionality. I’ve zipped up this project and you can grab it here. I’ll provide a few tips to getting this working in your environment:

1. Edit each of the plist files and change the path to the script depending on where you install this zipball
2. The script reads the com.sun:auto-snapshot property from each zfs file system and will only snapshot those which have this set to true. You can change this on the fly using the zfs command:

   # zfs set com.sun:auto-snapshot=true tank/prophead

3. Copy the plist files into /Library/LaunchDaemons and reboot or load them using launchctl. I found a nice utility for managing launched called LaunchControl and highly recommend it.

The auto-snapshot environment is easily controlled simply by changing or adding launchd plist files. The snapshots are organized and controlled by the -land -k flags:

$ ./zfs-snapshot.pl -h

    usage:

    ./zfs-snapshot.pl -l LABEL -k xx
    ./zfs-snapshot.pl -l LABEL -k xx [ -n ] [ -r USER@HOST ] [ -f TARGET-ZFS-FS ]
    ./zfs-snapshot.pl -h

    -n : dry run; print what would be done but do nothing
    -v : more verbose output
    -l : label to use for snapshot
    -k : number of snaps to keep
    -r : name of remote host where snapshot will be sent
    -f : what zfs file system to use on remote host
    -h : print this help

I’ve not coded the -r or the -f features yet as I still have to covert my file server to zfs. I’ll update this later this month after this is done. You can see how simple it is to create a scheme for snapshots:

• hourly run every 60*60 seconds with the following flags: zfs-snapshot.pl -l hourly -k 24
• daily run every 60*60*24 seconds with the following flags: zfs-snapshot.pl -l daily -k 7
• weekly run every 60*60*24*7 seconds with the following flags: zfs-snapshot.pl -l weekly -k 4

Final Thoughts

You should create a spare admin account and set the home directory for this user outside of the zfs filesystem. This will allow you to log in and upgrade zfs or perform maintenance tasks when zfs is not available. (Thanks Alex)

It’s a good feeling knowing my data is no longer subject to bit rot and I now can take advantage of all the wonderful features of ZFS. Please let me know how your migration goes and do provide feedback to the developers using the forum.