This answer and email message indicate that something called "OverlayFS" is available in Ubuntu 11.10 and will forcefully replace aufs in Ubuntu 12.04.
How do I use it? Where is its documentation?
Edit: Since writing this answer, some things have changed in overlayfs, namely the addition of a required parameter workdir, see totti's answer below for a detailed description of this new parameter.
I finally managed to find it. I found references to it in the kernel source, but for some reason it doesn't appear in the git tree on kernel.org. But! If you pull the Ubuntu kernel source like this: apt-get source linux-image-3.0.0-16-generic you can find it in linux-3.0.0/Documentation/overlayfs.txt. It is also available in the linux-doc package in /usr/share/doc/linux-doc/filesystems/overlayfs.txt.gz.
As the actual help documentation is more of a "how it works" instead of a "how to mount with it," here's a brief rundown (there is one example in the kernel documentation):
mount -t overlayfs -o [mount options] overlayfs [mountpoint for merged system]
Where [mount options] can be:
One thing that confused me at first, so I should probably clarify, is that mounting an overlayfs does not actually mount a filesystem. I was trying to mount a squashfs filesystem using an overlayfs mount, but that's not how it works. You must first mount the (in my case squashfs) filesystem to an arbitrary directory, then use overlayfs to merge the mount point (a directory) and another directory onto a tertiary directory (the overlayfs mount point)(edit: this "tertiary" directory can actually be the upperdir= directory). The tertiary directory is where you will see the merged filesystems (or directory trees - it's flexible).
I've been working on an Ubuntu hybrid boot disk where the base Ubuntu system exists as filesystem.squashfs and I have files called ubuntu.overlay kubuntu.overlay xubuntu.overlay and lubuntu.overlay. The .overlay files are base installs of said systems with the contents of filesystem.squashfs pruned (to save space). Then I modified the init scripts to overlay the correct distro's .overlay file (from a boot parameter) using overlayfs and the above options and it works like a charm!
These are the lines that I used in my init scripts (once all variables are translated):
mkdir -p /overlay
mount -t squashfs /cdrom/casper/ubuntu.overlay /overlay
mount -t overlayfs -o lowerdir=/filesystem.squashfs,upperdir=/overlay overlayfs /
Note that filesystem.squashfs above is a directory created by casper, not a file.
These three statements create an /overlay directory, mount a squashfs filesystem on the /overlay directory and then use OverlayFS to essentially merge the contents of /overlay over /.
In the process of re-building my live USB for each release, I use OverlayFS to save a bunch of time. I start off with a directory called ubuntu-base which contains the contents of the ubuntu-core image which is the most basic install. I will then create directories called ubuntu, kubuntu, lubuntu, and xubuntu.
Then, I use OverlayFS to make the files from the ubuntu-base show up in the individual directories. I would use something like this:
mount -t overlayfs -o lowerdir=ubuntu-base,upperdir=kubuntu overlayfs kubuntu
This makes the files from ubuntu-base show up in the kubuntu folder. Then, I can chroot to the kubuntu folder and do something like apt-get install kubuntu-desktop. Any changes that are made while in this OverlayFS mount will remain in the upper directory, in this case the kubuntu folder. Then, once I unmount the OverlayFS mount the files that really exist in ubuntu-base but are "mirrored" into the kubuntu folder vanish unless they have been changed. This keeps me from having to have multiple copies of the files in ubuntu-base while still being able to use them as if they physically exist in each location.
