Union Filesystems for MythTv

I have had a MythTv system running for some years now, having wanted one ever since I chanced across an install in the downtown Toronto apartment of a guy who was donating me an old box (200Mhz Pentium, without even a CD-ROM drive) for linux experimentation.

Myth is a free/open source DVR, or ‘TiVo for tightarsed techies’, which functions primarily as a video and music centre, but also for photo galleries, RSS reader, weather etc. Similar to Windows MCE but more powerful and as a consequence slightly more painful to maintain.

This maintenance is often refered to by the concept of ‘WAF’ or “Wife Acceptance Factor”, as failure to achieve WAF has been the undoing of many a techie project over the years. Thus any glitches or limits in functionality that may affect WAF are high on the list of things to iron out. I have managed to convert our entire house to a pure Ubuntu shop with minimal flaws, but I dread the occasional call I get at work when the Myth box has overheated and shutdown, and I still need to walk through some manual (vnc / mount) type things to get it back up.

One longstanding WAF issue relates to video folders in MythVideo. As the amount of media has increased, it has spread from the Myth backend to some nfs mounts on other boxes, and the horribly hackish way that had been achieved was folders symlinked as “Newer”,”Newest” etc., containing subdirectories of TV shows and movies, all showing up as separate trees in the root of the video section, and making navigation somewhat painful.

I was thinking of reorganising my files to accomodate this, copying all shows on one drive, documentaries on another, etc. This had the obvious flaw of relying on predetermining how large a given collection would grow, as well as just being pretty ungraceful. Raid/LVM was not an option, due to the multiple machines housing the files, and additionally without proper redundancy you run the risk of losing all files with a single disk failure. Buying new boxen was also not, due to the financial necessities that having an infant brings.

Then I encountered UnionFS. The idea sounded great – multiple filesystems / structures can be conjoined in to one tree, with the filesystem ‘overlay’ essentially representing them as one to the OS. Thus I could have 3 folders in different physical locations, all labeled ‘TV’, containing different content, and UnionFS would make one large ‘TV’ folder containing all of the files.

This seems to be a server technology that has primarily been used for the liveCD/diskless set, to enable such clever things as nonexistant root partitions being mounted from elsewhere. As such you would think it would be fairly well set up, documented, and reliable. Apparently not. Documentation is sparse, implementations sparser still, and apart from being installed as default in Ubuntu Gutsy I couldn’t get much further than a basic single mount command to play with. As for known bugs, much was made of using read-only branches as there are horrors which “could even RESULT IN DATA LOSS”, which is fine as these various disks are already accessible by the different machines adding content, and the test I did with a few simple local folders seemed ok. Until I added nfs into the mix.

On mounting a remote filesystem, and attempting to ls the tree, I was initially fine until I listed the ‘Movies’ directory. At this point the vnc server hung (and apparently the box). ssh was locked out. There was an almighty kernel ‘oops’. Apart from the general swearing, a few key phrases from the resulting messes I encountered follow:

• Unable to handle kernel NULL pointer dereference
• call trace
• sys_readahead
• mount: special device none does not exist
• on attempt to umount – “device is busy”

To cut a huge story slight less huge, I gave up after a few hours googling. I then tried aufs, “Another Union File System” as it is called, in the great tradition of badly named unix programs. This is supposed to be installed by default in the next Ubuntu LTS release (Hardy Heron), but then again, UnionFS was released in Gutsy and was pretty broken. Now I am no slowpoke these days with linux, but I couldn’t even work it due to lack of docs. “unknown filesystem type aufs” was one of the many errors I encountered, and following some compile-hell, I gave up.

Finally I encountered funionfs. This is mentioned as an afterthough on FUSE (Filesystem in Userspace), and as apparently is so with all of these union file systems, is horrendously documented, and relies on much prodding to check if it will work. The mount syntax is essentially similar to the other two discussed, but the crux of all of this ended up with the following simple method to get conjoined filesystems across nfs shares in Gutsy:

tom@Bender:~$ sudo apt-get install funionfs
tom@Bender:~$ mkdir ~/Myth
tom@Bender:~$ sudo mount -t fuse -o dirs=/mount/Zoidberg/Myth:/mount/Nibbler/Myth:/home/tom/Videos -o allow_other funionfs#NONE /home/tom/Myth

I am posting this in the hope it may increase WAF in your vicinity. The end-result is pretty damn seamless, and will make adding new media a simple process, even with fresh filestores. This sort of transparent filesystem joining is something I hope will become increasingly easy and transparent in the future, with Apple’s grey(white) boxes and networked optical drives etc currently leading the way in the super-user-friendly variety. Gutsy and MythBuntu are not the best of friends right now, and I will approach Hardy with trepidation due to the inevitable breakage that will occur. That said, Linux has come a long way since I tried net-installing a floppy-bootstrapped slackware on a P200 some years ago due to a machine which didn’t even have a CD-ROM drive.

*Update* Funnily enough, just a couple of days after posting this I see that Myth Tv 0.21 has added a feature called “Storage groups”. This suggests that it does a similar function at the application level, but I cannot yet find sensible docs to figure out if this is a better solution.

Related posts:

1. Exporting mail from Opera’s M2 email client and Importing to Thunderbird

Tags: Fuse, Linux, MythTV, Toys, Ubuntu