Home / Blog
Union Filesystems for MythTv
I have had a MythTv system running for some years now, having wanted one ever since I chanced across an install in the downtown Toronto apartment of a guy who was donating me an old box (200Mhz Pentium, without even a CD-ROM drive) for linux experimentation.
Myth is a free/open source DVR, or 'TiVo for tightarsed techies', which functions primarily as a video and music centre, but also for photo galleries, RSS reader, weather etc. Similar to Windows MCE but more powerful and as a consequence slightly more painful to maintain.
This maintenance is often refered to by the concept of 'WAF' or "Wife Acceptance Factor", as failure to achieve WAF has been the undoing of many a techie project over the years. Thus any glitches or limits in functionality that may affect WAF are high on the list of things to iron out. I have managed to convert our entire house to a pure Ubuntu shop with minimal flaws, but I dread the occasional call I get at work when the Myth box has overheated and shutdown, and I still need to walk through some manual (vnc / mount) type things to get it back up.
One longstanding WAF issue relates to video folders in MythVideo. As the amount of media has increased, it has spread from the Myth backend to some nfs mounts on other boxes, and the horribly hackish way that had been achieved was folders symlinked as "Newer","Newest" etc., containing subdirectories of TV shows and movies, all showing up as separate trees in the root of the video section, and making navigation somewhat painful.
I was thinking of reorganising my files to accomodate this, copying all shows on one drive, documentaries on another, etc. This had the obvious flaw of relying on predetermining how large a given collection would grow, as well as just being pretty ungraceful. Raid/LVM was not an option, due to the multiple machines housing the files, and additionally without proper redundancy you run the risk of losing all files with a single disk failure. Buying new boxen was also not, due to the financial necessities that having an infant brings.
Then I encountered UnionFS. The idea sounded great - multiple filesystems / structures can be conjoined in to one tree, with the filesystem 'overlay' essentially representing them as one to the OS. Thus I could have 3 folders in different physical locations, all labeled 'TV', containing different content, and UnionFS would make one large 'TV' folder containing all of the files.
This seems to be a server technology that has primarily been used for the liveCD/diskless set, to enable such clever things as nonexistant root partitions being mounted from elsewhere. As such you would think it would be fairly well set up, documented, and reliable. Apparently not. Documentation is sparse, implementations sparser still, and apart from being installed as default in Ubuntu Gutsy I couldn't get much further than a basic single mount command to play with. As for known bugs, much was made of using read-only branches as there are horrors which "could even RESULT IN DATA LOSS", which is fine as these various disks are already accessible by the different machines adding content, and the test I did with a few simple local folders seemed ok. Until I added nfs into the mix.
On mounting a remote filesystem, and attempting to ls the tree, I was initially fine until I listed the 'Movies' directory. At this point the vnc server hung (and apparently the box). ssh was locked out. There was an almighty kernel 'oops'. Apart from the general swearing, a few key phrases from the resulting messes I encountered follow:
- Unable to handle kernel NULL pointer dereference
- call trace
- sys_readahead
- mount: special device none does not exist
- on attempt to umount - "device is busy"
To cut a huge story slight less huge, I gave up after a few hours googling. I then tried aufs, "Another Union File System" as it is called, in the great tradition of badly named unix programs. This is supposed to be installed by default in the next Ubuntu LTS release (Hardy Heron), but then again, UnionFS was released in Gutsy and was pretty broken. Now I am no slowpoke these days with linux, but I couldn't even work it due to lack of docs. "unknown filesystem type aufs" was one of the many errors I encountered, and following some compile-hell, I gave up.
Finally I encountered funionfs. This is mentioned as an afterthough on FUSE (Filesystem in Userspace), and as apparently is so with all of these union file systems, is horrendously documented, and relies on much prodding to check if it will work. The mount syntax is essentially similar to the other two discussed, but the crux of all of this ended up with the following simple method to get conjoined filesystems across nfs shares in Gutsy:
tom@Bender:~$ sudo apt-get install funionfs
tom@Bender:~$ mkdir ~/Myth
tom@Bender:~$ sudo mount -t fuse -o dirs=/mount/Zoidberg/Myth:/mount/Nibbler/Myth:/home/tom/Videos -o allow_other funionfs#NONE /home/tom/Myth
I am posting this in the hope it may increase WAF in your vicinity. The end-result is pretty damn seamless, and will make adding new media a simple process, even with fresh filestores. This sort of transparent filesystem joining is something I hope will become increasingly easy and transparent in the future, with Apple's grey(white) boxes and networked optical drives etc currently leading the way in the super-user-friendly variety. Gutsy and MythBuntu are not the best of friends right now, and I will approach Hardy with trepidation due to the inevitable breakage that will occur. That said, Linux has come a long way since I tried net-installing a floppy-bootstrapped slackware on a P200 some years ago due to a machine which didn't even have a CD-ROM drive.
*Update* Funnily enough, just a couple of days after posting this I see that Myth Tv 0.21 has added a feature called "Storage groups". This suggests that it does a similar function at the application level, but I cannot yet find sensible docs to figure out if this is a better solution.
linux | posted by Tom - 2008-03-10 14:40:14
comment on this post | (0) comments already
Safari On Windows
Safari For WindowsIn a brief moment between projects today, I had a glance at the beeb's RSS feed, only to see "Apple announces Windows browser". So on I click, assuming I'll see some roadmap for a Safari Windows port as has been rumoured on and off for years. But No! No roadmap, but a full public beta.
Is it surprising that they have ported it? No, it's been rumoured for a while, and frankly anything that makes the Windows user more comfortable with Mac UI design will ease transitions. Well, that's as maybe, but what surprised me about this was that it was done totally by stealth. I am a bit of a news junkie, and always keep up to date with a huge range of tech site news - /., el Reg, and more frivolous ones - boingboing, digg etc. I haven't even heard a rumour of this until it drops on the BBC (via Jobs at a SanFran apple dev. conference).
Why do I care that they did a stealth release? Well, essentially no matter how well coded a browser is, there's always its own little quirks. As a webdeveloper, it helps to know the quirks for all browsers above a ~1% market share. The more browsers, the more competition which is great, but I prefer to pick up their idiosyncrasies as they gradually gain mindshare. Firefox was a good example, Opera has always been around, and now that Ubuntu is pushing forward on the desktop, it's worth taking a look at whether that has issues (Media integration with Firefox being one).
With this release though, there's a certain instant Mac fanbase trapped at work on Windows PCs who will jump at it. Then there's the inevitable... Jobs is a great marketer, so he wants a good marketshare what does he do? Integrate some minor part of iTunes into Safari, and BLAM. 5-10% marketshare from the iPod fanboi contingent.
First in my line of weird quirks?
For some reason unbeknownst to me, the default font that Safari chooses is a rather illegible "Brady Bunch" font from the depth of my font repository.
tech | posted by Tom - 2007-06-11 12:26:04
comment on this post | (0) comments already
Warning - visiting this web site may harm your computer!
Try this link, and click the top result: http://www.google.com/search?q=bugs+lyrics+lyricsandsongsYou should find that rather than immediately getting the lyrics to Pearl Jam's 'Bugs', you get a warning about potential bugs. If it is not operating in your local google, this link should show you the google error page I am getting.
___
Warning - visiting this web site may harm your computer!
You can learn more about harmful web content and how to protect your computer at StopBadware.org.
Suggestions:
* Return to the previous page and pick another result.
* Try another search to find what you're looking for.
Or you can continue to http://www.lyricsandsongs.com/song/817.html at your own risk.
Advisory provided by Google
____
Now I'm not sure where I stand on this. On the good side, this stops people from maliciously infecting machines if the user is foolish enough to use IE on the open internet - the online equivalent of procuring prostitutes in phuket without prophylactics - or they compulsively click "Yes please download my dodgy software"
The obvious 'con's of this are it is purely reliant on the quality of Badware.org's data, potentially open to abuse by interested parties, and also opens doors to further censorship potentially in the future...(Warning - visiting this web site may harm your Mind! - immediately occurs.)
I won't enter into a tedius discussion of potential liabilities GOOG could face as a result - that can be left to Slashdot - but I will say if you're worried about censorship of your search results, then Google is already doing that (although under litigious duress usually).
In all I'd still say it falls under 'do no evil'...but probably going to annoy me more than anything, as I'm now (apart from one sandboxed testing machine at work) a pure Ubuntu operation, so it's one more click for no gain.
If it cleans up the interweb a bit though, a good thing, though people will no doubt always find idiotic ways to infect themselves.
web | posted by Tom - 2006-11-24 11:57:21
comment on this post | (0) comments already
Exporting mail from Opera's M2 email client and Importing to Thunderbird
This is brief guide to Exporting email from Opera's M2 client and importing into Thunderbird, this guide assumes you're running on a windows system (XP for this example).
A nice Slovakian chap named Vello emailed me to ask if I had any idea how to do this. As it happens, when I was switching from Opera to Firefox I had cause to do exactly this - rather than keep the (admittedly rather nice) M2 client for my email, I decided to have a go with Thunderbird - the companion email client to Firefox.
Thunderbird only has an import function from set clients (Outlook, Outlook Express, Eudora, Communicator 4.x), and has no generic function for importing just 'Mail'. I had seen rumour that using Eudora as a go-between with Opera and Thunderbird would work, but it sounded convoluted. Turns out it's much more simple than that.
Disclaimer: This may not work in your setup. It was tested under Windows XP, Opera 9 & Thunderbird 1.5. Even with this setup there is a small chance you could lose your email, if this happens, it's not my fault :-D
Always make a backup of important things, mails, etc, before fiddling with your system, and make sure to backup the mbs files you export from opera
So...you have folders you would like to export from opera. I will use 3 folder for this example. 2 are standard Opera folders: "Received" + "Sent". One will be a custom folder: "Work" (you may not have custom folders, in which case it's even easier).
• First you export the folders you want from Opera (Right-click->Export), which will give you 3 files: Received.mbs , Sent.mbs , Work.mbs
These 'mbs' files are actually standard Unix mbox files, which are in fact what Thunderbird uses. Stupidly though, its 'import' utility is just from other programs, so we have to do this manually.
• Create your new account in Thunderbird...(tools->account settings->Add account)...put in all your settings...make sure you _Un-check_ the 'Use Global Folders' option if you want to keep multiple email accounts separate.
If you have a custom folder ('Work' in this example), create that now in your new account.
Now you close Thunderbird.
• If doing this on Windows XP, then navigate to folder that is like: c:/Documents And Settings/YOUR_USER/Application Data/Thunderbird/Profiles/xbsfhafaf.default/Mail/pop.yourPOPserver.com/
In this folder you will see sets of 2 files for each mail folder...there is one with '.msf' extension...this is just the data about the folder. Then there is one with no extension - this is the actual mails for that 'mail folder'.
• All we need to do is replace these 'no extension' files with our 'mbs' files, and rename them properly (keep a backup though ;-). So in my example:
- Delete Inbox (no extension) -----> copy+paste 'Received.mbs' into folder ----> Rename to 'Inbox'
- Delete Sent (no extension) -----> copy+paste 'Sent.mbs' into folder ----> Rename to 'Sent'
- Delete Work (no extension) -----> copy+paste 'Work.mbs' into folder ----> Rename to 'Work'
Now open Thunderbird, and all your mail should appear. Hurrah! for Tom you cry in wonderment.
It may all look 'unread'...so quickly 'right-click -> Mark folder as read' if you want to correct that.
This should hopefully have worked. If there's any bits that don't, send me details of your setup and what went wrong (tom_walsham AT yahoo DOT com - assuming your email still works ;-) , and I can modify these instructions accordingly.
Opera | posted by Tom - 2006-08-16 09:04:14
comment on this post | (1) comments already
Who do I use for my hosting?
So, I thought I'd stick in a quick piece about my webhost.Over 1 year in and Dreamhost continue to impress me with their service and value for money. A quick list of services (on their Cheap! plan $7.95/mo):
- 1 TB bandwidth (1000GB) per month. This increases by 8GB a week
- 20GB storage. This increases by 160MB a week
- Unlimited Domains Hosted
- Unlimited MySQL databases
- 3000 email accounts
- PHP4/5, Ruby on Rails,
- Full Unix Shell w/ crontab access
- Loads of 1-click installs: Wordpress, phpBB, mediaWiki, Gallery..etc...
- Analog powered stats on all domains and subdomains
- Real + Quicktime streaming
- 1 free .com domain with signup. ($9.95/yr registration for each additional)
They also add new services and bonuses with no additional fees every month
Anyway, I thought I'd plug it as the service is excellent, the downtime minimal, and the pipe is good. Also, as a bonus, if you sign up now, and use promo code TOMWALSHAM when prompted, you will get a $20 discount!
Sign up here:
web | posted by Tom - 2006-04-28 10:44:53
comment on this post | (0) comments already
powered by Walsham Blog v.0.1
