Why Arch and not Ubuntu? Why Linux at all?

Although my original intentions were to install Linux, I first tried XBMC on the AppleTV OS - mainly to see how it performed. I was quickly annoyed by the fact you cannot autolaunch XBMC anymore (you can, but it will lose focus with recent firmwares, and to my knowledge the AppleTV OS has no way of calling XBMC back to the foreground). I'm picky and demanding, so that's not the only thing that irritated me. The limited shell on OS X put me off too. Although the shell itself is more or less fine, the limited selection of programs (and their limited possibilities) was disappointing. The long-term maintainability is not all that either; you need to hack the AppleTV firmware each time a new one comes out; basically you'll be performing the same steps (installing the Broadcom CrystalHD kext, installing TurboKext Enabler, modifying startup files etc.) over and over. Although the AppleTV Patchstick creator takes a lot of work out of your hands, you still have to go through the same setup procedures. I for one am not looking forward to that, and if Apple decides to stop support for the device, you're on your own too (I know, I am a positive thinker). Pit that against a rolling release distro with a proven track record, the familiarity of a real Linux environment, the flexibility of the Arch build system, and you know the outcome... Also, what would be the fun in just buying a device and using it like the manufacturer intended you to? If you can't hack it, there's no fun in it!

Goal of this post

It's not my aim to explain to you how you install the Broadcom CrystalHD mini PCI-E card. There are tons of tutorials around detailing it nicely (and it's not that difficult either). If you're not going to install Linux with the patchstick, make sure you have Torx T8 and T10 screwdrivers handy though.
I am not going to walk you through installing Arch on the AppleTV either - I will describe how to partition the internal HD prior to installation, because the EFI firmware expects a GPT layout. I'll be honest about this: I am a chicken, and I'm lazy. If I could have gotten the installer to work from the patchstick (just like you can e.g. install Ubuntu from it with their netboot kernel & initrd), I would have documented that. However, the Arch installer always borked at some point (be it a non-existent optical drive that it insisted on detecting, or a /proc/mounts directory that was not there either), leaving you with a half-finished setup (and I am not going to dig through the installer scripts to find out what's been done and what hasn't). In the end I decided to take out the HD (lame, I know) and hook up the 2,5" P-ATA HD to one of my Linux boxes. Either way, if you do it right, and if you are even a tiny bit Linux-savvy, then this will likely be the only time you have to take the HD out - ever. I'd like to refer to the Arch wiki on how to install Arch from within a running Linux system. You don't need to be running Arch for this, although having Arch on your box will simplify the installation. It also makes a lot of sense to back up your AppleTV OS before you attempt to install Linux. Since this has been thoroughly documented as well I will not cover this. It goes without saying that you'll set up OpenSSH to manage your box remotely (unless you will use a keyboard and go sit in front of your HD TV).
Last but not least: I don't expect you to be able to think in assembler; you're expected to have some background in Linux though and knowledge of basic conventions. That, and a sane dose of critical thinking ;-).

Required packages

There are a few packages you'll need to get this going. A concise list of what you'll need:

• Libbsd: Some common BSD functions. Required to build Apple's file system tools (fsck, mkfs, etc.) The package is also available in the AUR, however, it installs the headers to a peculiar location where the compiler won't pick them up. My PKGBUILD fixes that problem.
• Hfsprogs: Tools to format and check partitions formatted with Apple's HFS/HFS+ file systems. Again, this PKGBUILD is different from the one in the AUR - apart from a few cosmetic changes, mine has been updated to the latest 332 version of Apple's tools (332.25), and it uses the Debian patches. I tried patching the latest (491 at the time of writing), but the patch by Mythic Beasts for 332.14 breaks on way too many spots for me to be able to fix it reliably. My C is only superficial, and messing around with file system code is not exactly my day-to-day job...
• Parted: At the time of writing the latest version is 2.2. I checked the code changes introduced by the patch for Parted 1.8.8 that enables support for the GUID of the AppleTV's recovery partition, and every single line is still there. This means you do not need anything besides a recent Parted. If you really want to stick with 'what works', you can use the pre-patched 1.8.8 provided by the atv-bootloader project.
• Dmg2img: Uncompresses Apple's compressed DMG files to the HFS+ IMG format. Needed to extract the boot.efi file from the AppleTV's firmware. You can grab it from the AUR. At the time of writing, the latest version is 1.6.1, but the PKGBUILD hasn't been updated yet.
• CrystalHD kernel driver: stable 1.0.3 release for the Arch LTS kernel.
• Crystal HD kernel driver: SVN version for the Arch LTS kernel (r137).
• CrystalHD library: stable 1.0.3 release.
• CrystalHD library: SVN release (r137).
• Atvtool: A utility that allows you to control the LED, the infrared receiver and the fan of the AppleTV. Atvtool can create the /dev/usb/hiddev0 device necessary for Lirc to handle the Apple remote. Note that atvtool cannot control the IR receiver and the LED simultaneously.
• Atvclient: Like atvtool, it controls the LED and IR receiver (but apparently not the fan, which seems to work fine though). You can find a git PKGBUILD here.Keep in mind you can't run atvclient and atvtool simultaneously. If you don't like running git versions, know the stable version does not support the so-called universal mode, which means you're only able to use the Apple Remote (or a universal remote with the Apple remote's buttons mapped, which can be confusing because you only use a few buttons).
• XBMC: Acclaimed and "award-winning free and open source (GPL) software media player and entertainment hub for digital media" [sic]. Might sound cocky, but guess what: they have the right to. You'll need an SVN build; be adventurous and build one yourself with the stuff in the AUR.

If you don't want to build yourself, you can always get the (i686) packages form my own repo. There are PKGBUILDs in the AUR for the crystalhd library, firmware and kernel driver - the GIT PKGBUILDs essentially use the same code as my SVN PKGBUILDs.

Obtaining the necessary files

You need a few AppleTV-specific EFI files to get Linux to boot on the AppleTV. EFI files from a full-blown OS X do not seem to work; the EFI firmware rejects them. You don't need to pry open your AppleTV to get access to the EFI executable though; you can get them from Apple itself - through the Apple TV firmware updates. If you're interested in the technical details, the AkwardTV Project has some in-depth information about how the boot.efi executable works.

• The AppleTV firmware
  At the time of writing the latest is 3.0.2. Apple has been progressively removing older updates; the oldest firmware you can get from its site is 2.4.0. If 3.0.2 also fell off the earth by the time you find this post, then you can track the firmware releases on this site.
  Convert the image using dmg2img.
  $ dmg2img 2Z694-6013-013.dmg atv.img:
  Mount the converted image and extract the boot.efi file. If you do this on a Linux Live CD, note loopback mounts do not work under unionfs. You can get around this by using a tmpfs partition, but be sure to have enough RAM - the uncompressed .img file is 699 MB, of which a little under 500 MB is actually being used according to df.
  $ mkdir atv-update
# mount -o loop -t hfsplus atv.img atv-update
# cp -ap atv-update/System/Library/CoreServices/boot.efi .
  Make sure permissions on the file and byte count are correct. According to the atv-bootloader guide, the byte count should be 298800, and the checksum 280323d8700e4cfef15116f7e50590e3:
  $ ls -l boot.efi
-rw-r--r-- 1 stijn users 298800 jan 30 11:36 boot.efi
$ md5sum boot.efi
280323d8700e4cfef15116f7e50590e3 boot.efi
  In fact, it turns out that Apple hasn't updated these files (same byte count, same checksum) since the first firmware update came out.
• The atv-bootloader
  To quote the project's homepage, atv-bootloader uses "principles from mach_linux_boot to boot a compiled-in Linux kernel and then finds and boots another Linux kernel using kexec (a user-land kernel bootloader). In addition, atv-bootloader translates several EFI structures into standard PC bios structures. This allows a standard Linux kernel to be booted without the numerous EFI patches required by both mach_linux_boot and mb_boot_tv." Download the latest version from their homepage.

Partitioning the HD

By now I am assuming you dug up a 25 to 40 pin IDE adapter, or managed to get hold of a 2,5" HD case. Either way, your OS has access to the disk. I am assuming it's the second HD (/dev/sdb). Note you do not really need the OSBoot and EFI partitions, include them if you want to play safe - the space loss is negligible. I assume, if you leave them out, you still need to observe the sector 40 boundary and simply need to start the first partition at that sector.

• First you have to zero the HD (according to the documentation of the atv-bootloader project the GUID won't change if you don't):
  # dd if=/dev/zero of=/dev/sdb bs=4096 count=1M
• Sync the partition tables and create the GPT:
  # partprobe /dev/sdb
# parted -s /dev/sdb mklabel gpt
• Find out the size of your HD (should be 40 GB (old model) or 160 GB (recent model), unless you use another HD):
  # parted -s /dev/sdb print|grep Disk
• Create a 25 MB EFI partition and set the boot flag. It is vital that the partition starts at the 40th sector:
  # parted -s /dev/sdb mkpart primary fat32 40s 25M
# parted -s /dev/sdb set 1 boot on
• Create a 25MB "Recovery" partition:
  # parted -s /dev/sdb mkpart primary HFS 25M 50M
# parted -s /dev/sdb set 2 atvrecv on
• Create a 25MB "OSBoot" partition:
  # parted -s /dev/sdb mkpart primary HFS 50M 75M
• Now we create the Linux partitions. Keep in mind only ext2/ext3 are supported by the atv-bootloader. You are of course free to pick your own layout. Since I'll be using ext4 for my partitions, I'll create a separate /boot partition (125 MB should be plenty, unless you want to play with a lot of kernels):
  # parted -s /dev/sdb mkpart primary ext3 75M 200M
  A root partition, a bit less than 10 GB should be more than sufficient:
  # parted -s /dev/sdb mkpart primary ext3 200M 10G
  A /var partition (3 GB is common):
  # parted -s /dev/sdb mkpart primary ext3 10G 13G
  A /home partition that fills the rest of the HD (with some space left for swap):
  # parted -s /dev/sdb mkpart primary ext3 13GB 159.5GB
  Finally, a swap partition. With 256 MB RAM, 512 MB should suffice, but you can make it 1 GB if you want (not that you'll have a lot of use for it, my system (old CrystalHD library) usually doesn't swap more than a few tens of MB). After that, sync the partition tables and print the layout to assure yourself you didn't mess up:
  # parted -s /dev/sdb mkpart primary linux-swap 159.5GB 160GB
# partprobe /dev/sdb
# LANG=C parted /dev/sdb print
Model: ATA Hitachi HTS54161 (scsi)
Disk /dev/sda: 160GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

  Number Start End Size File system Name Flags
1 20.5kB 25.0MB 25.0MB fat32 primary boot
2 25.0MB 50.0MB 25.0MB hfs+ primary atvrecv
3 50.0MB 75.0MB 25.0MB hfs+ primary
4 75.0MB 200MB 125MB ext3 primary boot
5 200MB 10.2GB 10.0GB ext4 primary
6 10.2GB 13.2GB 3000MB ext4 primary
7 13.2GB 159GB 146GB ext4 primary
8 159GB 160GB 1042MB linux-swap(v1) primary
  Note I set up 1 GB of swap, but that's probably a bit too much (and I don't feel like repartitioning and extending /home for an extra 512 MB). You probably might not see the file system types not listed; don't worry - that's because the partitions have not been formatted yet.
  If it all looks fine, you can format the partitions:
  # mkfs.msdos -F 32 -n EFI /dev/sdb1
# mkfs.hfsplus -v Recovery /dev/sdb2
# mkfs.hfsplus -v OSBoot /dev/sdb3
# mkfs.ext3 -b 4096 -L boot /dev/sdb4
# mkfs.ext4 -b 4096 -L root /dev/sdb5
# mkfs.ext4 -b 4096 -L var /dev/sdb6
# mkfs.ext4 -b 4096 -L home /dev/sdb7
# mkswap /dev/sdb8
# sync

Since I had no clue how big the partitions would need to be, I made a conservative guess. I have resized my setup after I found out the installation itself barely covers 1 GB:
[stijn@mnemosyne ~]$ LANG=C df -T|grep sd
/dev/sda5 ext4 3.0G 968M 1.9G 34% /
/dev/sda4 ext3 58M 13M 42M 24% /boot
/dev/sda6 ext4 1.6G 137M 1.4G 10% /var
/dev/sda7 ext4 142G 123G 13G 91% /home

Swap space is 256 MB now, but with the new CrystalHD API in mind (which, as davilla stated, uses 16 DMA buffers instead of 8, and will therefore require more RAM), you might consider allotting 512 MB. I'm not sure if a bigger swap partition (or file) is useful. If you already partitioned, don't worry; you don't need a swap partition per se, you can also use swap space, and reconfigure fstab accordingly, or combine both.

Installing the atv-bootloader

• Create temporary mount points and mount the Recovery and OSBoot partitions read/write:
  $ mkdir {osboot,recovery}
# fsck.hfs+ /dev/sdb2
# mount -o rw /dev/sdb2 recovery/
# fsck.hfs+ /dev/sdb3
# mount -o rw /dev/sdb3 osboot/
• Untar the recovery tarball you got from the atv-bootloader site and copy its contents into the directories. Make sure you preserve permissions:
  $ tar xf recovery-0.6.tar.gz
# cp -arp recovery/* osboot/
# cp -arp recovery/* recovery/
• The finishing touch: copy over the boot.efi file.
  # cp -ap boot.efi osboot/
# cp -ap boot.efi recovery/

The atv-bootloader will look for legacy grub configuration files by default. You do not have to install GRUB into the MBR but make sure you install the package itself with pacman; configure your menu.lst well and atv-bootloader will boot straight into Arch Linux. In case you might want to do some fiddling from within the atv-bootloader environment, you can edit the com.apple.Boot.plist file (in the recovery directory) and change atv-boot=auto to atv-boot=none. This way atv-bootloader will boot into a command-line environment, activate the eth0 interface and obtain an IP address through DHCP, and also activate a telnetd and print the network configuration so you can log in remotely.

There are a few things you need to set up to get the hardware recognised the right way. I'll cover this concisely, along with some (imho) interesting background information.

HDMI audio

Since the audio chip is very similar to the one found in iMacs, we'll tell the driver to use that model as a reference. You need to put this info in a text file under /etc/modprobe.d. There's only one requirement: make sure the name ends in .conf. If not, the driver will be probed without the configuration file being parsed. For clarity, I put all the AppleTV-specific module options in a file called appletv.conf:

[stijn@mnemosyne ~]$ grep imac /etc/modprobe.d/appletv.conf
options snd-hda-intel model=imac24

If you're interested in seeing what models can be specified, check the HD-Audio-Models.txt file that comes with the alsa-driver tarball.

If everything has been set up right, you should have a HDMI audio device:

[stijn@mnemosyne ~]$ aplay -l
**** List of PLAYBACK Hardware Devices ****
card 0: Intel [HDA Intel], device 0: ALC889A Analog [ALC889A Analog]
Subdevices: 1/1
Subdevice #0: subdevice #0
card 0: Intel [HDA Intel], device 1: ALC889A Digital [ALC889A Digital]
Subdevices: 1/1
Subdevice #0: subdevice #0
card 0: Intel [HDA Intel], device 3: ATI HDMI [ATI HDMI]
Subdevices: 1/1
Subdevice #0: subdevice #0

Notice the device (ironically) shows up as an ATi HDMI audio device. While this might seems utterly bizarre, it turns out the HDMI audio is handled by a Silicon Image SiI1390 chip, which is already supported by ALSA (you need at least 1.0.18a). Unfortunately, only the 100.14.19 version of the nVidia binary driver seems to allow HDMI audio. Newer drivers allow audio to be carried over HDMI - until X is loaded. The problem: the 100.14.19 driver won't build against more recent kernels (Arch is at 2.6.32 at the moment) and Xorg versions. If you're skeptical (it does sound a bit wimpy indeed), try playing a WAVE file over the HDMI device, before and after X has loaded:

[stijn@mnemosyne ~]$ aplay -D hw:0,3 *wav
Playing WAVE '50 Cent - Get Rich Or Die Tryin' - 01 - Intro.wav' : Signed 16 bit Little Endian, Rate 44100 Hz, Stereo

For now, I have decided to use the RCA jacks and connect them to my stereo amplifier; needless to say I would love to see HDMI audio working.

Getting the IR receiver to work

The AppleTV's IR receiver is a bit of a strange beast. Before firmware 2.3 it would only talk to Apple's own remote. Now you can also use other remotes (like the Logitech Harmony) with it - the so-called 'universal mode'. I have had no success setting up Lirc to work with the Apple remote (or my Logitech Harmony for that matter); I am a novice with Lirc however, so that doesn't mean you can't get it to work if you try. If you want to try Lirc, you'll need to use atvtool to get the /dev/usb/hiddev0 device Lirc wants to talk to. I have inquired about this on the XBMC forums, and apparently you can get the Apple remote to work (in legacy mode at least), but since you can only use the Apple remote for now, even with Lirc, there is no advantage to setting it all up.
For now I've settled for atvclient, which handles the Apple remote just fine. It appears support for universal mode just got added to the git tree :-). This should allow you to use your universal remote (e.g. a Logitech Harmony) with the AppleTV, with some extra buttons compared to the spartan and squeaky plastic thing Apple delivered. Atvclient will log any pressed keys, so make sure you either pipe the output to some log file or /dev/null. As of July 25th, you can use all the buttons on your Harmony remote thanks to the work done by Zeldi. Installing the latest atvclient-git package from AUR will get you up to date.

Controlling the LED

Running atvclient will also set the LED straight. Be gone, ye annoying blinking orange! You can use atvtool, but from the moment you use it to set up the IR receiver it will lose control of the LED.
While atvclient will just set the LED to static white (it will flash when it receives commands), you can have atvtool set it to whatever the LED supports (orange, white, blinking, static, ...).

Enabling the Broadcom CrystalHD card

This literally is a walk in the park. You install the kernel module, the userspace library and the firmware (which comes with the kernel module package, if you use my PKGBUILDs), load the kernelmodule, check dmesg if it got loaded properly, and all is set for XBMC to pick it up. Keep in mind driver and library need to be paired!

$ dmesg |grep -i crystal
Loading crystalhd 0.9.27
$ dmesg |grep -i broadcom
Broadcom 70012 Decoder 0000:02:00.0: enabling device (0000 -> 0002)
Broadcom 70012 Decoder 0000:02:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Broadcom 70012 Decoder 0000:02:00.0: irq 26 for MSI/MSI-X
Broadcom 70012 Decoder 0000:02:00.0: setting latency timer to 64
$ dmesg |grep -i bcm
Starting BCM70012 Device
Stopping BCM70012 Device
Starting BCM70012 Device
$ lspci -v|grep -i crystal
02:00.0 Multimedia controller: Broadcom Corporation BCM70012 Video Decoder [Crystal HD] (rev 01)
Kernel driver in use: crystalhd
Kernel modules: crystalhd
Note XBMC will not pick it up on the fly, it needs to be (re)started after the module got loaded. Also note you'll need an SVN build for XBMC since 9.11 does not support the card. Since the end of July, the long awaited successor (BCM970015) can be bought online. Apart from reduced power consumption (touted to be 1W or less), the card supports hardware acceleration for Xvid/DivX, and it seems the HD support has had a few updates too, like support for H@L4.2 whereas the BCM970012 'only' supported H@L4.1 (if that's Chinese, but you're nevertheless interested, you can find more info on h.264 profiles here). If you order the BCM970012 or BCM970015 with Logic Supply, be sure to use the Xbmc coupon code which gets you a 10 USD discount. It is unclear to me however whether the BCM970015 support is stable - the CrystalHD code for Linux definitely supports it, though. As for the drivers, I find the stable 1.0.3 release (old API) to be far more usable than the SVN trunk (even the r137, which should be more or less usable). I am sticking with 1.0.3 until the new API stabilises.

Configuring your TV

By default my TV ran at 1920x1080 with a 50 Hz refresh rate, although it can do 60 Hz. I could switch it to 60 Hz (which I did after talking to a dev on the Xbmc Linux support channel) but apparently there was a deeper problem - nVidia TwinView being enabled. Grepping Xorg.0.log will apparently not say anything about that (which I find pesky, but maybe it's nVidia's fault and not Xorg's, who knows?).
Anyway, xrandr can give you a hint:
[stijn@mnemosyne ~]$ DISPLAY=:0.0 xrandr
Screen 0: minimum 320 x 240, current 1920 x 1080, maximum 1920 x 1080
default connected 1920x1080+0+0 0mm x 0mm
1920x1080 50.0 56.0 57.0 58.0 59.0 60.0*
Note that I set the refresh rate already to 60 Hz manually in the Xbmc settings dialogs; however, you'll notice the refresh rate increments from 56 Hz to 60 Hz and only starts at 50 Hz (a good TV should offer you 24 Hz too). This could indicate you have TwinView enabled. nVidia's documentation < a href="http://http.download.nvidia.com/XFree86/Linux-x86/1.0-8178/README/appendix-g.html">states you have to enable it explicitly, but experience learns you have to disable it explicitly, by adding this line to the Screen section of your xorg.conf:
Option "DynamicTwinView" "False"
When you restart X/Xbmc/your system, you'll see the xrandr output is different:
[stijn@mnemosyne ~]$ DISPLAY=:0.0 xrandr
Screen 0: minimum 320 x 240, current 1920 x 1080, maximum 1920 x 1080
default connected 1920x1080+0+0 0mm x 0mm
1920x1080 60.0* 50.0 24.0
As you can see, there is now an option to use 24 Hz. You'll want to set your display to 60 Hz by default, and tell Xbmc to adjust the refresh rate when necessary (50 Hz for PAL and 24 Hz for a lot of movies). You do this by enabling the 'Adjust display refresh rate to match video' option you'll find under System > Video > Playback.

Launching XBMC directly

Once you assured yourself X is working right - in fact, apart from your keymap there's not much you need to specify in an Xorg.conf nowadays, and the AppleTV is no exception to that - you can set up X to log your user in directly, and configure ~/.xinitrc to launch XBMC automatically. Since it will fill the screen, you do not need a WM (which is welcome in an environment where RAM is scarce).
Modify /etc/inittab to boot into X by default and add this line:
x:5:respawn:/bin/su $USERNAME -l -c "/bin/bash --login -c startx >/dev/null 2>&1"
Replace $USERNAME with the user XBMC will be running as, and make sure the line above is the only uncommented line for runlevel 5. I assume you set up X correctly (after all, you're an Arch user) so we can do away with the wrapper script that /usr/bin/xbmc is (for the SVN package in the AUR, although the stable release in [community] seems to have the same script). Remove the script, and symlink the executable to /usr/bin directly:
ln -sv /usr/lib/xbmc/xbmc.bin /usr/bin/xbmc
As for ~/.xinitrc, add the following:
exec /usr/bin/xbmc --standalone --nolirc &> /dev/null
Because we do not launch Xbmc through the wrapper script, this will save us a few kB RAM, and we can use every bit on the AppleTV. The script does not only check for correct X configuration, it also performs some other checks; it's up to you to decide whether to keep it or not. It is, however, not necessary.

Shutting down the system from within XBMC

To be able to shut down from within XBMC directly, you need to make sure that your X session registers correctly with D-Bus. You can do that by putting the following code in your ~/.xinitrc:
if which dbus-launch >/dev/null && test -z "$DBUS_SESSION_BUS_ADDRESS"; then
eval `dbus-launch --sh-syntax --exit-with-session`
fi
You also need to make sure your XBMC session is launched the right way; you do this by prefixing ck-launch-session to the XBMC call in your ~/.xinitrc:
exec ck-launch-session /usr/bin/xbmc --standalone &> /dev/null
If it all works, you should see something like this in your log (check ~/.xbmc/temp/xbmc.old.log after a reboot):
06:52:10 T:3030374288 M:158814208 DEBUG: CAnnouncementManager - Announcement: Shutdown from xbmc
I have not tested suspend or hibernate, but they should work in the same manner.

Getting the diagnostics overlay to show

To get the overlay that shows CPU load, buffers and codec info, you need to modify the default keymap for the Apple remote. Copy the keymap to your homedir:
cp /usr/share/xbmc/system/keymaps/joystick.AppleRemote.xml ~/.xbmc/userdata/keymaps/keymap.xml.
The name of the destination file does not really matter; what matters is the suffix (.xml). Then apply this patch to get the overlay to show. After restarting XBMC you'll be able to call that OSD by holding your Menu button - just like you can call the playback OSD for holding your Play button.
For your Logitech Harmony, copy the corresponding keymap file:
cp /usr/share/xbmc/system/keymaps/joystick.Harmony.xml ~/.xbmc/userdata/keymaps/keymap.xml
After that, you should apply this patch, which lets you call the diagnostics overlay by pressing the Guide button during fullscreen video playback. You can of course change this; the trick is to find a button that you don't use (or want to be mapped differently) and reassign it.

Mapping Apple remote buttons on your Logitech Harmony

If you like to use your Logitech Harmony with the AppleTV, you'll notice the Logitech software simply maps the Apple buttons onto the Harmony remote. With tons of buttons on the Harmony remote, that's a bit irritating. Since the Menu button on the Apple Remote is a true wonder of versatility, you might want to remap it in the Logitech interface. Assign the Menu call to the Exit (and if you want, the Info button) on your Logitech remote and life gets a little bit easier. To me it's very counterintuitive to reach for that Menu buttton to exit (context) menus all the time. Make sure that, if you set up an Activity with the software, you do not only map the buttons in the device profile itself, but also in the Activity - if you fail to do so your activity will still only use the Menu button on the Harmony. While you're in there, you can always take a look at mapping other buttons (but the options are fairly limited).

Advancedsettings.xml

The ~/.xbmc/userdata/advancedsettings.xml file is where you can put a lot of custom settings. Here are some I put in:

• <bginfoloadermaxthreads>: I have set this to 5, a sane value if you have a relatively underpowered system. This controls the number of background info loader threads.
• <nodvdrom>: set to 1, since I have no optical device. The wiki does not say this is an Xbox-exclusive function, so it might still help (unlike e.g. cache settings which only work for Xbox, despite people looking for them all over on non-Xbox systems).
• <gputempcommand>echo "$(nvidia-settings -tq gpuCoreTemp) C"</gputempcommand>: This nice snippet will make the temperature of your GPU show up in the system info screen of XBMC. Ironic you can display the GPU temperature but not the CPU temperature on the AppleTV, no? ;-)
• <cleanonupdate>true</cleanonupdate>: This setting will make Xbmc perform a cleanup of the library every time you update it to add new media. Without this you'll have to perform a cleanup manually. For me, it doesn't really seem to work - I still have to run manual cleanups.

Performance tweaks

• Enable XvMC for hardware-accelerated MPEG-2 playback
  Before you set this up, be aware the application you use for video playback needs to support XvMC too for it to work. At the moment XBMC's playback (which is based on mplayer) engine does not; I understand you can patch mplayer (I think it does not support XvMC out of the box on Arch). You can find detailed instructions on the XBMC wiki on how to use an external player. Since the hardware is more than adequate to play back SD DivX/XviD, I am merely providing this info in case someone wants to play with it.
  The BCM970012 variety of the Broadcom CrystalHD card does not support MPEG-2 hardware acceleration. The nvidia binary driver does, however. Enabling it is pretty easy. First install the libxvmc library. Then add this to /etc/X11/XvMCConfig:
  libXvMCNVIDIA_dynamic.so.1
  If the file does not exist, create it.
  The BCM970015 version does support hardware-accelerated DivX/Xvid playback, but it is very difficult to find at the moment. So if you can wait a bit, I'd advise to hold out for that one instead of buying the BCM970012. Not only does it have extra codec support, apparently the design is also said to be better and more reliable.
• Pick the right skin
  Although XBMC has a pretty nice skin by default (make sure you try the different views available - by default the List view is pretty spartan), you might want to try some of the other skins out there. The low amount of RAM pretty much limits you to 'normal' skins. With heavy ones like e.g. Aeon (which looks very smooth), the transition effects are very choppy, and some screens take a lot of time to load. As you might notice even fanart will at times display with a delay of a few seconds.
• Compress fan art
  As of revision 28714, XBMC has out of the box support for DDS thumbnails, next to the traditional .tbn format. Using .dds should speed up the display of fanart; however, it's a) SVN, so it comes with the usual warning signs and b) a brand new feature so there might still be some hiccups; if you use it and encounter any problems you are kindly requested to report them as such. To use the feature, enable the <useddsfanart> flag in advancedsettings.xml. Further tweaks were introduced with revision 28272. Generating these compressed images takes a bit of time, but browsing your library will work considerably smoother afterwards. There seems to be a slight degradation in quality though, but you can adjust the compression settings.
• Limit possible C-states
  If you do any substantial transfers over your LAN, you'll notice they'll kick in with a healthy 10 MBps - standard Fast Ethernet speeds. It quickly goes downhill from there though; after a few minutes you can have speeds of a few KBps (you read that right). The problem here is the Realtek NIC which does a lot of offloading on the CPU - a practice which has prompted a lot of people to make a comparison with the dreaded winmodems. The CPU, however, will think there is nothing to do (there is only a network transfer going on) and switch to a lower C-state - thus bogging down the network transfer. It appears the driver for the Realtek NIC does not know about the C-state switching. Especially when you stream to the AppleTV (and you often will, since 160 GB isn't a lot of storage by today's standards), this can be very irritating.
  To fix this, you should load the processor module with the max_cstate=2 option. The best way to do this is to add it to the kernel line in your GRUB configuration file:
  kernel /vmlinuz26 root=/dev/sda5 ro processor.max_cstate=2

  To make sure it works, perform a check after you rebooted:
  [stijn@mnemosyne ~]$ grep max_cstate /proc/acpi/processor/CPU0/power
max_cstate: C2
  A normal system will go to C8:
  [stijn@hermes ~]$ grep max_cstate /proc/acpi/processor/CPU0/power
max_cstate: C8

• Optimise compilation for your CPU
  The CPU is essentially a Pentium M, and GCC has a specific -march=pentium-m switch available. If you don't need debugging, you could also add -fomit-frame-pointer. I plan to play with those myself but have not done it yet, nor have I done any tests.

Power consumption and heat production tweaks

• Clock back the CPU when possible
  Install cpufreq-utils, set it up to use the ondemand governor and set the frequencies in its config file. Make sure you add it to your DAEMONS array. Davilla, one of the XBMC devs, pointed out you might want to use the performance governor to prevent race conditions between CPU and the Broadcom Crystal HD card (that is, if I understood his explanation right). So if you see stuttering during playback (note initial playback of 720p/1080p material may stutter a bit, but should stop once you're a few minutes far), call the codec OSD (the transparent bar over the top of your screen like you can see here). The vq (XBMC's video buffer) value should constantly be up in the 90-100% range (or right below that, but not for long). Alternatively you could install cpufreqd and configure a setting for HD video playback. Unlike cpufreq-utils, cpufreqd is a lot more flexible. You can define usage patterns based on which application is running (since XBMC will be running all the time, this is a bit useless in this case), but I believe you can also define profiles based on CPU load (which could be interesting) and lots of other variables. It can run as a daemon as well. As a compromise, you could set the minimum frequency in cpufreq's config file to 800 MHz instead of the 600 MHz that the CPU can switch to.
• Eliminate redundant interrupts
  Powertop shows that the CPU spends most of its idle time in a C3 state (with the max_cstate option unset, that is), while the kernel generates over 800 interrupts per second. Compare that with my laptop which has been tweaked a bit in that regard and 'only' generates roughly 300 interrupts every second, while having wireless and bluetooth running.
  Almost 75% of the interrupts on the AppleTV is being generated by atvclient. The remainder is almost exclusively generated by nVidia's binary driver. Once I kill atvclient, it drops down to a mindboggling 22 interrupts per second - although Powertop still reports the IR receiver to be active 100% of the time. Since the receiver is connected to the USB bus, setting autosuspend on the usbcore module does not really help. There's only a tiny disadvantage: shutting down atvclient will effectively kill our IR remote - and that's not what we want. Despite the spectacular decrease in interrupts, the AppleTV does not seem to draw less power. We can do something about the nVidia interrupts though, by adding the following line to the Device section in xorg.conf:
  [stijn@mnemosyne ~]$ grep Interrupts /etc/X11/xorg.conf
Option "OnDemandVBlankInterrupts" "true"
  This should shave off some 60 wakeups per second. Apparently this might cause performance issues, so if your playback is affected, revert the setting. I myself am not using this setting for the moment.
• Decrease clock speeds of GPU and VRAM
  The GPU can apparently get into problems when doing MPEG-2 hardware-assisted decoding through XvMC. Lowering the clock speeds apparently helps with this (and will also lower power consumption, of course). Add this to a modprobe configuration file:
  [stijn@mnemosyne ~]$ grep nvidia /etc/modprobe.d/appletv.conf
options nvidia_new NVreg_RegistryDwords="PerfLevelSrc=0x2222" NVreg_Mobile=0
  This will tell the nVidia driver not to consider the GPU a mobile GPU, so it will not adjust clock speeds based on GPU load. Then we'll permanently lower the GPU and VRAM clock, by adding this at the end of our ~/.xinitrc:
  ( sleep 5 && sudo nvidia-settings -a GPUOverclockingState=1 ) &
( sleep 5 && sudo nvidia-settings -a GPU2DClockFreqs=200,800 ) &
  Not only does this save power (and makes your AppleTV produce less heat), with the Broadcom CrystalHD card even these lower clocks are more than adequate for 720p and 1080p playback.
• Sound device powersaving
  Enable sound powersaving by adding this line to a modprobe configuration file:
  [stijn@mnemosyne ~]$ grep power_save /etc/modprobe.d/appletv.conf
options snd-hda-intel power_save=10
• Increase VM dirty writeback time
  By increasing this value the kernel will need to access your HD less frequently. This is how it looks:
  <[stijn@mnemosyne ~]$ grep writeback /etc/sysctl.conf
# Increase VM dirty writeback time from 5 to 15s
vm.dirty_writeback_centisecs=1500
  Default is value 500.
• Make HD spin down faster
  You can use hdparm to tune your spindown intervals. Especially if you're streaming to your AppleTV this is handy since most of the stuff will be in RAM anyway, and I presume that, despite the 256 MB RAM, the streamed data is cached in RAM anyway. So eventually your HD will spin down. Especially if you play from the local HD though the benefits of faster spindown times are debatable.

Power consumption

With the lowest possible C-state locked to 2, my Cresta RCE-1106, a dirt cheap and yet quite accurate energy meter for being a consumer model, measured 12-14 kWh idle. When that limit was lifted, power consumption stayed (surprisingly) the same. Notice that these results were obtained with a lower GPU clock and the CPU running with the 'ondemand' governor, which clocks it back to 600 MHz whenever possible and also should lower the CPU voltage. XBMC's screensaver function kicks in after 3 minutes, and later on the screen goes completely blank. The values were recorded were done well after that. Under full load the device consumes less than 25W, which is not bad at all, especially for an x86 setup.

Known issues and design flaws

• PSU limitations
  The built-in PSU does not support a hardware shutdown; you can shutdown the OS but the device itself will not switch off, unlike a normal box. You can 'solve' this by using a power strip with its own power switch. I don't get why Apple did this - some say it has to be 'always-on' because it can also act as a backup device for Apple computers in the network through Time Capsule, but the simplicity model clearly shows its limits here. Wasn't California the state with the semi-permanent power outages a while ago?
• Lack of temperature diode
  To my knowledge there is no temperature diode in the Intel Crofton CPU. Which (unfortunately) makes sense, since it's based on Intel's second generation Pentium M 'Dothan', and that processor line lacks temperature diodes altogether.
• HDMI audio
  As discussed previously the only nVidia binary driver to date supporting audio over HDMI is 100.14.19, which is tied both to an older kernel and an older Xorg server. Expect to downgrade (and compile yourself) a lot of packages if you wish to go down this road. Oh, and breakage too ;-).
• Poor network performance
  As mentioned above, limiting the possible C-states barely impacts the power the AppleTV draws. To illustrate how much rubbish this Realtek NIC is, once more: a transfer over a 100 Mbps LAN will easily make your CPU spike at 70% or more during the file transfer. With 720p playback over the network and the Broadcom Crystal HD add-in card used, the CPU is used for 30%. Personally, I find this a huge turnoff - I'm not a 1080p addict, but this makes the device even less futureproof.
• Limited RAM
  Although XBMC runs just fine with 256 MB RAM, and Full HD playback is no problem, there's not that much headroom. As seen before installing heavy skins (especially Full HD ones) will make the AppleTV swap heavily (at one time to the point where XBMC got killed).

Stability with a rolling release: the LTS kernel

When I installed Arch on the AppleTV, the 2.6.33 kernel and 190.53 nVidia binary driver worked well together - Full HD playback was smooth. However, a few upgrades later (a newer 2.6.33 release and 195.xx nVidia drivers) HD playback was quite choppy. Since success can vary a lot depending on what kernels and drivers are available, I decided to switch to the LTS kernel (which just got bumped to 2.6.32) and build the 190.53 nVidia drivers for those. You can find packages here, if you're interested in PKGBUILDs you can find them here. To give you an idea about the stability: the previous kernel26-lts version (2.6.27.xx) has been around for almost ten months.

Minimising the memory footprint

Running a full-blown media center on a system with 256 MB RAM can be a bit of a chore if you want Full HD playback. While initial SVN build ran fine, with few swapping activity, if any, I noticed more recent builds (r32xxx) are more greedy. I have no hard data on this though - installing older releases would not be able to access the library and I don't know whether this affect RAM usage or not.
There is another factor at play though; Xbmc builds linked to the old CrystalHD library (1.0.3) are more memory-efficient than those linked to the library based on the new API (which davilla declared stable for end users with SVN revision 137). I do have hard numbers on this.
Memory consumption for Xbmc with CrystalHD 1.0.3 (old API):

[stijn@mnemosyne ~]$ free -m
             total       used       free     shared    buffers     cached
Mem:           243        239          3          0         11         91
-/+ buffers/cache:        136        106
Swap:          257          0        257

Memory consumption for Xbmc with CrystalHD SVN r137 (new API):

[stijn@mnemosyne ~]$ free -m
             total       used       free     shared    buffers     cached
Mem:           243        238          4          0          9         61
-/+ buffers/cache:        168         75
Swap:          257          1        256

The tips I rounded up are aimed at resident memory usage (running daemons for example), not at active programs. Here are a few things you can do to trim your memory usage:

• Replace OpenSSH with Dropbear
  Dropbear's memory usage is roughly half of OpenSSH's. In addition, Dropbear will only fork one instance per SSH connection, whereas OpenSSH forks two instances. There's a tradeoff of course - it isn't as full-featured as OpenSSH. It can handle OpenSSH's key files, so if you used keys before with OpenSSH, they should work with Dropbear just fine.
  If you need sftp and scp, you'll still have to keep OpenSSH around. Dropbear will pass any incoming 'calls' for sftp or scp to the OpenSSH binaries. Note that you do not need OpenSSH running for this; just installed.
  You can grab Dropbear from the AUR. Migrate wisely: make sure your Dropbear setup is fully functional (on another port than the one your OpenSSH daemon is running on) before you deactivate or remove OpenSSH, and that you added it to your daemons array.
• Limit the number of virtual terminals
  Linux, by default, creates six ttys you can log on to. Each tty needs a bit of RAM. You can disable five out of six, and if you need multiple terminals, you can install the screen utility.
• Replace Bash with a lighter shell
  I haven't tried this one yet, but I am planning to. The only angle is whether the startup scripts (especially the initscripts, but also the daemon scripts) have Bash-specific code in it or not. If they do, things will break - badly.
• Call Xbmc directly from within .xinitrc
  As explained above this will cut out a bash instance, so that will shave off a few kB.
• Tuning swap
  A last measure you can take is tuning swap. There are two values for that - vm.swappiness and vm.vfs_cache_pressure. Both can be set with the sysctl utility through its configuration file /etc/sysctl.conf. Default values are 60 for the former and 100 for the latter. Increasing swappiness will mean your system will swap out stuff more quickly. If you're on a desktop, you don't want this, but since all you'll probably use on this setup is Xbmc itself, this can be useful. It's up to you to decide what value you set it to. The maximum is 100. Most people will know the swappiness setting already. As for vfs_cache_pressure, according to Neil Horman's Understanding Virtual Memory In Red Hat
  Enterprise Linux 4, it "adjusts the bias on reclaiming inodes and dentries vs. reclaimation via swap and pagecache. The default setting of 100 provides a “fair” balance between the reclamaition of the often used data structures and swap/pagecache. Reducing this value biases the VM to prefer reclaimation of memory using swap and pagecache, while increasing it biases the VM to reclaim memory by flushing inodes and dentry structures." You can find the full document on RedHat's site. Even with that explanation, I am not sure I fully understand how it interacts with swap, so I won't recommend anything, except: try altering the values and see what benefits your system. The sysctl.conf values are vm.swappiness and vm.vfs_cache_pressure
• Disabling HAL
  Although HAL is being phased out and its functionality is being integrated in udev, devicekit e.a., most systems will still install or use it. Xorg does not need HAL anymore from 1.8 on, and Xbmc can use HAL, but it is not mandatory, so you can compile it without, which will earn you a few extra megabytes headroom.
• Compiling with -Os
  In addition, you could tell GCC to optimise for size by using -Os. I haven't tried this yet to see how this works out - it's pretty much a tradeoff between less RAM usage and better performance, so I leave that up to the reader.

Final thoughts

It has been an absolute nightmare to get this working like I wanted - mostly because it took me a while to realise the problem was with Mediatomb (or UPnP shares altogether, who knows?) and XBMC's library feature. However, from the pink screen issues I experienced the first time to finding out using UPnP shares and XBMC don't play nice, it was a true digital rollercoaster and I was glad to be on it. Once all that was sorted out, all the 'instability' problems (oom-killer killing XBMC during the library buildup), semi-random reboots (which I first thought were caused by overheating problems but will probably be the result of unstable SVN snapshots and the UPnP imbroglio too) disappeared, and the device has been running happily ever since. Library updates are quick, 1080p playback is smooth, memory usage is modest (with the old library). Of course the lack of HDMI audio is a bit of a letdown, but it's a choice - I could have gone with Ubuntu 8.04 (on which it is working) or with the AppleTV OS, but I chose not to. I have to bear the consequences ;-).

Acknowledgements

A big thanks goes to davilla, one of the XBMC devs. He (and others) put a lot of work into documenting the installation of Linux on the AppleTV and was instrumental in hacking Linux onto it. Without him and the XBMC community, we'd still be installing XBMC on Acer Revos (no flame intended ;-) ). I'd also like to thank Evinyatar for getting atvclient to support universal mode and bobo1on1 for pointing out the TwinView stuff and helping getting my TV configured right.

Last updated 2010-08-01