shameless self-promoting website
»Rejourn root
Posted Saturday, 11 December 2010 by Miklos
Tags: en frugalware

So we have two machines next to each other: genesis and helicon. The original idea years ago was to put a serial cable between the two so that it’s possible to manage the machines even in case ssh dies for some reason. Of course we never did that, because we were lazy - but in the meantime both machines were replaced by a newer mainboard, supporting Intel AMT.

In short, this is a poor man’s IP console, it supports serial-over-lan, so with a bit of configuration and a special client tool you can control grub, watch the bootup process (will you miss the risky few minutes after a reboot till it turned out if a remote machine boots up after a kernel upgrade or not?) and finally can even log in when ssh is not running.

First you need to set up grub to listen on the serial port as well. To do this, you need to know the device name and I/O port of your serial-over-lan port. The boxes mentioned above have one physical and one virtual serial port, so we need the later - that will be /dev/ttyS1. Now let’s grep in dmesg to get the port:

$ dmesg|grep 'ttyS1 at'
0000:00:03.3: ttyS1 at I/O 0xf1c0 (irq = 17) is a 16550A

Now we know everything to set up grub. Just prepend the following lines to your /boot/grub/menu.lst:

serial --port=0xf1c0 --speed=115200
terminal --timeout=30 console serial

The speed is AFAIK the same for each SoL port, so you can just copy&paste it.

It’s also recommended to change the timeout line to 30 as well, this way you will have half minute to press any key to tell grub you want to control is on the serial port and you’ll again have half minute to hit any key to edit the kernel parameters, etc. (The whole grub part is useful because this way in case you upgraded the machine and it fails to boot up, you can still boot using init=/bin/sh and fix the system.)

Second, you want to see the boot process, this way you can fix the system remotely, even if fsck fails and asks for root password, etc.

For this, you need to change your kernel parameters, so that messages sent to /dev/console will end up on the serial port.

I changed

root=/dev/md0 ro quiet vga=791


root=/dev/md0 ro vga=normal nomodeset console=ttyS1,115200

and that achieved this.

Third, you want to start a getty on ttyS1 once the machine booted up, in case ssh fails to start up. If you don’t use systemd yet, add the following line to /etc/inittab:

s1:12345:respawn:/sbin/agetty ttyS1 115200 vt100-nav

(Don’t forget to run init q to reload the config!)

Finally, you need amtterm on the client side - I plan to package it for Frugalware once 1.4pre2 is out. Till then you can just download, unpack and build the source tarball using make - no installation is necessary, you can run amtterm in-place. In my case I can’t connect directly to the machine as (for obvious reasons) the AMT is reachable from the local subnet only. So I build an ssh tunnel to the other machine and use socksify to pretend I run amtterm on the local subnet. The commandline I use is:

$ socksify ./amtterm -p <pass> <ip>

And that’s it - enjoy editing your kernel parameters, watching your boot process and login to the box even if ssh is not running and you don’t have IP console. :)

If you can’t try it out right now but you want to see it in action: grub in amtterm and boot in amtterm - without any virtualization! ;)