Perform the following kernel upgrade and FireWire modules install on all nodes in the cluster!

The next step is to obtain and install a new Linux kernel and the FireWire modules that support the use of IEEE1394 devices with multiple logins. This will require two separate downloads and installs: one for the new RHEL4 kernel and a second one that includes the supporting FireWire modules.

In a previous version of this guide, I included the steps to download a patched version of the Linux kernel (source code) and then compile it. Thanks to Oracle's Linux Projects Development Team , this is no longer a requirement. Oracle now provides a pre-compiled kernel for RHEL4 (which also works with CentOS!), that can simply be downloaded and installed. The instructions for downloading and installing the kernel and supporting FireWire modules are included in this section. Before going into the details of how to perform these actions, however, let's take a moment to discuss the changes that are required in the new kernel.

While FireWire drivers already exist for Linux, they often do not support shared storage. Typically when you logon to an OS, the OS associates the driver to a specific drive for that machine alone. This implementation simply will not work for our RAC configuration. The shared storage (our FireWire hard drive) needs to be accessed by more than one node. You need to enable the FireWire driver to provide nonexclusive access to the drive so that multiple servers—the nodes that comprise the cluster—will be able to access the same storage. This goal is accomplished by removing the bit mask that identifies the machine during login in the source code, resulting in nonexclusive access to the FireWire hard drive. All other nodes in the cluster login to the same drive during their logon session, using the same modified driver, so they too also have nonexclusive access to the drive.

Your implementation describes a dual node cluster (each with a single processor), each server running CentOS Enterprise Linux. Keep in mind that the process of installing the patched Linux kernel and supporting FireWire modules will need to be performed on both Linux nodes. CentOS Enterprise Linux 4.2 includes kernel 2.6.9-22.EL #1. We will need to download the OTN-supplied 2.6.9-11.0.0.10.3.EL #1 Linux kernel and the supporting FireWire modules from the following two URLs:

RHEL4 Kernels
FireWire Modules
Download one of the following files for the new RHEL 4 Kernel:

kernel-2.6.9-11.0.0.10.3.EL.i686.rpm - (for single processor)

or

kernel-smp-2.6.9-11.0.0.10.3.EL.i686.rpm - (for multiple processors)

Download one of the following files for the supporting FireWire Modules:

oracle-firewire-modules-2.6.9-11.0.0.10.3.EL-1286-1.i686.rpm - (for single processor)

or

oracle-firewire-modules-2.6.9-11.0.0.10.3.ELsmp-1286-1.i686.rpm - (for multiple processors)

Install the new RHEL 4 kernel, as root:

# rpm -ivh --force kernel-2.6.9-11.0.0.10.3.EL.i686.rpm - (for single processor)

or
# rpm -ivh --force kernel-smp-2.6.9-11.0.0.10.3.EL.i686.rpm - (for multiple processors)

Installing the new kernel using RPM will also update your GRUB (or lilo) configuration with the appropiate stanza and default boot option. There is no need to to modify your boot loader configuration after installing the new kernel.
Note: After installing the new kernel, do not proceed to install the supporting FireWire modules at this time! A reboot into the new kernel is required before the FireWire modules can be installed.

Reboot into the new Linux server:

At this point, the new RHEL4 kernel is installed. You now need to reboot into the new Linux kernel:

# init 6

Install the supporting FireWire modules, as root:

After booting into the new RHEL 4 kernel, you need to install the supporting FireWire modules package by running either of the following:

# rpm -ivh oracle-firewire-modules-2.6.9-11.0.0.10.3.EL-1286-1.i686.rpm - (for single processor)
- OR -
# rpm -ivh oracle-firewire-modules-2.6.9-11.0.0.10.3.ELsmp-1286-1.i686.rpm - (for multiple processors)

Add module options:

Add the following lines to /etc/modprobe.conf:

options sbp2 exclusive_login=0

It is vital that the parameter sbp2 exclusive_login of the Serial Bus Protocol module (sbp2) be set to zero to allow multiple hosts to login to and access the FireWire disk concurrently.
Perform the above tasks on the second Linux server:

With the new RHEL4 kernel and supporting FireWire modules installed on the first Linux server, move on to the second Linux server and repeat the same tasks in this section on it.

Connect FireWire drive to each machine and boot into the new kernel:

After performing the above tasks on both nodes in the cluster, power down both Linux machines:

===============================

# hostname
linux1

# init 0

===============================

# hostname
linux2

# init 0

===============================

After both machines are powered down, connect each of them to the back of the FireWire drive. Power on the FireWire drive. Finally, power on each Linux server and ensure to boot each machine into the new kernel.
Note: RHEL4 users will be prompted during the boot process on both nodes at the "Probing for New Hardware" section for your FireWire hard drive. Simply select the option to "Configure" the device and continue the boot process. If you are not prompted during the "Probing for New Hardware" section for the new FireWire drive, you will need to run the following commands and reboot the machine:

# modprobe -r sbp2
# modprobe -r sd_mod
# modprobe -r ohci1394
# modprobe ohci1394
# modprobe sd_mod
# modprobe sbp2
# init 6

Loading the FireWire stack:

In most cases, the loading of the FireWire stack will already be configured in the /etc/rc.sysinit file. The commands that are contained within this file that are responsible for loading the FireWire stack are:

# modprobe sbp2
# modprobe ohci1394

In older versions of Red Hat, this was not the case and these commands would have to be manually run or put within a startup file. With Red Hat Enterprise Linux 3 and later, these commands are already put within the /etc/rc.sysinit file and run on each boot.
Check for SCSI Device:

After each machine has rebooted, the kernel should automatically detect the disk as a SCSI device (/dev/sdXX). This section will provide several commands that should be run on all nodes in the cluster to verify the FireWire drive was successfully detected and being shared by all nodes in the cluster.

For this configuration, I was performing the above procedures on both nodes at the same time. When complete, I shutdown both machines, started linux1 first, and then linux2. The following commands and results are from my linux2 machine. Again, make sure that you run the following commands on all nodes to ensure both machine can login to the shared drive.

Let's first check to see that the FireWire adapter was successfully detected:

# lspci
00:00.0 Host bridge: Intel Corporation 82845G/GL[Brookdale-G]/GE/PE DRAM Controller/Host-Hub Interface (rev 01)
00:02.0 VGA compatible controller: Intel Corporation 82845G/GL[Brookdale-G]/GE Chipset Integrated Graphics Device (rev 01)
00:1d.0 USB Controller: Intel Corporation 82801DB/DBL/DBM (ICH4/ICH4-L/ICH4-M) USB UHCI Controller #1 (rev 01)
00:1d.1 USB Controller: Intel Corporation 82801DB/DBL/DBM (ICH4/ICH4-L/ICH4-M) USB UHCI Controller #2 (rev 01)
00:1d.2 USB Controller: Intel Corporation 82801DB/DBL/DBM (ICH4/ICH4-L/ICH4-M) USB UHCI Controller #3 (rev 01)
00:1d.7 USB Controller: Intel Corporation 82801DB/DBM (ICH4/ICH4-M) USB2 EHCI Controller (rev 01)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 81)
00:1f.0 ISA bridge: Intel Corporation 82801DB/DBL (ICH4/ICH4-L) LPC Interface Bridge (rev 01)
00:1f.1 IDE interface: Intel Corporation 82801DB (ICH4) IDE Controller (rev 01)
00:1f.3 SMBus: Intel Corporation 82801DB/DBL/DBM (ICH4/ICH4-L/ICH4-M) SMBus Controller (rev 01)
00:1f.5 Multimedia audio controller: Intel Corporation 82801DB/DBL/DBM (ICH4/ICH4-L/ICH4-M) AC'97 Audio Controller (rev 01)
01:04.0 Ethernet controller: Linksys NC100 Network Everywhere Fast Ethernet 10/100 (rev 11)
01:06.0 FireWire (IEEE 1394): Texas Instruments TSB43AB23 IEEE-1394a-2000 Controller (PHY/Link)
01:09.0 Ethernet controller: Broadcom Corporation BCM4401 100Base-T (rev 01)

Second, let's check to see that the modules are loaded:
# lsmod |egrep "ohci1394|sbp2|ieee1394|sd_mod|scsi_mod"
sd_mod 17217 0
sbp2 23948 0
scsi_mod 121293 2 sd_mod,sbp2
ohci1394 35784 0
ieee1394 298228 2 sbp2,ohci1394

Third, let's make sure the disk was detected and an entry was made by the kernel:
# cat /proc/scsi/scsi
Attached devices:
Host: scsi0 Channel: 00 Id: 01 Lun: 00
Vendor: Maxtor Model: OneTouch II Rev: 023g
Type: Direct-Access ANSI SCSI revision: 06

Now let's verify that the FireWire drive is accessible for multiple logins and shows a valid login:
# dmesg | grep sbp2
sbp2: $Rev: 1265 $ Ben Collins <bcollins@debian.org>
ieee1394: sbp2: Maximum concurrent logins supported: 2
ieee1394: sbp2: Number of active logins: 0
ieee1394: sbp2: Logged into SBP-2 device

From the above output, you can see that the FireWire drive I have can support concurrent logins by up to 2 servers. It is vital that you have a drive where the chipset supports concurrent access for all nodes within the RAC cluster.
One other test I like to perform is to run a quick fdisk -l from each node in the cluster to verify that it is really being picked up by the OS. Your drive may show that the device does not contain a valid partition table, but this is OK at this point of the RAC configuration.

# fdisk -l
Disk /dev/hda: 40.0 GB, 40000000000 bytes
255 heads, 63 sectors/track, 4863 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/hda1 * 1 13 104391 83 Linux
/dev/hda2 14 4863 38957625 8e Linux LVM

Disk /dev/sda: 300.0 GB, 300090728448 bytes
255 heads, 63 sectors/track, 36483 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 1 36483 293049666 c W95 FAT32 (LBA)

Rescan SCSI bus no longer required:

In older versions of the kernel, I would need to run the rescan-scsi-bus.sh script in order to detect the FireWire drive. The purpose of this script was to create the SCSI entry for the node by using the following command:

echo "scsi add-single-device 0 0 0 0" > /proc/scsi/scsi

With RHEL3 and RHEL4, this step is no longer required and the disk should be detected automatically.
Troubleshooting SCSI Device Detection:

If you are having troubles with any of the procedures (above) in detecting the SCSI device, you can try the following:

# modprobe -r sbp2
# modprobe -r sd_mod
# modprobe -r ohci1394
# modprobe ohci1394
# modprobe sd_mod
# modprobe sbp2

You may also want to unplug any USB devices connected to the server. The system may not be able to recognize your FireWire drive if you have a USB device attached!