Learn how to set up and configure an Oracle Real Applications Cluster for less than $1,500 (for development and testing only)

Overview

One of the most efficient ways to become familiar with Oracle Real Application Clusters (RAC) technology is to have access to an actual Oracle RAC cluster. In learning this new technology, you will soon start to realize the benefits Oracle RAC has to offer like fault tolerance, new levels of security, load balancing, and the ease of upgrading capacity. The challenge, however, is the price of the hardware required for a typical production RAC configuration. A small two-node cluster, for example, can run anywhere from $10,000 to well over $20,000. This cost would not even include shared storage, the heart of a production RAC environment.

For those who simply want to become familiar with Oracle RAC, this article provides a low-cost alternative for configuring an Oracle9i RAC system using commercial off-the-shelf components and downloadable software. The estimated cost for this configuration could be anywhere from $1,000 to $1,500. The system will comprise a dual-node cluster, both running Linux (Red Hat Linux Fedora Core 1 in this example) with a shared disk array based on IEEE1394 (FireWire) drive technology.

Please note that this is not the only way to build a low-cost Oracle9i RAC system. I have seen other solutions that utilize an implementation based on SCSI rather than FireWire for shared storage. In most cases, SCSI will cost more than our FireWire solution where a typical SCSI card is priced around $70 and an 80GB external SCSI drive will cost $700-$1,000. Keep in mind that some motherboards may already include built-in SCSI controllers.

It is important to note that this configuration should never be run in a production environment. In a production environment, fiber channel is the technology of choice, since it is the high-speed serial-transfer interface that can connect systems and storage devices in either point-to-point or switched topologies. FireWire is able to offer a low-cost alternative to fiber channel for testing and development, but it is not ready for production.

NOTE: At the time of this writing, I had not verified that these instructions will work with Oracle Database 10g. I will be providing a separate article in the next several months on how to perform a similar install using 10g.

Oracle9i Real Application Clusters (RAC) Introduction

Oracle Real Application Clusters (RAC) is the successor to Oracle Parallel Server (OPS). RAC allows multiple instances to access the same database (storage) simultaneously. RAC provides fault tolerance, load balancing, and performance benefits by allowing the system to scale out, and at the same time since all nodes access the same database, the failure of one instance will not cause the loss of access to the database.

At the heart of Oracle RAC is a shared disk subsystem. All nodes in the cluster must be able to access all of the data, redo log files, control files and parameter files for all nodes in the cluster. The data disks must be globally available in order to allow all nodes to access the database. Each node has its own redo log and control files, but the other nodes must be able to access them in order to recover that node in the event of a system failure.

Not all clustering solutions use shared storage. Some vendors use an approach known as a federated cluster, in which data is spread across several machines rather than shared by all. With Oracle RAC, however, multiple nodes use the same set of disks for storing data. With Oracle RAC, the data, redo log, control, and archived log files reside on shared storage on raw-disk devices or on a clustered file system. Oracle's approach to clustering leverages the collective processing power of all the nodes in the cluster and at the same time provides failover security.

Although it is not absolutely necessary, Oracle recommendeds that you install the Oracle Cluster File System (OCFS). OCFS makes disk management much easier for you by creating the same file system on all the nodes. This isn't necessary, but without OCFS, you will have to make all partitions manually. (NOTE: This article does not go into the details of installing or utilizing OCFS, but rather uses all manual methods for creating partitions and binding raw devices to those partitions.)

One of the main reasons why I do not use the Oracle Cluster File System for Red Hat Linux is that OCFS comes in the form of RPMs. All the RPM modules and the precompiled modules are tied to the Red Hat Enterprise Linux AS ($1,200) kernel-naming standard and will not load in the supplied 2.4.20 linked kernel.

The biggest difference between Oracle RAC and OPS is the addition of Cache Fusion. With OPS a request for data from one node to another required the data to be written to disk first, then the requesting node can read that data. With cache fusion, data is passed along with locks.

Pre-configured Oracle9i RAC solutions are available from vendors such as Dell, IBM and HP for production environments. This article, however, focuses on putting together your own Oracle9i RAC environment for development and testing by using Linux servers and a low cost shared disk solution; FireWire.

What software is necessary for RAC? Does it have a separate installation CD to order?

RAC is contained within the Oracle9i Database Enterprise Edition. (Oracle recently announced that RAC is now available in Oracle Database 10g Standard Edition as well.) If you install Oracle9i Enterprise Edition onto a cluster, and the Oracle Universal Installer (OUI) recognizes the cluster, you will be provided the option of installing RAC. Most UNIX platforms require an OSD installation for the necessary clusterware. For Intel platforms (Linux and Windows), Oracle provides the OSD software within the Oracle9i Enterprise Edition release.

Shared Storage Overview

Today, fiber-channel is one of the most popular solutions for shared storage. As mentioned earlier, fiber-channel is a high-speed serial-transfer interface that is used to connect systems and storage devices in either point-to-point or switched topologies. Protocols supported by fiber channel include SCSI and IP. Fiber channel configurations can support as many as 127 nodes and have a throughput of up to 2.12 gigabits per second. Fiber-channel, although, is very expensive. Just the fiber-channel switch alone can run as much as $1,000. This does not even include the fiber-channel storage array and high-end drives, which can reach prices of about $300 for a 36GB drive. A typical fiber-channel setup which includes fiber-channel cards for the servers, a basic setup is roughly $5,000, which does not include the cost of the servers that make up the cluster.

A less expensive alternative to fiber-channel is SCSI. SCSI technology provides acceptable performance for shared storage, but for administrators and developers who are accustomed to GPL-based Linux prices, even SCSI can come in over budget, at around $1,000 to $2,000 for a two-node cluster.

Another popular solution is the Sun NFS (Network File System). It can be used for shared storage but only if you are using a network appliance or something similar. Specifically, you need servers that guarantee direct I/O over NFS.

FireWire Technology

Developed by Apple Computer and Texas Instruments, FireWire is a cross-platform implementation of a high-speed serial data bus. With its high bandwidth, long distances (up to 100 meters in length) and high-powered bus, FireWire is being used in applications such as digital video (DV), professional audio, hard drives, high-end digital still cameras and home entertainment devices. Today, FireWire operates at transfer rates of up to 800 megabits per second while next generation FireWire calls for speeds to a theoretical bit rate to 1,600 Mbps and then up to a staggering 3,200 Mbps. That's 3.2 gigabits per second. This speed will make FireWire indispensable for transferring massive data files and for even the most demanding video applications, such as working with uncompressed high-definition (HD) video or multiple standard-definition (SD) video streams.

The following chart shows speed comparisons of the various types of disk interface. For each interface, I provide the maximum transfer rates in kilobits (kb), kilobytes (KB), megabits (Mb), and megabytes (MB) per second. As you can see, the capabilities of IEEE1394 compare very favorably with other available disk interface technologies.

Hardware & Costs

The hardware used to build our example Oracle9i RAC environment consists of two Linux servers and components that can be purchased at any local computer store or over the Internet.

A Brief Walk Through the Process

Before presenting the details of building our Oracle9i RAC system, I thought it would be beneficial to take a brief walk through the steps involved in building the environment. (See Figure 1.)

Our implementation describes a dual node cluster (each with a single processor), each server running Red Hat Linux Fedora Core 1. Note that most of the tasks within this document will need to be performed on both servers. I will indicate at the beginning of each section whether or not the task(s) should be performed on both nodes.

Install Red Hat Linux (Fedora Core 1)

After procuring the required hardware, it is time to start the configuration process. The first step in the process is to install the Red Hat Linux Fedora Core 1 software on both servers.

NOTE: This article does not provide detailed instructions for installing Red Hat Linux Fedora Core 1. For the purpose of this article, I choose to perform a Custom installation and then "Install Everything" when prompted for which products to install. Documentation for installing Red Hat Linux can be found at http://www.redhat.com/docs/manuals/.

Configure Network Settings

Configuring Public and Private Network

Let's start our Oracle RAC Linux configuration by ensuring the correct network configuration. In our two-node example, we will need to configure the network on both nodes.

The easiest way to configure network settings in RedHat Linux is via the program Network Configuration. This application can be started from the command-line as the "root" user id as follows:

# su -
# /usr/bin/redhat-config-network &

NOTE: Do not use DHCP naming as the interconnects need hard IP addresses!

Using the Network Configuration application, you will need to configure both NIC devices as well as the /etc/hosts file. Both of these tasks can be completed using the Network Configuration GUI. Notice that the /etc/hosts settings are the same for both nodes.

Our example configuration will use the following settings:

127.0.0.1        localhost      loopback
192.168.1.100    linux1
192.168.2.100    int-linux1
192.168.1.101    linux2
192.168.2.101    int-linux2

127.0.0.1        localhost      loopback
192.168.1.100    linux1
192.168.2.100    int-linux1
192.168.1.101    linux2
192.168.2.101    int-linux2

In the screenshots below, only node 1 (linux1) is shown. Ensure to make all the proper network settings to both nodes.

Figure 1: Network Configuration Screen, Node 1 (linux1)

Figure 2: Ethernet Device Screen, eth0 (linux1)

Figure 3: Ethernet Device Screen, eth1 (linux1)

Figure 4: Network Configuration Screen, /etc/hosts (linux1)
 

Adjusting Network Settings

With Oracle 9.2.0.1 and above, Oracle uses UDP as the default protocol on Linux for interprocess communication (IPC), such as cache fusion buffer transfers between instances within the RAC cluster.

Oracle strongly suggests to adjust the default and maximum send buffer size (SO_SNDBUF socket option) to 256KB, and the default and maximum receive buffer size (SO_RCVBUF socket option) to 256KB.

The receive buffers are used by TCP and UDP to hold received data until is is read by the application. The receive buffer cannot overflow because the peer is not allowed to send data beyond the buffer size window. This means that datagrams will be discarded if they don't fit in the socket receive buffer. This could cause the sender to overwhelm the receiver.

NOTE: The default and maximum window size can be changed in the /proc file system without reboot:

su - root

# Default setting in bytes of the socket receive buffer
sysctl -w net.core.rmem_default=262144

# Default setting in bytes of the socket send buffer
sysctl -w net.core.wmem_default=262144

# Maximum socket receive buffer size which may be set by using
# the SO_RCVBUF socket option
sysctl -w net.core.rmem_max=262144

# Maximum socket send buffer size which may be set by using 
# the SO_SNDBUF socket option
sysctl -w net.core.wmem_max=262144

You should make the above changes permanent by adding the following lines to the /etc/sysctl.conf file for each node in your RAC cluster:

net.core.rmem_default=262144
net.core.wmem_default=262144
net.core.rmem_max=262144
net.core.wmem_max=262144

Obtain and Install a Proper Linux Kernel

Overview

The next step is to obtain and install a new Linux kernel that supports the use of IEEE1394 devices with multiple logins. In previous releases of this article, I included the steps to download a patched version of the Linux kernel and then compile it. Thanks to Oracle's Linux Projects development group, this is no longer a requirement. They provide a pre-compiled kernel for Red Hat Enterprise Linux 3.0 (which also works with Fedora) that can simply be downloaded and installed. The instructions for downloading and installing the kernel are included in this section. Before going into the details of how to perform these actions, however, let's take a moment to discuss the changes that are required in the new kernel.

While FireWire drivers already exist for Linux, they often do not support shared storage. Normally, when you logon to an OS, the OS associates the driver to a specific drive for that machine alone. This implementation simply will not work for our RAC configuration. The shared storage (our FireWire hard drive) needs to be accessed by more than one node. We need to enable the FireWire driver to provide nonexclusive access to the drive so that multiple servers—the nodes that comprise the cluster— will be able to access the same storage. This task is accomplished by removing the bit mask that identifies the machine during login in the source code. This results in allowing nonexclusive access to the FireWire hard drive. All other nodes in the cluster login to the same drive during their logon session, using the same modified driver, so they too also have nonexclusive access to the drive.

I'm probably getting ahead of myself, but I want to cover several topics before diving into the details of installing our new Linux kernel. When we install our new Linux kernel (one that supports multiple logons to the FireWire drive) the system will detect and recognize the FireWire attached drive as a SCSI device. You will be able to use standard OS tools to partition the disk, create a file system, and so on. For Oracle9i RAC, you must make partitions for all the files and bind raw devices to those partitions. This article will make use of Logical Volume Manager (LVM) to make all needed paritions (actually to be known as logical partitions) on the FireWire shared drive.

Our implementation describes a dual node cluster (each with a single processor), each server running Red Hat Linux Fedora Core 1. Keep in mind that the process of installing the patched Linux kernel will need to be performed on both Linux nodes. Red Hat Linux Fedora Core 1 includes kernel linux-2.4.22-1.2115.nptl; we will need to download the Oracle-supplied 2.4.21-9.0.1 Linux kernel from the following URL: http://oss.oracle.com/projects/firewire/files.

Perform the following procedures on both nodes in the cluster:

1. Download one of the following files:

   kernel-2.4.21-9.0.1.ELorafw1.i686.rpm - for single processor
   
   - OR -

   kernel-smp-2.4.21-9.0.1.ELorafw1.i686.rpm - for multiple processors

2. Make a backup of your GRUB configuration file:

   In most cases you will be using GRUB for your boot loader. Before actually installing the new kernel ensure to backup a copy of your /etc/grub.conf file:

   # cp /etc/grub.conf /etc/grub.conf.original

3. Install the new kernel, as user root:

   # rpm -ivh --force kernel-2.4.21-9.0.1.ELorafw1.i686.rpm - for single processor

   - OR -

   # rpm -ivh --force kernel-smp-2.4.21-9.0.1.ELorafw1.i686.rpm  - for multiple processors

   NOTE: Installing the new kernel using RPM will also undate your grub or lilo configuration with the appropiate stanza. There is no need to add any new stanza to your boot loader configuration unless you want to have your old kernel image available.

   The following is a listing of my /etc/grub.conf file before and then after the kernel install. As you can see, the install that I did put in another stanza for the 2.4.21-9.0.1.ELorafw1 kernel. If you want, you can change the entry (default) in the new file so that the new kernel will be the default one booted. By default, the installer keeps your old kernel the default one by setting it to default=1.

   Original /etc/grub.conf File for Fedora Core 1

   # grub.conf generated by anaconda
   #
   # Note that you do not have to rerun grub after making changes to this file
   # NOTICE:  You have a /boot partition.  This means that
   #          all kernel and initrd paths are relative to /boot/, eg.
   #          root (hd0,0)
   #          kernel /vmlinuz-version ro root=/dev/hda3
   #          initrd /initrd-version.img
   #boot=/dev/hda
   default=0
   timeout=10
   splashimage=(hd0,0)/grub/splash.xpm.gz
   title Fedora Core (2.4.22-1.2115.nptl)
         root (hd0,0)
         kernel /vmlinuz-2.4.22-1.2115.nptl ro root=LABEL=/ rhgb
         initrd /initrd-2.4.22-1.2115.nptl.img

   Newly Configured /etc/grub.conf File for Fedora Core 1 After Kernel Install

   # grub.conf generated by anaconda
   #
   # Note that you do not have to rerun grub after making changes to this file
   # NOTICE:  You have a /boot partition.  This means that
   #          all kernel and initrd paths are relative to /boot/, eg.
   #          root (hd0,0)
   #          kernel /vmlinuz-version ro root=/dev/hda3
   #          initrd /initrd-version.img
   #boot=/dev/hda
   default=0
   timeout=10
   splashimage=(hd0,0)/grub/splash.xpm.gz
   title Fedora Core (2.4.21-9.0.1.ELorafw1)
           root (hd0,0)
           kernel /vmlinuz-2.4.21-9.0.1.ELorafw1 ro root=LABEL=/ rhgb
           initrd /initrd-2.4.21-9.0.1.ELorafw1.img
   title Fedora Core (2.4.22-1.2115.nptl)
           root (hd0,0)
           kernel /vmlinuz-2.4.22-1.2115.nptl ro root=LABEL=/ rhgb
           initrd /initrd-2.4.22-1.2115.nptl.img

4. Add module options:

   Add the following lines to /etc/modules.conf:

   options sbp2 sbp2_exclusive_login=0
   post-install sbp2 insmod sd_mod
   post-remove sbp2 rmmod sd_mod

   It is vital that the parameter sbp2_exclusive_login of the Serial Bus Protocol module (sbp2) be set to zero to allow multiple hosts to login to and access the FireWire disk concurrently. The second line ensures the SCSI disk driver module (sd_mod) is loaded as well since (sbp2) requires the SCSI layer. The core SCSI support module (scsi_mod) will be loaded automatically if (sd_mod) is loaded—there is no need to make a separate entry for it.

5. Reboot machine

   Reboot your machine into the new kernel. Ensure the firewire (ieee1394) pci cards are plugged into the machine!

6. Load the firewire stack

   In most cases, the loading of the FireWire stack will already be configured in the /etc/rc.sysinit file. The commands that are contained within this file that are responsible for loading the FireWire stack are:

   # modprobe ohci1394
   # modprobe sbp2

   In older versions of Red Hat, this was not the case and these commands would have to be manually run or put within a startup file. With Fedora Core 1 and higher, these commands are already put within the /etc/rc.sysinit file and run on each boot.
    
7. Rescan SCSI bus

   In older versions of the kernel, I would need to run the rescan-scsi-bus.sh script in order to detect the FireWire drive. The purpose of this script was to create the SCSI entry for the node by using the following command:

   echo "scsi add-single-device 0 0 0 0" > /proc/scsi/scsi

   With Fedora Core 1, the disk should be detected automatically.

8. Check for SCSI Device

   After you have rebooted the machine, the kernel should automatically detect the disk as a SCSI device (/dev/sdXX). This section will provide several commands that should be run on both nodes in the cluster to ensure the FireWire drive was successfully detected.

   For this configuration, I was performing the above procedures on both nodes at the same time. When complete, I shutdown both machines, started linux1 first, and then linux2. The following commands and results are from my linux2 machine. Again, make sure that you run the following commands on both nodes to ensure both machine can login to the shared drive.

   Let's first check to see that the FireWire adapter was successfully detected:

   # lspci
   00:00.0 Host bridge: Intel Corp. 82845 845 (Brookdale) Chipset Host Bridge (rev 11)
   00:01.0 PCI bridge: Intel Corp. 82845 845 (Brookdale) Chipset AGP Bridge (rev 11)
   00:1d.0 USB Controller: Intel Corp. 82801DB USB (Hub #1) (rev 01)
   00:1d.1 USB Controller: Intel Corp. 82801DB USB (Hub #2) (rev 01)
   00:1d.2 USB Controller: Intel Corp. 82801DB USB (Hub #3) (rev 01)
   00:1d.7 USB Controller: Intel Corp. 82801DB USB2 (rev 01)
   00:1e.0 PCI bridge: Intel Corp. 82801BA/CA/DB/EB PCI Bridge (rev 81)
   00:1f.0 ISA bridge: Intel Corp. 82801DB LPC Interface Controller (rev 01)
   00:1f.1 IDE interface: Intel Corp. 82801DB Ultra ATA Storage Controller (rev 01)
   00:1f.3 SMBus: Intel Corp. 82801DB/DBM SMBus Controller (rev 01)
   01:00.0 VGA compatible controller: nVidia Corporation NV34 [GeForce FX 5200] (rev a1)
   02:00.0 Ethernet controller: Linksys Network Everywhere Fast Ethernet 10/100 model NC100 (rev 11)
   02:01.0 FireWire (IEEE 1394): Texas Instruments TSB12LV26 IEEE-1394 Controller (Link)
   02:05.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+ (rev 10)
   02:07.0 Multimedia audio controller: C-Media Electronics Inc CM8738 (rev 10)

   Second, let's check to see that the modules are loaded:

   # lsmod |egrep "ohci1394|sbp2|ieee1394|sd_mod|scsi_mod"
   sd_mod                 13808   0
   sbp2                   20556   0
   scsi_mod              109864   3  [sg sd_mod sbp2]
   ohci1394               28904   0  (unused)
   ieee1394               63652   0  [sbp2 ohci1394]

   Third, let's make sure the disk was detected and an entry was made by the kernel:

   # cat /proc/scsi/scsi
   Attached devices:
   Host: scsi0 Channel: 00 Id: 00 Lun: 00
     Vendor: Maxtor   Model: OneTouch         Rev: 0200
     Type:   Direct-Access

   Now let's ensure the FireWire drive is accessible for multiple logins and shows a valid login:

   # dmesg | grep sbp2
   ieee1394: sbp2: Query logins to SBP-2 device successful
   ieee1394: sbp2: Maximum concurrent logins supported: 3
   ieee1394: sbp2: Number of active logins: 2
   ieee1394: sbp2: Logged into SBP-2 device
   ieee1394: sbp2: Node[01:1023]: Max speed [S400] - Max payload [2048]
   ieee1394: sbp2: Reconnected to SBP-2 device
   ieee1394: sbp2: Node[01:1023]: Max speed [S400] - Max payload [2048]

   From the above output, you can see that the FireWire drive we have can support concurrent logins by up to 3 servers. It is vital that you have a drive where the chipset supports concurrent access for all nodes within the RAC cluster.

9. Troubleshoot SCSI Device Detection

   If you are having troubles with any of the procedures (above) in detecting the SCSI device, you can try the following:

   # modprobe -r sbp2
   # modprobe -r sd_mod
   # modprobe -r ohci1394
   # modprobe ohci1394
   # modprobe sd_mod
   # modprobe sbp2

Create "oracle" User and Directories (both nodes)

Let's continue our example by creating the UNIX dba group and oracle userid along with all appropriate directories.

# mkdir /u01
# mkdir /u01/app

# groupadd -g 115 dba

# useradd -u 175 -g 115 -d /u01/app/oracle -s /bin/bash -c "Oracle Software Owner" -p oracle oracle

NOTE: When you are setting the Oracle environment variables for each RAC node, ensure to assign each RAC node a unique Oracle SID!

For this example, I used:

• linux1 : ORACLE_SID=orcl1
• linux2 : ORACLE_SID=orcl2

You can check the available space in /tmp by running the following command:

# df -k /tmp
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/hda3             36384656   6224240  28312140  19% /

# su -
# mkdir /<AnotherFilesystem>/tmp
# chown root.root /<AnotherFilesystem>/tmp
# chmod 1777 /<AnotherFilesystem>/tmp
# export TEMP=/<AnotherFilesystem>/tmp     # used by Oracle
# export TMPDIR=/<AnotherFilesystem>/tmp   # used by Linux programs
                                           #   like the linker "ld"

# su -
# rmdir /<AnotherFilesystem>/tmp
# unset TEMP
# unset TMPDIR

After creating the "oracle" UNIX userid on both nodes, ensure that the environment is setup correctly by using the following .bash_profile:

# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
      . ~/.bashrc
fi

alias ls="ls -FA"

# User specific environment and startup programs
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/9.2.0

# Each RAC node must have a unique ORACLE_SID. (i.e. orcl1, orcl2,...)
export ORACLE_SID=orcl1

export PATH=.:${PATH}:$HOME/bin:$ORACLE_HOME/bin
export PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin
export ORACLE_TERM=xterm
export TNS_ADMIN=$ORACLE_HOME/network/admin
export ORA_NLS33=$ORACLE_HOME/ocommon/nls/admin/data
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib
export CLASSPATH=$ORACLE_HOME/JRE
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib
export THREADS_FLAG=native
export TEMP=/tmp
export TMPDIR=/tmp
export LD_ASSUME_KERNEL=2.4.1

Creating Partitions on the Shared FireWire Storage Device (one node)

Overview

It is time to create the physical and logical volumes to be used by the Logical Volume Manager (LVM). (For a more detailed view of managing the LVM, see my article Managing Physical & Logical Volumes.) The following table lists the mappings of logical partition to tablespace that we will be accomplishing in this section of the document: