A note of my Oracle 10gR2 RAC installation on RHEL based on this guide .

• RHEL 4.8 x86-64
• Oracle 10gR2
• 2 VMWare VM, 4g RAM each
• NFS as shared storage

Very simple home-brew implemention and for evaluation only.

Suppose we have Two wanna-be nodes in lan, both with 2 NICs:

ifconfig | grep -A 1 eth

Server A:

eth0      Link encap:Ethernet  HWaddr 00:0C:29:20:55:96
          inet addr:10.10.10.11  Bcast:10.10.10.255  Mask:255.255.255.0
--
eth1      Link encap:Ethernet  HWaddr 00:0C:29:20:55:A0
          inet addr:16.173.245.242  Bcast:16.173.247.255  Mask:255.255.248.0

Server B:

eth0      Link encap:Ethernet  HWaddr 00:0C:29:20:55:96
          inet addr:10.10.10.12  Bcast:10.10.10.255  Mask:255.255.255.0
--
eth1      Link encap:Ethernet  HWaddr 00:0C:29:20:55:A0
          inet addr:16.173.245.247  Bcast:16.173.247.255  Mask:255.255.248.0

eth0 will be used as inner communicator between nodes, eth1 will be interface for connections, and will later be attached a pair of extra IPs as VIP exposed by Oracle RAC.

Other than packages included in default RHEL 4.8 x86-64 installation these packages is need:

rpm -Uvh setarch-1*
rpm -Uvh compat-libstdc++-33-3*
rpm -Uvh make-3*
rpm -Uvh glibc-2*
rpm -Uvh glibc-headers-2.3*
rpm -Uvh glibc-devel-2.3*
rpm -Uvh kernel-devel-2.6*
rpm -Uvh glibc-kernheaders-2.4*
rpm -Uvh compat-glibc-headers-2.3*
rpm -Uvh openmotif-2*
rpm -Uvh compat-db-4*
rpm -Uvh gcc-3*
rpm -Uvh libaio-0*
rpm -Uvh rsh-*
rpm -Uvh compat-gcc-32-3*
rpm -Uvh compat-gcc-32-c++-3*
rpm -Uvh openmotif21*
rpm -Uvh sysstat*
rpm -Uvh libaio-0.3.*
rpm -Uvh libaio-devel*

All I used are those shipped with DVD. RPM is just stupid…

/etc/hosts will contain host⇒ip maping to meet our plan:

# cat /etc/hosts 
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1               testrac1.localdomain localhost
 
10.10.10.11             testrac1-priv.localdomain
16.173.245.242          testrac1.localdomain
16.173.245.243          testrac1-vip.localdomain
 
10.10.10.12             testrac2-priv.localdomain
16.173.245.247          testrac2.localdomain
16.173.245.249          testrac2-vip.localdomain

VIP IP will be created by Oracle Clusterware Installer.

Set hostname as our plan: testrac1:

sed 's/\(^HOSTNAME=\).*/\1testrac1/g' /etc/sysconfig/network
/bin/hostname testrac1
/sbin/service syslog restart

testrac2:

sed 's/\(^HOSTNAME=\).*/\1testrac2/g' /etc/sysconfig/network
/bin/hostname testrac2
/sbin/service syslog restart

Note that rebooting is NOT REQUIRED to change system hostname. At this time we should make sure system time of all the nodes are synchronized. A ntp server is STRONGLY recommended, at least set the clocks by hand to reduce difference.

Just like any Oracle Installation, Add the following suggested lines to the /etc/sysctl.conf:

kernel.shmall = 2097152
kernel.shmmax = 2147483648
kernel.shmmni = 4096
# semaphores: semmsl, semmns, semopm, semmni
kernel.sem = 250 32000 100 128
#fs.file-max = 65536
net.ipv4.ip_local_port_range = 1024 65000
#net.core.rmem_default=262144
#net.core.rmem_max=262144
#net.core.wmem_default=262144
#net.core.wmem_max=262144

# Additional and amended parameters suggested by Kevin Closson
net.core.rmem_default = 524288
net.core.wmem_default = 524288
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.ipfrag_high_thresh=524288
net.ipv4.ipfrag_low_thresh=393216
net.ipv4.tcp_rmem=4096 524288 16777216
net.ipv4.tcp_wmem=4096 524288 16777216
net.ipv4.tcp_timestamps=0
net.ipv4.tcp_sack=0
net.ipv4.tcp_window_scaling=1
net.core.optmem_max=524287
net.core.netdev_max_backlog=2500
sunrpc.tcp_slot_table_entries=128
sunrpc.udp_slot_table_entries=128
net.ipv4.tcp_mem=16384 16384 16384

And apply the change:

/sbin/sysctl -p

• Add the following lines to the /etc/security/limits.conf file:

*               soft    nproc   2047
*               hard    nproc   16384
*               soft    nofile  1024
*               hard    nofile  65536

• Add a line to the /etc/pam.d/login file, if it does not already exist:

cp /etc/pam.d/login /etc/pam.d/login_old
echo 'session    required     pam_limits.so' >> /etc/pam.d/login

• Disable SELinux as always:

cp /etc/selinux/config /etc/selinux/config_old
sed -i 's/^SELINUX=.*/SELINUX=disabled/g' /etc/selinux/config

• Set the hangcheck kernel module parameters and load it:

cp /etc/modprobe.conf /etc/modprobe.conf_old
echo 'options hangcheck-timer hangcheck_tick=30 hangcheck_margin=180' >> /etc/modprobe.conf
modprobe -v hangcheck-timer

• New groups and users for Oracle Instance:

groupadd oinstall
groupadd dba
groupadd oper
 
# Choose an unoccupied UID
useradd -u 504 –g oinstall -G dba -d /home/oracle -r oracle
passwd oracle

• Generate SSH Keys on each node:

su - oracle
mkdir ~/.ssh
chmod 700 ~/.ssh
/usr/bin/ssh-keygen -t rsa # Accept the default settings.
exit

• Exchange keys between nodes

# testrac1:
su - oracle
cd ~/.ssh
cat id_rsa.pub >> authorized_keys
scp authorized_keys testrac2:/home/oracle/.ssh/
exit

# testrac2:
su - oracle
cd ~/.ssh
cat id_rsa.pub >> authorized_keys
scp authorized_keys rac1:/home/oracle/.ssh/
exit

su - oracle
ssh testrac1 date
ssh testrac2 date
ssh testrac1.localdomain date
ssh testrac2.localdomain date
exec /usr/bin/ssh-agent $SHELL
/usr/bin/ssh-add

On both nodes, add this to ~/.bash_profile. Note that Oracle SID should differ from each nodes.

# Oracle Settings  catching up OMF defaults
TMP=/tmp; export TMP
TMPDIR=$TMP; export TMPDIR
 
ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE
ORACLE_HOME=$ORACLE_BASE/product/10.2.0/db_1; export ORACLE_HOME
ORACLE_SID=RAC1; export ORACLE_SID
ORACLE_TERM=xterm; export ORACLE_TERM
CRS_BASE=/u01/app/crs; export CRS_BASE
CRS_HOME=$CRS_BASE/product/10.2.0/db_1; export CRS_HOME
PATH=/usr/sbin:$PATH; export PATH
PATH=$ORACLE_HOME/bin:$PATH; export PATH
PATH=$CRS_HOME/bin:$PATH; export PATH
 
LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH
 
if [ $USER = "oracle" ]; then
  if [ $SHELL = "/bin/ksh" ]; then
    ulimit -p 16384
    ulimit -n 65536
  else
    ulimit -u 16384 -n 65536
  fi
fi

Node testrac2's ORACLE_SID is set to RAC2.

We are using NFS. For simplicity I use testrac1 as NFS Server, thus on testrac1:

• Create dirs

mkdir /shared_config
mkdir /shared_crs
mkdir /shared_home
mkdir /shared_data

• Add them to /etc/exports:

/shared_config    *(rw,sync,no_wdelay,insecure_locks,no_root_squash)
/shared_crs       *(rw,sync,no_wdelay,insecure_locks,no_root_squash)
/shared_home      *(rw,sync,no_wdelay,insecure_locks,no_root_squash)
/shared_data      *(rw,sync,no_wdelay,insecure_locks,no_root_squash)

• And export the shares:

chkconfig nfs on
service nfs restart

• On both node, create directories for Oracle Instance:

# Again, catching up OMF defaults
mkdir -p /u01/app/crs/product/10.2.0/crs
mkdir -p /u01/app/oracle/product/10.2.0/db_1
mkdir -p /u01/oradata
mkdir -p /u01/shared_config
chown -R oracle:oinstall /u01/app /u01/app/oracle /u01/oradata /u01/shared_config
chmod -R 775 /u01/app /u01/app/oracle /u01/oradata /u01/shared_config

• Make mount permanent in /etc/fstab on both nodes:

testrac1:/shared_config /u01/shared_config                   nfs  rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,noac,vers=3,timeo=600  0 0
testrac1:/shared_crs    /u01/app/crs/product/10.2.0/crs      nfs  rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,vers=3,timeo=600,actimeo=0  0 0
testrac1:/shared_home   /u01/app/oracle/product/10.2.0/db_1  nfs  rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,vers=3,timeo=600,actimeo=0  0 0
testrac1:/shared_data   /u01/oradata                         nfs  rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,vers=3,timeo=600  0 0

• Mount NFS Shares:

mount /u01/shared_config
mount /u01/app/crs/product/10.2.0/crs
mount /u01/app/oracle/product/10.2.0/db_1
mount /u01/oradata

• Shared CRS Configuration and Voting Disk files can be created now:

touch /u01/shared_config/ocr_configuration
touch /u01/shared_config/voting_disk

• Enforce permissions:

chown -R oracle:oinstall /u01/shared_config
chown -R oracle:oinstall /u01/app/crs/product/10.2.0/crs
chown -R oracle:oinstall /u01/app/oracle/product/10.2.0/db_1
chown -R oracle:oinstall /u01/oradata

Before installing the clusterware, run “runcluvfy.sh” utility in the clusterware media to perform a check and correct any issue.
CPIO package can be extracted like this:

cpio -idmv < MEDIA.cpio

/mnt/clusterware/runcluvfy/runcluvfy.sh stage -pre crsinst -n rac1,rac2 -verbose

• Login to testrac1 as oracle and run installer:

./runInstaller

• Accept the default inventory location.

/u01/app/oracle/oraInventory

• Name and path for the Oracle Clusterware Home and click the “Next” button.

/u01/app/crs/product/10.2.0/crs

• Run the prerequisite checks. Correct failure and retry.

• The “Specify Cluster Configuration” screen shows only the testrac1 node, Click the “Add” button to add testrac2
  • Public Node Name: testrac2.localdomain
  • Private Node Name: testrac2-priv.localdomain
  • Virtual Host Name: testrac2-VIP.localdomain

• The “Specific Network Interface Usage” screen defines how each network interface will be used

In our case, eth0 of each node will be used as private and eth1 as public

• In “Specify OCR Location” screen, click the “External Redundancy” option, enter”/u01/ocr_configuration” we touched earlier as the OCR Location.

• In “Specify Voting Disk Location” screen, click the “External Redundancy” option, enter ”/u01/voting_disk” we touched earlier as the Voting Disk Location

• On the “Summary” screen, click the “Install”

• After installation completes, run “orainstRoot.sh” and “root.sh” scripts as root on both nodes as installer requested.

Warnings like “WARNING: directory '/u01/app/crs/product/10.2.0' isnot owned by root” is Okay,
The scripts expect separate directory structure for the clusterware but ours is too simple

• Network interface changed by installer:

Checking the 2 NICs:

ifconfig | grep -A 1 eth

Server A:

eth0      Link encap:Ethernet  HWaddr 00:0C:29:20:55:96
          inet addr:10.10.10.11  Bcast:10.10.10.255  Mask:255.255.255.0
--
eth1      Link encap:Ethernet  HWaddr 00:0C:29:20:55:A0
          inet addr:16.173.245.242  Bcast:16.173.247.255  Mask:255.255.248.0
--
eth1:1    Link encap:Ethernet  HWaddr 00:0C:29:20:55:A0
          inet addr:16.173.245.243  Bcast:16.173.247.255  Mask:255.255.248.0

Server B:

eth0      Link encap:Ethernet  HWaddr 00:0C:29:20:55:96
          inet addr:10.10.10.12  Bcast:10.10.10.255  Mask:255.255.255.0
--
eth1      Link encap:Ethernet  HWaddr 00:0C:29:20:55:A0
          inet addr:16.173.245.247  Bcast:16.173.247.255  Mask:255.255.248.0
--
eth1:1    Link encap:Ethernet  HWaddr 00:0C:29:20:55:A0
          inet addr:16.173.245.249  Bcast:16.173.247.255  Mask:255.255.248.0

An additional virtual IP is attatched by Oracle Clusterware Installer.

DB Instance installation should be similar to single server install, some notable points, production enviroments might be differ:

• Login to testrac1 as oracle and run installer:

./runInstaller

• Installation Type: Enterprise

• Home Detail:
  • Name: db_1
  • Path: /u01/app/oracle/product/10.2.0/db_1

• Select the “Cluster Install” option and select both RAC

• Run prerequisite check

• Configuration option: Install database Software only

• Install

• run “root.sh” scripts as root on both nodes, as installer requested.

DB Creation should be similar to single server install, some notable points, production enviroments might be differ:

• Login to RAC1 as the oracle user and use the DBCA GUI.

dbca

• Create cluster database, highlight both RAC nodes.

• Enter the values “RAC.WORLD” and “RAC” for the Global Database Name and SID Prefix

• Select the “Cluster File System” option, since we are using NFS.

• Select the “Use Oracle-Managed Files” option, enter ”/u01/oradata/” as the database location

• Check the “Specify Flash Recovery Area” option and accept the default location

• Select the “Custom” memory management option and accept the default settings

Nothing else notable…

Once the DB installation is complete, TNS Listener shall be configured. Refer to the following files:

• $ORACLE_HOME/network/admin/listener.ora
• $ORACLE_HOME/network/admin/tnsnames.ora

The listener.ora shall indecate that their are two listeners listening on private host/ip we specified, 10.10.10.11 in my case:

grep 'HOST =' $ORACLE_HOME/network/admin/listener.ora

displays:

        (ADDRESS = (PROTOCOL = TCP)(HOST = testrac2-vip.localdomain)(PORT = 1521)(IP = FIRST))
        (ADDRESS = (PROTOCOL = TCP)(HOST = 10.10.10.12)(PORT = 1521)(IP = FIRST))
        (ADDRESS = (PROTOCOL = TCP)(HOST = testrac1-vip.localdomain)(PORT = 1521)(IP = FIRST))
        (ADDRESS = (PROTOCOL = TCP)(HOST = 10.10.10.11)(PORT = 1521)(IP = FIRST))

The tnsnames.ora shall indecate that we can connect through each listener or through a load balanced connection to the service:

grep -A 4 'LISTENERS' $ORACLE_HOME/network/admin/tnsnames.ora 

displays:

LISTENERS_RAC =
  (ADDRESS_LIST =
    (ADDRESS = (PROTOCOL = TCP)(HOST = rac1-vip.localdomain)(PORT = 1521))
    (ADDRESS = (PROTOCOL = TCP)(HOST = rac2-vip.localdomain)(PORT = 1521))
  )

$ sqlplus / AS sysdba
SQL> CONN sys/password@testrac1 AS SYSDBA
Connected.
SQL> SELECT instance_name, host_name FROM v$instance;
 
INSTANCE_NAME    HOST_NAME
----------------
----------------------------------------------------------------
RAC1             testrac1.localdomain
 
SQL> CONN sys/password@testrac2 AS SYSDBA
Connected.
SQL> SELECT instance_name, host_name FROM v$instance;
 
INSTANCE_NAME    HOST_NAME
----------------
----------------------------------------------------------------
RAC2             testrac2.localdomain
 
SQL> CONN sys/password@testrac AS SYSDBA
Connected.
SQL> SELECT instance_name, host_name FROM v$instance;
 
INSTANCE_NAME    HOST_NAME
----------------
----------------------------------------------------------------
RAC1             testrac1.localdomain
 
SQL>

• Use srvctl to check RAC status:

$ srvctl config database -d RAC
testrac1 RAC1 /u01/app/oracle/product/10.2.0/db_1
testrac2 RAC2 /u01/app/oracle/product/10.2.0/db_1
$
 
$ srvctl status database -d RAC
Instance RAC1 is running on node testrac1
Instance RAC2 is running on node testrac2
$

• The V$ACTIVE_INSTANCES view:

SQL> SELECT * FROM v$active_instances;
 
INST_NUMBER INST_NAME
-----------
------------------------------------------------------------
1 testrac1.localdomain:RAC1
2 testrac2.localdomain:RAC2
 
SQL>

One kick to Start/Stop All Instances and their services with SRVCTL:

$ srvctl start database -d RAC
$ srvctl stop database -d RAC

While sometimes TNS Listeners won't start… just start them manually

lsnrctl start

Sometimes ONS has to be started manually either:

onsctl start

Oracle RAC Installation itself is straight-forward

This can be complicated depends on:

• Oracle Version
• Network complicity and scale
• OS(X availablity and Kernel Version, commercial support, etc. )
• Storage Solutions

Given time I shall try Oracle 11g on more recent OSs, or even toying with my Gentoo…