Tuesday, 18 October 2011

Easy Step By Step installation Of RAC On Linux

Oracle Linux 6 installation as you would for a basic server. A general pictorial guide to the installation can be found here. More specifically, it should be a server installation with a minimum of 4G+ swap, firewall and SELinux disabled and the following package groups installed:
  • Base System > Base
  • Base System > Client management tools
  • Base System > Compatibility libraries
  • Base System > Hardware monitoring utilities
  • Base System > Large Systems Performance
  • Base System > Network file system client
  • Base System > Performance Tools


  • Base System > Perl Support
  • Servers > Server Platform
  • Servers > System administration tools
  • Desktops > Desktop
  • Desktops > Desktop Platform
  • Desktops > Fonts
  • Desktops > General Purpose Desktop
  • Desktops > Graphical Administration Tools
  • Desktops > Input Methods
  • Desktops > X Window System
  • Applications > Internet Browser
  • Development > Additional Development
  • Development > Development Tools

    To be consistent with the rest of the article, the following information should be set during the installation:
    • hostname: ol6-112-rac1.localdomain
    • IP Address eth0: 192.168.0.111 (public address)
    • Default Gateway eth0: 192.168.0.1 (public address)
    • IP Address eth1: 192.168.1.111 (private address)
    • Default Gateway eth1: none
    You are free to change the IP addresses to suit your network, but remember to stay consistent with those adjustments throughout the rest of the article.
    Once the basic installation is complete, install the following packages whilst logged in as the root user. This includes the 64-bit and 32-bit versions of some packages. The commented out packages are those already installed if you have followed the suggested package selection. There is no suitable "oracle-validated" package available at this time, so the packages need to be installed manually.
    # From Oracle Linux 6 DVD
    cd /media/cdrom/Server/Packages
    #rpm -Uvh binutils-2.*
    #rpm -Uvh compat-libstdc++-33*
    #rpm -Uvh elfutils-libelf-0.*
    #rpm -Uvh libaio-0.*
    rpm -Uvh libaio-devel-0.*
    #rpm -Uvh sysstat-9.*
    #rpm -Uvh glibc-2.*
    #rpm -Uvh glibc-common-2.*
    #rpm -Uvh glibc-devel-2.* glibc-headers-2.*
    rpm -Uvh ksh-2*
    #rpm -Uvh make-3.*
    #rpm -Uvh libgcc-4.*
    #rpm -Uvh libstdc++-4.*
    rpm -Uvh libstdc++-4.*.i686*
    #rpm -Uvh libstdc++-devel-4.*
    #rpm -Uvh gcc-4.*x86_64*
    #rpm -Uvh gcc-c++-4.*x86_64*
    #rpm -Uvh --allfiles elfutils-libelf-0*x86_64* elfutils-libelf-devel-0*x86_64*
    rpm -Uvh elfutils-libelf-0*i686* elfutils-libelf-devel-0*i686*
    rpm -Uvh libtool-ltdl*i686*
    rpm -Uvh ncurses*i686*
    rpm -Uvh readline*i686*
    rpm -Uvh unixODBC*
    cd /
    eject
    
    
    # Install the following package from the Oracle grid media after you've defined groups.
    cd /your/path/to/grid/rpm
    rpm -Uvh cvuqdisk*

    Oracle Installation Prerequisites

    Perform the following steps whilst logged into the "rac1" virtual machine as the root user.
    If you are not using DNS, the "/etc/hosts" file must contain the following information.
    127.0.0.1       localhost.localdomain   localhost
    # Public
    192.168.0.111   ol6-112-rac1.localdomain        ol6-112-rac1
    192.168.0.112   ol6-112-rac2.localdomain        ol6-112-rac2
    # Private
    192.168.1.111   ol6-112-rac1-priv.localdomain   ol6-112-rac1-priv
    192.168.1.112   ol6-112-rac2-priv.localdomain   ol6-112-rac2-priv
    # Virtual
    192.168.0.113   ol6-112-rac1-vip.localdomain    ol6-112-rac1-vip
    192.168.0.114   ol6-112-rac2-vip.localdomain    ol6-112-rac2-vip
    # SCAN
    192.168.0.115   ol6-112-scan.localdomain ol6-112-scan
    192.168.0.116   ol6-112-scan.localdomain ol6-112-scan
    192.168.0.117   ol6-112-scan.localdomain ol6-112-scan
    Note. The SCAN address should not really be defined in the hosts file. Instead is should be defined on the DNS to round-robin between 3 addresses on the same subnet as the public IPs. For this installation, we will compromise and use the hosts file.
    If you are using DNS, then only the first line must be present in the "/etc/hosts" file. The other entries are defined in the DNS, as described here. Having said that, I normally include everything except the SCAN entries when using DNS.
    Add or amend the following lines to the "/etc/sysctl.conf" file.
    fs.aio-max-nr = 1048576
    fs.file-max = 6815744
    #kernel.shmall = 2097152 
    #kernel.shmmax = 1054504960
    kernel.shmmni = 4096
    # semaphores: semmsl, semmns, semopm, semmni
    kernel.sem = 250 32000 100 128
    net.ipv4.ip_local_port_range = 9000 65500
    net.core.rmem_default=262144
    net.core.rmem_max=4194304
    net.core.wmem_default=262144
    net.core.wmem_max=1048586
    Run the following command to change the current kernel parameters.
    /sbin/sysctl -p
    Add the following lines to the "/etc/security/limits.conf" file.
    oracle              soft    nproc   2047
    oracle              hard    nproc   16384
    oracle              soft    nofile  4096
    oracle              hard    nofile  65536
    oracle              soft    stack   10240
    Add the following lines to the "/etc/pam.d/login" file, if it does not already exist.
    session    required     pam_limits.so
    Disable secure linux by editing the "/etc/selinux/config" file, making sure the SELINUX flag is set as follows.
    SELINUX=disabled
    Either configure NTP, or make sure it is not configured so the Oracle Cluster Time Synchronization Service (ctssd) can synchronize the times of the RAC nodes. In this case we will deconfigure NTP.
    # service ntpd stop
    Shutting down ntpd:                                        [  OK  ]
    # chkconfig ntpd off
    # mv /etc/ntp.conf /etc/ntp.conf.orig
    # rm /var/run/ntpd.pid
    If you are using NTP, you must add the "-x" option into the following line in the "/etc/sysconfig/ntpd" file.
    OPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid"
    Then restart NTP.
    # service ntpd restart
    Create the new groups and users.
    groupadd -g 1000 oinstall
    groupadd -g 1200 dba
    useradd -u 1100 -g oinstall -G dba oracle
    passwd oracle
    Create the directories in which the Oracle software will be installed.
    mkdir -p  /u01/app/11.2.0/grid
    mkdir -p /u01/app/oracle/product/11.2.0/db_1
    chown -R oracle:oinstall /u01
    chmod -R 775 /u01/
    Login as the oracle user and add the following lines at the end of the .bash_profile file.
    # Oracle Settings
    TMP=/tmp; export TMP
    TMPDIR=$TMP; export TMPDIR
    
    ORACLE_HOSTNAME=ol6-112-rac1.localdomain; export ORACLE_HOSTNAME
    ORACLE_UNQNAME=RAC; export ORACLE_UNQNAME
    ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE
    GRID_HOME=/u01/app/11.2.0/grid; export GRID_HOME
    DB_HOME=$ORACLE_BASE/product/11.2.0/db_1; export DB_HOME
    ORACLE_HOME=$DB_HOME; export ORACLE_HOME
    ORACLE_SID=RAC1; export ORACLE_SID
    ORACLE_TERM=xterm; export ORACLE_TERM
    BASE_PATH=/usr/sbin:$PATH; export BASE_PATH
    PATH=$ORACLE_HOME/bin:$BASE_PATH; export PATH
    
    LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
    CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH
    
    if [ $USER = "oracle" ]; then
      if [ $SHELL = "/bin/ksh" ]; then
        ulimit -p 16384
        ulimit -n 65536
      else
        ulimit -u 16384 -n 65536
      fi
    fi
    
    alias grid_env='. /home/oracle/grid_env'
    alias db_env='. /home/oracle/db_env'
    Create a file called "/home/oracle/grid_env" with the following contents.
    ORACLE_SID=+ASM1; export ORACLE_SID
    ORACLE_HOME=$GRID_HOME; export ORACLE_HOME
    PATH=$ORACLE_HOME/bin:$BASE_PATH; export PATH
    
    LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
    CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH
    Create a file called "/home/oracle/db_env" with the following contents.
    ORACLE_SID=RAC1; export ORACLE_SID
    ORACLE_HOME=$DB_HOME; export ORACLE_HOME
    PATH=$ORACLE_HOME/bin:$BASE_PATH; export PATH
    
    LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
    CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH
    Once the "/home/oracle/grid_env" has been run, you will be able to switch between environments as follows.
    $ grid_env
    $ echo $ORACLE_HOME
    /u01/app/11.2.0/grid
    $ db_env
    $ echo $ORACLE_HOME
    /u01/app/oracle/product/11.2.0/db_1
    $

    Install Guest Additions

    Log into the VM as the root user and add the "divider=10" option to the kernel boot options in "/etc/grub.conf" file to reduce the idle CPU load. The entry should look something like this.
    # grub.conf generated by anaconda
    #
    # Note that you do not have to rerun grub after making changes to this file
    # NOTICE:  You have a /boot partition.  This means that
    #          all kernel and initrd paths are relative to /boot/, eg.
    #          root (hd0,0)
    #          kernel /vmlinuz-version ro root=/dev/mapper/vg_ol6112rac1-lv_root
    #          initrd /initrd-[generic-]version.img
    #boot=/dev/sda
    default=0
    timeout=5
    splashimage=(hd0,0)/grub/splash.xpm.gz
    hiddenmenu
    title Oracle Linux Server-uek (2.6.32-100.34.1.el6uek.x86_64)
            root (hd0,0)
            kernel /vmlinuz-2.6.32-100.34.1.el6uek.x86_64 ro root=/dev/mapper/vg_ol6112rac1-lv_root rd_LVM_LV=vg_ol6112rac1/lv_root rd_LVM_LV=vg_ol6112rac1/lv_swap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=uk  rhgb quiet divider=10
            initrd /initramfs-2.6.32-100.34.1.el6uek.x86_64.img
    title Oracle Linux Server (2.6.32-131.0.15.el6.x86_64)
            root (hd0,0)
            kernel /vmlinuz-2.6.32-131.0.15.el6.x86_64 ro root=/dev/mapper/vg_ol6112rac1-lv_root rd_LVM_LV=vg_ol6112rac1/lv_root rd_LVM_LV=vg_ol6112rac1/lv_swap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=uk crashkernel=auto rhgb quiet divider=10
            initrd /initramfs-2.6.32-131.0.15.el6.x86_64.img
    Click on the "Devices > Install Guest Additions" menu option at the top of the VM screen, then run the following commands.
    cd cd /media/VBOXADDITIONS_4.1.4_74291
    export MAKE='/usr/bin/gmake –i'
    sh ./VBoxLinuxAdditions.run
    The VM will need to be restarted for the additions to be used properly. The next section requires a shutdown so no additional restart is needed at this time.

    Create Shared Disks

    Shut down the "rac1" virtual machine using the following command.
    # shutdown -h now
    Create 4 sharable virtual disks and associate them as virtual media using one of the following sets of commands on the host server.
    $ mkdir -p /u04/VirtualBox/ol6-112-rac
    $ cd /u04/VirtualBox/ol6-112-rac
    $
    $ # Create the disks and associate them with VirtualBox as virtual media.
    $ VBoxManage createhd --filename asm1.vdi --size 5120 --format VDI --variant Fixed
    $ VBoxManage createhd --filename asm2.vdi --size 5120 --format VDI --variant Fixed
    $ VBoxManage createhd --filename asm3.vdi --size 5120 --format VDI --variant Fixed
    $ VBoxManage createhd --filename asm4.vdi --size 5120 --format VDI --variant Fixed
    $
    $ # Connect them to the VM.
    $ VBoxManage storageattach ol6-112-rac1 --storagectl "SATA Controller" --port 1 --device 0 --type hdd --medium asm1.vdi --mtype shareable
    $ VBoxManage storageattach ol6-112-rac1 --storagectl "SATA Controller" --port 2 --device 0 --type hdd --medium asm2.vdi --mtype shareable
    $ VBoxManage storageattach ol6-112-rac1 --storagectl "SATA Controller" --port 3 --device 0 --type hdd --medium asm3.vdi --mtype shareable
    $ VBoxManage storageattach ol6-112-rac1 --storagectl "SATA Controller" --port 4 --device 0 --type hdd --medium asm4.vdi --mtype shareable
    $
    $ # Make shareable.
    $ VBoxManage modifyhd asm1.vdi --type shareable
    $ VBoxManage modifyhd asm2.vdi --type shareable
    $ VBoxManage modifyhd asm3.vdi --type shareable
    $ VBoxManage modifyhd asm4.vdi --type shareable
    Start the "ol6-112-rac1" virtual machine by clicking the "Start" button on the toolbar. When the server has started, log in as the root user so you can configure the shared disks. The current disks can be seen by issuing the following commands.
    # cd /dev
    # ls sd*
    sda  sda1  sda2  sdb  sdc  sdd  sde
    #
    Use the "fdisk" command to partition the disks sdb to sdf. The following output shows the expected fdisk output for the sdb disk.
    # fdisk /dev/sdb
    Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
    Building a new DOS disklabel with disk identifier 0x62be91cf.
    Changes will remain in memory only, until you decide to write them.
    After that, of course, the previous content won't be recoverable.
    
    Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
    
    WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
             switch off the mode (command 'c') and change display units to
             sectors (command 'u').
    
    Command (m for help): n
    Command action
       e   extended
       p   primary partition (1-4)
    p
    Partition number (1-4): 1
    First cylinder (1-652, default 1): 
    Using default value 1
    Last cylinder, +cylinders or +size{K,M,G} (1-652, default 652): 
    Using default value 652
    
    Command (m for help): w
    The partition table has been altered!
    
    Calling ioctl() to re-read partition table.
    Syncing disks.
    #
    In each case, the sequence of answers is "n", "p", "1", "Return", "Return" and "w".
    Once all the disks are partitioned, the results can be seen by repeating the previous "ls" command.
    # cd /dev
    # ls sd*
    sda  sda1  sda2  sdb  sdb1  sdc  sdc1  sdd  sdd1  sde  sde1
    #
    Configure your UDEV rules, as shown here.
    The SCSI ID of my disks are displayed below.
    # /sbin/scsi_id -g -u /dev/sdb1
    1ATA_VBOX_HARDDISK_VBbfd37c1c-9b7c9336
    # /sbin/scsi_id -g -u /dev/sdc1
    1ATA_VBOX_HARDDISK_VB46bd30ca-ef5d5d6d
    # /sbin/scsi_id -g -u /dev/sdd1
    1ATA_VBOX_HARDDISK_VB5d431382-68a070ad
    # /sbin/scsi_id -g -u /dev/sde1
    1ATA_VBOX_HARDDISK_VB2ceafd7d-e56ad1d8
    #
    Using these, the "/etc/udev/rules.d/99-oracle-asmdevices.rules" file contained the following entries.
    KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u /dev/$name", RESULT=="1ATA_VBOX_HARDDISK_VBbfd37c1c-9b7c9336", NAME="asm-disk1", OWNER="oracle", GROUP="dba", MODE="0660"
    KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u /dev/$name", RESULT=="1ATA_VBOX_HARDDISK_VB46bd30ca-ef5d5d6d", NAME="asm-disk2", OWNER="oracle", GROUP="dba", MODE="0660"
    KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u /dev/$name", RESULT=="1ATA_VBOX_HARDDISK_VB5d431382-68a070ad", NAME="asm-disk3", OWNER="oracle", GROUP="dba", MODE="0660"
    KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u /dev/$name", RESULT=="1ATA_VBOX_HARDDISK_VB2ceafd7d-e56ad1d8", NAME="asm-disk4", OWNER="oracle", GROUP="dba", MODE="0660"
    Load updated block device partition tables.
    # /sbin/partprobe /dev/sdb1
    # /sbin/partprobe /dev/sdc1
    # /sbin/partprobe /dev/sdd1
    # /sbin/partprobe /dev/sde1
    Test the rules are working as expected.
    # /sbin/udevadm test /block/sdb/sdb1
    Reload the UDEV rules and start UDEV.
    # /sbin/udevadm control --reload-rules
    # /sbin/start_udev
    The disks should now be visible and have the correct ownership using the following command. If they are not visible, your UDEV configuration is incorrect and must be fixed before you proceed.
    # ls -al /dev/asm*
    brw-rw---- 1 oracle dba 8, 17 Oct 12 14:39 /dev/asm-disk1
    brw-rw---- 1 oracle dba 8, 33 Oct 12 14:38 /dev/asm-disk2
    brw-rw---- 1 oracle dba 8, 49 Oct 12 14:39 /dev/asm-disk3
    brw-rw---- 1 oracle dba 8, 65 Oct 12 14:39 /dev/asm-disk4
    #
    The shared disks are now configured for the grid infrastructure.

    Clone the Virtual Machine

    Later versions of VirtualBox allow you to clone VMs, but these also attempt to clone the shared disks, which is not what we want. Instead we must manually clone the VM.
    Shut down the "ol6-112-rac1" virtual machine using the following command.
    # shutdown -h now
    Manually clone the "ol6-112-rac1.vdi" disk using the following commands on the host server.
    • mkdir -p /u03/VirtualBox/ol6-112-rac2
      

    • VBoxManage clonehd /u01/VirtualBox/ol6-112-rac1/ol6-112-rac1.vdi /u03/VirtualBox/ol6-112-rac2/ol6-112-rac2.vdi
     Now create the hardisk in the same way as you did in "ol6-112-rac1", with the exception of using an existing rac2.vdi virtual hard drive.

    Remember to add the second network adaptor as you did on the "ol6-112-rac1" VM. When the VM is created, attach the shared disks to this VM.


    Next Blog I ll tell you the steps about Installing the Grid Infrastructure