Production Cluster Part One

During the week of April 2nd, 2007 started the process of installing Oracle Cluster 10g R2 on six Itanium Blades running HP-UX 11.23.

The first pass at installing the software resulted in five blades forming and joining the Cluster without any problems. We had trouble getting node (blade) 5 to join.

After spending a few hours troubleshooting we decided it would be beneficial to drop (delete) the fifth node from the cluster configuration and use the clusterware software options to add it back in. Accidentally, during the process to drop the fifth node we accidentally deleted the first node which also happened to be the master node at the time.

At that point I decided to de-install the Oracle clusterware software and start over. If you have ever used the Oracle Installer to de-install software you are aware that the majority of the times the Oracle Installer leaves directories behind. In the case of Oracle Clusterware and the fact that after the two root.sh scripts are executed by the root user before you can proceed with uninstalling the Oracle software via Oracle Installer you need to change the owner of CRS_ORACLE_HOME back to oracle:oinstall. The Installer also needs access to /var/opt/oracle, /usr/local/bin and /etc/oratab. Surprisingly, the Installer actually removed the bulk of the installed software, however, we still have to remove some directories under $CRS_ORACLE_HOME or /opt/app/oracle/product/10.2.0/crs/.., had the root user remove /var/opt/oracle all together as well as remove /etc/oratb and the three shell programs in /usr/local/bin – dbhome, coraenv and oraenv. Last but not the least important because you may not be able to successfully re-install the software if you don’t zero out the voting disk and cluster registry raw devices. If you are using OCFS2 then you may not have to clear or erase these two key file resources but if you are using raw devices you MUST do this.