So I’ve been having a tough time of it lately on things I should know better. We’ve all been there before.
I’m trying to recreate a 2-node RAC testbed on my laptop. This will be Oracle 220.127.116.11 on Oracle Linux 6.5 with VirtualBox 5.0. Should be pretty simple, right? I even wrote instructions on how I did this back in April of this year. I’m following the same exact steps in my document on the same exact laptop but yet I still have problems. The only two things that have changed is I am going directly to 18.104.22.168 (my doc was 22.214.171.124 I believe) and I now have VB 5.0 on my laptop.
I have my 2 virtual nodes created and ready to go. I fire up the OUI to start my installation of Grid Infrastructure. When the OUI gets to the linking phase, I get an error message.
Error in invoking target ‘irman ioracle’ of makefile.
We’ve all been at a crossroads in life. You have a choice to make. Do I go left or do I go right? Unfortunately for me, I turned the wrong direction and wasted a few weeks of my spare time. At this point, I had a decision to make. Do I do exactly as the error said or do I rely on my experience? I blindly ignored the error and relied on my experience. Silly me.
Just two months ago I had the exact same error when I couldn’t get GI 126.96.36.199 to compile on a testbed. That testbed was running on VMWare ESX hosts. I filed a SR with Oracle Support and they let me know that my compile issue was because I did not have enough RAM devoted to each virtual machine. Being a virtual environment, this was easy enough to fix. I had my SysAdmin bump up the RAM and swap space on the virtual machines and GI 188.8.131.52 compiled exactly as promised. So I naturally assumed that I was running into the same issue here. On my laptop, I kept bumping up RAM. I expanded swap space. I even went so far as to rebuild the nodes from scratch. I spent the last two weeks down the road of experience and I found it to be bumpy, scratchy, dusty and very unpleasant.
The road I should have taken was to do explicitly as the pop-up box said…read the log file for more details. When I finally got past my stubbornness in thinking I knew the answer, I read the log file. I found the following messages towards the end.
INFO: - Linking Oracle
INFO: rm -f /u01/app/crs184.108.40.206/rdbms/lib/oracle
INFO: /usr/bin/ld: cannot find -ljavavm12 collect2: ld returned 1 exit status
INFO: make: *** [/u01/app/crs220.127.116.11/rdbms/lib/oracle] Error 1
Well its completely obvious now! A library file is missing. A quick check on the Internet easily led me to the solution. I’m not alone in this issue, but for some reason the OUI is not copying libjavavm12.a to $GRID_HOME/lib as it should be. With that pop-up box still sitting there, I issue the following on the node.
[oracle@host01 ~]$ export GRID_HOME=/u01/app/crs18.104.22.168 [oracle@host01 ~]$ cp $GRID_HOME/javavm/jdk/jdk7/lib/libjavavm12.a $GRID_HOME/lib
I then went back to the pop-up box and pressed the RETRY button. Sure enough, the damn thing compiled and the OUI finished up its work.
UPDATE: I had the same issue when installing the RDBMS 22.214.171.124 software on the cluster. I used the same exact fix to get oracle to compile correctly for the database software as well.
But I was not done. As anyone who has installed Grid Infrastructure knows, they need to run the $GRID_HOME/root.sh script on all nodes later on in the process. When I attempted this on the first node, I received a “segmentation fault” error. Turns out, there is a problem (and again, I’m not alone here) with perl in GI 126.96.36.199 installations. Even the following will receive a segmentation fault:
The solution is to re-install Perl in $GRID_HOME. I found a blog entry by Laurent Leturgez which describes exactly how this is done. I never would have figured this out on my own. After re-installing Perl on all my nodes, the root.sh script ran just fine.
UPDATE: I had the same issue when installing the RDBMS 188.8.131.52 software on the cluster. I used the same exact fix to get perl to run without a segmentation fault.
Like all of us, I rely on my experience to save me lots of time. Things I learned a month or a year ago are applied today. Sometimes, experience gets in our way and takes us down a road we wished remained untravelled.