How to Change the public IP Address in an Oracle RAC

 

Though this task is well documented (see [1] and [2]) there are still some pitfalls and quirks that can cause problems. Such as..

 Should a particular task be done with the root or oracle user? Should it be performed on both nodes?

This is a step-by-step description how to perform the change of the public IP address in the Oracle two node RAC. The Oracle version is 10.2.0.3. The OS used in this example is Linux Centos 2.6.9-42.ELsmp

 

Note that due to the change of the private IP address the virtual IP address (VIP) may be required to change as well as it must remain in the same subnet as the public IP address.

 

The configuration

 

 

Host

Current IP Address

Changed IP Address

Private IP

hst1

192.168.1.101  

192.168.2.121  

Private IP

hst2

192.168.1.100  

192.168.2.120  

VIP

hst1-vip

192.168.1.111  

192.168.2.111  

VIP

hst2-vip

192.168.1.110  

192.168.2.110  

 

 

 

Note that the actual value of IP addresses and host names was changed.

 

The basic steps

 

The change of the private IP address is performed in the following basic steps

 

1) Shut down everything except the CRS stack

2) Change the public interface

3) Modify the VIP address

4) Shut down CRS

5) Modify IP address on OS level and reconfigure /etc/hosts, listener,..

6) Restart

 

The order of the steps is very important. E.g. step 3) can be performed only if the CRS stack is up. If you perform step 5) prematurely, CRS will go down automatically (probably trying some reboots beforehand). This will effectively block step 3) to be done.

 

Shut Down Everything Except the CRS Stack

 

The database and nodeapps (on all nodes) are stopped.

 

 

[oracle@hst1 ~]$ srvctl stop database -d mydb

[oracle@hst1 ~]$ srvctl stop nodeapps -n hst1

[oracle@hst1 ~]$ srvctl stop nodeapps -n hst2

 

Note that if you use ASM, it must be stopped as well.

 

After that we verify the status.

 

[oracle@hst1 ~]$ srvctl status database -d mydb

Instance MYDB1 is not running on node hst2

Instance MYDB2 is not running on node hst1

[oracle@hst1 ~]$ srvctl status nodeapps -n hst1

VIP is not running on node: hst1

GSD is not running on node: hst1

Listener is not running on node: hst1

ONS daemon is not running on node: hst1

[oracle@hst1 ~]$ srvctl status nodeapps -n hst2

VIP is not running on node: hst2

GSD is not running on node: hst2

Listener is not running on node: hst2

ONS daemon is not running on node: hst2

 

Everything is OK, we can go to the next step.

 

Change the Public Interface

 

First let us have a look on the actual status

 

[oracle@hst1 ~]$ oifcfg getif

eth0  192.168.1.0  global  public

eth1  192.168.2.0  global  cluster_interconnect

 

We need to change the interface eth0.

As there is no modify command, we will delete and redefine the interface.

 

 

[oraclu@hst1 ~]$ oifcfg delif -global eth0

[oraclu@hst1 ~]$ oifcfg setif -global eth0/192.168.2.0:public

 

the exact syntax of the setif command can be found in [3]

 

Note that the CRS installation user (here oraclu) must be used for this operation. Otherwise an error message is issued:

 

PROC-5: User does not have permission to perform a cluster registry operation on

 this key. Authentication error [User [oraclu] does not match with initialized u

ser] [0]

PRIF-11: cluster registry error

 

Finally, we check if the action was done successfully

 

[oracle@hst1 ~]$ oifcfg getif

eth0  192.168.2.0  global  public

eth1  192.168.2.0  global  cluster_interconnect

 

 

Modify the VIP Address

 

As already mentioned, due to the fact that we changed the subnet of the public IP address, we must change the VIP address as well.

The following modify statement should be used.

 

[root@hst1 ~]# /appl/oracle/product/10.2.0/db_1/bin/srvctl modify nodeapps -n hst2 -A 192.168.2.110/255.255.255.0/eth0

[root@hst1 ~]# /appl/oracle/product/10.2.0/db_1/bin/srvctl modify nodeapps -n hst1 -A 192.168.2.111/255.255.255.0/eth0

 

Note that the root user should be used for this action.

The variable ORACLE_HOHE must be initialised.

Otherwise one of the following errors will be raised.

 

PRKO-2117 : This command should be executed as the system privilege user.

 

 

****ORACLE_HOME environment variable not set!

    ORACLE_HOME should be set to the main

    directory that contains Oracle products.

    Set and export ORACLE_HOME, then re-run.

 

 

Shut Down CRS

 

The stop command must be performed on all nodes:

 

crsctl stop crs

 

Modify the IP Address on OS Level

 

Modify the public IP address (eth0). In Centos with application / system setting / network

Perform the change on all nodes.

Modify /etc/hosts and listener.ora files if required.

 

Restart

 

Reboot all nodes and verify the status.

Shortly after reboot we can see...

 

[oracle@hst2 ~]$ /sbin/ifconfig -a | egrep '(eth|Mask)'

eth0      Link encap:Ethernet  HWaddr CA:FE:CA:FE:CA:FE

          inet addr:192.168.2.120  Bcast:192.168.2.255  Mask:255.255.255.0

eth0:1    Link encap:Ethernet  HWaddr CA:FE:CA:FE:CA:FE

          inet addr:192.168.2.111  Bcast:192.168.2.255  Mask:255.255.255.0

eth0:2    Link encap:Ethernet  HWaddr CA:FE:CA:FE:CA:FE

          inet addr:192.168.2.110  Bcast:192.168.2.255  Mask:255.255.255.0

eth1      Link encap:Ethernet  HWaddr CA:FE:CA:FE:CA:F1

          inet addr:192.168.2.100  Bcast:192.168.2.255  Mask:255.255.255.0

          inet addr:127.0.0.1  Mask:255.0.0.0

 

This means both VIP's eth0:1 and eth0:2 are switched to one host. This is not good, normally this appears if one host is down and the second host takes over the VIP.

 

Fortunately, after a few seconds we see..

 

[oracle@hst2 ~]$ /sbin/ifconfig -a | egrep '(eth|Mask)'

eth0      Link encap:Ethernet  HWaddr CA:FE:CA:FE:CA:FE

          inet addr:192.168.2.120  Bcast:192.168.2.255  Mask:255.255.255.0

eth0:2    Link encap:Ethernet  HWaddr CA:FE:CA:FE:CA:FE

          inet addr:192.168.2.110  Bcast:192.168.2.255  Mask:255.255.255.0

eth1      Link encap:Ethernet  HWaddr CA:FE:CA:FE:CA:F1

          inet addr:192.168.2.100  Bcast:192.168.2.255  Mask:255.255.255.0

          inet addr:127.0.0.1  Mask:255.0.0.0

 

[oracle@hst1 ~]$ /sbin/ifconfig -a | egrep '(eth|Mask)'

eth0      Link encap:Ethernet  HWaddr CA:FE:CA:FE:CA:F2

          inet addr:192.168.2.121  Bcast:192.168.2.255  Mask:255.255.255.0

eth0:1    Link encap:Ethernet  HWaddr CA:FE:CA:FE:CA:F2

          inet addr:192.168.2.111  Bcast:192.168.2.255  Mask:255.255.255.0

eth1      Link encap:Ethernet  HWaddr CA:FE:CA:FE:CA:F3

          inet addr:192.168.2.101  Bcast:192.168.2.255  Mask:255.255.255.0

          inet addr:127.0.0.1  Mask:255.0.0.0

 

I.e. the VIP's are distributed correctly.

The configuration of IP addresses is as required.

Let’s have a final look at the status of the RAC.

 

[oracle@hst1 ~]$ srvctl status database -d mydb

Instance MYDB1 is running on node hst2

Instance MYDB2 is running on node hst1

 

[oracle@hst1 ~]$ srvctl status nodeapps -n hst1

VIP is running on node: hst1

GSD is running on node: hst1

Listener is running on node: hst1

ONS daemon is running on node: hst1

 

[oracle@hst2 ~]$ srvctl status database -d mydb

Instance MYDB1 is running on node hst2

Instance MYDB2 is running on node hst1

[oracle@hst2 ~]$ srvctl status nodeapps -n hst1

VIP is running on node: hst1

GSD is running on node: hst1

Listener is running on node: hst1

ONS daemon is running on node: hst1

 

 

[root@hst1 ~]#  /appl/oraclu/product/10.2.0/crs/bin/crs_stat -t

Name           Type           Target    State     Host

------------------------------------------------------------

ora....B1.inst application    ONLINE    ONLINE    hst2

ora....B2.inst application    ONLINE    ONLINE    hst1

ora....DB1.srv application    ONLINE    ONLINE    hst2

ora....DB2.srv application    ONLINE    ONLINE    hst1

ora....BTAF.cs application    ONLINE    ONLINE    hst2

ora.MYDB.db    application    ONLINE    ONLINE    hst2

ora....B2.lsnr application    ONLINE    ONLINE    hst2

ora.hst2.gsd   application    ONLINE    ONLINE    hst2

ora.hst2.ons   application    ONLINE    ONLINE    hst2

ora.hst2.vip   application    ONLINE    ONLINE    hst2

ora....B1.lsnr application    ONLINE    ONLINE    hst1

ora.hst1.gsd   application    ONLINE    ONLINE    hst1

ora.hst1.ons   application    ONLINE    ONLINE    hst1

ora.hst1.vip   application    ONLINE    ONLINE    hst1

 

 

Everything is up. We are ready!

 

 

 

References

 

[1] http://forums.oracle.com/forums/thread.jspa?messageID=2161448

 

[2] Metalink Note:276434.1

https://metalink.oracle.com/metalink/plsql/showdoc?db=NOT&id=276434.1&blackframe=1

 

[3] Metalink Note:283684.1

https://metalink.oracle.com/metalink/plsql/showdoc?db=NOT&id=283684.1&blackframe=1 

 

(Account is required for Metalink access)

 

Jaromir D.B. Nemec is a freelancer specializing in data warehouse and integration solutions. He can be reached under http://www.db-nemec.com