当我远程过去的时候，发现节点2已经没有任何oracle用户的进程，且concurrent的vg没有激活，HACMP的服务也offline。
另一个节点Oracle的实例是正常的，且有部分服务器进程依然在工作，但是本地监听器出现了故障，导致新的连接无法连接到实例，通过crs_stat -t看到两个实例的监听也都是OFFLINE状态。
在节点上并没有发现有LISTENER进程，且手动杀掉了所有的服务器进程，在oracle用户下启动监听时收到以下的报错：
$ lsnrctl start listener_cdfy740a
LSNRCTL for IBM/AIX RISC System/6000: Version 11.1.0.6.0 - Production on 20-NOV-2014 20:09:09
Copyright (c) 1991, 2007, Oracle.? All rights reserved.
Starting /oracle/app/oracle/product/11.1.0/db_1/bin/tnslsnr: please wait...
TNSLSNR for IBM/AIX RISC System/6000: Version 11.1.0.6.0 - Production
System parameter file is /oracle/app/oracle/product/11.1.0/db_1/network/admin/listener.ora
Log messages written to /oracle/app/oracle/diag/tnslsnr/cdfy740a/listener_cdfy740a/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.107.64.1)(PORT=1521)))
Error listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=10.107.64.1)(PORT=1521)(IP=FIRST)))
TNS-12542: TNS:address already in use
?TNS-12560: TNS:protocol adapter error
? TNS-00512: Address already in use
? IBM/AIX RISC System/6000 Error: 67: Address already in use
Listener failed to start. See the error message(s) above...
10.107.64.1是该节点的vip地址，下面是RAC环境的hosts配置：
10.107.64.1? ? vip1
10.107.64.2? ? vip2
10.107.64.3? ? cdfy740a
10.107.64.4? ? cdfy740b
172.201.201.1? prv1
172.201.201.2? prv2
手动停掉该节点的nodeapps服务：
cdfy740a@root[/oracle/app/11.1.0/crs/bin]./srvctl stop nodeapps -n cdfy740a
成功停止后，VIP在主机层面已经消失：
cdfy740a@root[/oracle/app/11.1.0/crs/bin]ifconfig -a | more
en0: flags=1e080863,c0
? ? ? ? inet 172.200.200.1 netmask 0xffffff00 broadcast 172.200.200.255
? ? ? ? tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0
en1: flags=1e080863,c0
? ? ? ? inet 172.201.201.1 netmask 0xffffff00 broadcast 172.201.201.255
? ? ? ? tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0
en4: flags=5e080863,c0
? ? ? ? inet 10.107.64.3 netmask 0xffffff00 broadcast 10.107.64.255
? ? ? ? tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0
lo0: flags=e08084b,c0
? ? ? ? inet 127.0.0.1 netmask 0xff000000 broadcast 127.255.255.255
? ? ? ? inet6 ::1%1/0
? ? ? ? tcp_sendspace 131072 tcp_recvspace 131072 rfc1323 1
再次启动节点nodeapps服务：
cdfy740a@root[/oracle/app/11.1.0/crs/bin]./srvctl start nodeapps -n cdfy740a
CRS-1006: No more members to consider
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:LSNRCTL for IBM/AIX RISC System/6000: Version 11.1.0.6.0 - Production on 20-NOV-2014 20:13:07
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:Copyright (c) 1991, 2007, Oracle.? All rights reserved.
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:Starting /oracle/app/oracle/product/11.1.0/db_1/bin/tnslsnr: please wait...
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:TNSLSNR for IBM/AIX RISC System/6000: Version 11.1.0.6.0 - Production
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:System parameter file is /oracle/app/oracle/product/11.1.0/db_1/network/admin/listener.ora
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:Log messages written to /oracle/app/oracle/diag/tnslsnr/cdfy740a/listener_cdfy740a/alert/log.xml
cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:Listening on: (DESCRIPTION