10G RAC CRS 服务状态 UNKNOWN

crs_stop -all 命令后遇到 UNKNOWN

RAC中执行crs_stop -all后提示一些服务停不了，再用crs_stat -t查看状态的时候，停不掉的服务会显示状态是UNKNOWN，
服务在UNKNOWN状态下用 crs_start 不能启动服务

例如：
[oracle@rac1 ~]$ crs_stop -all
Attempting to stop `ora.rac1.gsd` on member `rac1`
Attempting to stop `ora.rac1.ons` on member `rac1`
Attempting to stop `ora.rac2.gsd` on member `rac2`
Attempting to stop `ora.ora9i.db` on member `rac1`
Attempting to stop `ora.rac2.ons` on member `rac2`
Stop of `ora.rac1.gsd` on member `rac1` succeeded.
Stop of `ora.rac2.gsd` on member `rac2` succeeded.
Stop of `ora.rac2.ons` on member `rac2` succeeded.
Stop of `ora.rac1.ons` on member `rac1` succeeded.
Stop of `ora.ora9i.db` on member `rac1` succeeded.
Attempting to stop `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1`
Attempting to stop `ora.rac2.LISTENER_RAC2.lsnr` on member `rac2`
`ora.rac1.LISTENER_RAC1.lsnr` on member `rac1` has experienced an unrecoverable failure.
Human intervention required to resume its availability.
`ora.ora9i.ora9i1.inst` is already OFFLINE.
`ora.rac2.LISTENER_RAC2.lsnr` on member `rac2` has experienced an unrecoverable failure.
Human intervention required to resume its availability.
`ora.ora9i.ora9i2.inst` is already OFFLINE.
Attempting to stop `ora.rac1.ASM1.asm` on member `rac1`
`ora.rac1.ASM1.asm` on member `rac1` has experienced an unrecoverable failure.
Human intervention required to resume its availability.
Attempting to stop `ora.rac2.ASM2.asm` on member `rac2`
Attempting to stop `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1`
`ora.rac2.ASM2.asm` on member `rac2` has experienced an unrecoverable failure.
Human intervention required to resume its availability.
`ora.rac1.LISTENER_RAC1.lsnr` on member `rac1` has experienced an unrecoverable failure.
Human intervention required to resume its availability.
Attempting to stop `ora.rac1.ASM1.asm` on member `rac1`
`ora.rac1.ASM1.asm` on member `rac1` has experienced an unrecoverable failure.
Human intervention required to resume its availability.
Attempting to stop `ora.rac2.LISTENER_RAC2.lsnr` on member `rac2`
Attempting to stop `ora.rac1.vip` on member `rac1`
`ora.rac2.LISTENER_RAC2.lsnr` on member `rac2` has experienced an unrecoverable failure.
Human intervention required to resume its availability.
Attempting to stop `ora.rac2.ASM2.asm` on member `rac2`
`ora.rac2.ASM2.asm` on member `rac2` has experienced an unrecoverable failure.
Human intervention required to resume its availability.
Attempting to stop `ora.rac2.vip` on member `rac2`
Stop of `ora.rac1.vip` on member `rac1` succeeded.
Stop of `ora.rac2.vip` on member `rac2` succeeded.
CRS-0216: Could not stop resource 'ora.ora9i.ora9i1.inst'.

CRS-0216: Could not stop resource 'ora.ora9i.ora9i2.inst'.

CRS-0216: Could not stop resource 'ora.rac1.ASM1.asm'.

CRS-0216: Could not stop resource 'ora.rac1.LISTENER_RAC1.lsnr'.

CRS-0216: Could not stop resource 'ora.rac1.vip'.

CRS-0216: Could not stop resource 'ora.rac2.ASM2.asm'.

CRS-0216: Could not stop resource 'ora.rac2.LISTENER_RAC2.lsnr'.

CRS-0216: Could not stop resource 'ora.rac2.vip'.

在UNKNOWN状态下，使用crs_start -all命令不会把RAC启动起来，启动的时候会报错。

[oracle@rac1 ~]$ crs_start -all
Attempting to start `ora.rac1.vip` on member `rac1`
Attempting to start `ora.rac2.vip` on member `rac2`
Attempting to start `ora.rac2.gsd` on member `rac2`
Attempting to start `ora.rac2.ons` on member `rac2`
Attempting to start `ora.rac1.gsd` on member `rac1`
Attempting to start `ora.rac1.ons` on member `rac1`
`ora.rac2.gsd` on member `rac2` has experienced an unrecoverable failure.
Human intervention required to resume its availability.
`ora.rac2.ons` on member `rac2` has experienced an unrecoverable failure.
Human intervention required to resume its availability.
Attempting to start `ora.ora9i.db` on member `rac1`
`ora.rac1.ons` on member `rac1` has experienced an unrecoverable failure.
Human intervention required to resume its availability.
`ora.rac1.gsd` on member `rac1` has experienced an unrecoverable failure.
Human intervention required to resume its availability.
`ora.ora9i.db` on member `rac1` has experienced an unrecoverable failure.
Human intervention required to resume its availability.
Start of `ora.rac1.vip` on member `rac1` succeeded.
CRS-1028: Dependency analysis failed because of:
'Resource in UNKNOWN state: ora.rac1.ASM1.asm'

Start of `ora.rac2.vip` on member `rac2` succeeded.
CRS-1028: Dependency analysis failed because of:
'Resource in UNKNOWN state: ora.rac2.ASM2.asm'

CRS-0215: Could not start resource 'ora.ora9i.db'.

CRS-0223: Resource 'ora.ora9i.ora9i1.inst' has placement error.

CRS-0223: Resource 'ora.ora9i.ora9i2.inst' has placement error.

CRS-0215: Could not start resource 'ora.rac1.gsd'.

CRS-0215: Could not start resource 'ora.rac1.ons'.

CRS-0215: Could not start resource 'ora.rac2.gsd'.

CRS-0215: Could not start resource 'ora.rac2.ons'.

此时crs_stat -t的状态还是有很多的UNKNOWN
（烦、烦、烦，rac中真奇怪，正常启动关闭服务时，
  状态只有online和offline两种（启动与未启动），结果RAC中给你来了个UNKNOWN）：
[oracle@rac1 ~]$ crs_stat -t
Name           Type           Target    State     Host       
------------------------------------------------------------
ora.ora9i.db   application    ONLINE    UNKNOWN   rac1       
ora....i1.inst application    OFFLINE   OFFLINE              
ora....i2.inst application    OFFLINE   OFFLINE              
ora....SM1.asm application    OFFLINE   UNKNOWN   rac1       
ora....C1.lsnr application    OFFLINE   UNKNOWN   rac1       
ora.rac1.gsd   application    ONLINE    UNKNOWN   rac1       
ora.rac1.ons   application    ONLINE    UNKNOWN   rac1       
ora.rac1.vip   application    ONLINE    ONLINE    rac1       
ora....SM2.asm application    OFFLINE   UNKNOWN   rac2       
ora....C2.lsnr application    OFFLINE   UNKNOWN   rac2       
ora.rac2.gsd   application    ONLINE    UNKNOWN   rac2       
ora.rac2.ons   application    ONLINE    UNKNOWN   rac2       
ora.rac2.vip   application    ONLINE    ONLINE    rac2       
[oracle@rac1 ~]$ sqlplus /nolog

SQL*Plus: Release 10.2.0.1.0 - Production on Thu Aug 7 13:42:36 2008

Copyright (c) 1982, 2005, Oracle.  All rights reserved.

SQL> conn /as sysdba
Connected to an idle instance.
显示数据库已经连接不上；

此时可有两种方法来解决：

方法一：先禁用crs服务，后启用crs服务
（相当于从新启动系统，因为crs是随系统自动启动的），需要在两台节点上都执行；

具体步骤：
节点1：
1、[root@rac1 ~]# /etc/init.d/init.crs stop
Shutting down Oracle Cluster Ready Services (CRS):
Stopping resources.
Successfully stopped CRS resources
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.
Shutdown has begun. The daemons should exit soon.
2、[root@rac1 ~]# su - oracle
[oracle@rac1 ~]$ crs_stat -t
CRS-0184: Cannot communicate with the CRS daemon.

3、[oracle@rac1 ~]$ exit
logout
[root@rac1 ~]# /etc/init.d/init.crs start
Startup will be queued to init within 90 seconds.
4、[root@rac1 ~]# su - oracle
[oracle@rac1 ~]$ crs_stat -t
Name           Type           Target    State     Host       
------------------------------------------------------------
ora.ora9i.db   application    ONLINE    ONLINE    rac1       
ora....i1.inst application    ONLINE    ONLINE    rac1       
ora....i2.inst application    ONLINE    ONLINE    rac2       
ora....SM1.asm application    ONLINE    ONLINE    rac1       
ora....C1.lsnr application    ONLINE    ONLINE    rac1       
ora.rac1.gsd   application    ONLINE    ONLINE    rac1       
ora.rac1.ons   application    ONLINE    ONLINE    rac1       
ora.rac1.vip   application    ONLINE    ONLINE    rac1       
ora....SM2.asm application    ONLINE    ONLINE    rac2       
ora....C2.lsnr application    ONLINE    ONLINE    rac2       
ora.rac2.gsd   application    ONLINE    ONLINE    rac2       
ora.rac2.ons   application    ONLINE    ONLINE    rac2       
ora.rac2.vip   application    ONLINE    ONLINE    rac2       
[oracle@rac1 ~]$
 

节点2：
1、[root@rac2 ~]# /etc/init.d/init.crs stop
Shutting down Oracle Cluster Ready Services (CRS):
Stopping resources.
Successfully stopped CRS resources
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.
Shutdown has begun. The daemons should exit soon.
2、[root@rac2 ~]# su - oracle
[oracle@rac2 ~]$ crs_stat -t
CRS-0184: Cannot communicate with the CRS daemon.

[oracle@rac2 ~]$ exit
logout
3、[root@rac2 ~]# /etc/init.d/init.crs start
Startup will be queued to init within 90 seconds.
4、[root@rac2 ~]# su - oracle
[oracle@rac2 ~]$ crs_stat -t
Name           Type           Target    State     Host       
------------------------------------------------------------
ora.ora9i.db   application    ONLINE    ONLINE    rac1       
ora....i1.inst application    ONLINE    ONLINE    rac1       
ora....i2.inst application    ONLINE    ONLINE    rac2       
ora....SM1.asm application    ONLINE    ONLINE    rac1       
ora....C1.lsnr application    ONLINE    ONLINE    rac1       
ora.rac1.gsd   application    ONLINE    ONLINE    rac1       
ora.rac1.ons   application    ONLINE    ONLINE    rac1       
ora.rac1.vip   application    ONLINE    ONLINE    rac1       
ora....SM2.asm application    ONLINE    ONLINE    rac2       
ora....C2.lsnr application    ONLINE    ONLINE    rac2       
ora.rac2.gsd   application    ONLINE    ONLINE    rac2       
ora.rac2.ons   application    ONLINE    ONLINE    rac2       
ora.rac2.vip   application    ONLINE    ONLINE    rac2       
[oracle@rac2 ~]$

方法二：可以使用crs_stop -f先强制停止服务，在启动服务（在一台节点上运行即可）；
具体步骤：

[oracle@rac2 ~]$ crs_stop -f ora.rac1.ons
Attempting to stop `ora.rac1.ons` on member `rac1`
Stop of `ora.rac1.ons` on member `rac1` succeeded.
[oracle@rac2 ~]$ crs_start -f ora.rac1.ons
Attempting to start `ora.rac1.ons` on member `rac1`
Start of `ora.rac1.ons` on member `rac1` succeeded.
[oracle@rac2 ~]$ crs_stop -f ora.rac2.ons
Attempting to stop `ora.rac2.ons` on member `rac2`
Stop of `ora.rac2.ons` on member `rac2` succeeded.
[oracle@rac2 ~]$ crs_start -f ora.rac2.ons
Attempting to start `ora.rac2.ons` on member `rac2`
Start of `ora.rac2.ons` on member `rac2` succeeded.
[oracle@rac2 ~]$ crs_stop -f ora.rac1.gsd
Attempting to stop `ora.rac1.gsd` on member `rac1`
Stop of `ora.rac1.gsd` on member `rac1` succeeded.
[oracle@rac2 ~]$ crs_start -f ora.rac1.gsd
Attempting to start `ora.rac1.gsd` on member `rac1`
Start of `ora.rac1.gsd` on member `rac1` succeeded.
[oracle@rac2 ~]$ crs_stop -f ora.rac2.gsd
Attempting to stop `ora.rac2.gsd` on member `rac2`
Stop of `ora.rac2.gsd` on member `rac2` succeeded.
[oracle@rac2 ~]$ crs_start -f ora.rac2.gsd
Attempting to start `ora.rac2.gsd` on member `rac2`
Start of `ora.rac2.gsd` on member `rac2` succeeded.
[oracle@rac2 ~]$ crs_stop -f ora.rac1.LISTENER_RAC1.lsnr
Attempting to stop `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1`
Stop of `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1` succeeded.
[oracle@rac2 ~]$ crs_start -f ora.rac1.LISTENER_RAC1.lsnr
Attempting to start `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1`
Start of `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1` succeeded.
[oracle@rac2 ~]$ crs_stop -f ora.rac2.LISTENER_RAC2.lsnr
Attempting to stop `ora.rac2.LISTENER_RAC2.lsnr` on member `rac2`
Stop of `ora.rac2.LISTENER_RAC2.lsnr` on member `rac2` succeeded.
[oracle@rac2 ~]$ crs_start -f ora.rac2.LISTENER_RAC2.lsnr
Attempting to start `ora.rac2.LISTENER_RAC2.lsnr` on member `rac2`
Start of `ora.rac2.LISTENER_RAC2.lsnr` on member `rac2` succeeded.
[oracle@rac2 ~]$ crs_stop -f ora.rac1.ASM1.asm
Attempting to stop `ora.rac1.ASM1.asm` on member `rac1`
Stop of `ora.rac1.ASM1.asm` on member `rac1` succeeded.
[oracle@rac2 ~]$ crs_start -f ora.rac1.ASM1.asm
Attempting to start `ora.rac1.ASM1.asm` on member `rac1`
Start of `ora.rac1.ASM1.asm` on member `rac1` succeeded.
[oracle@rac2 ~]$ crs_stop -f ora.rac2.ASM2.asm
Attempting to stop `ora.rac2.ASM2.asm` on member `rac2`
Stop of `ora.rac2.ASM2.asm` on member `rac2` succeeded.
[oracle@rac2 ~]$ crs_start -f ora.rac2.ASM2.asm
Attempting to start `ora.rac2.ASM2.asm` on member `rac2`
Start of `ora.rac2.ASM2.asm` on member `rac2` succeeded.
[oracle@rac2 ~]$ crs_stop -f ora.ora9i.ora9i1.inst
`ora.ora9i.ora9i1.inst` is already OFFLINE.
CRS-0216: Could not stop resource 'ora.ora9i.ora9i1.inst'.

[oracle@rac2 ~]$ crs_start -f ora.ora9i.ora9i1.inst
Attempting to start `ora.ora9i.ora9i1.inst` on member `rac1`
Start of `ora.ora9i.ora9i1.inst` on member `rac1` succeeded.
[oracle@rac2 ~]$ crs_stop -f ora.ora9i.ora9i2.inst
`ora.ora9i.ora9i2.inst` is already OFFLINE.
CRS-0216: Could not stop resource 'ora.ora9i.ora9i2.inst'.

[oracle@rac2 ~]$ crs_start -f ora.ora9i.ora9i2.inst
Attempting to start `ora.ora9i.ora9i2.inst` on member `rac2`
Start of `ora.ora9i.ora9i2.inst` on member `rac2` succeeded.
[oracle@rac2 ~]$ crs_stop -f ora.ora9i.db
Attempting to stop `ora.ora9i.db` on member `rac2`
Stop of `ora.ora9i.db` on member `rac2` succeeded.
[oracle@rac2 ~]$ crs_start -f ora.ora9i.db
Attempting to start `ora.ora9i.db` on member `rac1`
Start of `ora.ora9i.db` on member `rac1` succeeded.
 
 
[oracle@rac2 ~]$ crs_stat -t
Name           Type           Target    State     Host       
------------------------------------------------------------
ora.ora9i.db   application    ONLINE    ONLINE    rac1       
ora....i1.inst application    ONLINE    ONLINE    rac1       
ora....i2.inst application    ONLINE    ONLINE    rac2       
ora....SM1.asm application    ONLINE    ONLINE    rac1       
ora....C1.lsnr application    ONLINE    ONLINE    rac1       
ora.rac1.gsd   application    ONLINE    ONLINE    rac1       
ora.rac1.ons   application    ONLINE    ONLINE    rac1       
ora.rac1.vip   application    ONLINE    ONLINE    rac1       
ora....SM2.asm application    ONLINE    ONLINE    rac2       
ora....C2.lsnr application    ONLINE    ONLINE    rac2       
ora.rac2.gsd   application    ONLINE    ONLINE    rac2       
ora.rac2.ons   application    ONLINE    ONLINE    rac2       
ora.rac2.vip   application    ONLINE    ONLINE    rac2

总结：在rac环境中如果使用crs_stop -all去停rac服务时，很有可能会出现crs中的一些服务的状态是UNKNOWN，这时我们可以采用两种方法去重新启动crs服务：

1：转换到root用户下用/etc/init.d/init.crs stop先禁用crs，然后再用/etc/init.d/init.crs start去启用crs，启用crs后会自动启动crs的一系列服务，注意此种方法需要在两台节点上都执行；

2：是用crs_stop -f  参数把crs中状态是UNKNOWN的服务关掉，然后再用crs_start -f （加一个-f的参数）启动所有的服务就可以。参考：http://www.rachelp.nl/index_kb.php?menu=articles&actie=show&id=31

要分别在两个节点上执行；

在启动的过程中最好检测着crs、ASM和数据库的日志：

crs日志：
[oracle@rac1 ~]$ tail -f /u01/app/oracle/product/10.2.0/crs_1/log/rac1/alertrac1.log
[oracle@rac2 ~]$ tail -f /u01/app/oracle/product/10.2.0/crs_1/log/rac2/alertrac2.log

ASM日志：
[oracle@rac1 ~]$ tail -f /u01/app/oracle/admin/+ASM/bdump/alert_+ASM1.log
[oracle@rac2 ~]$ tail -f /u01/app/oracle/admin/+ASM/bdump/alert_+ASM2.log
数据库日志：
[oracle@rac1 ~]$ tail -f /u01/app/oracle/admin/ora9i/bdump/alert_ora9i1.log
[oracle@rac2 ~]$ tail -f /u01/app/oracle/admin/ora9i/bdump/alert_ora9i2.log