整理一下前段时间遇到的ORA-15064错误。
有一套测试环境是11gR2的rac使用ASM，应用在测试时发现一个节点宕机了，检查了一下alert日志如下错误：
Mon Jun 04 23:27:32 2012
NOTE: ASMB terminatingErrors in file /soft/Oracle/diag/rdbms/pt1db/pt1db2/trace/pt1db2_asmb_13828564.trc:
ORA-15064: communication failure with ASM instance
ORA-03113: end-of-file on communication channel
Process ID:
Session ID: 23 Serial number: 1161
Errors in file /soft/oracle/diag/rdbms/pt1db/pt1db2/trace/pt1db2_asmb_13828564.trc:
ORA-15064: communication failure with ASM instance
ORA-03113: end-of-file on communication channel
Process ID:
Session ID: 23 Serial number: 1161
ASMB (ospid: 13828564): terminating the instance due to error 15064
Termination issued to instance processes. Waiting for the processes to exit
Mon Jun 04 23:27:43 2012
Instance termination failed to kill one or more processes
Instance terminated by ASMB, pid = 13828564
错误的trace文件内容如下：
[pt1rac2@root]# cat /soft/oracle/diag/rdbms/pt1db/pt1db2/trace/pt1db2_asmb_13828564.trc
Trace file /soft/oracle/diag/rdbms/pt1db/pt1db2/trace/pt1db2_asmb_13828564.trcOracle Database 11g Enterprise Edition Release 11.2.0.2.0 – 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
ORACLE_HOME = /soft/oracle/product/11.2.0.2/dbh
System name: AIX
Node name: pt1rac2
Release: 1
Version: 6
Machine: 00F770E14C00
Instance name: pt1db2
Redo thread mounted by this instance: 0
Oracle process number: 27
Unix process pid: 13828564, image: oracle@pt1rac2 (ASMB)
*** 2012-06-04 16:19:14.483
*** SESSION ID:(2998.1) 2012-06-04 16:19:14.483
*** CLIENT ID:() 2012-06-04 16:19:14.483
*** SERVICE NAME:() 2012-06-04 16:19:14.483
*** MODULE NAME:() 2012-06-04 16:19:14.483
*** ACTION NAME:() 2012-06-04 16:19:14.483
NOTE: initiating MARK startup
*** 2012-06-04 16:19:18.290
Mode of disk 0 (DATA_0000) changing from 0x7 to 0x7
*** 2012-06-04 23:07:12.286
Mode of disk 0 (DATA_0000) changing from 0x7 to 0x7
*** 2012-06-04 23:27:32.492
NOTE: ASMB terminating
ORA-15064: communication failure with ASM instance
ORA-03113: end-of-file on communication channel
Process ID:
Session ID: 23 Serial number: 1161
error 15064 detected in background process
ORA-15064: communication failure with ASM instance
ORA-03113: end-of-file on communication channel
Process ID:
Session ID: 23 Serial number: 1161
kjzduptcctx: Notifying DIAG for crash event
—– Abridged Call Stack Trace —–
ksedsts()+644<-kjzdssdmp()+444<-kjzduptcctx()+272<-kjzdicrshnfy()+96<-ksuitm()+1284<-ksbrdp()+4508<-opirip()+1216<-opidrv()+1156<-sou2o()+192<-opimai_real()+308<-ssthrdmain()+340<-main()+216<-__start()+112
—– End of Abridged Call Stack Trace —–
*** 2012-06-04 23:27:32.854
ASMB (ospid: 13828564): terminating the instance due to error 15064
*** 2012-06-04 23:27:43.387
Instance termination failed to kill one or more processes
可知的确是由于ASM意外关闭才导致的instance关闭，ASM意外关闭的原因继续查。
准备生产一下awr报告看一下，结果发现由于数据关闭了，没有awr可看。
一下数据说明数据库被重启过几次，且经过与应用人员确认，这些重启的时间点他们都在做压力测试，因此初步怀疑是和性能有关，继续查。
SnapInstance DB Name Snap Id Snap Started Level
———— ———— ——— —————— —–
pt1db2 PT1DB 503 02 Jun 2012 16:00 1
504 02 Jun 2012 17:00 1
505 04 Jun 2012 09:43 1
506 04 Jun 2012 11:00 1
507 04 Jun 2012 12:00 1
508 04 Jun 2012 13:00 1
509 04 Jun 2012 14:00 1
510 04 Jun 2012 15:00 1
511 04 Jun 2012 16:00 1
512 04 Jun 2012 17:00 1
513 04 Jun 2012 18:00 1
514 04 Jun 2012 19:00 1
515 04 Jun 2012 20:00 1
516 04 Jun 2012 21:00 1
517 04 Jun 2012 22:00 1
518 04 Jun 2012 23:00 1
522 05 Jun 2012 01:20 1
523 05 Jun 2012 01:30 1
524 05 Jun 2012 01:45 1
先查看ocssd.log日志
2012-06-04 23:26:03.126: [ CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:26:03.126: [ CSSD][1029]clssgmAllocProc: (11328bcd0) allocated
2012-06-04 23:26:03.127: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 11328bcd0 – 0,1,2,3,4
2012-06-04 23:26:03.127: [ CSSD][1029]clssgmClientConnectMsg: Connect from con(2ea97f) proc(11328bcd0) pid(15335558) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:26:03.127: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:26:03.274: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:26:03.274: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:26:03.309: [ CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:26:03.309: [ CSSD][1029]clssgmAllocProc: (1132d30d0) allocated
2012-06-04 23:26:03.310: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1132d30d0 – 0,1,2,3,4
2012-06-04 23:26:03.310: [ CSSD][1029]clssgmClientConnectMsg: Connect from con(2ea9fa) proc(1132d30d0) pid(9306434/9306434) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:26:03.310: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:26:03.315: [ CSSD][1029]clssscSelect: cookie accept request 1132d30d0
2012-06-04 23:26:03.315: [ CSSD][1029]clssscevtypSHRCON: getting client with cmproc 1132d30d0
2012-06-04 23:26:03.316: [ CSSD][1029]clssgmRegisterClient: proc(57/1132d30d0), client(1/11313ef70)
2012-06-04 23:26:03.316: [ CSSD][1029]clssgmRegisterShared: grp DG_DATA, mbr 1, type 1
。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。
2012-06-04 23:26:29.658: [ CSSD][1029]clssgmDestroyProc: cleaning up proc(112f32d30) con(2ea049) skgpid ospid 35520704 with 0 clients, refcount 0
2012-06-04 23:26:29.658: [ CSSD][1029]clssgmDiscEndpcl: gipcDestroy 2ea049
2012-06-04 23:26:29.675: [ CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:26:29.675: [ CSSD][1029]clssgmAllocProc: (113242130) allocated
2012-06-04 23:26:29.676: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 113242130 – 0,1,2,3,4
2012-06-04 23:26:29.676: [ CSSD][1029]clssgmClientConnectMsg: Connect from con(2eadef) proc(113242130) pid(10879472) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:26:29.676: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:26:30.226: [ CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:26:30.226: [ CSSD][1029]clssgmAllocProc: (1132a3390) allocated
2012-06-04 23:26:30.227: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1132a3390 – 0,1,2,3,4
2012-06-04 23:26:30.227: [ CSSD][1029]clssgmClientConnectMsg: Connect from con(2eae64) proc(1132a3390) pid(35520706) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:26:30.228: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:26:30.542: [ CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:26:30.542: [ CSSD][1029]clssgmAllocProc: (1132ad030) allocated
2012-06-04 23:26:30.544: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1132ad030 – 0,1,2,3,4
2012-06-04 23:26:30.544: [ CSSD][1029]clssgmClientConnectMsg: Connect from con(2eaede) proc(1132ad030) pid(8585490/8585490) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:26:30.544: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:26:30.553: [ CSSD][1029]clssscSelect: cookie accept request 1132ad030
2012-06-04 23:26:30.553: [ CSSD][1029]clssscevtypSHRCON: getting client with cmproc 1132ad030
2012-06-04 23:26:30.553: [ CSSD][1029]clssgmRegisterClient: proc(63/1132ad030), client(1/1132ad370)
2012-06-04 23:26:30.554: [ CSSD][1029]clssgmRegisterShared: grp DG_DATA, mbr 1, type 1
2012-06-04 23:26:30.554: [ CSSD][1029]clssgmQueueShare: (1132ad510) target global grock DG_DATA member 1 type 1 queued from client (1132ad370), global grock DG_DATA, refcount 15
2012-06-04 23:26:30.554: [ CSSD][1029]clssgmRegisterShared: global grock DG_DATA member 1 share type 1, refcount 15
2012-06-04 23:26:30.557: [ CSSD][1029]clssscSelect: cookie accept request 1132ad030
2012-06-04 23:26:30.557: [ CSSD][1029]clssscevtypSHRCON: getting client with cmproc 1132ad030
2012-06-04 23:26:30.557: [ CSSD][1029]clssgmRegisterClient: proc(63/1132ad030), client(2/1132a3d90)
2012-06-04 23:26:30.558: [ CSSD][1029]clssgmRegisterShared: grp DB+ASM, mbr 1, type 1
2012-06-04 23:26:30.558: [ CSSD][1029]clssgmQueueShare: (1132ad910) target global grock DB+ASM member 1 type 1 queued from client (1132a3d90), global grock DB+ASM, refcount 17
2012-06-04 23:26:30.558: [ CSSD][1029]clssgmRegisterShared: global grock DB+ASM member 1 share type 1, refcount 17
2012-06-04 23:26:30.636: [ CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:26:30.637: [ CSSD][1029]clssgmAllocProc: (11333ba70) allocated
2012-06-04 23:26:30.638: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 11333ba70 – 0,1,2,3,4
2012-06-04 23:26:30.638: [ CSSD][1029]clssgmClientConnectMsg: Connect from con(2eaf6f) proc(11333ba70) pid(8913188/8913188) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:26:30.638: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:26:30.650: [ CSSD][1029]clssscSelect: cookie accept request 11333ba70
2012-06-04 23:26:30.650: [ CSSD][1029]clssscevtypSHRCON: getting client with cmproc 11333ba70
2012-06-04 23:26:30.650: [ CSSD][1029]clssgmRegisterClient: proc(64/11333ba70), client(1/112e8e550)
2012-06-04 23:26:30.652: [ CSSD][1029]clssgmRegisterShared: grp DG_DATA, mbr 1, type 1
2012-06-04 23:26:30.652: [ CSSD][1029]clssgmQueueShare: (11328ed90) target global grock DG_DATA member 1 type 1 queued from client (112e8e550), global grock DG_DATA, refcount 16
2012-06-04 23:26:30.652: [ CSSD][1029]clssgmRegisterShared: global grock DG_DATA member 1 share type 1, refcount 16
2012-06-04 23:26:30.766: [ CSSD][1029]clssscSelect: cookie accept request 11333ba70
2012-06-04 23:26:30.766: [ CSSD][1029]clssscevtypSHRCON: getting client with cmproc 11333ba70
2012-06-04 23:26:30.766: [ CSSD][1029]clssgmRegisterClient: proc(64/11333ba70), client(2/11328eff0)
2012-06-04 23:26:30.772: [ CSSD][1029]clssgmRegisterShared: grp DB+ASM, mbr 1, type 1
2012-06-04 23:26:30.772: [ CSSD][1029]clssgmQueueShare: (11328f190) target global grock DB+ASM member 1 type 1 queued from client (11328eff0), global grock DB+ASM, refcount 18
2012-06-04 23:26:30.773: [ CSSD][1029]clssgmRegisterShared: global grock DB+ASM member 1 share type 1, refcount 18
2012-06-04 23:26:31.382: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:26:31.382: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:26:34.394: [ CSSD][1029]clssscMonitorThreads clssnmvDiskPingThread not scheduled for 145718 msecs
2012-06-04 23:26:35.392: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:26:35.392: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:26:36.396: [ CSSD][1029]clssscMonitorThreads clssnmvWorkerThread not scheduled for 47678 msecs
2012-06-04 23:26:38.571: [ CSSD][2834]clssnmvDiskCheck: (/dev/rhdisk1) No I/O completed after 75% maximum time, 200000 ms, will be considered unusable in 49104 ms
2012-06-04 23:26:39.408: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:26:39.409: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:26:42.413: [ CSSD][1029]clssscMonitorThreads clssnmvDiskPingThread not scheduled for 153738 msecs
2012-06-04 23:26:43.413: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:26:43.413: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:26:44.414: [ CSSD][1029]clssscMonitorThreads clssnmvWorkerThread not scheduled for 55696 msecs
2012-06-04 23:26:47.421: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:26:47.421: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:26:50.433: [ CSSD][1029]clssscMonitorThreads clssnmvDiskPingThread not scheduled for 161758 msecs
2012-06-04 23:26:51.431: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:26:51.431: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:26:52.439: [ CSSD][1029]clssscMonitorThreads clssnmvWorkerThread not scheduled for 63721 msecs
2012-06-04 23:26:55.442: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:26:55.442: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:26:58.445: [ CSSD][1029]clssscMonitorThreads clssnmvDiskPingThread not scheduled for 169771 msecs
2012-06-04 23:26:59.455: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:26:59.455: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:27:00.493: [ CSSD][1029]clssscMonitorThreads clssnmvWorkerThread not scheduled for 71776 msecs
2012-06-04 23:27:03.470: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:27:03.470: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:27:06.473: [ CSSD][1029]clssscMonitorThreads clssnmvDiskPingThread not scheduled for 177799 msecs
2012-06-04 23:27:07.475: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:27:07.476: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:27:08.192: [ CSSD][1029]clssgmDeadProc: proc 112f35f50
2012-06-04 23:27:08.192: [ CSSD][1029]clssgmDestroyProc: cleaning up proc(112f35f50) con(2ea340) skgpid ospid 10551654 with 0 clients, refcount 0
2012-06-04 23:27:08.192: [ CSSD][1029]clssgmDiscEndpcl: gipcDestroy 2ea340
2012-06-04 23:27:08.192: [ CSSD][1029]clssscMonitorThreads clssnmvWorkerThread not scheduled for 79475 msecs
2012-06-04 23:27:08.498: [ CSSD][1029]clssgmDeadProc: proc 1131b4b10
2012-06-04 23:27:08.498: [ CSSD][1029]clssgmDestroyProc: cleaning up proc(1131b4b10) con(2ea3c4) skgpid ospid 10355074 with 0 clients, refcount 0
2012-06-04 23:27:08.498: [ CSSD][1029]clssgmDiscEndpcl: gipcDestroy 2ea3c4
2012-06-04 23:27:08.569: [ CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:27:08.569: [ CSSD][1029]clssgmAllocProc: (1131fe670) allocated
2012-06-04 23:27:08.571: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1131fe670 – 0,1,2,3,4
2012-06-04 23:27:08.571: [ CSSD][1029]clssgmClientConnectMsg: Connect from con(2eb1f8) proc(1131fe670) pid(10551656) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:27:08.571: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:27:08.637: [ CSSD][2834]clssnmvDiskCheck: (/dev/rhdisk1) No I/O completed after 90% maximum time, 200000 ms, will be considered unusable in 19038 ms
2012-06-04 23:27:08.809: [ CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:27:08.810: [ CSSD][1029]clssgmAllocProc: (1132593f0) allocated
2012-06-04 23:27:08.811: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1132593f0 – 0,1,2,3,4
2012-06-04 23:27:08.811: [ CSSD][1029]clssgmClientConnectMsg: Connect from con(2eb278) proc(1132593f0) pid(10355076) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:27:08.811: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:27:08.855: [ CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:27:08.855: [ CSSD][1029]clssgmAllocProc: (113259730) allocated
2012-06-04 23:27:08.857: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 113259730 – 0,1,2,3,4
2012-06-04 23:27:08.858: [ CSSD][1029]clssgmClientConnectMsg: Connect from con(2eb2ec) proc(113259730) pid(8192508/8192508) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:27:08.858: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:27:08.867: [ CSSD][1029]clssscSelect: cookie accept request 113259730
2012-06-04 23:27:08.867: [ CSSD][1029]clssscevtypSHRCON: getting client with cmproc 113259730
2012-06-04 23:27:08.867: [ CSSD][1029]clssgmRegisterClient: proc(65/113259730), client(1/112f35f50)
2012-06-04 23:27:08.868: [ CSSD][1029]clssgmRegisterShared: grp DG_DATA, mbr 1, type 1
2012-06-04 23:27:08.868: [ CSSD][1029]clssgmQueueShare: (11334b430) target global grock DG_DATA member 1 type 1 queued from client (112f35f50), global grock DG_DATA, refcount 17
2012-06-04 23:27:08.868: [ CSSD][1029]clssgmRegisterShared: global grock DG_DATA member 1 share type 1, refcount 17
2012-06-04 23:27:08.872: [ CSSD][1029]clssscSelect: cookie accept request 113259730
2012-06-04 23:27:08.872: [ CSSD][1029]clssscevtypSHRCON: getting client with cmproc 113259730
2012-06-04 23:27:08.872: [ CSSD][1029]clssgmRegisterClient: proc(65/113259730), client(2/113215ed0)
2012-06-04 23:27:08.874: [ CSSD][1029]clssgmRegisterShared: grp DB+ASM, mbr 1, type 1
2012-06-04 23:27:08.874: [ CSSD][1029]clssgmQueueShare: (11334b690) target global grock DB+ASM member 1 type 1 queued from client (113215ed0), global grock DB+ASM, refcount 19
2012-06-04 23:27:08.874: [ CSSD][1029]clssgmRegisterShared: global grock DB+ASM member 1 share type 1, refcount 19
2012-06-04 23:27:09.000: [ CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:27:09.000: [ CSSD][1029]clssgmAllocProc: (11334b8f0) allocated
2012-06-04 23:27:09.001: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 11334b8f0 – 0,1,2,3,4
2012-06-04 23:27:09.001: [ CSSD][1029]clssgmClientConnectMsg: Connect from con(2eb37d) proc(11334b8f0) pid(8061184/8061184) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:27:09.001: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:27:09.010: [ CSSD][1029]clssscSelect: cookie accept request 11334b8f0
2012-06-04 23:27:09.010: [ CSSD][1029]clssscevtypSHRCON: getting client with cmproc 11334b8f0
2012-06-04 23:27:09.010: [ CSSD][1029]clssgmRegisterClient: proc(66/11334b8f0), client(1/1131b5310)
2012-06-04 23:27:09.011: [ CSSD][1029]clssgmRegisterShared: grp DG_DATA, mbr 1, type 1
2012-06-04 23:27:09.011: [ CSSD][1029]clssgmQueueShare: (11334bc30) target global grock DG_DATA member 1 type 1 queued from client (1131b5310), global grock DG_DATA, refcount 18
2012-06-04 23:27:09.011: [ CSSD][1029]clssgmRegisterShared: global grock DG_DATA member 1 share type 1, refcount 18
2012-06-04 23:27:09.014: [ CSSD][1029]clssscSelect: cookie accept request 11334b8f0
2012-06-04 23:27:09.014: [ CSSD][1029]clssscevtypSHRCON: getting client with cmproc 11334b8f0
2012-06-04 23:27:09.014: [ CSSD][1029]clssgmRegisterClient: proc(66/11334b8f0), client(2/11334be90)
2012-06-04 23:27:09.015: [ CSSD][1029]clssgmRegisterShared: grp DB+ASM, mbr 1, type 1
2012-06-04 23:27:09.015: [ CSSD][1029]clssgmQueueShare: (113350230) target global grock DB+ASM member 1 type 1 queued from client (11334be90), global grock DB+ASM, refcount 20
2012-06-04 23:27:09.015: [ CSSD][1029]clssgmRegisterShared: global grock DB+ASM member 1 share type 1, refcount 20
2012-06-04 23:27:11.480: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:27:11.480: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:27:14.494: [ CSSD][1029]clssscMonitorThreads clssnmvDiskPingThread not scheduled for 185820 msecs
2012-06-04 23:27:15.500: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:27:15.500: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:27:16.504: [ CSSD][1029]clssscMonitorThreads clssnmvWorkerThread not scheduled for 87787 msecs
2012-06-04 23:27:19.519: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:27:19.519: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:27:20.285: [ CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:27:20.285: [ CSSD][1029]clssgmAllocProc: (1132b9290) allocated
2012-06-04 23:27:20.294: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1132b9290 – 0,1,2,3,4
2012-06-04 23:27:20.294: [ CSSD][1029]clssgmClientConnectMsg: Connect from con(2eb49d) proc(1132b9290) pid(8257826) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:27:20.294: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:27:20.299: [ CSSD][1029]clssnmGetNodeNumber: pt1rac2
2012-06-04 23:27:20.457: [ CSSD][1029]clssgmDeadProc: proc 1132b9290
2012-06-04 23:27:20.457: [ CSSD][1029]clssgmDestroyProc: cleaning up proc(1132b9290) con(2eb49d) skgpid ospid 8257826 with 0 clients, refcount 0
2012-06-04 23:27:20.457: [ CSSD][1029]clssgmDiscEndpcl: gipcDestroy 2eb49d
2012-06-04 23:27:20.999: [ CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:27:20.999: [ CSSD][1029]clssgmAllocProc: (1132b2650) allocated
2012-06-04 23:27:21.000: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1132b2650 – 0,1,2,3,4
2012-06-04 23:27:21.000: [ CSSD][1029]clssgmClientConnectMsg: Connect from con(2eb51a) proc(1132b2650) pid(7733742) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:27:21.000: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:27:21.005: [ CSSD][1029]clssnmGetNodeNumber: pt1rac1
2012-06-04 23:27:21.289: [ CSSD][1029]clssgmDeadProc: proc 1132b2650
2012-06-04 23:27:21.290: [ CSSD][1029]clssgmDestroyProc: cleaning up proc(1132b2650) con(2eb51a) skgpid ospid 7733742 with 0 clients, refcount 0
2012-06-04 23:27:21.290: [ CSSD][1029]clssgmDiscEndpcl: gipcDestroy 2eb51a
2012-06-04 23:27:22.001: [ CSSD][1029]clssscMonitorThreads clssnmvDiskPingThread not scheduled for 193327 msecs
2012-06-04 23:27:23.533: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:27:23.534: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:27:24.945: [ CSSD][1029]clssgmDeadProc: proc 113248030
2012-06-04 23:27:24.945: [ CSSD][1029]clssgmDestroyProc: cleaning up proc(113248030) con(2ea626) skgpid ospid 9699686 with 0 clients, refcount 0
2012-06-04 23:27:24.945: [ CSSD][1029]clssgmDiscEndpcl: gipcDestroy 2ea626
2012-06-04 23:27:25.142: [ CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:27:25.143: [ CSSD][1029]clssgmAllocProc: (1132b2650) allocated
2012-06-04 23:27:25.143: [ CSSD][1029]clssscMonitorThreads clssnmvWorkerThread not scheduled for 96425 msecs
2012-06-04 23:27:25.144: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1132b2650 – 1,2,3,4
2012-06-04 23:27:25.144: [ CSSD][1029]clssgmClientConnectMsg: Connect from con(2eb5c3) proc(1132b2650) pid(9699688) version 11:2:1:4, properties: 1,2,3,4
2012-06-04 23:27:25.144: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:27:25.323: [ CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:27:25.324: [ CSSD][1029]clssgmAllocProc: (1132b7e50) allocated
2012-06-04 23:27:25.326: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1132b7e50 – 0,1,2,3,4
2012-06-04 23:27:25.326: [ CSSD][1029]clssgmClientConnectMsg: Connect from con(2eb63a) proc(1132b7e50) pid(7733746/7733746) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:27:25.326: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:27:25.485: [ CSSD][1029]clssscSelect: cookie accept request 1132b7e50
2012-06-04 23:27:25.485: [ CSSD][1029]clssscevtypSHRCON: getting client with cmproc 1132b7e50
2012-06-04 23:27:25.485: [ CSSD][1029]clssgmRegisterClient: proc(67/1132b7e50), client(1/113153d90)
2012-06-04 23:27:25.487: [ CSSD][1029]clssgmRegisterShared: grp DG_DATA, mbr 1, type 1
2012-06-04 23:27:25.487: [ CSSD][1029]clssgmQueueShare: (113338450) target global grock DG_DATA member 1 type 1 queued from client (113153d90), global grock DG_DATA, refcount 19
2012-06-04 23:27:25.487: [ CSSD][1029]clssgmRegisterShared: global grock DG_DATA member 1 share type 1, refcount 19
2012-06-04 23:27:25.495: [ CSSD][1029]clssscSelect: cookie accept request 1132b7e50
2012-06-04 23:27:25.495: [ CSSD][1029]clssscevtypSHRCON: getting client with cmproc 1132b7e50
2012-06-04 23:27:25.495: [ CSSD][1029]clssgmRegisterClient: proc(67/1132b7e50), client(2/1132b42b0)
2012-06-04 23:27:25.497: [ CSSD][1029]clssgmRegisterShared: grp DB+ASM, mbr 1, type 1
2012-06-04 23:27:25.497: [ CSSD][1029]clssgmQueueShare: (11334c0f0) target global grock DB+ASM member 1 type 1 queued from client (1132b42b0), global grock DB+ASM, refcount 21
2012-06-04 23:27:25.497: [ CSSD][1029]clssgmRegisterShared: global grock DB+ASM member 1 share type 1, refcount 21
2012-06-04 23:27:27.542: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:27:27.542: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:27:27.680: [ CSSD][2834](:CSSNM00058:)clssnmvDiskCheck: No I/O completions for 200006 ms for voting file /dev/rhdisk1)
2012-06-04 23:27:27.693: [ CSSD][2834]clssnmvDiskAvailabilityChange: voting file /dev/rhdisk1 now offline
2012-06-04 23:27:27.693: [ CSSD][2834](:CSSNM00018:)clssnmvDiskCheck: Aborting, 0 of 1 configured voting disks available, need 1
2012-06-04 23:27:27.694: [ CSSD][2834]###################################
2012-06-04 23:27:27.694: [ CSSD][2834]clssscExit: CSSD aborting from thread clssnmvDiskPingMonitorThread
2012-06-04 23:27:27.694: [ CSSD][2834]###################################
2012-06-04 23:27:27.694: [ CSSD][2834](:CSSSC00012:)clssscExit: A fatal error occurred and the CSS daemon is terminating abnormally
2012-06-04 23:27:27.695: [ CSSD][2834]
—– Call Stack Trace —–
2012-06-04 23:27:27.723: [ CSSD][2834]calling call entryargument values in hex
2012-06-04 23:27:27.723: [ CSSD][2834]location type point(? means dubious value)
2012-06-04 23:27:27.723: [ CSSD][2834]——————– ——– ——————– —————————-
2012-06-04 23:27:27.724: [ CSSD][2834]clssscExit()+692 bl 100067114 67800000678 ? 1000EB40C ?
2012-06-04 23:27:27.724: [ CSSD][2834] 110091190 ? 900000000BA4D14 ?
2012-06-04 23:27:27.724: [ CSSD][2834] 000000000 ? 013960031 ?
2012-06-04 23:27:27.724: [ CSSD][2834] 013960031 ? 000000000 ?
2012-06-04 23:27:27.724: [ CSSD][2834]clssnmvDiskCheck()+ call clssscExit() 111E38B90 ? 200000002 ?
2012-06-04 23:27:27.724: [ CSSD][2834]2576 000000001 ? 1000F52A0 ?
2012-06-04 23:27:27.724: [ CSSD][2834] 110091190 ? 013960031 ?
2012-06-04 23:27:27.724: [ CSSD][2834] 013960031 ? 000000000 ?
2012-06-04 23:27:27.724: [ CSSD][2834]clssnmvDiskPingMoni call clssnmvDiskCheck() 111E38B90 ? 110BF7FB0 ?
2012-06-04 23:27:27.724: [ CSSD][2834]torThread()+392 112468680 ? 000000000 ?
2012-06-04 23:27:27.724: [ CSSD][2834] 000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [ CSSD][2834] 013960031 ? 000000000 ?
2012-06-04 23:27:27.725: [ CSSD][2834]clssscthrdmain()+20 call clssnmvDiskPingMoni 111E38B90 ? 000000000 ?
2012-06-04 23:27:27.725: [ CSSD][2834]8 torThread() 000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [ CSSD][2834] 000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [ CSSD][2834] 000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [ CSSD][2834]_pthread_body()+240 call clssscthrdmain() 111E38B90 ? 000000000 ?
2012-06-04 23:27:27.725: [ CSSD][2834] 000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [ CSSD][2834] 000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [ CSSD][2834] 000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [ CSSD][2834]fffffffffffffffc call _pthread_body() 000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [ CSSD][2834] 000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [ CSSD][2834] 000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [ CSSD][2834] 000000000 ? 000000000 ?
2012-06-04 23:27:27.726: [ CSSD][2834]
2012-06-04 23:27:27.726: [ CSSD][2834]——————— Binary Stack Dump ———————
2012-06-04 23:27:27.726: [ CSSD][2834]
2012-06-04 23:27:27.726: [ CSSD][2834]========== FRAME [1] (clssscExit()+692 -> 100067114) ==========
2012-06-04 23:27:27.726: [ CSSD][2834]defined by frame pointers 0x1124683f0 and 0x112468260
2012-06-04 23:27:27.726: [ CSSD][2834]CALL TYPE: bl ERROR SIGNALED: no CALLER: clssscExit
2012-06-04 23:27:27.726: [ CSSD][2834]
Dump of memory from 0x112468260 to 0x1124683f0
2012-06-04 23:27:27.726: [ CSSD][2834]112468260 00000001 124683F0 44000224 000003D7 […..F..D..$….]
2012-06-04 23:27:27.726: [ CSSD][2834]112468270 00000001 000031D8 00000000 00000083 [……1………]
2012-06-04 23:27:27.726: [ CSSD][2834]112468280 00000000 00000029 00000001 100182B0 […….)……..]
2012-06-04 23:27:27.726: [ CSSD][2834]112468290 00000678 00000678 00000001 000EB40C […x…x……..]
2012-06-04 23:27:27.726: [ CSSD][2834]1124682A0 00000001 10091190 09000000 00BA4D14 […………..M.]
2012-06-04 23:27:27.726: [ CSSD][2834]1124682B0 00000000 00000000 00000000 13960031 [……………1]
2012-06-04 23:27:27.727: [ CSSD][2834]1124682C0 00000000 13960031 00000000 00000000 […….1……..]
2012-06-04 23:27:27.727: [ CSSD][2834]1124682D0 00000001 1008FF50 00000001 1002BD70 […….P…….p]
2012-06-04 23:27:27.727: [ CSSD][2834]1124682E0 00000000 00000001 43525300 12468368 [……..CRS..F.h]
2012-06-04 23:27:27.727: [ CSSD][2834]1124682F0 00000001 00000000 00000001 100181F0 […………….]
2012-06-04 23:27:27.727: [ CSSD][2834]112468300 00000001 10018208 00000001 100181F0 […………….]
2012-06-04 23:27:27.727: [ CSSD][2834]112468310 00000001 10018208 00000001 11E38B90 […………….]
2012-06-04 23:27:27.727: [ CSSD][2834]112468320 00000001 11E38B90 00000000 00000000 […………….]
2012-06-04 23:27:27.727: [ CSSD][2834]112468330 00000000 00000000 00000000 00000000 […………….]
2012-06-04 23:27:27.728: [ CSSD][2834] Repeat 3 times
2012-06-04 23:27:27.728: [ CSSD][2834]112468370 00000000 00000000 00000000 13960031 [……………1]
2012-06-04 23:27:27.728: [ CSSD][2834]112468380 09000000 00C27160 00000000 00000000 [……q`……..]
2012-06-04 23:27:27.728: [ CSSD][2834]112468390 00000001 1002BD70 00000001 117B6590 […….p…..{e.]
2012-06-04 23:27:27.728: [ CSSD][2834]1124683A0 00000001 00000001 00000000 00000000 […………….]
2012-06-04 23:27:27.728: [ CSSD][2834]1124683B0 00000000 00000000 00000646 00000646 [………..F…F]
2012-06-04 23:27:27.728: [ CSSD][2834]1124683C0 00000001 12468428 00000001 117C6C84 […..F.(…..|l.]
2012-06-04 23:27:27.728: [ CSSD][2834]1124683D0 00000001 117C6C84 00000001 1008FF50 […..|l……..P]
2012-06-04 23:27:27.728: [ CSSD][2834]1124683E0 00000000 00000000 00000001 000F24E8 […………..$.]
2012-06-04 23:27:27.728: [ CSSD][2834]========== FRAME [2] (clssnmvDiskCheck()+2576 -> clssscExit()) ==========
2012-06-04 23:27:27.728: [ CSSD][2834]defined by frame pointers 0x112468600 and 0x1124683f0
2012-06-04 23:27:27.729: [ CSSD][2834]CALL TYPE: call ERROR SIGNALED: no CALLER: clssnmvDiskCheck
2012-06-04 23:27:27.729: [ CSSD][2834]
Dump of memory from 0x1124683f0 to 0x112468600
2012-06-04 23:27:27.729: [ CSSD][2834]1124683F0 00000001 12468600 09001000 A0688640 […..F…….h.@]
2012-06-04 23:27:27.729: [ CSSD][2834]112468400 00000001 0002CD94 4400B22B 00000000 [……..D..+….]
2012-06-04 23:27:27.729: [ CSSD][2834]112468410 00000001 001058B8 00000001 100182B0 [……X………]
2012-06-04 23:27:27.729: [ CSSD][2834]112468420 00000001 11E38B90 00000002 00000002 […………….]
2012-06-04 23:27:27.729: [ CSSD][2834]112468430 00000000 00000001 00000001 000F52A0 […………..R.]
2012-06-04 23:27:27.729: [ CSSD][2834]112468440 00000001 10091190 00000000 13960031 [……………1]
2012-06-04 23:27:27.729: [ CSSD][2834]112468450 00000000 13960031 00000000 00000000 […….1……..]
2012-06-04 23:27:27.730: [ CSSD][2834]112468460 00000001 1008FF50 00000001 118508F0 […….P……..]
2012-06-04 23:27:27.730: [ CSSD][2834]112468470 00000001 117B6590 8C91440C 00030D46 […..{e…D….F]
2012-06-04 23:27:27.730: [ CSSD][2834]112468480 00000001 12C508B0 00000000 00030D40 [……………@]
2012-06-04 23:27:27.730: [ CSSD][2834]112468490 00010001 9DD81012 D0E54F97 BFEE0F2C [……….O….,]
2012-06-04 23:27:27.730: [ CSSD][2834]1124684A0 AF8AA88C 00000000 2200088B 00000000 [……..”…….]
2012-06-04 23:27:27.730: [ CSSD][2834]1124684B0 09000000 014F0814 09001000 A0688640 […..O…….h.@]
2012-06-04 23:27:27.730: [ CSSD][2834]1124684C0 00030D40 1045F530 00000004 00000004 […@.E.0……..]
2012-06-04 23:27:27.730: [ CSSD][2834]1124684D0 00000001 00105DE8 00000001 000F2BD4 [……]…….+.]
2012-06-04 23:27:27.730: [ CSSD][2834]1124684E0 00000001 000F2BD4 00030D40 000F24E8 [……+….@..$.]
2012-06-04 23:27:27.731: [ CSSD][2834]1124684F0 00000001 00000001 2200022B 00000000 [……..”..+….]
2012-06-04 23:27:27.731: [ CSSD][2834]112468500 00000000 00000000 FFFFFFF5 000F24E8 […………..$.]
2012-06-04 23:27:27.731: [ CSSD][2834]112468510 00000001 12468600 09001000 A0688640 […..F…….h.@]
2012-06-04 23:27:27.731: [ CSSD][2834]112468520 00000001 00070DFC 00000001 117C6C84 [………….|l.]
2012-06-04 23:27:27.731: [ CSSD][2834]112468530 00000001 117C6C84 00000001 100182B0 […..|l………]
2012-06-04 23:27:27.731: [ CSSD][2834]112468540 00000001 1045F530 00000004 00000004 […..E.0……..]
2012-06-04 23:27:27.731: [ CSSD][2834]112468550 00000001 00105DE8 00000000 00000009 [……]………]
2012-06-04 23:27:27.731: [ CSSD][2834]112468560 00000000 000003E8 00000000 000003E8 […………….]
2012-06-04 23:27:27.731: [ CSSD][2834]112468570 00000000 13960031 00000000 00000040 […….1…….@]
2012-06-04 23:27:27.732: [ CSSD][2834]112468580 00000001 1008FF50 00000001 117B6590 […….P…..{e.]
2012-06-04 23:27:27.732: [ CSSD][2834]112468590 00000001 1002BD70 00000001 117B65A0 […….p…..{e.]
2012-06-04 23:27:27.732: [ CSSD][2834]1124685A0 00000001 117B66A0 FFFFFFB2 000003D4 […..{f………]
2012-06-04 23:27:27.732: [ CSSD][2834]1124685B0 00000001 117C6C84 00000001 117C6C84 […..|l……|l.]
2012-06-04 23:27:27.732: [ CSSD][2834] Repeat 1 times
2012-06-04 23:27:27.732: [ CSSD][2834]1124685D0 00000001 000F2BD4 00000001 000F2BD4 [……+…….+.]
2012-06-04 23:27:27.732: [ CSSD][2834]1124685E0 00000000 00000000 00000001 12C508B0 […………….]
2012-06-04 23:27:27.732: [ CSSD][2834]1124685F0 00000000 00000000 00000001 000F24E8 […………..$.]
2012-06-04 23:27:27.732: [ CSSD][2834]========== FRAME [3] (clssnmvDiskPingMonitorThread()+392 -> clssnmvDiskCheck()) ==========
2012-06-04 23:27:27.732: [ CSSD][2834]defined by frame pointers 0x1124686f0 and 0x112468600
2012-06-04 23:27:27.732: [ CSSD][2834]CALL TYPE: call ERROR SIGNALED: no CALLER: clssnmvDiskPingMonitorThread
2012-06-04 23:27:27.732: [ CSSD][2834]
Dump of memory from 0x112468600 to 0x1124686f0
2012-06-04 23:27:27.733: [ CSSD][2834]112468600 00000001 124686F0 00000000 00000000 […..F……….]
2012-06-04 23:27:27.733: [ CSSD][2834]112468610 00000001 0002CFAC 09001000 A0688640 [………….h.@]
2012-06-04 23:27:27.733: [ CSSD][2834]112468620 00000000 00000000 00000001 100182B0 […………….]
2012-06-04 23:27:27.733: [ CSSD][2834]112468630 00000001 11E38B90 00000001 10BF7FB0 […………….]
2012-06-04 23:27:27.733: [ CSSD][2834]112468640 00000001 12468680 00000000 00000000 […..F……….]
2012-06-04 23:27:27.733: [ CSSD][2834]112468650 00000000 00000000 00000000 00000000 […………….]
2012-06-04 23:27:27.733: [ CSSD][2834]112468660 00000000 13960031 00000000 00000000 […….1……..]
2012-06-04 23:27:27.733: [ CSSD][2834]112468670 00000001 10BF7FB0 00000001 117B6590 [………….{e.]
2012-06-04 23:27:27.733: [ CSSD][2834]112468680 00030D40 8C914024 000003E8 100182B0 […@..@$……..]
2012-06-04 23:27:27.733: [ CSSD][2834]112468690 09000000 014F0814 00000000 00000000 […..O……….]
2012-06-04 23:27:27.734: [ CSSD][2834]1124686A0 00000000 00000000 00000000 00000001 […………….]
2012-06-04 23:27:27.734: [ CSSD][2834]1124686B0 000003E8 00000000 00000000 00000000 […………….]
2012-06-04 23:27:27.734: [ CSSD][2834]1124686C0 00000000 00000000 000003E8 000003E8 […………….]
2012-06-04 23:27:27.734: [ CSSD][2834]1124686D0 000003E8 1008FF50 00000001 1002BD70 […….P…….p]
2012-06-04 23:27:27.734: [ CSSD][2834]1124686E0 00000001 10BF97D0 00000001 12468A78 [………….F.x]
2012-06-04 23:27:27.734: [ CSSD][2834]========== FRAME [4] (clssscthrdmain()+208 -> clssnmvDiskPingMonitorThread()) ==========
2012-06-04 23:27:27.734: [ CSSD][2834]defined by frame pointers 0x112468790 and 0x1124686f0
2012-06-04 23:27:27.734: [ CSSD][2834]CALL TYPE: call ERROR SIGNALED: no CALLER: clssscthrdmain
2012-06-04 23:27:27.734: [ CSSD][2834]
Dump of memory from 0x1124686f0 to 0x112468790
2012-06-04 23:27:27.734: [ CSSD][2834]1124686F0 00000001 12468790 00000000 00000000 […..F……….]
2012-06-04 23:27:27.734: [ CSSD][2834]112468700 00000001 0000A294 00000000 00000000 […………….]
2012-06-04 23:27:27.735: [ CSSD][2834]112468710 00000000 00000000 00000001 100182B0 […………….]
2012-06-04 23:27:27.735: [ CSSD][2834]112468720 00000001 11E38B90 00000000 00000000 […………….]
2012-06-04 23:27:27.735: [ CSSD][2834]112468730 00000000 00000000 00000000 00000000 […………….]
2012-06-04 23:27:27.735: [ CSSD][2834] Repeat 2 times
2012-06-04 23:27:27.735: [ CSSD][2834]112468760 00000001 1008FF50 00000001 1002BD70 […….P…….p]
2012-06-04 23:27:27.735: [ CSSD][2834]112468770 00000000 00000000 00000000 00000000 […………….]
2012-06-04 23:27:27.735: [ CSSD][2834]112468780 00000000 00000000 00000001 1008FF50 [……………P]
2012-06-04 23:27:27.735: [ CSSD][2834]========== FRAME [5] (_pthread_body()+240 -> clssscthrdmain()) ==========
2012-06-04 23:27:27.735: [ CSSD][2834]defined by frame pointers 0x112468810 and 0x112468790
2012-06-04 23:27:27.735: [ CSSD][2834]CALL TYPE: call ERROR SIGNALED: no CALLER: _pthread_body
2012-06-04 23:27:27.735: [ CSSD][2834]
Dump of memory from 0x112468790 to 0x112468890
2012-06-04 23:27:27.736: [ CSSD][2834]112468790 00000001 12468810 00000000 00000000 […..F……….]
2012-06-04 23:27:27.736: [ CSSD][2834]1124687A0 09000000 00B7BD54 00000000 00000000 […….T……..]
2012-06-04 23:27:27.736: [ CSSD][2834]1124687B0 00000000 00000000 09001000 A0288350 [………….(.P]
2012-06-04 23:27:27.736: [ CSSD][2834]1124687C0 00000001 11E38B90 00000000 00000000 […………….]
2012-06-04 23:27:27.736: [ CSSD][2834]1124687D0 00000000 00000000 00000000 00000000 […………….]
2012-06-04 23:27:27.737: [ CSSD][2834] Repeat 11 times
2012-06-04 23:27:27.737: [ CSSD][2834]========== FRAME [6] (fffffffffffffffc -> _pthread_body()) ==========
2012-06-04 23:27:27.737: [ CSSD][2834]defined by frame pointers 0x0 and 0x112468810
2012-06-04 23:27:27.737: [ CSSD][2834]CALL TYPE: call ERROR SIGNALED: no CALLER: fffffffffffffffc
2012-06-04 23:27:27.737: [ CSSD][2834]
—– Argument/Register Address Dump —–
2012-06-04 23:27:27.737: [ CSSD][2834]Argument/Register addr=0x1000eb40c.
Dump of memory from 0x1000eb3cc to 0x1000eb50c
2012-06-04 23:27:27.737: [ CSSD][2834]1000EB3C0 6C206572 [l er]
2012-06-04 23:27:27.737: [ CSSD][2834]1000EB3D0 726F7220 6F636375 72726564 20616E64 [ror occurred and]
2012-06-04 23:27:27.737: [ CSSD][2834]1000EB3E0 20746865 20435353 20646165 6D6F6E20 [ the CSS daemon ]
2012-06-04 23:27:27.737: [ CSSD][2834]1000EB3F0 69732074 65726D69 6E617469 6E672061 [is terminating a]
2012-06-04 23:27:27.737: [ CSSD][2834]1000EB400 626E6F72 6D616C6C 79004942 283A4353 [bnormally.IB(:CS]
2012-06-04 23:27:27.737: [ CSSD][2834]1000EB410 53534330 30303132 3A290049 636C7373 [SSC00012:).Iclss]
2012-06-04 23:27:27.738: [ CSSD][2834]1000EB420 73634578 69743A20 53746172 74696E67 [scExit: Starting]
2012-06-04 23:27:27.738: [ CSSD][2834]1000EB430 20435253 4420636C 65616E75 70004942 [ CRSD cleanup.IB]
2012-06-04 23:27:27.738: [ CSSD][2834]1000EB440 636C7373 73634578 69743A20 43525344 [clssscExit: CRSD]
2012-06-04 23:27:27.738: [ CSSD][2834]1000EB450 20636C65 616E7570 20737461 74757320 [ cleanup status ]
2012-06-04 23:27:27.738: [ CSSD][2834]1000EB460 25640049 636C7373 73634578 69743A20 [%d.IclssscExit: ]
2012-06-04 23:27:27.738: [ CSSD][2834]1000EB470 43525344 20636C65 616E7570 20666169 [CRSD cleanup fai]
2012-06-04 23:27:27.738: [ CSSD][2834]1000EB480 6C656420 77697468 20256400 636C7373 [led with %d.clss]
2012-06-04 23:27:27.738: [ CSSD][2834]1000EB490 73634578 69743A20 43525344 20636C65 [scExit: CRSD cle]
2012-06-04 23:27:27.738: [ CSSD][2834]1000EB4A0 616E7570 20737563 63657373 66756C6C [anup successfull]
2012-06-04 23:27:27.739: [ CSSD][2834]1000EB4B0 7920636F 6D706C65 74656400 636C7373 [y completed.clss]
2012-06-04 23:27:27.739: [ CSSD][2834]1000EB4C0 73634578 69743A20 61626F72 7420616C [scExit: abort al]
2012-06-04 23:27:27.739: [ CSSD][2834]1000EB4D0 72656164 79207365 74202564 0049424D [ready set %d.IBM]
2012-06-04 23:27:27.739: [ CSSD][2834]1000EB4E0 41535345 52542025 73202564 0049424D [ASSERT %s %d.IBM]
2012-06-04 23:27:27.739: [ CSSD][2834]1000EB4F0 636C7373 73632E63 0049424D 283A4353 [clsssc.c.IBM(:CS]
2012-06-04 23:27:27.739: [ CSSD][2834]1000EB500 53534330 30303133 3A29636C [SSC00013:)cl]
2012-06-04 23:27:27.739: [ CSSD][2834]Argument/Register addr=0x110091190.
Dump of memory from 0x110091150 to 0x110091290
2012-06-04 23:27:27.739: [ CSSD][2834]110091150 00000000 00000000 00000000 00000000 […………….]
2012-06-04 23:27:27.739: [ CSSD][2834] Repeat 2 times
2012-06-04 23:27:27.739: [ CSSD][2834]110091180 00000000 00000000 00000001 00000001 […………….]
2012-06-04 23:27:27.740: [ CSSD][2834]110091190 2F736F66 742F6772 69642F70 726F6475 [/soft/grid/produ]
2012-06-04 23:27:27.740: [ CSSD][2834]1100911A0 63742F31 312E322E 302E322F 6769682F [ct/11.2.0.2/gih/]
2012-06-04 23:27:27.740: [ CSSD][2834]1100911B0 6C6F672F 70743172 6163322F 63737364 [log/pt1rac2/cssd]
2012-06-04 23:27:27.740: [ CSSD][2834]1100911C0 2F6F6373 73642E6C 6F670000 00000000 [/ocssd.log……]
2012-06-04 23:27:27.740: [ CSSD][2834]1100911D0 00000000 00000000 00000000 00000000 […………….]
2012-06-04 23:27:27.741: [ CSSD][2834] Repeat 11 times
2012-06-04 23:27:27.741: [ CSSD][2834]Argument/Register addr=0x900000000ba4d14.
Dump of memory from 0x900000000ba4cd4 to 0x900000000ba4e14
2012-06-04 23:27:27.741: [ CSSD][2834]900000000BA4CD0 38600000 4BFD5608 00000000 [8`..K.V…..]
2012-06-04 23:27:27.741: [ CSSD][2834]900000000BA4CE0 000C0000 00000000 00000000 00000000 […………….]
2012-06-04 23:27:27.741: [ CSSD][2834]900000000BA4CF0 00000000 00000000 00000000 00000000 […………….]
2012-06-04 23:27:27.741: [ CSSD][2834]900000000BA4D00 348D8278 41810008 38800000 E8630000 [4..xA…8….c..]
2012-06-04 23:27:27.741: [ CSSD][2834]900000000BA4D10 38A00008 81840230 7C0004AC 398CFFFF [8……0|…9…]
2012-06-04 23:27:27.741: [ CSSD][2834]900000000BA4D20 E9040110 38E00000 65081000 7D251828 [….8…e…}%.(]
2012-06-04 23:27:27.741: [ CSSD][2834]900000000BA4D30 7C894000 40860014 7CE5192D 40A2FFF0 [|.@.@…|..-@…]
2012-06-04 23:27:27.742: [ CSSD][2834]900000000BA4D40 91840230 4E800020 7C632A14 4BFD4694 […0N.. |c*.K.F.]
2012-06-04 23:27:27.742: [ CSSD][2834]900000000BA4D50 00000000 000C0000 00000000 00000000 […………….]
2012-06-04 23:27:27.742: [ CSSD][2834]900000000BA4D60 00000000 00000000 00000000 00000000 […………….]
2012-06-04 23:27:27.742: [ CSSD][2834] Repeat 1 times
2012-06-04 23:27:27.742: [ CSSD][2834]900000000BA4D80 7D251828 64871000 2C890000 40860014 [}%.(d…,…@…]
2012-06-04 23:27:27.742: [ CSSD][2834]900000000BA4D90 7CE5192D 4C00012C 4D820020 4BFFFFE4 [|..-L..,M.. K…]
2012-06-04 23:27:27.742: [ CSSD][2834]900000000BA4DA0 7C862378 7C851A14 7CC53378 4BFD5534 [|.#x|…|.3xK.U4]
2012-06-04 23:27:27.742: [ CSSD][2834]900000000BA4DB0 00000000 000C0000 00000000 00000000 […………….]
2012-06-04 23:27:27.742: [ CSSD][2834]900000000BA4DC0 00000000 00000000 00000000 00000000 […………….]
2012-06-04 23:27:27.743: [ CSSD][2834] Repeat 3 times
2012-06-04 23:27:27.743: [ CSSD][2834]900000000BA4E00 7D251828 64871000 2C890000 40860014 [}%.(d…,…@…]
2012-06-04 23:27:27.743: [ CSSD][2834]900000000BA4E10 7CE5192D [|..-]
2012-06-04 23:27:27.743: [ CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.743: [ CSSD][2834]000000000 00000000 00000000 00000000 00000000 […………….]
2012-06-04 23:27:27.744: [ CSSD][2834] Repeat 15 times
2012-06-04 23:27:27.744: [ CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.744: [ CSSD][2834]000000000 00000000 00000000 00000000 00000000 […………….]
2012-06-04 23:27:27.745: [ CSSD][2834] Repeat 15 times
2012-06-04 23:27:27.745: [ CSSD][2834]Argument/Register addr=0x111e38b90.
Dump of memory from 0x111e38b50 to 0x111e38c90
2012-06-04 23:27:27.745: [ CSSD][2834]111E38B50 00000001 00000001 8C8FC179 000003E8 [………..y….]
2012-06-04 23:27:27.745: [ CSSD][2834]111E38B60 8C914038 8C913A22 0000000B 00000000 [..@8..:”……..]
2012-06-04 23:27:27.745: [ CSSD][2834]111E38B70 00000000 00000000 00000000 00000000 […………….]
2012-06-04 23:27:27.745: [ CSSD][2834]111E38B80 5B5B5B5B 00000000 00000000 000001E0 [[[[[…………]
2012-06-04 23:27:27.745: [ CSSD][2834]111E38B90 00000001 1008FF50 00000001 10BF7FB0 […….P……..]
2012-06-04 23:27:27.746: [ CSSD][2834]111E38BA0 00000001 11E3A320 00000001 11E389A0 [……. ……..]
2012-06-04 23:27:27.746: [ CSSD][2834]111E38BB0 00000B12 00000B12 636C7373 6E6D7644 [……..clssnmvD]
2012-06-04 23:27:27.746: [ CSSD][2834]111E38BC0 69736B50 696E674D 6F6E6974 6F725468 [iskPingMonitorTh]
2012-06-04 23:27:27.746: [ CSSD][2834]111E38BD0 72656164 00000000 00000000 00000000 [read…………]
2012-06-04 23:27:27.746: [ CSSD][2834]111E38BE0 00000000 00000000 00000000 00000000 […………….]
2012-06-04 23:27:27.746: [ CSSD][2834]111E38BF0 00000000 00000000 00020000 00000000 […………….]
2012-06-04 23:27:27.746: [ CSSD][2834]111E38C00 00000001 1045F530 00000001 11E38B90 […..E.0……..]
2012-06-04 23:27:27.746: [ CSSD][2834]111E38C10 00000001 100179B0 00000000 00000000 [……y………]
2012-06-04 23:27:27.746: [ CSSD][2834]111E38C20 00000000 00000000 00000000 00000000 […………….]
2012-06-04 23:27:27.747: [ CSSD][2834] Repeat 4 times
2012-06-04 23:27:27.747: [ CSSD][2834]111E38C70 00000000 00000001 00000000 00000005 […………….]
2012-06-04 23:27:27.747: [ CSSD][2834]111E38C80 00000000 00001677 00000000 00000000 […….w……..]
2012-06-04 23:27:27.747: [ CSSD][2834]Argument/Register addr=0x1.
Dump of memory from 0x0 to 0x101
2012-06-04 23:27:27.747: [ CSSD][2834]000000000 00000000 00000000 00000000 00000000 […………….]
2012-06-04 23:27:27.748: [ CSSD][2834] Repeat 15 times
2012-06-04 23:27:27.748: [ CSSD][2834]000000100 7C7243A6 [|rC.]
2012-06-04 23:27:27.748: [ CSSD][2834]Argument/Register addr=0x1000f52a0.
Dump of memory from 0x1000f5260 to 0x1000f53a0
2012-06-04 23:27:27.748: [ CSSD][2834]1000F5260 6B436865 636B3A20 41626F72 74696E67 [kCheck: Aborting]
2012-06-04 23:27:27.748: [ CSSD][2834]1000F5270 2C202564 206F6620 25642025 7320766F [, %d of %d %s vo]
2012-06-04 23:27:27.748: [ CSSD][2834]1000F5280 74696E67 20646973 6B732061 7661696C [ting disks avail]
2012-06-04 23:27:27.749: [ CSSD][2834]1000F5290 61626C65 2C206E65 65642025 75004942 [able, need %u.IB]
2012-06-04 23:27:27.749: [ CSSD][2834]1000F52A0 283A4353 534E4D30 30303138 3A290049 [(:CSSNM00018:).I]
2012-06-04 23:27:27.749: [ CSSD][2834]1000F52B0 636C7373 6E6D7644 69736B50 696E674D [clssnmvDiskPingM]
2012-06-04 23:27:27.749: [ CSSD][2834]1000F52C0 6F6E6974 6F725468 72656164 3A20206E [onitorThread: n]
2012-06-04 23:27:27.749: [ CSSD][2834]1000F52D0 6F20766F 74696E67 20646973 6B2C0049 [o voting disk,.I]
2012-06-04 23:27:27.749: [ CSSD][2834]1000F52E0 636C7373 6E6D7644 69736B50 4D543A20 [clssnmvDiskPMT: ]
2012-06-04 23:27:27.749: [ CSSD][2834]1000F52F0 736C6565 70696E67 20666F72 20256420 [sleeping for %d ]
2012-06-04 23:27:27.749: [ CSSD][2834]1000F5300 6D730049 636C7373 6E6D7653 63616E43 [ms.IclssnmvScanC]
2012-06-04 23:27:27.749: [ CSSD][2834]1000F5310 6F6D706C 6574696F 6E733A20 4661696C [ompletions: Fail]
2012-06-04 23:27:27.749: [ CSSD][2834]1000F5320 75726520 2564206E 6F746966 79696E67 [ure %d notifying]
2012-06-04 23:27:27.750: [ CSSD][2834]1000F5330 20636C69 656E7473 206F6620 776F726B [ clients of work]
2012-06-04 23:27:27.750: [ CSSD][2834]1000F5340 20636F6D 706C6574 696F6E2E 0049424D [ completion..IBM]
2012-06-04 23:27:27.750: [ CSSD][2834]1000F5350 636C7373 6E6D7653 63616E43 6F6D706C [clssnmvScanCompl]
2012-06-04 23:27:27.750: [ CSSD][2834]1000F5360 6574696F 6E733A20 636F6D70 6C657465 [etions: complete]
2012-06-04 23:27:27.750: [ CSSD][2834]1000F5370 64202564 20697465 6D730049 636C7373 [d %d items.Iclss]
2012-06-04 23:27:27.750: [ CSSD][2834]1000F5380 6E6D7652 6561644C 65617365 3A20656E [nmvReadLease: en]
2012-06-04 23:27:27.750: [ CSSD][2834]1000F5390 74657200 636C7373 6E6D7652 6561644C &nbs