oraclerac重启分析

oraclerac重启 Oracle rac 重分析启 一.故障现象 Oracle rac 现现始半月现点交替重～目前个两启2天就重启,并启没且重的现候有什现高现现二.现境介现 台两oracle linux 64位系现～10.2.0.5 64位系现～存现现DELL ps6000 [oracle@dbasc1 ~]$ oifcfg getif eth0 30.30.0.0 global public eth1 192.168.0.0 global cluster_interconnect eth2 10.10.0.0 global cluster_interconnect eth4 10.10.0.0 global cluster_interconnect SQL> select * from GV$CONFIGURED_INTERCONNECTS; INST_ID NAME IP_ADDRESS IS_PUBLIC SOURCE ---------- --------------- ---------------- --------- ------------------------------- 2 eth1 192.168.0.2 NO Oracle Cluster Repository 2 eth2 10.10.10.102 NO Oracle Cluster Repository 2 eth4 10.10.10.103 NO Oracle Cluster Repository 2 eth0 30.30.30.20 YES Oracle Cluster Repository 1 eth1 192.168.0.1 NO Oracle Cluster Repository 1 eth2 10.10.10.100 NO Oracle Cluster Repository 1 eth4 10.10.10.101 NO Oracle Cluster Repository 1 eth0 30.30.30.10 YES Oracle Cluster Repository 8 rows selected SQL> select INST_ID,PUB_KSXPIA,PICKED_KSXPIA,NAME_KSXPIA,IP_KSXPIA 2 from x$ksxpia; INST_ID PUB_KSXPIA PICKED_KSXPIA NAME_KSXPIA IP_KSXPIA ---------- ---------- ----------------------------------- --------------- ---------------- 1 N OCR eth1 192.168.0.1 1 N OCR eth2 10.10.10.100 1 N OCR eth4 10.10.10.101 1 Y OCR eth0 30.30.30.10 SQL> oradebug setmypidSQL> oradebug ipc Trace如下; /u01/app/oracle/admin/asc/udump/asc1_ora_5396.trcOracle Database 10g Enterprise Edition Release 10.2.0.5.0 - 64bit Production With the Partitioning, Real Application Clusters, OLAP, Data Miningand Real Application Testing options ORACLE_HOME = /u01/app/oracle/product/10.2.0/db_1System name:Linux Node name:dbasc1 Release:2.6.18-194.el5xen Version:#1 SMP Mon Mar 29 22:22:00 EDT 2010 Machine:x86_64 Instance name: asc1 Redo thread mounted by this instance: 1 Oracle process number: 46 Unix process pid: 5396, image: oracle@dbasc1 (TNS V1-V3)*** 2011-12-02 11:10:49.327 *** ACTION NAME:() 2011-12-02 11:10:49.327 *** MODULE NAME:(sqlplus@dbasc1 (TNS V1-V3)) 2011-12-02 11:10:49.327*** SERVICE NAME:(SYS$USERS) 2011-12-02 11:10:49.327*** SESSION ID:(3244.1653) 2011-12-02 11:10:49.327Dump of unix-generic skgm context areaflags 000000e7 realmflags 0000000f mapsize 00000800 protectsize 00001000 lcmsize 00001000 seglen 00008000 largestsize 0000040000000000 smallestsize 0000000001000000 stacklimit 0x7fff6bcca860 stackdir -1 mode 640 magic acc01ade Handle: 0xcfac160 `/u01/app/oracle/product/10.2.0/db_1asc1' Dump of unix-generic realm handle `/u01/app/oracle/product/10.2.0/db_1asc1', flags = 00000000 Area #0 `Fixed Size' containing Subareas 0-0 Total size 0000000000207290 Minimum Subarea size 00000000 Area Subarea Shmid Stable Addr Actual Addr 0 0 2555912 0x00000060000000 0x00000060000000 Subarea size Segment size 0000000000208000 0000000400010000 Area #1 `Variable Size' containing Subareas 2-2 Total size 00000003ff000000 Minimum Subarea size 01000000 Area Subarea Shmid Stable Addr Actual Addr 1 2 2555912 0x00000061000000 0x00000061000000 Subarea size Segment size 00000003ff000000 0000000400010000 Area #2 `Redo Buffers' containing Subareas 1-1 Total size 0000000000df8000 Minimum Subarea size 00000000 Area Subarea Shmid Stable Addr Actual Addr 2 1 2555912 0x00000060208000 0x00000060208000 Subarea size Segment size 0000000000df8000 0000000400010000 Area #3 `skgm overhead' containing Subareas 3-3 Total size 0000000000009000 Minimum Subarea size 00000000 Area Subarea Shmid Stable Addr Actual Addr 3 3 2555912 0x00000460000000 0x00000460000000 Subarea size Segment size 0000000000009000 0000000400010000 Dump of Solaris-specific skgm context sharedmmu 00000000 shareddec 0 used region 0: start 0000000040000000 length 000000047fff40000000Maximum processes: = 3000 Number of semaphores per set: = 187 Semaphores key overhead per set: = 4 User Semaphores per set: = 183 Number of semaphore sets: = 17 Semaphore identifiers: = 17 Semaphore List= 262146 -------------- system semaphore information ------------------- Shared Memory Segments -------- key shmid owner perms bytes nattch status 0x00000000 2392066 root 644 80 2 0x00000000 2424836 root 644 16384 2 0x00000000 2457605 root 644 280 2 0x00fa5a34 2523143 oracle 640 130056192 16 0x81c8eca0 2555912 oracle 660 17179934720 153 ------ Semaphore Arrays -------- key semid owner perms nsems 0x57aff660 131073 oracle 640 44 0x53b1ce2c 262146 oracle 660 187 0x53b1ce2d 294915 oracle 660 187 0x53b1ce2e 327684 oracle 660 187 0x53b1ce2f 360453 oracle 660 187 0x53b1ce30 393222 oracle 660 187 0x53b1ce31 425991 oracle 660 187 0x53b1ce32 458760 oracle 660 187 0x53b1ce33 491529 oracle 660 187 0x53b1ce34 524298 oracle 660 187 0x53b1ce35 557067 oracle 660 187 0x53b1ce36 589836 oracle 660 187 0x53b1ce37 622605 oracle 660 187 0x53b1ce38 655374 oracle 660 187 0x53b1ce39 688143 oracle 660 187 0x53b1ce3a 720912 oracle 660 187 0x53b1ce3b 753681 oracle 660 187 0x53b1ce3c 786450 oracle 660 187 ------ Message Queues -------- key msqid owner perms used-bytes messages ksxpdmp: facility 0 (?) (0x0, (nil)) counts 0, 0 ksxpdmp: Dumping the osd context (brief) SKGXPCTX: 0x0xcfd43c0 ctx WAIT HISTORY Wait Time Time since Fast reaps Wait Type Return Code (ms) prev wait(ms) before --------- -------------- ----------- --------- -----------0 0 0 NORMAL invalid status code0 0 0 NORMAL invalid status code0 0 0 NORMAL invalid status code0 0 0 NORMAL invalid status code0 0 0 NORMAL invalid status code0 0 0 NORMAL invalid status code0 0 0 NORMAL invalid status code0 0 0 NORMAL invalid status code0 0 0 NORMAL invalid status code0 0 0 NORMAL invalid status code0 0 0 NORMAL invalid status code 0 0 0 NORMAL invalid status code0 0 0 NORMAL invalid status code0 0 0 NORMAL invalid status code0 0 0 NORMAL invalid status code0 0 0 NORMAL invalid status codewait delta 19 sec (19844 msec) ctx ts 0x0 last ts 0x0 user cpu time since last wait 0 sec 0 ticks system cpu time since last wait 0 sec 0 ticks locked 1 blocked 0 timed wait receives 0 fast reaps since last wait 0 admno 0x771fe85a admport: SSKGXPT 0xcfd67b0 flags SSKGXPT_READPENDING socket no 7 IP 192.168.0.1 UDP 51784 last bytes received: 32824 context timestamp 0 done Queue no completed requests port Queue no ports connection Queue no pending connect disconnect operations sends waiting to be transmitted no sends waiting to be transmitted pending ack Queue no send requests pending ack Mapped regions Region[0] Id 1321025581 Base Address 0x61000000 Size -16777216 key 246783003 rgnport 0xcfd5aa4 lbuf 0 nrgns 1 flags 1 SSKGXPT 0xcfd5aa4 flags socket no 12 IP 10.10.10.100 UDP 33644 ksxpdmp: Dumping the ksxp contexts ksxpdmp: Dumping ksxp context 0x2b925ae7b6d8 client 2 Dump of memory from 0x00002B925AE7B6D8 to 0x00002B925AE7B9682B925AE7B6D0 0CFD8778 00000000 [x.......]2B925AE7B6E0 0CFD2398 00000000 02000000 00000000 [.#..............]2B925AE7B6F0 00000000 00000000 5AE7B6F8 00002B92 [...........Z.+..]2B925AE7B700 5AE7B6F8 00002B92 5AE7B708 00002B92 [...Z.+.....Z.+..]2B925AE7B710 5AE7B708 00002B92 5AE7B718 00002B92 [...Z.+.....Z.+..]2B925AE7B720 5AE7B718 00002B92 00000000 00000000 [...Z.+..........]2B925AE7B730 00000000 00000000 5AE7B738 00002B92 [........8..Z.+..]2B925AE7B740 5AE7B738 00002B92 00000002 00000000 [8..Z.+..........]2B925AE7B750 00000000 00000000 00000000 00000000 [................] Repeat 8 times 2B925AE7B7E0 00000002 00000000 00000000 00000000 [................]2B925AE7B7F0 00000000 00000000 00000000 00000000 [................] Repeat 7 times 2B925AE7B870 00000000 00000000 00000002 00000000 [................]2B925AE7B880 00000000 00000000 00000000 00000000 [................] Repeat 8 times 2B925AE7B910 00000002 00000000 00000000 00000000 [................]2B925AE7B920 00000000 00000000 5AE7B928 00002B92 [........(..Z.+..]2B925AE7B930 5AE7B928 00002B92 00D238F0 00000000 [(..Z.+...8......]2B925AE7B940 5AE7B5A0 00002B92 05655B0C 00000000 [...Z.+...[e.....]2B925AE7B950 05654378 00000000 8878EE0F 00000000 [xCe.......x.....]2B925AE7B960 00000000 00000000 [........] ksxpdmp: Dumping ksxp context 0xcfd8778 client 1 Dump of memory from 0x000000000CFD8778 to 0x000000000CFD8A0800CFD8770 0CFD2398 00000000 [.#......]00CFD8780 5AE7B6D8 00002B92 00000000 00000000 [...Z.+..........]00CFD8790 00000000 00000000 0CFD8798 00000000 [................] 00CFD87A0 0CFD8798 00000000 0CFD87A8 00000000 [................] 00CFD87B0 0CFD87A8 00000000 0CFD87B8 00000000 [................] 00CFD87C0 0CFD87B8 00000000 00000000 00000000 [................] 00CFD87D0 00000000 00000000 0CFD87D8 00000000 [................] 00CFD87E0 0CFD87D8 00000000 00000001 00000000 [................] 00CFD87F0 00000000 00000000 00000000 00000000 [................] Repeat 8 times 00CFD8880 00000002 00000000 00000000 00000000 [................] 00CFD8890 00000000 00000000 00000000 00000000 [................] Repeat 7 times 00CFD8910 00000000 00000000 00000002 00000000 [................] 00CFD8920 00000000 00000000 00000000 00000000 [................] Repeat 8 times 00CFD89B0 00000002 00000000 00000000 00000000 [................] 00CFD89C0 00000000 00000000 0CFF1928 00000000 [........(.......] 00CFD89D0 0CFF1928 00000000 0103A460 00000000 [(.......`.......] 00CFD89E0 00000000 00000000 056A690C 00000000 [.........ij.....] 00CFD89F0 00000000 00000000 8878EE0F 00000000 [..........x.....] 00CFD8A00 00000000 00000000 [........] ksxpdmp: Done dumping the ksxp contexts ksxpdmp: Dumping pending request queue ksxpdmp: Done dumping the pending request queue 三.初步分析现程 具现生故障的现现现序此次略～现现弄现了体 1 、现现reason 1,具现明如下,体 Here, you can see the reason for the reconfiguration event. The most common reasons would be 1, 2, or 3. Reason 1 means that the NM initiated the reconfiguration event, as typically seen when a node joins or leaves a cluster. A reconfiguration event is initiated with reason 2 when an instance death is detected. How is an instance death detected? Every instance updates the control file with a heartbeat through its Checkpoint (CKPT) process. If heartbeat information is not present for x amount of time, the instance is considered to be dead and the Instance Membership Recovery (IMR) process initiates reconfiguration. This type of reconfiguration is commonly seen when significant time changes occur across nodes, the node is starved for CPU or I/O times, or some problems occur with the shared storage. A reason 3 reconfiguration event is due to a communication failure. Communication channels are established between the Oracle processes across the nodes. This communication occurs over the interconnect. Every message sender expects an acknowledgment message from the receiver. If a message is not received for a timeout period, then a “communication failure” is assumed. This is more relevant for UDP, as Reliable Shared Memory (RSM), Reliable DataGram protocol (RDG), and Hyper Messaging Protocol (HMP) do not need it, since the acknowledgment mechanisms are built into the cluster communication and protocol itself. When the block is sent from one instance to another using wire, especially when unreliable protocols such as UDP are used, it is best to get an acknowledgment message from the receiver. The acknowledgment is a simple side channel message that is normally required for most of the UNIX systems where UDP is used as the default IPC protocol. When user-mode IPC protocols such as RDG (on HP Tru64 UNIX TruCluster) or HP HMP are used, the additional messaging can be disabled by setting _reliable_block_sends=TRUE. For Windows-based systems, it is always recommended to leave the default value as is.2、现看DELL ps6000日志 INFO 11-12-4 14:52:32 ps6000-1 iSCSI login to target '10.10.10.14:3260, iqn.2001-05.com.equallogic:0-8a0906-c2827630a-58b0000000c4e3fa-flash' from initiator '10.10.10.102:39526, iqn.1994-05.com.redhat:452355c1463' successful using standard-sized frames. NOTE: More than one initiator is now logged in to the target. INFO 11-12-4 14:52:32 ps6000-1 iSCSI login to target '10.10.10.12:3260, iqn.2001-05.com.equallogic:0-8a0906-c2a27630a-ef3000000154e3fa-test2' from initiator '10.10.10.102:49500, iqn.1994-05.com.redhat:452355c1463' successful using standard-sized frames. NOTE: More than one initiator is now logged in to the target. INFO 11-12-4 14:52:32 ps6000-1 iSCSI login to target '10.10.10.13:3260, iqn.2001-05.com.equallogic:0-8a0906-0c427630a-48b000000184e3fb-ocr' from initiator '10.10.10.102:42678, iqn.1994-05.com.redhat:452355c1463' successful using standard-sized frames. NOTE: More than one initiator is now logged in to the target. INFO 11-12-4 14:52:32 ps6000-1 iSCSI login to target '10.10.10.12:3260, iqn.2001-05.com.equallogic:0-8a0906-0c427630a-05e0000001b4e3fb-vote' from initiator '10.10.10.102:49498, iqn.1994-05.com.redhat:452355c1463' successful using standard-sized frames. NOTE: More than one initiator is now logged in to the target. INFO 11-12-4 14:52:25 ps6000-1 iSCSI login to target '10.10.10.14:3260, iqn.2001-05.com.equallogic:0-8a0906-c2827630a-add0000000f4e3fa-backup' from initiator '10.10.10.102:39518, iqn.1994-05.com.redhat:452355c1463' successful using standard-sized frames. NOTE: More than one initiator is now logged in to the target. INFO 11-12-4 14:52:25 ps6000-1 iSCSI login to target '10.10.10.11:3260, iqn.2001-05.com.equallogic:0-8a0906-ad727630a-ed0000000094e3fa-data' from initiator '10.10.10.102:43887, iqn.1994-05.com.redhat:452355c1463' successful using standard-sized frames. NOTE: More than one initiator is now logged in to the target. INFO 11-12-4 14:52:25 ps6000-1 iSCSI login to target '10.10.10.11:3260, iqn.2001-05.com.equallogic:0-8a0906-c2927630a-ba1000000124e3fa-test1' from initiator '10.10.10.102:43886, iqn.1994-05.com.redhat:452355c1463' successful using standard-sized frames. NOTE: More than one initiator is now logged in to the target. INFO 11-12-4 14:52:10 ps6000-1 iSCSI login to target '10.10.10.11:3260, iqn.2001-05.com.equallogic:0-8a0906-c2a27630a-ef3000000154e3fa-test2' from initiator '10.10.10.103:36997, iqn.1994-05.com.redhat:452355c1463' successful using standard-sized frames. NOTE: More than one initiator is now logged in to the target. INFO 11-12-4 14:52:10 ps6000-1 iSCSI login to target '10.10.10.11:3260, iqn.2001-05.com.equallogic:0-8a0906-c2827630a-58b0000000c4e3fa-flash' from initiator '10.10.10.103:36996, iqn.1994-05.com.redhat:452355c1463' successful using standard-sized frames. NOTE: More than one initiator is now logged in to the target. INFO 11-12-4 14:52:10 ps6000-1 iSCSI login to target '10.10.10.11:3260, iqn.2001-05.com.equallogic:0-8a0906-0c427630a-48b000000184e3fb-ocr' from initiator '10.10.10.103:36995, iqn.1994-05.com.redhat:452355c1463' successful using standard-sized frames. NOTE: More than one initiator is now logged in to the target. INFO 11-12-4 14:52:10 ps6000-1 iSCSI login to target '10.10.10.11:3260, iqn.2001-05.com.equallogic:0-8a0906-0c427630a-05e0000001b4e3fb-vote' from initiator '10.10.10.103:36994, iqn.1994-05.com.redhat:452355c1463' successful using standard-sized frames. NOTE: More than one initiator is now logged in to the target. INFO 11-12-4 14:52:07 ps6000-1 iSCSI login to target '10.10.10.11:3260, iqn.2001-05.com.equallogic:0-8a0906-ad727630a-ed0000000094e3fa-data' from initiator '10.10.10.103:37000, iqn.1994-05.com.redhat:452355c1463' successful using standard-sized frames. NOTE: More than one initiator is now logged in to the target. INFO 11-12-4 14:52:07 ps6000-1 iSCSI login to target '10.10.10.11:3260, iqn.2001-05.com.equallogic:0-8a0906-c2927630a-ba1000000124e3fa-test1' from initiator '10.10.10.103:36999, iqn.1994-05.com.redhat:452355c1463' successful using standard-sized frames. NOTE: More than one initiator is now logged in to the target. INFO 11-12-4 14:52:07 ps6000-1 iSCSI login to target '10.10.10.11:3260, iqn.2001-05.com.equallogic:0-8a0906-c2827630a-add0000000f4e3fa-backup' from initiator '10.10.10.103:36998, iqn.1994-05.com.redhat:452355c1463' successful using standard-sized frames. NOTE: More than one initiator is now logged in to the target. INFO 11-12-4 14:49:18 ps6000-1 iSCSI session to target '10.10.10.11:3260, iqn.2001-05.com.equallogic:0-8a0906-0c427630a-05e0000001b4e3fb-vote' from initiator '10.10.10.102:47937, iqn.1994-05.com.redhat:452355c1463' was closed. iSCSI initiator connection failure. No response on connection for 6 seconds. INFO 11-12-4 14:49:18 ps6000-1 iSCSI session to target '10.10.10.14:3260, iqn.2001-05.com.equallogic:0-8a0906-c2827630a-58b0000000c4e3fa-flash' from initiator '10.10.10.103:51813, iqn.1994-05.com.redhat:452355c1463' was closed. iSCSI initiator connection failure. No response on connection for 6 seconds. INFO 11-12-4 14:49:18 ps6000-1 iSCSI session to target '10.10.10.11:3260, iqn.2001-05.com.equallogic:0-8a0906-ad727630a-ed0000000094e3fa-data' from initiator '10.10.10.102:47931, iqn.1994-05.com.redhat:452355c1463' was closed. iSCSI initiator connection failure. No response on connection for 6 seconds. INFO 11-12-4 14:49:17 ps6000-1 iSCSI session to target '10.10.10.13:3260, iqn.2001-05.com.equallogic:0-8a0906-ad727630a-ed0000000094e3fa-data' from initiator '10.10.10.103:36059, iqn.1994-05.com.redhat:452355c1463' was closed. iSCSI initiator connection failure. No response on connection for 6 seconds. INFO 11-12-4 14:49:17 ps6000-1 iSCSI session to target '10.10.10.13:3260, iqn.2001-05.com.equallogic:0-8a0906-c2927630a-ba1000000124e3fa-test1' from initiator '10.10.10.102:43696, iqn.1994-05.com.redhat:452355c1463' was closed. iSCSI initiator connection failure. No response on connection for 6 seconds. INFO 11-12-4 14:49:17 ps6000-1 iSCSI session to target '10.10.10.14:3260, iqn.2001-05.com.equallogic:0- 8a0906-c2827630a-add0000000f4e3fa-backup' from initiator '10.10.10.102:55345, iqn.1994-05.com.redhat:452355c1463' was closed. iSCSI initiator connection failure. No response on connection for 6 seconds. INFO 11-12-4 14:49:16 ps6000-1 iSCSI session to target '10.10.10.11:3260, iqn.2001-05.com.equallogic:0- 8a0906-0c427630a-48b000000184e3fb-ocr' from initiator '10.10.10.103:46666, iqn.1994-05.com.redhat:452355c1463' was closed. iSCSI initiator connection failure. No response on connection for 6 seconds. INFO 11-12-4 14:49:16 ps6000-1 iSCSI session to target '10.10.10.11:3260, iqn.2001-05.com.equallogic:0- 8a0906-c2a27630a-ef3000000154e3fa-test2' from initiator '10.10.10.103:46664, iqn.1994-05.com.redhat:452355c1463' was closed. iSCSI initiator connection failure. No response on connection for 6 seconds. INFO 11-12-4 14:49:16 ps6000-1 iSCSI session to target '10.10.10.11:3260, iqn.2001-05.com.equallogic:0- 8a0906-c2827630a-58b0000000c4e3fa-flash' from initiator '10.10.10.102:47938, iqn.1994-05.com.redhat:452355c1463' was closed. iSCSI initiator connection failure. No response on connection for 6 seconds. INFO 11-12-4 14:49:14 ps6000-1 iSCSI session to target '10.10.10.11:3260, iqn.2001-05.com.equallogic:0- 8a0906-c2827630a-add0000000f4e3fa-backup' from initiator '10.10.10.103:46668, iqn.1994-05.com.redhat:452355c1463' was closed. iSCSI initiator connection failure. No response on connection for 6 seconds. INFO 11-12-4 14:49:14 ps6000-1 iSCSI session to target '10.10.10.11:3260, iqn.2001-05.com.equallogic:0- 8a0906-c2927630a-ba1000000124e3fa-test1' from initiator '10.10.10.103:46667, iqn.1994-05.com.redhat:452355c1463' was closed. iSCSI initiator connection failure. No response on connection for 6 seconds. INFO 11-12-4 14:49:13 ps6000-1 iSCSI session to target '10.10.10.12:3260, iqn.2001-05.com.equallogic:0- 8a0906-c2a27630a-ef3000000154e3fa-test2' from initiator '10.10.10.102:52886, iqn.1994-05.com.redhat:452355c1463' was closed. iSCSI initiator connection failure. No response on connection for 6 seconds. INFO 11-12-4 14:49:13 ps6000-1 iSCSI session to target '10.10.10.13:3260, iqn.2001-05.com.equallogic:0- 8a0906-0c427630a-48b000000184e3fb-ocr' from initiator '10.10.10.102:43703, iqn.1994-05.com.redhat:452355c1463' was closed. iSCSI initiator connection failure. No response on connection for 6 seconds. INFO 11-12-4 14:49:13 ps6000-1 iSCSI session to target '10.10.10.12:3260, iqn.2001-05.com.equallogic:0-8a0906- 0c427630a-05e0000001b4e3fb-vote' from initiator '10.10.10.103:60849, iqn.1994-05.com.redhat:452355c1463' was closed. iSCSI initiator connection failure. No response on connection for 6 seconds. 感现现始现接存现的现候要3秒才能现接成功～现不现致个会会disktimeout～出现点踢 现现现迫～现有些分析此次略 四、现现 我现现现是同步的～初步想法是脚本现现是不是存现先写个down现致的disktimeout～ 希望同志现有现现的现现我提供思路～我分析分析～现现日志在共享～现现大家帮 , 关于 oracle rac 关关关点的逐 rac中现于node被现逐现而现致reboot的情。况 首先现现rac中的心跳, 现于oracle clusterware中的心跳有现～如下,两 1. Disk heartbeat (voting device) - IOT 2. Network heartbeat (across the interconnect) - misscount 现里的disk hearbeat是指votedisk心跳～我现都知道votedisk是仲裁现～ 那现到底有什现作用,它呢Oracle文有现现描述的,档 votedisk, The Voting Disk is used by the Oracle cluster manager in various layers. The Node Monitor (NM)uses the Voting Disk for the Disk Hearbeat, which is essential in the detection and resolution of cluster "split brain". NM monitors the Voting Disk for other competing sub-clusters and uses it for the eviction phase. Hence the availability from the Voting Disk is critical for the operation of the Oracle Cluster Manager. The shared volumes created for the OCR and the voting disk should be configured using RAID to protect against media failure. This requires the use of an external cluster volume manager, cluster file system, or storage hardware that provides RAID protection. Disk heartbeat, Each node writes a disk heartbeat to each voting disk once per second。Each node reads their kill block once per second, if the kill block is overwritten node commits suicide.During reconfig (join or leave) CSSD monitors all nodes and determines whether a node has a disk heartbeat, including those with no network heartbeat.If no disk heartbeat within I/O timeout (MissCount during cluster reconfiguration) then node is declared as dead. Voting disk needs to be mirrored, should it become unavailable, cluster will come down. If an I/O error is reported immediately on access to the vote disk, we immediately mark the vote disk as offline so it isn't at the party anymore. So now we have (in our case) just two voting disks available. We do keep retrying access to that dead disk, and if it becomes available again and the data is uncorrupted we mark it online again. If a second vote disk suffered an I/O error in the window that the first disk was marked offline. So now we don't have quorum. Bang reboot. 网现心跳就不多现了～指的是rac私心跳。如下一段摘自现,网网 Voting files are used by CSS to ensure data integrity of the database by detecting and resolving network problems that could lead to a split-brain, so must be accessible at all times. There are other techniques used by other cluster managers, like quorum server, and quorum disks which function differently, but serve the same purpose. Note that a majority of vote disks, i.e. N/2 + 1, must be accessible by each node to ensure that all pairs node have at least one voting file that they both see, which allows proper resolution of network issues; this is to address the possible complaint that 2 voting files provide redundancy, so a third should not be necessary. During normal processing, each node writes a disk heartbeat once per second and also reads its kill block once per second. When the kill block indicates that the node has been evicted, the node exits, causing a node reboot.As long as we have enough voting disks online, the node can survive, but when the number of offline voting disks is greater than or equal to the number of online voting disks, the Cluster Communication Service daemon will fail resulting in a reboot. The rationale for this is that as long as each node is required to have a majority of voting disks online, there is guaranteed to be one voting disk that both nodes in a 2 node pair can see. 上面提到了现于votedisk(仲裁现)～当cluster中有现点出现故障现～offline的votedisk个数必现小于存活votedisk个数～ 否现现致存活的现点会reboot。 下面再现现重要的现来几个参数. misscount, 现心跳可以现失的次网数(现位是秒) 不同平台和版本的misscount默现现是不一现的～现现如下表格, 10g (R1 OS 11g &R2) Linux6030 Unix3030 VMS3030 Window3030s 另外如果使用了第三方cluster现件现～那现misscount现默现现即600s～现里现裂也是第三方cluster现件完成的。来 diskhearbeat即disktimeout～在10.2.0.1+版本以后(打了patch 4896338)默现现是200s。 disktimeout也现现称DTO～但是文上又把档DTO现分现现～如下,两 -- SDTO～是short disk time out的现～现点添加或现除现称即cluster需要现行reconfigure的现现。 -- LDTO～是指正常的rac操作中允现votedisk i/o完成超现的现现。 rebootime,在10g~11g中默现都现3s～是即rac出现现裂或现点被现逐的现候～现现点在将会rebootime现现被重。内启 文中提到档node的现逐从10201版本以后～不在根据DTO来决定～而是基于disktimeout～ 默现情下～况misscount的现小于disktimeout。 那现, 在什现情下现致况会node被现逐,如下呢: ? Node is not pinging via the network heartbeat? Node is not pinging the Voting disk? Node is hung/busy and is unable to perform either of the earlier tasks 根据文中的描述～现现现的档翻来node reboot条件表格, Reboot ;是否重Network PingDisk Ping启, 在misscount现 完成内在misscount现完成内否 Disk ping现现超现misscount现～但在misscount现完成内否是小于disktimeout现。 在misscount现完成内Disk ping现现超现disktimeout现是 Network ping现现超现misscount在misscount现完成内是现 修改如上现的如下,几个参数 $ORA_CRS_HOME/bin/crsctl set css misscount where is the maximum i/o latency to the voting disk +1 second10.2.0.1+版本～如果现用了patch4896338～那现现需要有如下的操作: $CRS_HOME/bin/crsctl set css reboottime [-force] ( is seconds)$CRS_HOME/bin/crsctl set css disktimeout [-force] ( is seco最后需要现明一下的是～在使用了第三方集群如cluster现件以后～oracle就不再推修改荐misscount～ 可以引现在的现现。潜oracle是现现解现的, Do not change default misscount values if you are running Vendor Clusterware along with Oracle Clusterware.The default values for misscount should not be changed when using vendor clusterware. Modifying misscount inthis environment may cause clusterwide outages and potential corruptions.如上的信息大家可以参几个考如下的mos文,档 10g RAC- Steps To Increase CSS Misscount- Reboottime and DisktimeoutCSS Timeout Computation in Oracle Clusterware Reconfiguring the CSS disktimeout of 10gR2 Clusterware for Proper LUN Failover of the Dell MD3000i iSCSI Storage [ID 462616.1]