RAC: Recover lost voting and OCR Disks from backup

We have a lost voting disks issue yesterday, this occurs due to incorrect aliases or the device mapper persistent names have been changed after storage upgradation/migration.

CRS was not up and showing the error below Obviously the CSSD , cssd.log

No voting disks founds

So, we have lost the voting and OCR disks.

Here is the situational tasks. In normal condition (10g)

    1) We know that OCR is backed up automatically.

    2) Deconfigure and configure the CRS (long procedure)

From 11gr2 onwards, Before pen down the procedure some thing you must know is

    1) Voting disk also will be backed up automatically to ocr backup. (Many of us does not know this)

    2) Clusterware can be startup in exclusive mode i.e no crs (as you know crsd starts later along with other cluster resources)

    3) OCR and voting disks are part of ASM, means you can recreate the diskgroups without CRS by manually starting the ASM and assign that diskgroup to CRS as OCR/Voting Disk

Okay, now see how this works, the situational tasks

Assuming you have not started the cluster stack and only hasd is running.

    1) Start CRS in exclusive mode in any of the node.

        root@rac2 ~]# crsctl start crs -excl
        CRS-4123: Oracle High Availability Services has been started.
        CRS-2672: Attempting to start ‘ora.gipcd’ on ‘rac2′
        CRS-2672: Attempting to start ‘ora.mdnsd’ on ‘rac2′
        CRS-2676: Start of ‘ora.gipcd’ on ‘rac2′ succeeded
        CRS-2676: Start of ‘ora.mdnsd’ on ‘rac2′ succeeded
        CRS-2672: Attempting to start ‘ora.gpnpd’ on ‘rac2′
        CRS-2676: Start of ‘ora.gpnpd’ on ‘rac2′ succeeded
        CRS-2672: Attempting to start ‘ora.cssdmonitor’ on ‘rac2′
        CRS-2676: Start of ‘ora.cssdmonitor’ on ‘rac2′ succeeded
        CRS-2672: Attempting to start ‘ora.cssd’ on ‘rac2′
        CRS-2679: Attempting to clean ‘ora.diskmon’ on ‘rac2′
        CRS-2681: Clean of ‘ora.diskmon’ on ‘rac2′ succeeded
        CRS-2672: Attempting to start ‘ora.diskmon’ on ‘rac2′
        CRS-2676: Start of ‘ora.diskmon’ on ‘rac2′ succeeded
        CRS-2676: Start of ‘ora.cssd’ on ‘rac2′ succeeded
        CRS-2672: Attempting to start ‘ora.ctssd’ on ‘rac2′
        CRS-2672: Attempting to start ‘ora.drivers.acfs’ on ‘rac2′
        CRS-2676: Start of ‘ora.drivers.acfs’ on ‘rac2′ succeeded
        CRS-2676: Start of ‘ora.ctssd’ on ‘rac2′ succeeded
        CRS-2672: Attempting to start ‘ora.asm’ on ‘rac2′
        CRS-2676: Start of ‘ora.asm’ on ‘rac2′ succeeded
        CRS-2672: Attempting to start ‘ora.crsd’ on ‘rac2′
        CRS-2676: Start of ‘ora.crsd’ on ‘rac2′ succeeded

2) Add the new disks to asm diskgroup

            oracle@rac2 ~]$ sqlplus / as sysasm
            SQL*Plus: Release 11.2.0.1.0 Production on Wed Jun 01 17:23:56 2013
            Copyright (c) 1982, 2009, Oracle.  All rights reserved.

            Connected to:
            Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 – 64bit Production
            With the Real Application Clusters and Automatic Storage Management options

            SQL> create diskgroup DATA external redundancy disk ‘ORCL:DATA’ attribute ‘COMPATIBLE.ASM’ = ’11.2′;
            Diskgroup Created.

3) Identify the latest backup

            Identify the latest OCR backup.
            root@oracle ~]# /u01/app/11.2.0/grid/bin/ocrconfig -showbackup

            oracle      2013/06/01 09:00:30     /u01/app/11.2.0/grid/cdata/oracle/backup00.ocr
            oracle      2013/06/06 05:37:29     /u01/app/11.2.0/grid/cdata/oracle/backup01.ocr
            oracle      2013/06/06 01:37:27     /u01/app/11.2.0/grid/cdata/oracle/backup02.ocr
            oracle      2013/05/31 01:37:21   

  /u01/app/11.2.0/grid/cdata/oracle/day.ocr
            oracle         2013/05/24 13:37:19     /u01/app/11.2.0/grid/cdata/oracle/week.ocr
       

4) Restore the OCR from automatic backup

                [root@rac2 ~]# ocrconfig -restore /u01/app/11.2.0/grid/cdata/oracle/backup00.ocr
       

5) Start the CRS in exclusive mode

            crsctl start res ora.crsd -init

            CRS-2672: Attempting to start ‘ora.crsd’ on ‘rac2′
            CRS-2676: Start of ‘ora.crsd’ on ‘rac2′ succeeded

6) Replace the voting disk from automatic backup

            [root@rac2 ~]# crsctl replace votedisk +DATA
            Successful addition of voting disk ahshuehki8489020msjsjsj
            Successfully replaced voting disk group with +DATA.
            CRS-4266: Voting file(s) successfully replaced

7) Stop the crs and restart

            $CRS_HOME/bin/crsctl stop crs -f

            $CRS_HOME/bin/crsctl start crs

8) Start the crs in other nodes  

        $CRS_HOME/bin/crsctl start crs

9) Verify the cluster

        # $GRID_HOME/bin/crsctl check cluster -all

            rac1:
            CRS-4537: Cluster Ready Services is online
            CRS-4529: Cluster Synchronization Services is online
            CRS-4533: Event Manager is online
            rac2:
            CRS-4537: Cluster Ready Services is online
            CRS-4529: Cluster Synchronization Services is online
            CRS-4533: Event Manager is online

To note, just in case,

If your SPFILE of ASM is in ocr/votingdisk diskgroup, you need to create temporary parameters for asm and then start asm and then finally add this spfile to Datagroup.
<pre>
        Sample Parameters
         *.asm_power_limit=1
         *.diagnostic_dest=’/u01/app/oragrid’
         *.instance_type=’asm’
         *.large_pool_size=12M
         *.remote_login_passwordfile=’EXCLUSIVE’

       

SQL> create spfile=’+CRS’ from pfile=’/tmp/asm_pfile.ora’;

Hope this help you :)