RMAN ERRORS


Problem Description
While starting Oracle RMAN backup whenever we use catalog database as repository it fails with following errors:
$rman target / catalog catdba@RMAN

Recovery Manager: Release 11.2.0.3.0 - Production on Tue Nov 22 22:30:47 2011

Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.

connected to target database: SPMDP (DBID=2694055214)
recovery catalog database Password:
connected to recovery catalog database
PL/SQL package CATDBA.DBMS_RCVCAT version 11.02.00.02 in RCVCAT database is not current
PL/SQL package CATDBA.DBMS_RCVMAN version 11.02.00.02 in RCVCAT database is not current

Depending on the version of the RMAN-catalog schema version there might notice following errors.
PL/SQL package RMAN.DBMS_RCVCAT version 11.01.00.07 in RCVCAT database is not current
PL/SQL package RMAN.DBMS_RCVMAN version 11.01.00.07 in RCVCAT database is not current
or,
PL/SQL package RMAN.DBMS_RCVCAT version 11.02.00.01 in RCVCAT database is not current
PL/SQL package RMAN.DBMS_RCVMAN version 11.02.00.01 in RCVCAT database is not current

Whenever it is issued "register database" it fails with following error:
RMAN> register database;

DBGSQL: RCVCAT> begin dbms_rcvman.dumpPkgState('RCVMAN after sqlerror'); end;
DBGSQL: sqlcode = 6550

DBGSQL: RCVCAT> begin dbms_rcvcat.dumpPkgState('RCVCAT after sqlerror');end;
DBGSQL: sqlcode = 6550
database registered in recovery catalog
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03008: error while performing automatic resync of recovery catalog
RMAN-10015: error compiling PL/SQL program
RMAN-10014: PL/SQL error 0 on line 1655 column 12: Statement ignored
RMAN-10014: PL/SQL error 306 on line 1655 column 12: wrong number or types of arguments in call to 'CHECKTABLESPACE'

Resync catalog also fails with following error:
RMAN> resync catalog;

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of resync command on default channel at 11/22/2011 23:09:07
RMAN-10015: error compiling PL/SQL program
RMAN-10014: PL/SQL error 0 on line 1655 column 12: Statement ignored
RMAN-10014: PL/SQL error 306 on line 1655 column 12: wrong number or types of arguments in call to 'CHECKTABLESPACE'

Recent Changes
The target database has been recently upgrades and so RMAN executable is upgraded but catalog database is not aware of this upgrade.

Cause of the Problem
The problem happened due to version mismatch between the RMAN-executable and the RMAN-catalog SCHEMA.

Solution of the Problem
Solution 01: 
Upgrade the RMAN catalog SCHEMA. Start the RMAN-executable from the ORACLE_HOME which has been upgraded. There is only a connection to the CATALOG required.
A connection to the TARGET is optional.

For example issue, 
$ rman catalog $RMAN_USERID/$RMAN_PASSWD@$RMAN_CONN_STR
RMAN> upgrade catalog;
Solution 02: 
Don't use catalog for backup information. So run backup without connecting catalog database.
$ rman target /
$ backup database;
Posted by Arju at 9:33 PM 0 comments http://img1.blogblog.com/img/icon18_email.gif Labels: RMAN
Reactions: 
Wednesday, November 2, 2011
Problem Description
In Oracle database version 11.2g RMAN Backup on RAC database fails with following errors while doing controlfile autobackup. 
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of Control File and SPFILE Autobackup command on
ORA_DISK_1 channel at 301/10/2011 12:10:12
ORA-00245: control file backup operation failed
While backing up through Sql*plus ORA-00245 error also returned. 
alter database backup controlfile to '/oracle/cntl_file'
ERROR at line 1:
ORA-00245: control file backup operation failed
Cause of the Problem
From Oracle documentation we get following information about the error ORA-00245.

ORA-00245 "control file backup failed; target is likely on a local file system"
// *Cause: Failed to create a control file backup because some process
// signaled an error during backup creation. This is likely caused
// by the backup target being on a local file system so it could not
// be accessed by other instances. It can also be caused by other
// I/O errors to the backup target. Any process of any instance that
// starts a read/write control file transaction must have access
// to the backup control file during backup creation.
// *Action: Check alert files of all instances for further information.

RMAN creates a copy of the control file for read consistency, this is the snapshot controlfile. Due to the changes made to the controlfile backup mechanism in 11gR2 any instances in the cluster may write to the snapshot controlfile. Therefore, the snapshot controlfile file needs to be visible to all instances.

The same happens when a backup of the controlfile is created directly from sqlplus any instance in the cluster may write to the backup controfile file.

In 11gR2 onwards, the controlfile backup happens without holding the control file enqueue. For non-RAC database, this doesn't change anything.

But, for RAC database, the snapshot controlfile location must be in a shared file system that will be accessible from all the nodes. So if you put your snapshot controlfile in local file system and if that file are not accessible to other nodes of the RAC database you will likely face Oracle error ORA-00245.

Solution of the Problem
The snapshot controlfile must be accessible by all nodes of a RAC database. If the snapshot controlfile does not reside in on a shared device this error will raise.

1. Check the existing snapshot controlfile location:

rman> show all;

2. Configure the snapshot controlfile to a shared disk:

rman> CONFIGURE SNAPSHOT CONTROLFILE NAME TO 'shared_disk/snapcf.f'; # default
Posted by Arju at 3:03 AM 0 comments http://img1.blogblog.com/img/icon18_email.gif Labels: ControlfilesRMAN
Reactions: 
Sunday, October 30, 2011
Problem Description
After switchover operation standby database now becomes primary database and now database backups, crosscheck archivelog all on new primary database fails with error RMAN-20070. 
RMAN> CROSSCHECK archivelog all;

starting full resync of recovery catalog
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of crosscheck command at 30/10/2011 12:13:00
RMAN-03014: implicit resync of recovery catalog failed
RMAN-03009: failure of full resync command on default channel at 30/10/2011 12:13:00
RMAN-20070: invalid log history sequence#
Cause of the Problem
This issue is due to Oracle Bug:4755799 RMAN-20070 ERROR ON PRIMARY AFTER DATAGUARD SWITCHOVER

RMAN-20070 can occur during RMAN resync after a dataguard switchover. Also this bug can appear on using CROSSCHECK as it also implies a resync.
This bug remains in,
9.2.0.4 to 9.2.0.8
10.1 to 10.1.0.4
10.2.0.1 to 10.2.0.2
However, this issue does not affect 10.1.0.5, 10.2.0.3 or higher, and 11g.

Solution of the Problem
Workaround:
Disable the rlh_u2 constraint after connecting to the recovery catalog schema.

$ sqlplus rman/rman@catalog_db
SQL> alter table rlh modify constraint rlh_u2 disable ;

Fix:
- Upgrade to 11g, 10.2.0.3 or 10.1.0.5 (fixed versions)
- Apply patch 4755799 if it is available for your platform and release.
Posted by Arju at 3:36 AM 0 comments http://img1.blogblog.com/img/icon18_email.gif Labels: BugRMAN
Reactions: 
Saturday, October 29, 2011
Problem Description
Backup of Oracle database using RMAN with catalog fails with RMAN-20032:
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of show command at 30/10/2011 10:21:12
RMAN-03014: implicit resync of recovery catalog failed
RMAN-03009: failure of full resync command on default channel at 30/10/2011 10:21:12
RMAN-20032: checkpoint change# too low

Cause of the Problem
Database was resorted from the cold backup. Catalog can not be reseted for this change.
Reset database will not work as open resertlogs is a prerequisite for reset.

Solution of the Problem
$ rman TARGET / CATALOG rman/password@catalog
RMAN> UNREGISTER DATABASE NOPROMPT;
RMAN> REGISTER DATABASE; 
Posted by Arju at 11:50 PM 1 comments http://img1.blogblog.com/img/icon18_email.gif Labels: RMAN
Reactions: 
Problem Description
During Oracle RMAN database backup ORA-30036 error is raised. 
starting full resync of recovery catalog
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of show command at 30/10/2011 10:32:12
RMAN-03014: implicit resync of recovery catalog failed
RMAN-03009: failure of full resync command on default channel at 30/10/2011 10:32:12
ORA-30036: unable to extend segment by 8 in undo tablespace 'UNDOTBS1'
The funny thing is error is not reported in alert log and does not generate trace files.

Cause of the Problem
The problem happened due to oracle bug 8416270 - ORA-30036 DURING RMAN RESYNC.

Solution of the Problem
Workaround of this bug is to turn off undo autotune hidden parameter in init.ora:

_undo_autotune = false

If you are using spfile, then you have to issue,

alter system set "_undo_autotune" = false scope=spfile;
shutdown immediate
startup


Note that, with _undo_autotune=false, statistics tracking on UNDO is no longer available (for example, in V$UNDOSTAT).
Posted by Arju at 10:51 PM 0 comments http://img1.blogblog.com/img/icon18_email.gif Labels: BugRMAN
Reactions: 
Problem Description
RMAN backups that use catalog database fails with following deadlock errors.
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of backup plus archivelog command at 30/10/2011 12:00:40
RMAN-03014: implicit resync of recovery catalog failed
RMAN-03009: failure of partial resync command on default channel at 30/10/2011 12:00:40
ORA-00060: deadlock detected while waiting for resource
Dead lock trace file contains the following statement

DELETE FROM CKP WHERE DBINC_KEY = :B1 AND CKP_KEY IN (SELECT CKP_KEY1 FROM
(SELECT CKP_KEY CKP_KEY1 FROM CKP WHERE DBINC_KEY = :B1 ) CKP1, (SELECT :B2
CKP_KEY2 FROM DUAL UNION SELECT :B3 FROM DUAL UNION SELECT NVL(MAX(CKP_KEY),0) F
ROM CKP WHERE DBINC_KEY=:B1 UNION SELECT START_CKP_KEY FROM TSATT WHERE DBINC_KE
Y = :B1 UNION SELECT NVL(END_CKP_KEY,0) FROM TSATT WHERE DBINC_KEY = :B1 UNION S
ELECT START_CKP_KEY FROM DFATT WHERE DBINC_KEY = :B1 UNION SELECT NVL(END_CKP_KE
Y,0) FROM DFATT WHERE DBINC_KEY = :B1 UNION SELECT START_CKP_KEY FROM TFATT WHER
E DBINC_KEY = :B1 UNION SELECT NVL(END_CKP_KEY,0) FROM TFATT WHERE DBINC_KEY = :
B1 ) CKP2 WHERE CKP_KEY1 = CKP_KEY2(+) AND CKP_KEY2 IS NULL)

Cause of the Problem
Oracle is buggy product. And this issue is due to Bug 6830296.

Solution of Problem
Workaround
In RMAN catalog database create the following indexes :

CREATE INDEX tfatt_i_sck on tfatt(start_ckp_key) ;
CREATE INDEX tfatt_i_eck on tfatt(end_ckp_key) ;

You can also check for availability of one off patch 6830296.
Posted by Arju at 10:09 PM 0 comments http://img1.blogblog.com/img/icon18_email.gif Labels: BugRMAN
Reactions: 
Problem Description
Oracle database backup and maintenance commands fail with following errors:
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of show command at 30/10/2011 19:26:33
RMAN-03014: implicit resync of recovery catalog failed
RMAN-03009: failure of full resync command on default channel at 30/10/2011 19:26:33
ORA-01580: error creating control backup file /u011/app/oracle/admin/BDDBA/snapctl/
ORA-27037: unable to obtain file status
SVR4 Error: 20: Not a directory
Additional information: 6
Cause of the Problem
The error occurred due to wrong name in snapshot control file. You have to specify full name of snapshot control file including file name. By default snapshot controlfile name is '$ORACLE/HOME/dbs/snapcf_$ORACLE_SID.f'. Instead of snapshot controlfile name if only the directory is given, you will see above errors.

For example if you specify following command in RMAN you will face this error.
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/u011/app/oracle/admin/BDDBA/snapctl';

Because here file name is not given, only the location is specified in the configure snapshot controlfile name.

Solution of the Problem 
Connect to RMAN with nocatalog option and then set the snapshot controlfile to the proper location and specify name.

$ rman target / nocatalog
RMAN> CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/u011/app/oracle/admin/BDDBA/snapctl/snapcf_ora10g.f';


Once the snapshot controlfile name is set properly; the backups or maintenance operation can be performed successfully without any error.
Posted by Arju at 7:53 PM 0 comments http://img1.blogblog.com/img/icon18_email.gif Labels: RMAN
Reactions: 
Tuesday, March 15, 2011
Problem Description
Database version is 10.2.0.3 and it is 5 nodes RAC on Red Hat Enterprise Linux. RMAN backup is integrated with Veritas Netbackup 5.1 (MP6). Whenever run backup operation is going to be hanged for long periods of time and then generates error:

ORA-3135: connection lost contact

once if we kill the backup job everything goes back to normal.

Investigation
From the OEM ADDM it is found followings:

FINDING 1: 100% impact (98213 seconds)
---------------------------------------------
Wait class "Scheduler" was consuming significant database time.

Cause of the Problem
This is Oracle bug specifically Bug 6874858 Poor performance with Resource Manager when RMAN is running. When RMAN is running with Resource Manager enabled, the performance of the system may degrade. Users may have problems logging in. When this occurs large waits will be seen with the wait event "resmgr:cpu quantum" with low CPU utilization.

Solution of the Problem
Workaround you can disable Oracle Resource Manager.
However this bug is fixed in following versions:

- 11.2.0.1 (Base Release)
- 11.1.0.7 (Server Patch Set)
- 10.2.0.5 (Server Patch Set)
- 11.1.0.6 Patch 11 on Windows Platforms
- 10.2.0.4 Patch 41 on Windows Platforms

So upgrade to your Oracle database is another solution.
Posted by Arju at 11:18 PM 0 comments http://img1.blogblog.com/img/icon18_email.gif Labels: RMAN
Reactions: 
Wednesday, December 22, 2010
Problem Description
RMAN backup failed with error message RMAN-03009, ORA-19510 and ORA-27046.
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on channel1 channel at 20/12/2010 01:22:21
ORA-19510: failed to set size of 262143 blocks for file "/backup/database/prod/hot_backup.rman" (blocksize=)
ORA-27046: file size is not a multiple of logical block size
Additional information: 2

Cause of the Problem
Note that, there are several variants of this ORA-19510 error. You can search for ORA-19510 within my blog. In this case problem started due to the operating system size limit. RMAN backup piece size exceed the max file size limit of the O/S.

Solution of the Problem
Make sure that rman backup piece won't exceed the O/S permitted filesize limit. You can restrict the rman backup piece using the MAXPIECESIZE option.
For example in RMAN prompt issue,

RMAN> Configure channel device type disk MAXPIECESIZE = 2G; 
And then run your backup command again.

RMAN> backup database ;
Posted by Arju at 10:13 PM 0 comments http://img1.blogblog.com/img/icon18_email.gif Labels: RMAN
Reactions: 
Monday, December 20, 2010
Problem Description
While taking RMAN backup it fails with following message.
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on channel1 channel at 12/20/2010 01:30:21
ORA-19510: failed to set size of 6410 blocks for file "/backup/PROD_DATABASE/rman/PROD_tdatabase_s2398_LVL_1" (blocksize=8192)
ORA-27044: unable to write the header block of file
HP-UX Error: 2: No such file or directory
Additional information: 7
ORA-19510: failed to set size of 4294967295 blocks for file "/backup/PROD_DATABASE/rman/PROD_tdatabase_s2398_LVL_1" (blocksize=8192)
ORA-27044: unable to write the header block of file


Cause of the Problem
The backup fails as there is no space available on the disk.

Solution of the Problem
Check the space where you are taking RMAN backup. For example, in this case we see backup is generated in the location /backup/PROD_DATABASE/rman/ so check disk space usage or free disk space remained into /backup partition. You can check space usage on unix file system by using du -sh * comamnd. You can also issue,
$df -h
to see free space available.
In order to take backup you need to delete unnecessary files from the partition.
Posted by Arju at 1:51 AM 0 comments http://img1.blogblog.com/img/icon18_email.gif Labels: RMAN
Reactions: 
Problem Description
While taking RMAN backup it fails with following errors:
RMAN-03009: failure of backup command on DISK1 channel at 09/19/2005 19:51:02
ORA-19510: failed to set size of 268930 blocks for file
"/backup/rman/backup/full/rman_complete_backup" (blocksize=8192)
ORA-27037: unable to obtain file status
IBM AIX RISC System/6000 Error: 2: No such file or directory
Additional information: 5
Cause of the Problem
The problem occurred because backup piece is being removed from disk before the backup finished. That means backup piece is deleted before RMAN completely finishes creating it. In fact there was a scheduled cron purge job that conflicted with the RMAN backup and was removing the piece before RMAN finished creating it.

Solution of the Problem
The solution is schedule the cron purge job to run at a different time to allow the RMAN backup job to complete or remove the cron purge job entirely.
Posted by Arju at 12:33 AM 0 comments http://img1.blogblog.com/img/icon18_email.gif Labels: RMAN
Reactions: 
Tuesday, October 19, 2010
Problem Description
Starting backup at 19-OCT-2010 00:31:37
current log archived
released channel: channel1
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of backup command at 10/19/2010 00:32:03
RMAN-06059: expected archived log not found, lost of archived log compromises recoverability
ORA-19625: error identifying file /u01/app/oracle/admin/ORACLE/arch/arch_94049.log
ORA-27037: unable to obtain file status
HP-UX Error: 2: No such file or directory
Additional information: 3
Cause of the Problem
The problem happened because Oracle archivelog files were deleted at OS level.

Solution of the Problem
Run following commands in your RMAN prompt to resolve the issue,
Rman> Crosscheck copy of archivelog all;
Rman> Crosscheck archivelog all;
Rman> Resync catalog;
Rman> delete force obsolete;
Rman> delete expired archivelog all ;

After you run the above commands in your RMAN prompt RMAN will unlink all archived log entry from its repository. But due to oracle bug
- if still it can't delete or,
- delete force obsolete and delete expired archivelog all fails with segmentation errors or,
- delete is succeed but still you face RMAN-03002, RMAN-06059, ORA-19625 and ORA-27037 error then do the following:


Rman> Change archivelog '{location of archivelog which is deleted}' UNCATALOG ; 

Please note the first archive log name would be present in the error message ORA-19625:

For example in our case our error output was:
RMAN-06059: expected archived log not found, lost of archived log compromises recoverability
ORA-19625: error identifying file /u01/app/oracle/admin/ORACLE/arch/arch_94049.log
ORA-27037: unable to obtain file status
HP-UX Error: 2: No such file or directory
So run the following command,

Rman> Change archivelog '/u01/app/oracle/admin/ORACLE/arch/arch_94049.log' uncatalog;

Run the archive log backup command check if you still get the error

Keeping specify the archive log file name reported in ORA-19625 till backup of archive log goes fine.

If there is so many archivelog you deleted from specify then alternatively you can do the following as each time specifying the name will take much time.

Rman> Change archivelog all uncatalog ; 

Please note the above command will uncatalog the information about the Archive log from catalog database.
Posted by Arju at 9:58 PM 0 comments http://img1.blogblog.com/img/icon18_email.gif Labels: BackupRMAN
Reactions: 
Monday, October 18, 2010
Problem Description
You can run the following command to create a physical standby database from the primary database.
run{
allocate channel prmy1 type disk;
allocate channel prmy2 type disk;
allocate auxiliary channel stby1 type disk;
duplicate target database for standby from active database
spfile
parameter_value_convert 'BDDIPDC','BDDIPDRS'
set db_file_name_convert='/BDDIPDC/','/BDDIPDRS/'
set log_file_name_convert='/BDDIPDC/','/BDDIPDRS/'
set 'db_unique_name'='BDDIPDRS'
set control_files='+DATA/BDDIPDRS/control01.ctl','+DATA/BDDIPDRS/control02.ctl'
set db_recovery_file_dest='+RECOVERY/BDDIPDRS'
set DB_RECOVERY_FILE_DEST_SIZE='500G'
nofilenamecheck;
}
So you have one primary and one physical standby database. Now you want to add another standby database in your Data Guard environment. And you issued following command but it failed with
RMAN-20242
rman target sys/sys@bddipdc auxiliary sys/sys@bddipdrs
DUPLICATE TARGET DATABASE FOR STANDBY FROM ACTIVE DATABASE DORECOVER SPFILE
SET db_unique_name = 'BDDIPDRS'
SET FAL_CLIENT = 'BDDIPDRS'
SET FAL_SERVER = 'BDDIPDC'
set log_archive_dest_1 = 'LOCATION=USE_DB_RECOVERY_FILE_DEST valid_for=(all_logfiles,all_roles) db_unique_name=x11106_stby2'
set log_archive_dest_2 = 'db_unique_name=BDDIPDC SERVICE=BDDIPDC valid_for=(online_logfile, primary_role) REOPEN=60 OPTIONAL LGWR SYNC AFFIRM'
CONTROL_FILES='+DATA/BDDIPDRS/control01.dbf'
set DB_FILE_NAME_CONVERT='/BDDIPDC/','/BDDIPDRS/'
set LOG_FILE_NAME_CONVERT='/BDDIPDC/','/BDDIPDRS/'
NOFILENAMECHECK;
Following is the error output,
RMAN-8162: executing Memory Script

RMAN-3090: Starting backup at 01-NOV-10
RMAN-12016: using channel ORA_DISK_1
RMAN-571: ===========================================================
RMAN-569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-571: ===========================================================
RMAN-3002: failure of Duplicate Db command at 11/01/2010 04:26:39
RMAN-3015: error occurred in stored script Memory Script
RMAN-20242: specification does not match any archived log in the recovery
catalog
Cause of the Problem
The problem happened due to oracle bug. If the source database has standby destination, then while copying archived logs from the source database the bug will fire. Oracle named this bug as Bug 6603587.

Solution of the Problem
Solution 01: This bug is fixed in Oracle 11.2 and in patchset 11.1.0.7. So upgrade your current oracle software is a solution.

Solution 02: For DUPLICATE DATABASE FOR STANDBY FROM ACTIVE, simply avoid using the DORECOVER clause. Logs will ship naturally as part of the standby log shipping.

Solution 03: For DUPLICATE TARGET DATABASE ... FROM ACTIVE, defer all standby destinations prior to executing the command. For example, issue
alter system set LOG_ARCHIVE_DEST_STATE_2=DEFER scope=both sid='*';
where LOG_ARCHIVE_DEST_STATE_2 is standby destination.
Posted by Arju at 12:50 AM 0 comments http://img1.blogblog.com/img/icon18_email.gif Labels: Data GuardRMAN
Reactions: 
Saturday, July 31, 2010
Problem Description
While duplicating database using RMAN it fails with ORA-01092: "ORACLE instance terminated. Disconnection forced" and ora-03113 error reported.
RMAN-00601: fatal error in recovery manager
RMAN-03004: fatal error during execution of command
RMAN-10039: error encountered while polling for RPC completion on channel clone_default
RMAN-10006: error running SQL statement: select act from x$ksusex where sid=:1 and serial=:2
RMAN-10002: ORACLE error: ORA-03113: end-of-file on communication channel
RMAN-03002: failure of Duplicate Db command at 02/30/2010 01:21:18
RMAN-03015: error occurred in stored script Memory Script
RMAN-06136: ORACLE error from auxiliary database:
ORA-01092: ORACLE instance terminated. Disconnection forced
If we look for the auxiliary database alert.log file then it shows that Undo Tablespace 'UNDOTBS1' does not exist or of wrong type.

Cause of the Problem
The UNDO_TABLESPACE parameter of auxiliary database is not same as target database. In the source database undo tablespace is set to UNDOTBS2 while in target database undo tablespace is set to UNDOTBS1. The undo tablespace specified in the auxiliary database parameter file does not exist.

Solution of the Problem
Make sure that the undo tablespace for auxiliary database is same as of target database.
Edit the init.ora or alter the UNDO_TABLESPACE parameter of the auxiliary instance and set
undo_tablespace = UNDOTBS2
- restart the auxiliary instance in nomount mode and repeat the duplicate database command.
Posted by Arju at 8:12 AM 0 comments http://img1.blogblog.com/img/icon18_email.gif Labels: RMAN
Reactions: 
Problem Description
Duplicating database using RMAN fails with following errors.
channel channel1: starting piece 1 at 29-JUL-2010 03:49:19
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-00601: fatal error in recovery manager
RMAN-03004: fatal error during execution of command
RMAN-10039: error encountered while polling for RPC completion on channel channel1
RMAN-10006: error running SQL statement: select action from gv$session where sid=:1 and serial#=:2 and inst_id=:3
RMAN-10002: ORACLE error: ORA-3114: not connected to ORACLE
RMAN-10041: Could not re-create polling channel context following failure.
RMAN-10024: error setting up for rpc polling
RMAN-10005: error opening cursor
RMAN-10002: ORACLE error: ORA-3114: not connected to ORACLE
RMAN-03002: failure of Duplicate Db command at 02/23/2007 11:11:50
RMAN-03015: error occurred in stored script Memory Script
RMAN-06136: ORACLE error from auxiliary database: ORA-1092: ORACLE instance
terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-39700: database must be opened with UPGRADE option

Cause of the Problem
The errors occurred because the Oracle versions for both the target database home and the auxiliary database home are not same.

Solution of the Problem
The solution is to upgrade the auxiliary database server binaries to match the target database server. Also if there is one off patches that exist on the target database, you also need to apply on the auxiliary database in order to make the duplication work.
Posted by Arju at 6:40 AM 0 comments http://img1.blogblog.com/img/icon18_email.gif Labels: RMAN
Reactions: 
Thursday, June 10, 2010
Problem Description
Whenever I run "crosscheck archivelog all" it failed with message "validation failed for archived log" like below.
RMAN> crosscheck archivelog all;

released channel: ORA_DISK_1
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=323 devtype=DISK
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_10/o1_mf_1_3_5bkc1o5q_.arc recid=6 stamp=697208725
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_10/o1_mf_1_4_5bkd60x4_.arc recid=7 stamp=697209883
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_10/o1_mf_1_5_5bkd78q9_.arc recid=8 stamp=697209922
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_10/o1_mf_1_6_5bkf2rwx_.arc recid=9 stamp=697210802
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_10/o1_mf_1_7_5blbdsqs_.arc recid=10 stamp=697240820
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_10/o1_mf_1_8_5blc2tvx_.arc recid=11 stamp=697241524
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_11/o1_mf_1_9_5bm1t7hk_.arc recid=12 stamp=697264802
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_11/o1_mf_1_10_5bmqwzr6_.arc recid=13 stamp=697287417
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_11/o1_mf_1_11_5bnjqg4n_.arc recid=14 stamp=697312840
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_11/o1_mf_1_12_5bnytoy5_.arc recid=15 stamp=697327280
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_12/o1_mf_1_13_5boplo7l_.arc recid=16 stamp=697351600
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_12/o1_mf_1_14_5bp6vv6w_.arc recid=17 stamp=697368277
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_12/o1_mf_1_15_5bpdtlfn_.arc recid=18 stamp=697374380
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_13/o1_mf_1_16_5br3nh19_.arc recid=19 stamp=697430505
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_13/o1_mf_1_17_5brvlghg_.arc recid=20 stamp=697455015
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_14/o1_mf_1_18_5btzdcv5_.arc recid=21 stamp=697524454
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_14/o1_mf_1_19_5bw0drf0_.arc recid=22 stamp=697558259
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_14/o1_mf_1_20_5bwvyksr_.arc recid=23 stamp=697586476
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_15/o1_mf_1_21_5bxmsqql_.arc recid=24 stamp=697610898
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_15/o1_mf_1_22_5bz0h5on_.arc recid=25 stamp=697656639
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_15/o1_mf_1_23_5bzjbvg2_.arc recid=26 stamp=697672884
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_16/o1_mf_1_24_5c0lgk9r_.arc recid=27 stamp=697707819
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_16/o1_mf_1_25_5c24o21l_.arc recid=28 stamp=697759227
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_17/o1_mf_1_26_5c30s9py_.arc recid=29 stamp=697788035
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_17/o1_mf_1_27_5c4s0wyy_.arc recid=30 stamp=697845623
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_18/o1_mf_1_28_5c5k0jrj_.arc recid=31 stamp=697870186
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_18/o1_mf_1_29_5c6z1g0p_.arc recid=32 stamp=697917319
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_18/o1_mf_1_30_5c7mqfl8_.arc recid=33 stamp=697938502
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_19/o1_mf_1_31_5c89jkhw_.arc recid=34 stamp=697960811
validation failed for archived log
archive log filename=/u01/app/oracle/archivelog/2009_09_19/o1_mf_1_32_5c9ggk5p_.arc recid=35 stamp=697998634
validation succeeded for archived log
archive log filename=/u01/app/oracle/archivelog/2009_12_02/o1_mf_1_343_5kdfc1sv_.arc recid=36 stamp=704560058
validation succeeded for archived log
archive log filename=/u01/app/oracle/archivelog/2009_12_02/o1_mf_1_344_5kdfy3pd_.arc recid=37 stamp=704560667
validation succeeded for archived log
archive log filename=/u05/archive/%t_%s_%r.arc1_345_697204983.dbf recid=38 stamp=704560901
validation succeeded for archived log
archive log filename=/u05/archive/1_346_697204983.dbf recid=39 stamp=704561258
validation succeeded for archived log
archive log filename=/u05/archive/1_347_697204983.dbf recid=40 stamp=704561835
validation succeeded for archived log
archive log filename=/u05/archive/1_348_697204983.dbf recid=41 stamp=704561992
validation succeeded for archived log
archive log filename=/u05/archive/1_349_697204983.dbf recid=42 stamp=704562987
validation succeeded for archived log
archive log filename=/u05/archive/1_350_697204983.dbf recid=43 stamp=704562987
validation succeeded for archived log
archive log filename=/u05/archive/1_351_697204983.dbf recid=44 stamp=704562992
validation succeeded for archived log
archive log filename=/u05/archive/1_352_697204983.dbf recid=45 stamp=704563257
validation succeeded for archived log
archive log filename=/u05/archive/1_353_697204983.dbf recid=46 stamp=704563321
validation succeeded for archived log
archive log filename=/u05/archive/1_354_697204983.dbf recid=47 stamp=704563408
validation succeeded for archived log
archive log filename=/u05/archive/1_355_697204983.dbf recid=48 stamp=704563434
validation succeeded for archived log
archive log filename=/u05/archive/1_356_697204983.dbf recid=49 stamp=704584836
validation succeeded for archived log
archive log filename=/u05/archive/1_357_697204983.dbf recid=50 stamp=704586204
validation succeeded for archived log
archive log filename=/u05/archive/1_358_697204983.dbf recid=51 stamp=704611511
validation succeeded for archived log
archive log filename=/u05/archive/1_359_697204983.dbf recid=52 stamp=704648675
validation succeeded for archived log
archive log filename=/u05/archive/1_360_697204983.dbf recid=53 stamp=704650176
validation succeeded for archived log
archive log filename=/u05/archive/1_361_697204983.dbf recid=54 stamp=704650675
validation succeeded for archived log
archive log filename=/u05/archive/1_362_697204983.dbf recid=55 stamp=704671244
validation succeeded for archived log
archive log filename=/u05/archive/1_363_697204983.dbf recid=56 stamp=704692487
validation succeeded for archived log
archive log filename=/u05/archive/1_364_697204983.dbf recid=57 stamp=704708839
validation succeeded for archived log
archive log filename=/u05/archive/1_365_697204983.dbf recid=59 stamp=704736263
validation succeeded for archived log
archive log filename=/u05/archive/1_366_697204983.dbf recid=58 stamp=704736262
validation succeeded for archived log
archive log filename=/u05/archive/1_367_697204983.dbf recid=60 stamp=704736266
validation succeeded for archived log
archive log filename=/u05/archive/1_368_697204983.dbf recid=61 stamp=704737398
Crosschecked 56 objects
Cause of the Problem
The problem happened because archive log file destination was changed or someone deleted the archive log files manually using OS commands.

Solution of the Problem
The solution is to delete the expired archive log files. So do following,
$ rman
RMAN> connect target /
RMAN> crosscheck archivelog all;
RMAN> delete expired archivelog all;
Posted by Arju at 12:49 AM 1 comments http://img1.blogblog.com/img/icon18_email.gif Labels: RMAN
Reactions: 
Tuesday, March 2, 2010
About RMAN Environment
If you say about RMAN Environment, then there comes the following components.
1) RMAN executable
2) Recovery catalog database
3) Recovery catalog schema in the recovery catalog database
4) Target database
5) Auxiliary database (used for duplicate, duplicate for standby or tablespace point-in-time recovery)

Each component has a release number. Oracle uses up to 5 digits to indicate a version of the release, but only up to 4 are significant for RMAN compatibility purposes. For example, to check the release number of RMAN executable just issue, rman on command prompt like below.
E:\Documents and Settings\Arju>rman

Recovery Manager: Release 10.2.0.1.0 - Production on Tue Mar 2 16:16:59 2010

Copyright (c) 1982, 2005, Oracle. All rights reserved.


Here we see the version is 10.2.0.1.0

Determine the Recovery Catalog Schema Version

Use SQL*Plus to connect to the recovery catalog database as the catalog owner. For example, enter:

$ sqlplus rman@catdb
Query the rcver catalog table. For example, run this query:

SQL> SELECT * FROM rcver;

VERSION
------------
09.00.01.00
10.02.01.00
11.01.00.03
Note that, if multiple versions are listed, then the last row is the current version, and the rows before it are prior versions. In the preceding example, the current recovery catalog schema version is 11.1 and the previous version was 10.2.

Note that for releases 10.2 and later, the last two digits in the rcver output indicate patch level. For earlier releases, they are always zeros.

RMAN Compatibility Matrix
In general, the rules of RMAN compatibility are as follows:

1) You can create an 8.x or 9.x recovery catalog schema in any Oracle database release 8.1.x (or higher), and a 10.0.1 (or higher) recovery catalog schema in any Oracle database release 9.0.1 (or higher).

2) The version of an auxiliary database instance must be equal to the version of the RMAN client.

3) Any release of Oracle database can restore backup sets and copies created by any prior release back to Oracle8i.

4) The RMAN executable version should be the same as the target database. There are some exceptions regarding this rule.

5) The RMAN catalog schema version must be greater than or equal to the RMAN executable.

6) The RMAN catalog is backwards compatible with target databases from earlier releases.

7) While backing up an Oracle Database 10g or later database with the Oracle9i RMAN client, you cannot include a control file that was created using COMPATIBLE=10.0.0 in a datafile backup set. The workaround is to turn control file autobackup ON.

RMAN Compatibility table is shown below.







Target/Auxiliary
Database
RMAN Executable
Catalog Database
Catalog Schema
8.0.6
8.0.6
>=8.1.7
>=8.0.6
8.1.7
8.0.6.1
>=8.1.7
>=8.1.7
8.1.7
8.1.7
>=8.1.7
>=RMAN executable
8.1.7.4
8.1.7.4
>=8.1.7
8.1.7.4
8.1.7.4
8.1.7.4
>=8.1.7
>=9.0.1.4
9.0.1
9.0.1
>=8.1.7
>= RMAN executable
9.2.0
>=9.0.1.3 and <= Target database
>=8.1.7
>= RMAN executable
10.1.0
>=9.0.1.3 and <= Target database
>=9.0.1
>= RMAN executable
10.2.0
>=9.0.1.3 and <= target database executable
>=9.0.1
>= RMAN executable
11.1.0
>=9.0.1.3 and <= target database executable
>=9.0.1
>= RMAN executable
11.2.0
>=9.0.1.3 and <= target database executable
>=9.0.1
>= RMAN executable
Reactions: 
Wednesday, February 17, 2010
Problem Description
RMAN backup completed successfully but backup logs shows warning message as,
RMAN-06900: WARNING: unable to generate V$RMAN_STATUS or V$RMAN_OUTPUT row
RMAN-06901: WARNING: disabling update of the V$RMAN_STATUS and V$RMAN_OUTPUT rows ORACLE error from target database:
ORA-19921: maximum number of 64 rows exceeded

A variant version of warning message is like below,

connected to target database: PROD1 (DBID=3378321056)
RMAN-06900: WARNING: unable to generate V$RMAN_STATUS or V$RMAN_OUTPUT row
RMAN-06901: WARNING: disabling update of the V$RMAN_STATUS and V$RMAN_OUTPUT rows
ORACLE error from target database:
ORA-19921: maximum number of 128 rows exceeded

Cause of the Problem
The RMAN-06900 RMAN-06901 ORA-19921 error stack indicates oracle RMAN bug. ORA-19921: maximum number of 64 rows exceeded is oracle bug 4659734 and ORA-19921: MAXIMUM NUMBER OF 128 ROWS EXCEEDED is oracle Bug 8264365. This bug fires when the RMAN output is too huge because it couldn't log the output in v$rman_output.

Solution of the Problem
If your RMAN goes well and you only see warning message in your RMAN log then simply you can ignore these warnings. The bug "ORA-19921: maximum number of 64 rows exceeded" is fixed in Oracle 10.2.0.4 version and in 11g. So to solve it upgrade your oracle version or apply oracle patch. If you already have oracle version 10.2.0.4 + and hit this bug then in fact you should do nothing and just to wait for oracle to resolve the issue.

Related Documents
http://arjudba.blogspot.com/2010/01/rman-00571-rman-00569-rman-00571-rman.html
http://arjudba.blogspot.com/2010/01/rman-06900-rman-06901-ora-19921-maximum.html

Reactions: 
Problem Description
RMAN backup fails with ORA-00204 ORA-00202 ORA-27071 HP-UX Error: 9: Bad file number as below.

Recovery Manager: Release 11.1.0.7.0 - Production on Fri Jan 29 00:01:15 2010
Copyright (c) 1982, 2007, Oracle. All rights reserved.
RMAN-06900: WARNING: unable to generate V$RMAN_STATUS or V$RMAN_OUTPUT row
RMAN-06901: WARNING: disabling update of the V$RMAN_STATUS and V$RMAN_OUTPUT rows
ORACLE error from target database:
ORA-00204: error in reading (block 1, # blocks 1) of control file
ORA-00202: control file: '/SID/oradata/APEXP/control01.ctl'
ORA-27071: unable to seek to desired position in file
HP-UX Error: 9: Bad file number

An another version of error message appear while starting RMAN is,
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-00554: initialization of internal recovery manager package failed
RMAN-06003: ORACLE error from target database:
ORA-00204: error in reading (block 3, # blocks 1) of control file
ORA-00202: control file: '/SID/oradata/APEXP/control01.ctl'
ORA-27071: unable to seek to desired position in file
HP-UX Error: 9: Bad file number Additional information: 2

Cause of the Problem
The RMAN-06900, RMAN-06901 error occurs when the RMAN couldn't log the output in $rman_output. The actual investigation of the error comes in the associated additional error messages. If you look for additional error message we will see ORA-27071, HP-UX Error: 9: Bad file number. The word 'HP-UX Error: 9: Bad file number' indicates that error is coming from operating system. The OS user who is invoking RMAN don't have OS permission on the control file and hence HP-UX generates bad file number error.

Solution of the Problem
The user who is invoking RMAN don't have write permission on the controlfile. You must need to change permission from operating system. You can do so by issuing,
$chmod 666 /SID/oradata/APEXP/control01.ctl

After you change the permission now try to invoke RMAN command again.
Related Documents:
Reactions: 
Saturday, January 30, 2010
Problem Description
RMAN backups has been successfully completed but from the backup logs the following error have been generated.
Recovery Manager: Release 11.1.0.7.0 - Production on Fri Jan 29 00:01:15 2010

Copyright (c) 1982, 2007, Oracle. All rights reserved.

RMAN-06900: WARNING: unable to generate V$RMAN_STATUS or V$RMAN_OUTPUT row
RMAN-06901: WARNING: disabling update of the V$RMAN_STATUS and V$RMAN_OUTPUT rows
ORACLE error from target database:
ORA-19921: maximum number of 64 rows exceeded

A variation of the above error is,
Recovery Manager: Release 11.1.0.7.0 - Production on Fri Jan 29 00:01:15 2010

Copyright (c) 1982, 2007, Oracle. All rights reserved.

RMAN-06900: WARNING: unable to generate V$RMAN_STATUS or V$RMAN_OUTPUT row
RMAN-06901: WARNING: disabling update of the V$RMAN_STATUS and V$RMAN_OUTPUT rows
ORACLE error from target database:
ORA-19921: maximum number of 128 rows exceeded

Analysis And Solution of the Problem
As soon as you get error message RMAN-06900 and RMAN-06901 you immediately look for associated error messages. The associated error message should tell you more information and you need to look for those messages in order to solve this error. For example here we are getting additional error "ORA-19921: maximum number of 128 rows exceeded". So our solution will lie on ORA-19921.

RMAN gives a warning message of RMAN-6900, RMAN-6901 ORA-19921 when the output is too huge and oracle is not able to write the log output into v$rman_output. There are several reasons when oracle will not be able to write the log output into V$RMAN_OUTPUT. For example, one of your control file becomes corrupted, hence oracle is unable to write the the log output to RMAN view/table. The another major reasons cause this problem to happen is due to oracle bug.

You get oracle error "ORA-19921: maximum number of 64 rows exceeded" due to oracle Bug 465973.
You get oracle error "ORA-19921: MAXIMUM NUMBER OF 128 ROWS EXCEEDED" due to oracle Bug 8264365.

The bug 465973 is fixed in Oracle 10.2.0.4 and Oracle 11G. So if you see oracle only gives warning message and backup successfully done then you can simply ignore error messages or upgrade oracle or apply patch where bug is fixed.

However if you see your backup is not done due to these errors then immediately check for additional messages. If it happened due to controlfiles then take care of those errors.

Related Documents: