Building a physical standby from a standby when the primary is unreachable

The documented path for adding a physical standby is unambiguous. Connect RMAN with TARGET on the primary and AUXILIARY on the new host, run DUPLICATE TARGET DATABASE FOR STANDBY FROM ACTIVE DATABASE, and let Data Guard Broker manage the resulting configuration. That path assumes the primary can reach the new standby host over the network.

In this case it could not. The new standby host, use1pcrmor002.adcorp.example.com, had connectivity only to the existing standby host. The primary host, hqdc1pcrmor001.adcorp.example.com, had no route to the new host at all. Redo would only ever need to travel from the existing standby, CRMPROD_STBY, to the new standby, CRMPROD_S, and never from the primary, CRMPROD_PRIM. That single constraint is the premise for everything that follows, and it shapes the RMAN TARGET connection, the incarnation problem, and whether the broker can manage the new leg at all.

Why the standard path is unavailable

Current Oracle documentation states the restriction plainly: you cannot connect RMAN to a standby database and use DUPLICATE … FOR STANDBY to create an additional standby. TARGET must be the original primary. This is a live restriction in current releases, not a stale note carried forward from older documentation. With no route to CRMPROD_PRIM, TARGET could only be the existing standby, and that is exactly the configuration the restriction forbids.

Three approaches were tried against that restriction, in order.

Opening the source standby read-only

The reasoning was that opening the standby might change how RMAN evaluated it as a TARGET. It did not. With managed recovery cancelled and CRMPROD_STBY open read-only, RMAN continued to enforce the restriction regardless of open mode. The DATABASE_ROLE was still PHYSICAL STANDBY, and read-only open does nothing to change that. The duplicate was refused on the same grounds as before.

Converting the source standby to a snapshot standby

This one moves the role label, and it lets the duplicate proceed, which is what makes it dangerous. Converting CRMPROD_STBY to a snapshot standby genuinely changes DATABASE_ROLE away from PHYSICAL STANDBY to SNAPSHOT STANDBY, so RMAN’s role check passes and the duplicate runs to completion.

The problem is what a snapshot standby is underneath. Converting to snapshot standby opens the database read/write, and that open requires an implicit RESETLOGS, which creates a new incarnation branch. Any duplicate built during the snapshot window inherits datafiles stamped with that branch’s RESETLOGS_SCN. When the source is later converted back to physical standby, the snapshot is discarded by flashing the branch away, and that branch ceases to exist. The new standby, built from datafiles that belong to it, is now orphaned.

This was confirmed empirically. The new standby surfaced ORA-19909, datafile belongs to an orphan incarnation, on recovery. There is no RESET DATABASE TO INCARNATION path out of it, because the orphaned branch was never recorded as an ancestor incarnation anywhere in the primary’s lineage. It only ever existed transiently on the source standby, for the duration of the snapshot, and it left no trace in CRMPROD_PRIM’s incarnation history. There was nothing to reset to. The new standby had to be discarded and rebuilt.

Cancelling recovery and leaving the source mounted, never opened

Managed recovery was cancelled, the source was left mounted, and it was never opened in any mode. The duplicate ran, completed, and held up under inspection. On the new standby, v$database_incarnation showed a clean, unbroken lineage matching the primary, with no orphan branch anywhere in it.

The working theory is that RMAN’s restriction, and the incarnation damage seen with the snapshot, are both keyed to whether the source has been activated rather than to its literal role label. A snapshot standby has been opened read/write and has resetlogs, so it has branched. A standby that is only ever mounted has not been activated, has not run resetlogs, and has not branched, so the datafiles it hands to the duplicate carry the primary’s incarnation unmodified. This is offered as a working theory consistent with the observed results, not a documented mechanism. What is not in question is the outcome: the mounted, never-opened source produced a standby with intact lineage, and the snapshot source did not.

The rest of this post is the full build that followed from that third approach.

The build, phase by phase

Phase 1: prepare the new standby host

The new standby must register statically, because the instance will not be up to self-register when RMAN first connects to the auxiliary.

LISTENER =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCP)(HOST = use1pcrmor002.adcorp.example.com)(PORT = 1521))
      (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1521))
    )
  )

SID_LIST_LISTENER =
  (SID_LIST =
    (SID_DESC =
      (GLOBAL_DBNAME = CRMPROD_S)
      (ORACLE_HOME = /oracle/product/19.0.0.0/dbhome_1)
      (SID_NAME = CRMPROD_S)
    )
    (SID_DESC =
      (GLOBAL_DBNAME = CRMPROD_S_DGMGRL)
      (ORACLE_HOME = /oracle/product/19.0.0.0/dbhome_1)
      (SID_NAME = CRMPROD_S)
    )
  )

lsnrctl start
lsnrctl status

TNS resolution. The existing standby needs an alias for the new standby, and the new standby needs aliases for both the existing standby and the primary, even though the primary path is never used for transport.

CRMPROD_S =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = use1pcrmor002.adcorp.example.com)(PORT = 1521))
    (CONNECT_DATA = (SERVER = DEDICATED)(SERVICE_NAME = CRMPROD_S))
  )

The password file must match the source database’s SYS credentials, because RMAN active duplication authenticates the auxiliary with the same password.

export ORACLE_SID=CRMPROD_S
cd $ORACLE_HOME/dbs
orapwd file=orapw$ORACLE_SID password='<same as source>' entries=15 force=y

Minimal pfile, directories, and a nomount start.

cat > initCRMPROD_S.ora << 'EOF'
db_name='CRMPROD'
service_names='CRMPROD_S'
memory_target=4G
EOF

mkdir -p /oracle/admin/CRMPROD_S/adump
mkdir -p /mnt/crmproddb/logs_01/fast_recovery_area
mkdir -p /mnt/crmproddb/data_01/CRMPROD_S
mkdir -p /mnt/crmproddb/undo_01/CRMPROD_S
mkdir -p /mnt/crmproddb/data_01/oractrl

STARTUP NOMOUNT PFILE='/oracle/product/19.0.0.0/dbhome_1/dbs/initCRMPROD_S.ora';

Phase 2: put the source standby into the right state

Cancel managed recovery on the existing standby without touching its open mode. Do not open it. This is the state that produced the clean incarnation above.

-- broker-managed source:
DGMGRL> EDIT DATABASE CRMPROD_STBY SET STATE = 'APPLY-OFF';

-- non-broker source:
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;

Phase 3: run the duplicate from the existing standby

TARGET is the existing standby, AUXILIARY is the new standby.

run {
allocate channel prm1 type disk;
allocate auxiliary channel stb1 type disk;
allocate auxiliary channel stb2 type disk;
allocate auxiliary channel stb3 type disk;
allocate auxiliary channel stb4 type disk;
allocate auxiliary channel stb5 type disk;
allocate auxiliary channel stb6 type disk;
allocate auxiliary channel stb7 type disk;
allocate auxiliary channel stb8 type disk;
allocate auxiliary channel stb9 type disk;
allocate auxiliary channel stb10 type disk;
  duplicate target database
    for standby
    from active database
    dorecover
    nofilenamecheck
    spfile
      parameter_value_convert 'CRMPROD_STBY','CRMPROD_S'
      set db_name='CRMPROD'
      set db_unique_name='CRMPROD_S'
      set service_names='CRMPROD_S'
      set db_recovery_file_dest='/mnt/crmproddb/logs_01/fast_recovery_area'
      set db_file_name_convert='/oracle/oradata/crmprod/','/mnt/crmproddb/data_01/CRMPROD_S/','/oracle/oraundo/','/mnt/crmproddb/undo_01/CRMPROD_S/'
      set log_file_name_convert='/oracle/oraredoa/','/mnt/crmproddb/redoa_01/CRMPROD_S/','/oracle/oraredob/','/mnt/crmproddb/redob_01/CRMPROD_S/'
      set db_recovery_file_dest_size='100G'
      set control_files='/mnt/crmproddb/data_01/oractrl/control01.ctl','/mnt/crmproddb/data_01/CRMPROD_S/control02.ctl','/mnt/crmproddb/logs_01/fast_recovery_area/control03.ctl'
      set audit_file_dest='/oracle/admin/CRMPROD_S/adump'
      set diagnostic_dest='/oracle'
      set log_archive_config='DG_CONFIG=(CRMPROD_PRIM,CRMPROD_STBY,CRMPROD_S)'
      set log_archive_dest_1='LOCATION=USE_DB_RECOVERY_FILE_DEST VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=CRMPROD_S';
}

nohup rman target sys/"<pwd>"@CRMPROD_STBY auxiliary sys/"<pwd>"@CRMPROD_S \
  cmdfile stby.cmd log stby.log &

Three details in this run matter later. db_file_name_convert and log_file_name_convert take paired old-path and new-path strings, and a convert pair that does not exactly match the directory tree created in Phase 1, including a stray extra path segment, produces converted paths that do not land where intended. service_names has to be explicitly SET, or the duplicate inherits the source standby’s service name rather than the new standby’s. standby_file_management is deliberately left unset here and handled explicitly during the redo log rebuild in Phase 5. There is no FOR FARSYNC and no dg_broker_start, because the broker does not manage this leg at all.

Phase 4: resume recovery on the source standby

Once the duplicate completes, restart managed recovery on the existing standby so it closes the redo gap that built up while it was paused. The size of that gap is proportional to how long the duplicate ran.

DGMGRL> EDIT DATABASE CRMPROD_STBY SET STATE = 'APPLY-ON';

Phase 5: rebuild the redo log groups

RMAN’s DUPLICATE reference documentation is specific about which files are re-created during active duplication. Online redo logs are always re-created. Standby redo logs are re-created only, in the documentation’s own wording, when FOR STANDBY is specified and they are defined on the primary database. TARGET here was the existing standby, not the primary, so that condition was never met. The new standby’s controlfile ended up with standby redo log group entries pointing at inherited, unconverted paths with no backing files ever created. The online redo logs were created, being unconditional, but at the old unconverted paths as well, because the file-name-convert pattern had not taken effect for either log type in that run.

Set manual file management first so Oracle does not try to auto-manage files mid-rebuild.

ALTER SYSTEM SET standby_file_management='MANUAL';

Starting state, showing online groups 1 to 3 and standby groups 4 to 7, all at old paths.

SELECT group#, member, type, status FROM v$logfile ORDER BY 1;

GROUP#  MEMBER                        TYPE     STATUS
1       /oracle/oraredoa/redo1a.log   ONLINE
1       /oracle/oraredob/redo1b.log   ONLINE
2       /oracle/oraredoa/redo2a.log   ONLINE
2       /oracle/oraredob/redo2b.log   ONLINE
3       /oracle/oraredoa/redo3a.log   ONLINE
3       /oracle/oraredob/redo3b.log   ONLINE
4       /oracle/oraredoa/stby04.log   STANDBY
4       /oracle/oraredob/stby04.log   STANDBY
5       /oracle/oraredoa/stby05.log   STANDBY
5       /oracle/oraredob/stby05.log   STANDBY
6       /oracle/oraredoa/stby06.log   STANDBY
6       /oracle/oraredob/stby06.log   STANDBY
7       /oracle/oraredoa/stby07.log   STANDBY
7       /oracle/oraredob/stby07.log   STANDBY

The primary’s online redo logs were 3,221,225,984 bytes, confirmed on the primary before recreating anything. This exact byte count matters, and the reason is in the sizing note below.

SELECT group#, bytes FROM v$log;   -- 3221225984

Online groups 1 to 3 existed as real files, so a plain drop and add worked immediately. They are recreated at the exact byte count rather than a rounded unit.

ALTER DATABASE DROP LOGFILE GROUP 1;
ALTER DATABASE ADD LOGFILE GROUP 1
  ('/mnt/crmproddb/redoa_01/CRMPROD_S/redo1a.log',
   '/mnt/crmproddb/redob_01/CRMPROD_S/redo1b.log') SIZE 3221225984;
-- repeat for groups 2 and 3

Standby groups 4 to 7 did not exist as files, so the first drop failed on the missing file.

SQL> ALTER DATABASE DROP STANDBY LOGFILE GROUP 4;
ORA-00313: open failed for members of log group 4 of thread 1
ORA-00312: online log 4 thread 1: '/oracle/oraredob/stby04.log'
ORA-27037: unable to obtain file status ... No such file or directory

Clearing the group failed too, on a different error, but a second drop immediately after the failed clear succeeded cleanly. The reason the failed clear unblocks the subsequent drop is not confirmed. This is an observed, repeatable sequence rather than an explained one.

ALTER DATABASE DROP STANDBY LOGFILE GROUP 4;   -- fails, ORA-00313
ALTER DATABASE CLEAR LOGFILE GROUP 4;          -- fails, ORA-00344
ALTER DATABASE DROP STANDBY LOGFILE GROUP 4;   -- succeeds
ALTER DATABASE ADD STANDBY LOGFILE GROUP 4
  ('/mnt/crmproddb/redoa_01/CRMPROD_S/stby04.log',
   '/mnt/crmproddb/redob_01/CRMPROD_S/stby04.log') SIZE 3221225984;
-- repeat the same three-step sequence for groups 5, 6, 7

The sizing note. Standby redo logs must match the primary’s online redo log size to the exact byte, not approximately. Recreating with SIZE 3G produces logs of 3,221,225,472 bytes, 512 bytes short of the source’s 3,221,225,984, because 3G is interpreted as exactly 3 times 1024 cubed rather than as the source’s actual byte count. RFS rejects undersized standby redo logs silently. It does not error in a way that stops the database. It falls back to archived-log shipping and logs a single line, RFS: No SRLs created for T-1. Redo still moves by archived log, so nothing fails loudly, and real-time apply is lost until someone notices. Using the literal byte count from v$log, as above, is what makes RFS accept them.

Restore automatic file management and verify all seven groups show two members each under the new mount paths with nothing INVALID.

ALTER SYSTEM SET standby_file_management='AUTO';
SELECT group#, member, type, status FROM v$logfile ORDER BY type, group#;

Phase 6: configure the cascade manually

Data Guard Broker requires every database in a configuration to be reachable from every other database, not only along the redo transport path. The documentation is explicit that the connect identifier for each database must allow all other databases in the configuration to reach it. That is a full mesh requirement. Because the primary cannot reach the new standby at all, the new standby cannot be a broker-managed member, even though redo only ever needs to flow from the existing standby to the new standby and never from the primary. Transport and apply are therefore configured directly, outside the broker.

On the existing standby, the source of the cascade:

ALTER SYSTEM SET LOG_ARCHIVE_DEST_3=
  'SERVICE=CRMPROD_S ASYNC VALID_FOR=(STANDBY_LOGFILES,STANDBY_ROLE) DB_UNIQUE_NAME=CRMPROD_S'
  SCOPE=BOTH;
ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_3=ENABLE SCOPE=BOTH;

VALID_FOR=(STANDBY_LOGFILES,STANDBY_ROLE) is what makes this real-time cascading rather than log-switch-triggered. Redo lands in the new standby’s standby redo logs as the existing standby receives it, not after an archive log switch. On both the existing standby and the primary, log_archive_config lists all three databases.

ALTER SYSTEM SET log_archive_config='DG_CONFIG=(CRMPROD_PRIM,CRMPROD_STBY,CRMPROD_S)' SCOPE=BOTH;

Phase 7: start recovery on the new standby and verify

ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT;

-- on the existing standby: confirm shipping to the new standby
SELECT dest_id, status, destination, error FROM v$archive_dest_status WHERE dest_id=3;

-- on the new standby: confirm redo arriving and applying
SELECT process, status, sequence#, thread# FROM v$managed_standby;
SELECT sequence#, applied FROM v$archived_log ORDER BY sequence# DESC FETCH FIRST 5 ROWS ONLY;

Findings

RMAN DUPLICATE … FOR STANDBY requires TARGET to be the primary. This is a current restriction, and it holds regardless of the source standby’s open mode.
Opening the source standby read-only does not lift the restriction. Converting it to a snapshot standby lifts the restriction but produces an orphan incarnation (ORA-19909) with no RESET DATABASE TO INCARNATION recovery, because the branch never existed in the primary’s lineage. Leaving the source mounted and never opening it produces a standby with clean, unbroken lineage.
Data Guard Broker requires full mesh connectivity among all members. A standby the primary cannot reach cannot be broker-managed, even when redo only ever flows standby to standby. Transport was configured manually with LOG_ARCHIVE_DEST_n and VALID_FOR=(STANDBY_LOGFILES,STANDBY_ROLE).
DUPLICATE … FOR STANDBY does not create standby redo log files when TARGET is not the primary. The groups register in the controlfile at inherited paths with no files on disk, surfacing later as ORA-00313 and ORA-27037. Online redo logs are created but land at the unconverted paths when the file-name-convert did not take effect for logs in that run.
Dropping a phantom standby redo log group failed with ORA-00313, and clearing it failed with ORA-00344, but issuing the drop immediately after the failed clear succeeded. Observed and repeatable, mechanism unconfirmed.
Standby redo logs must match the primary’s online redo log size to the exact byte. SIZE 3G produced logs 512 bytes short of the source’s 3,221,225,984 bytes, and RFS silently rejected them and fell back to archived-log shipping, logging “RFS: No SRLs created for T-1” with no stopping error. Recreating at the literal byte count restored real-time apply.

Have a question about your Oracle environment?

The discovery call is free, 20 minutes, and focused entirely on your situation. If an assessment does not make sense, I will tell you that in the first five minutes.

Book a discovery call See services and pricing