When your RMAN duplicate fails at 94 percent on a 110TB ExaCS database

Two days is what an active RMAN duplicate over the WAN costs on a 110TB database, and two days is exactly how far in we were when it died at roughly 94 percent complete. Oracle 19.25, ExaCS, building a standby from a live primary across a wide area network. The error stack was short and arrived at the worst possible moment.

RMAN-03002: failure of Duplicate Db command
RMAN-05501: aborting duplication of target database
RMAN-03015: error occurred in stored script Memory Script
ORA-19875: multisection restore or copy not complete for file <datafile_name>

The instinct in the room was immediate and close to unanimous. Start over. Clear the destination, rerun the duplicate, wait another two days. I understand that instinct, because a failed duplicate feels like a poisoned duplicate and the clean slate feels like the safe move. But look at what that decision actually costs before you commit to it. Two more days of WAN transfer for 110TB of data. Two more days without a standby. Two more days of a maintenance window that was never scoped for a second attempt. And no guarantee at all that it does not fail at 94 percent again for the same reason, because at that moment nobody in the room could explain what the reason was. Restarting a job you do not yet understand is not a recovery plan. It is a coin flip that takes forty eight hours to land.

What ORA-19875 is telling you

ORA-19875 fires when a multisection restore or copy operation does not complete cleanly. On a database this size RMAN does not move a large datafile as a single unit. It breaks that file into sections and copies the sections in parallel across the available channels, which is the only sane way to move a multi terabyte file over a link with real latency. What the error is reporting is that for one file, one or more of those sections did not finish. Oracle’s internal accounting for that file is incomplete, so it refuses to call the file done.

Read that carefully, because the wording matters more than it first appears. The error is not telling you the duplicate is corrupt. It is not telling you the destination is unusable. It is telling you that Oracle’s checkpoint for a specific file is incomplete. That is a statement about bookkeeping on one file, not a verdict on the ninety four percent of work that already landed on disk. Everything that completed is still sitting on the standby destination, whole and accounted for. The question worth asking is not whether the job failed. It obviously did. The question is whether Oracle can be made to recognize what it already finished.

The recovery path

The sequence that worked was deliberate about not disturbing anything that had already succeeded. First, create a pfile from the standby spfile before touching a single other thing, because everything after this step is easier to undo when you can put the parameters back exactly as they were. Then shut down the standby instance cleanly. Remove the spfile that was created as part of the duplicate process, and let it go without ceremony, because Oracle will regenerate it. Start the instance nomount using the pfile you just created. Strip any spfile related entries out of the RMAN duplicate command file so the rerun is not fighting the state you just cleared. Then rerun the duplicate.

What happens next is the part worth watching. Rather than pulling 110TB from the primary again, RMAN inspects the destination, recognizes the datafiles that were already duplicated successfully, and uses them where they sit.

Using previous duplicated file +DATA/STBY/DATAFILE/system.301.1174382915 for datafile 1
Using previous duplicated file +DATA/STBY/DATAFILE/sysaux.302.1174383077 for datafile 2
Using previous duplicated file +DATA/STBY/DATAFILE/undotbs1.303.1174383211 for datafile 3
...
Using previous duplicated file +DATA/STBY/DATAFILE/appdata.887.1174561903 for datafile 612

Those “Using previous duplicated file” lines are the confirmation you are looking for. They mean RMAN is reconciling rather than restarting, and that the only work left on the wire is the sections that genuinely did not finish. What was framed as another two day transfer became a short one.

What this tells you about RMAN duplicate state

Oracle’s multisection duplicate tracks progress at the file level, not just at the job level. When you rerun a failed duplicate against the same target, the same catalog, and the same channel configuration, RMAN has enough information to reconcile what is already on disk against what still needs to be copied. That reconciliation is what saves the restart.

I want to be precise about the status of that claim. This is not a documented guarantee. It is behavior that held in this case, on 19.25, on ExaCS, with the configuration unchanged between the failed run and the rerun. The prudent interpretation is that it is worth attempting before you commit to a full restart, and that the restart remains your fallback if the rerun does not pick up prior state. You are risking the length of one rerun attempt against the certainty of two more days. That is a favorable trade almost every time.

One practical note before you try it. Verify that the datafiles already sitting on the standby destination are intact, and confirm you have enough space for whatever sections still need to be copied. If either condition is not met the rerun will fail fast, which is its own form of useful information, but it is better to know going in than to find out an hour later.

On databases at this scale, knowing which failures are recoverable without a restart is the difference between a two hour recovery and a two day one.