Every DBA has a debug session that starts as a five minute task and turns into an afternoon. This one started with a gold image and ended with a lesson about a daemon most of us never think about until it quietly refuses to run our binaries.
The setup
I was building an Oracle 19c gold image for fleet deployment. The source home on orasrv-src-01 was already patched to the RU I wanted, so the plan was simple: capture the home, drop it on the targets, relink, done. The officially supported path for this is runInstaller with -createGoldImage, and that is where the afternoon began to go sideways.
The supported gold image command refused to run. It threw INS-32707 claiming it needed 1.2 TB of free space to capture a home that was, by any honest measurement, about 25 GB.
[FATAL] [INS-32707] The location specified for storing the gold image does not have enough space.
CAUSE: The disk space available at the specified location is less than the required 1258291 MB.
ACTION: Provide a location with sufficient disk space.
A 1.2 TB requirement for a 25 GB home is not a real number, it is a bug in the space calculation. I did not have the appetite to chase an installer sizing defect on a deadline, so I reached for the older methodology that has shipped in every Oracle home for years: clone.pl. It is deprecated in 19c, but deprecated is not the same as removed, and it was sitting right there in $ORACLE_HOME/clone/bin. Tar the home on the source, extract it on the target, run clone.pl, run root.sh. I have done this dance a hundred times.
The error
I extracted the tar archive on the target into /oracle/product/19.0.0.0/dbhome_1, changed into the clone directory, and went to run clone.pl. Nothing. Before I even got to the Perl script, I wanted to confirm the extracted Perl interpreter worked, so I ran the most trivial command possible:
$ /oracle/product/19.0.0.0/dbhome_1/perl/bin/perl -v
bash: /oracle/product/19.0.0.0/dbhome_1/perl/bin/perl: Operation not permitted
Operation not permitted. Not “permission denied,” which would be EACCES and would point at a mode or ownership problem. This was EPERM, a different animal, and it applied to every binary in the extracted home. I could cat the file, I could read it, the bits were all there. I just could not execute it.
Diagnosis
The nice thing about EPERM on execve is that the suspect list is short when the obvious causes are already excluded. File permissions were correct (oracle owned it, 0755 on the binary), and EPERM rather than EACCES told me permissions were not the story anyway. So I started working through the kernel-level reasons a read-execute that should succeed gets refused.
First, mount options. A home mounted noexec would produce exactly this symptom. I checked findmnt and the fstab entry:
$ findmnt /oracle
TARGET SOURCE FSTYPE OPTIONS
/oracle /dev/mapper/vg_oracle-lv_oracle xfs rw,nodev,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota
rw, nodev, relatime, and importantly no noexec. The filesystem was perfectly happy to execute code. Ruled out.
Next, SELinux, which was my leading theory because this is the classic tar-does-not-preserve-context failure. getenforce confirmed we were enforcing, and the file’s security context confirmed the problem:
$ getenforce
Enforcing
$ ls -Z /oracle/product/19.0.0.0/dbhome_1/perl/bin/perl
unconfined_u:object_r:default_t:s0 /oracle/product/19.0.0.0/dbhome_1/perl/bin/perl
default_t is SELinux’s shrug. It is the label the kernel applies when it has no policy telling it what a file is supposed to be, and under enforcing mode a lot of domains are not allowed to execute a default_t file. Tar extraction does not carry the source contexts, so the whole extracted home landed as default_t. This looked like an open and shut case, so I relabeled:
$ restorecon -R -v /oracle/product/19.0.0.0/dbhome_1
$ semanage fcontext -a -t usr_t "/oracle(/.*)?"
$ restorecon -R -v /oracle/product/19.0.0.0/dbhome_1
$ /oracle/product/19.0.0.0/dbhome_1/perl/bin/perl -v
bash: /oracle/product/19.0.0.0/dbhome_1/perl/bin/perl: Operation not permitted
No change. Same EPERM. And here is the detail that made me stop trusting my SELinux theory: if SELinux had been the one denying execution, it would have logged an AVC. It always does. So I checked the audit log for any recent access vector denial:
$ ausearch -m avc -ts recent
<no matches>
Nothing. An empty AVC search under enforcing mode, on a binary that is being denied execution, means SELinux is not the thing denying it. That was the moment the SELinux theory fell apart, because you cannot have a silent SELinux denial. If it denies, it audits.
I quickly cleared the last cheap suspect, the immutable attribute, which would also produce EPERM-flavored failures on certain operations:
$ lsattr /oracle/product/19.0.0.0/dbhome_1/perl/bin/perl
-------------------- /oracle/product/19.0.0.0/dbhome_1/perl/bin/perl
Clean. No i flag, no a flag, nothing.
At this point I had ruled out the filesystem, SELinux, and the immutable bit, and I still had an EPERM at execution. So I stopped guessing and traced the syscall directly:
$ strace -f -e trace=execve /oracle/product/19.0.0.0/dbhome_1/perl/bin/perl -v 2>&1 | head -20
execve("/oracle/product/19.0.0.0/dbhome_1/perl/bin/perl", ["/oracle/product/19.0.0.0/dbhome_"..., "-v"], 0x7ffd70a2a360 /* 50 vars */) = -1 EPERM (Operation not permitted)
strace: exec: Operation not permitted
+++ exited with 1 +++
The execve syscall itself returns EPERM. Not an open, not an mmap, the exec call. On a modern hardened Oracle Linux or RHEL 8/9 box, an EPERM at execve with no SELinux AVC points at one thing, and it is not SELinux at all. It is fapolicyd, the file access policy daemon.
fapolicyd is an application allowlisting service that Red Hat ships and that shows up enabled on hardened builds, CIS-benchmarked images, and a lot of enterprise gold OS images. It blocks execution of any file that is not in its trust database. The trust database is populated primarily from the RPM database, so anything installed by a package is trusted automatically. Anything that arrives by any other route (tar, cp, scp, rsync, a build script, an extracted ORACLE_HOME) has no trust record and gets blocked at execve. And critically, fapolicyd denials do not go to the SELinux audit trail. There is no AVC, no dmesg line, no entry in /var/log/messages. They land in the daemon’s own log. That silence is exactly why this eats an afternoon.
A tar-based clone is the textbook trigger. You extract a home full of binaries that no RPM ever installed, and the allowlist does precisely what it was designed to do.
Resolution
Confirming the theory took one command each. The daemon was running, and its log showed the denials on the Oracle binaries:
$ systemctl status fapolicyd
● fapolicyd.service - File Access Policy Daemon
Loaded: loaded (/usr/lib/systemd/system/fapolicyd.service; enabled; vendor preset: enabled)
Active: active (running)
$ journalctl -u fapolicyd --no-pager | tail -5
fapolicyd[2118]: rule=9 dec=deny_audit perm=execute auid=54321 pid=48122 exe=/usr/bin/bash file=/oracle/product/19.0.0.0/dbhome_1/perl/bin/perl
There it was: dec=deny_audit, perm=execute, on the exact file that had been failing. Now that I knew what I was fighting, the fix was straightforward, but the order matters.
I stopped the daemon for the duration of the clone. I did not want to trust binaries that were about to be rewritten:
$ systemctl stop fapolicyd
With fapolicyd out of the way, clone.pl ran to completion and root.sh followed:
$ cd /oracle/product/19.0.0.0/dbhome_1/clone/bin
$ perl clone.pl ORACLE_HOME=/oracle/product/19.0.0.0/dbhome_1 ORACLE_HOME_NAME=OraDB19Home1 ORACLE_BASE=/oracle
...
Copy files in progress.
Copy files successful.
...
The cloning of OraDB19Home1 was successful.
$ /oracle/product/19.0.0.0/dbhome_1/root.sh
Then, and only then, I added the finished home to the trust database. The reason for waiting is important: clone.pl relinks the Oracle binaries as part of its work, which changes their on-disk content and therefore their hashes. fapolicyd trusts files by hash. If you add the home to the trust database before the relink, every hash you registered is stale the moment clone.pl finishes, and you are right back to EPERM. So trust the home after clone.pl has done its rewriting:
$ fapolicyd-cli --file add /oracle/product/19.0.0.0/dbhome_1
$ fapolicyd-cli --update
$ systemctl start fapolicyd
Finally, I verified that the Oracle tooling executed cleanly with fapolicyd back in enforcing mode:
$ sqlplus -V
SQL*Plus: Release 19.0.0.0.0 - Production
Version 19.24.0.0.0
$ $ORACLE_HOME/OPatch/opatch version
OPatch Version: 12.2.0.1.42
OPatch succeeded.
Both ran, under an active fapolicyd, because the home was now in the trust database with hashes that matched the post-relink binaries.
Lessons
A few things from this one are worth carrying into the next fleet build.
An EPERM from execve, on a filesystem without noexec and with correct file permissions, has a short suspect list: SELinux, fapolicyd, or the immutable attribute. Diagnose them in that order. getenforce plus ausearch -m avc clears or confirms SELinux, systemctl status fapolicyd plus journalctl -u fapolicyd clears or confirms fapolicyd, and lsattr clears the immutable bit. If you internalize that triage, this failure goes from an afternoon to about four minutes.
fapolicyd is silent by default. This is the single most important takeaway. It produces no AVCs, nothing in dmesg, nothing in /var/log/messages. If you are chasing an EPERM and your SELinux checks come up clean (enforcing, but an empty AVC search on a file that is clearly being denied), fapolicyd is your next stop, not your last resort. The absence of an AVC is itself the clue.
tar-based ORACLE_HOME clones sidestep two different trust mechanisms at once. They do not inherit SELinux contexts, and they do not register with fapolicyd, because both of those are populated by the process that creates the files, and tar is not that process. This is the real, undocumented argument for the supported gold image path. runInstaller -createGoldImage with INSTALL_DB_SWONLY creates the files fresh on the target through the installer, which carries proper contexts and executes through a trusted flow, so you never hit either wall. When the supported path is not throwing a bogus INS-32707 at you, it is genuinely the better tool, and this is a large part of why.
Deprecated is not removed, but it is a countdown. clone.pl still works in 19c and it saved this deployment, but I am on borrowed time with it and so is anyone else using it. I documented the choice in the change ticket, including why I fell back to it (the INS-32707 sizing bug) so that when we migrate to the supported path the reasoning is traceable and nobody has to reverse-engineer why the fleet was built with a deprecated tool.
Target readiness belongs in the automation. Every check I ran by hand (SELinux mode, fapolicyd status, mount options, inventory state) is a preflight assertion that should run before the first file is extracted. I burned this debug cycle on target one. I did not want to burn it again on target three of fifteen, so the fapolicyd trust step and the readiness checks went into the deployment scripting before I touched the next box. The best time to discover a hardened OS control is once, on the first host, in a controlled way, not repeatedly and by surprise across a fleet.
