ORA-27369: Job of Type EXECUTABLE Failed With Exit Code

ORA-27369: Job of Type EXECUTABLE Failed With Exit Code

Error Overview

Error Text: ORA-27369: job of type EXECUTABLE failed with exit code: NNN

ORA-27369 occurs when a DBMS_SCHEDULER job of type EXECUTABLE (running an external OS command or script) returns a non-zero exit code. The error message includes the exit code and a description, allowing the DBA to identify whether the script failed, was killed, or hit a permission issue.

Common Exit Code Meanings

Exit Code	Meaning
1	General script error
2	Misuse of shell builtin
126	Command found but not executable
127	Command not found
128+N	Killed by signal N (e.g., 137 = SIGKILL)
130	Script terminated by Ctrl+C (SIGINT)
137	Killed by SIGKILL (often OOM)
139	Segmentation fault
255	Exit status out of range

Common Causes

Script Logic Errors

• Script itself returns non-zero
• Required input file missing
• Application-specific failure
• Database connection from script failed

Permission and Path Issues

• Script not executable
• oracle user (or job credential) lacks permissions
• Required binary not in PATH
• File ownership mismatch

Environment Problems

• ORACLE_HOME not set in scheduler context
• Library path missing
• Locale/timezone differences
• Environment variables not propagated

Credential Issues (12c+)

• Required credential not configured
• Credential expired
• OS user not authorized
• extjob configuration missing (older versions)

Resource Limits

• Memory limit triggered SIGKILL (exit 137)
• Disk full mid-execution
• Network unavailable for remote commands

Diagnostic Steps

Check Job Status and Output

-- Last run details
SELECT job_name, status, error#, additional_info,
       run_duration, output
FROM dba_scheduler_job_run_details
WHERE job_name = 'MY_EXTERNAL_JOB'
ORDER BY actual_start_date DESC
FETCH FIRST 5 ROWS ONLY;

-- Currently running and recent
SELECT job_name, state, last_start_date, last_run_duration,
       last_executions, failures
FROM dba_scheduler_jobs
WHERE job_name = 'MY_EXTERNAL_JOB';

Inspect Job Definition

-- Show job action and credential
SELECT job_name, job_type, job_action,
       credential_owner, credential_name,
       destination
FROM dba_scheduler_jobs
WHERE job_name = 'MY_EXTERNAL_JOB';

-- For jobs using Job Class
SELECT job_class_name, resource_consumer_group,
       service, logging_level
FROM dba_scheduler_job_classes
WHERE job_class_name IN (
    SELECT job_class FROM dba_scheduler_jobs
    WHERE job_name = 'MY_EXTERNAL_JOB');

Test Script Manually

# Run as oracle user (or job credential user)
sudo -u oracle /path/to/script.sh
echo "Exit: $?"

# Test with full PATH
sudo -u oracle bash -c 'PATH=/usr/local/bin:/usr/bin /path/to/script.sh'

# Check executable
ls -la /path/to/script.sh
file /path/to/script.sh

Inspect Job Log Output

-- Output captured by scheduler
SELECT log_id, job_name, log_date, status,
       additional_info, run_duration
FROM dba_scheduler_job_run_details
WHERE job_name = 'MY_EXTERNAL_JOB'
AND log_date > SYSDATE - 1
ORDER BY log_date DESC;

Resolution Steps

1. Decode Exit Code

-- Get specific exit code
SELECT job_name, error#, additional_info
FROM dba_scheduler_job_run_details
WHERE job_name = 'MY_EXTERNAL_JOB'
AND status = 'FAILED'
ORDER BY log_date DESC
FETCH FIRST 1 ROW ONLY;

Map exit code to common cause table above.

2. Fix Script Permissions

# Make executable
chmod 755 /path/to/script.sh

# Set correct ownership
chown oracle:oinstall /path/to/script.sh

# For files accessed by script
chmod 644 /path/to/data/input.csv
chown oracle:oinstall /path/to/data/input.csv

3. Set Up Credential (12c+)

-- Create credential for OS user
BEGIN
    DBMS_CREDENTIAL.CREATE_CREDENTIAL(
        credential_name => 'OS_ORACLE_CRED',
        username        => 'oracle',
        password        => 'os_password'
    );
END;
/

-- Assign to job
BEGIN
    DBMS_SCHEDULER.SET_ATTRIBUTE(
        name      => 'MY_EXTERNAL_JOB',
        attribute => 'credential_name',
        value     => 'OS_ORACLE_CRED'
    );
END;
/

-- Verify
SELECT job_name, credential_owner, credential_name
FROM dba_scheduler_jobs
WHERE job_name = 'MY_EXTERNAL_JOB';

4. Add Environment Setup to Script

#!/bin/bash
# Always source environment in script
. /etc/profile
. /home/oracle/.bash_profile

export ORACLE_HOME=/u01/app/oracle/product/19c/dbhome_1
export ORACLE_SID=PROD
export PATH=$ORACLE_HOME/bin:$PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib

# Now run actual logic
sqlplus -s /nolog <<EOF
CONNECT / AS SYSDBA
EXEC my_procedure;
EXIT
EOF

5. Configure External Job Settings (Older Versions)

For Oracle 11g, ensure extjobo/extjob is configured:

# As root, configure extjob
cd $ORACLE_HOME/rdbms/admin
./externaljob.ora.sample > $ORACLE_HOME/rdbms/admin/externaljob.ora

# Set in $ORACLE_HOME/rdbms/admin/externaljob.ora
run_user = oracle
run_group = oinstall

# Set permissions
chmod 640 $ORACLE_HOME/rdbms/admin/externaljob.ora
chmod 6750 $ORACLE_HOME/bin/extjobo
chown root:oinstall $ORACLE_HOME/bin/extjobo

6. Increase Resource Limits

/etc/security/limits.conf

# If exit code 137 (SIGKILL, often OOM)
# Increase memory available to oracle user
oracle  soft  memlock  16777216
oracle  hard  memlock  16777216

-- Or restrict job concurrency
BEGIN
    DBMS_SCHEDULER.SET_ATTRIBUTE(
        name      => 'MY_JOB_CLASS',
        attribute => 'resource_consumer_group',
        value     => 'BATCH_GROUP'
    );
END;
/

7. Add Error Handling to Script

#!/bin/bash
set -euo pipefail  # exit on error, unset var, pipe fail

LOG=/tmp/myjob_$(date +%Y%m%d_%H%M%S).log
exec > "$LOG" 2>&1

trap 'echo "Error at line $LINENO with exit $?"; exit 1' ERR

# Your logic here
echo "Starting job at $(date)"
# ...

echo "Completed successfully"
exit 0

Common Scenarios

Scenario 1: Script Not Found

ORA-27369: job of type EXECUTABLE failed with exit code: 127 (Command not found)

Fix: Use absolute path in job_action. Verify script exists at that path for oracle user.

Scenario 2: Permission Denied

ORA-27369: job of type EXECUTABLE failed with exit code: 126

Fix: chmod +x script.sh, verify ownership, check directory traversal permissions.

Scenario 3: Out of Memory

ORA-27369: job of type EXECUTABLE failed with exit code: 137

Fix: Script killed by OOM. Reduce memory usage or schedule outside peak hours.

Scenario 4: SQL*Plus in Script Fails

ORA-27369: job of type EXECUTABLE failed with exit code: 1

Fix: Add set echo on; whenever sqlerror exit failure; to capture SQL errors. Source Oracle environment in script.

Sample Output

SQL> SELECT job_name, status, additional_info
  2  FROM dba_scheduler_job_run_details
  3  WHERE job_name = 'BACKUP_JOB'
  4  ORDER BY log_date DESC FETCH FIRST 1 ROW ONLY;

JOB_NAME      STATUS    ADDITIONAL_INFO
------------- --------- ------------------------------------------
BACKUP_JOB    FAILED    EXTERNAL_LOG_ID="job_12345_678",
                        ORA-27369: job of type EXECUTABLE failed
                        with exit code: 127

SQL> SELECT job_action FROM dba_scheduler_jobs
  2  WHERE job_name = 'BACKUP_JOB';

JOB_ACTION
--------------------------------------------------------------
/u01/scripts/backup.sh

# Test manually
[oracle@host]$ /u01/scripts/backup.sh
-bash: /u01/scripts/backup.sh: No such file or directory

# Path was wrong; fix it
SQL> BEGIN
  2    DBMS_SCHEDULER.SET_ATTRIBUTE(
  3      name      => 'BACKUP_JOB',
  4      attribute => 'job_action',
  5      value     => '/u01/app/scripts/backup.sh'
  6    );
  7  END;
  8  /

Prevention Strategies

Always Use Absolute Paths

-- Good
job_action => '/u01/app/scripts/etl.sh'

-- Bad - relies on PATH
job_action => 'etl.sh'

Source Environment in Scripts

#!/bin/bash
# Always start with environment setup
[ -f /etc/profile ] && . /etc/profile
[ -f $HOME/.bash_profile ] && . $HOME/.bash_profile

export ORACLE_HOME=/u01/app/oracle/product/19c/dbhome_1
export ORACLE_SID=PROD
export PATH=$ORACLE_HOME/bin:/usr/local/bin:$PATH

Capture Output and Exit Codes

-- Configure job to log output
BEGIN
    DBMS_SCHEDULER.SET_ATTRIBUTE(
        name => 'MY_JOB',
        attribute => 'logging_level',
        value => DBMS_SCHEDULER.LOGGING_FULL
    );
END;
/

Test Job Manually First

# Always run script as scheduler will
sudo -u oracle bash -c '
export ORACLE_HOME=/u01/app/oracle/product/19c/dbhome_1
export PATH=$ORACLE_HOME/bin:/usr/bin
/u01/scripts/myjob.sh
echo "Exit: $?"
'

Monitoring

-- Daily failed job report
SELECT job_name, COUNT(*) AS failures,
       MAX(log_date) AS last_failure
FROM dba_scheduler_job_run_details
WHERE status = 'FAILED'
AND log_date > SYSDATE - 1
GROUP BY job_name
ORDER BY failures DESC;

-- Email alert on failure
BEGIN
    DBMS_SCHEDULER.SET_ATTRIBUTE(
        name => 'MY_JOB',
        attribute => 'raise_events',
        value => DBMS_SCHEDULER.JOB_FAILED
    );
END;
/

Related Errors

• ORA-27370: Job slave failed to launch
• ORA-27371: Invalid scheduler credentials
• ORA-27300: OS system dependent operation failed
• ORA-27302: Failure occurred at skgpspawn
• ORA-12011: Execution of jobs failed

Troubleshooting Checklist

• Capture exit code from dba_scheduler_job_run_details
• Decode exit code (1=script error, 127=not found, 137=killed)
• Verify script path is absolute and correct
• Check script is executable by job credential user
• Validate environment setup in script
• Configure credential for 12c+ external jobs
• Enable full logging on job class
• Test script manually as oracle user