Cron job exit codes: what 1, 2, 126, 127 and 137 mean

Every command ends with a number, its exit code. By convention, 0 means it went well; the other values usually mean an error, but what each one means depends on the program. Cron doesn’t show it anywhere by default, but it’s the quickest clue to why a job failed: some codes point straight to the cause.

How to see the exit code

• In your terminal, right after running the command: echo $?. To run it as cron would, see how to run a cron job manually.

• In the crontab, write it to a log after the command:

  0 3 * * * /path/to/job.sh >> /var/log/job.log 2>&1; echo "exit $?" >> /var/log/job.log

• In cron’s log, on Debian and Ubuntu, cron can also log the jobs that end with an error. Add -L 5 to EXTRA_OPTS in /etc/default/cron (1 logs the start of each job, as by default, and 4 the ones that fail) and restart cron. Where that log is depends on the system: see where to find the cron logs.

What each code means

• 0: it went well. If the job still didn’t do what it should, the script ended without noticing the error (more on that below).
• 1: a general error. The program decides what it means: read its output. Many programs, like grep when it finds nothing, also use it for a result that isn’t an error.
• 2: wrong use or a missing file. A shell syntax error, an option the program doesn’t know, or, for ls and other commands, a file that doesn’t exist.
• 124: timeout stopped it. The command was wrapped in timeout and took longer than allowed.
• 126: the command was found but couldn’t run. Check:
  • that the script is executable: chmod +x /path/to/job.sh;
  • that the path is a file, not a directory;
  • that its file system isn’t mounted with noexec (findmnt -T /path/to/job.sh shows its options).
• 127: the command wasn’t found. The most common one in cron. Check, in this order:
  • the PATH. Cron’s is usually just /usr/bin:/bin, so a command in /usr/local/bin, or installed with nvm or pyenv, isn’t found. Use the absolute path, as in /usr/local/bin/node, or set PATH in the crontab.
  • the script’s first line, like #!/bin/bash: the interpreter it names must exist.
  • the line endings. A script saved on Windows fails with /bin/bash^M: bad interpreter. Fix it with sed -i 's/\r$//' /path/to/job.sh.
• 128 plus a number: a signal ended it. The number is the signal:
  • 137 (128 + 9, SIGKILL): something killed it without letting it finish. Often it’s the kernel’s OOM killer when memory runs out (look for Out of memory in dmesg or journalctl -k), a container’s memory limit, or a kill -9.
  • 143 (128 + 15, SIGTERM): something asked it to stop: a server shutdown, a restart of the service or the container, or a kill.
  • 130 (128 + 2, SIGINT): Ctrl+C. It doesn’t happen in cron, but it does when you run the job by hand.
• 255: depends on the program. ssh, for example, returns it when it can’t connect, and a program that exits with -1 ends with 255.

The code goes from 0 to 255: a script that ends with exit 256 ends with 0.

When the code says 0 but the job failed

• A pipe keeps the code of its last command. In /path/to/job.sh | tee job.log, the code is the one of tee, which almost always ends well. In bash, set -o pipefail keeps the error of any command in the pipe.
• A script keeps the code of its last command, unless it stops at the first error. In bash, start it with set -e, or check each step.
• A list keeps the code of its last command. In job.sh; cleanup.sh, the code is the one of cleanup.sh. With job.sh && cleanup.sh, the cleanup only runs if the job went well, and the code is the one of whichever failed.

Get the code with each run

The exit code tells you why a job failed only if someone sees it. A monitor that receives it with each run alerts you when it isn’t 0, with the code in the alert, and keeps the history of the codes of each run.