Skip to content
Tutorials Center
My learning

Topics

Lesson 21 of 78 ยท Workloads

Jobs and CronJobs

Run work to completion with Kubernetes Jobs and schedule it with CronJobs: completions, parallelism, backoffLimit, TTL, indexed jobs and concurrencyPolicy.

Deployments, StatefulSets and DaemonSets all describe things that should keep running forever. A lot of real work is the opposite: process this file, run this database migration, send tonight's report. That work starts, finishes and should then be left alone. If you wrapped it in a Deployment, Kubernetes would "helpfully" restart it the moment it exited successfully, because to a Deployment an exited container is a failure.

A Job is the workload type for run-to-completion tasks. It creates one or more Pods, retries them when they fail, and records the Job as complete once enough Pods have exited successfully. A CronJob is a Job factory: on a schedule it stamps out a new Job from a template. In this lesson you will learn the knobs that matter, then build two stand-ins in Python: a cron expression matcher and a retry simulation.

There is no cluster on this site. The YAML and kubectl commands below are read-only. The Python examples are stand-ins that reproduce the logic (schedule matching, retry counting), not the real controllers.

Two objects, one idea

Job Runs Pods until enough of them succeed.
CronJob Creates a new Job on a schedule.

A CronJob owns Jobs, and a Job owns Pods.

1 / 2

A first Job

This Job computes a value once and exits. It is a read-only example.

YAML
apiVersion: batch/v1
kind: Job
metadata:
  name: report
spec:
  backoffLimit: 4
  ttlSecondsAfterFinished: 3600
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: report
          image: python:3.13-slim
          command: ["python", "-c", "print(sum(range(1_000_000)))"]

Two details are required and often forgotten:

  • restartPolicy must be Never or OnFailure. The Pod default, Always, is rejected for Jobs, because a Pod that restarts itself forever can never finish. With Never, a failed Pod stays around (so you can read its logs) and the Job creates a new Pod for the retry. With OnFailure, the kubelet restarts the container inside the same Pod.
  • You do not write a selector. The Job controller labels its Pods automatically (for example with batch.kubernetes.io/job-name) and finds them by that label.

Typical commands, also read-only:

Terminal
kubectl apply -f job.yaml
kubectl get jobs
kubectl wait --for=condition=complete job/report --timeout=120s
kubectl logs job/report

kubectl logs job/report picks one of the Job's Pods for you. When there were several retries, list the Pods with kubectl get pods -l batch.kubernetes.io/job-name=report and read the one you care about.

completions, parallelism and the three shapes of Job

Two numbers describe how much work a Job does and how fast:

Field Meaning Default
completions How many Pods must finish successfully 1
parallelism How many Pods may run at the same time 1

Combining them gives three common patterns:

  1. Single run. Leave both at 1. One Pod runs; when it succeeds, the Job is done. Database migrations look like this.
  2. Fixed completion count. completions: 10, parallelism: 3 runs up to three Pods at a time until ten have succeeded. Each Pod must work out for itself what to do, which is where indexed Jobs help (below).
  3. Work queue. Set parallelism but leave completions unset. The Pods pull items from an external queue, and the Job is complete once at least one Pod has succeeded and all Pods have terminated. This pattern needs the workers to coordinate, so it is the rarest.

Near the end of a fixed-count Job the controller never starts more Pods than are still needed. With completions: 5 and parallelism: 2, the last round runs just one Pod.

Retries: backoffLimit and deadlines

Pods fail: the process crashes, the node disappears, an image cannot be pulled. The Job controller counts failures and retries with an exponential delay that starts around ten seconds and doubles up to a cap of six minutes. When the number of failures passes backoffLimit (default 6), the Job is marked Failed with the reason BackoffLimitExceeded and the controller stops creating Pods.

Two related fields:

  • activeDeadlineSeconds is a wall-clock limit for the whole Job. When it runs out, Kubernetes terminates the running Pods and fails the Job (DeadlineExceeded), even if the retry budget is not used up. It protects you from a job that hangs instead of failing.
  • podFailurePolicy lets you treat certain exit codes differently. For example, an exit code that means "bad input, do not retry" can fail the Job immediately instead of burning through the backoff limit, and failures caused by node disruption can be ignored so they do not count. It is the right tool when "retry everything six times" is wrong.

Keep this in mind when you write the container: a Job gives you at-least-once execution. A Pod can be killed after it did the work but before it reported success, and then a second Pod does the same work. Make your tasks idempotent (safe to run twice), for example with "insert if not exists" rather than blind inserts.

Cleaning up: ttlSecondsAfterFinished

A finished Job is not deleted. Its Pods stay (finished, taking no CPU) so you can read the logs. Thousands of old Jobs slow the API server down and clutter kubectl get. Set ttlSecondsAfterFinished and a controller deletes the Job, and its Pods, that many seconds after it completes or fails. 0 means "immediately". A common value is a few hours: long enough to debug a failure, short enough to keep the cluster tidy. CronJobs have their own history limits, shown later.

Indexed Jobs

When a Job has many completions, each Pod needs to know which slice it owns. Set completionMode: Indexed and every Pod gets a unique index from 0 to completions - 1. The index is available in the environment variable JOB_COMPLETION_INDEX and in a Pod annotation.

YAML
apiVersion: batch/v1
kind: Job
metadata:
  name: resize-images
spec:
  completionMode: Indexed
  completions: 8
  parallelism: 4
  backoffLimit: 2
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: worker
          image: example/resizer:1.0
          command: ["sh", "-c", "resize --shard $JOB_COMPLETION_INDEX --of 8"]

The Job tracks completion per index. If shard 3 fails and is retried, shards 0 to 2 stay done. A newer field, backoffLimitPerIndex, gives each index its own retry budget so one bad shard does not exhaust the budget for all the others. Indexed Jobs are the standard way to split a batch job without an external queue.

CronJobs

A CronJob adds a schedule and a Job template:

YAML
apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-report
spec:
  schedule: "30 2 * * 1-5"
  timeZone: "Europe/Berlin"
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 300
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 1
  jobTemplate:
    spec:
      backoffLimit: 2
      ttlSecondsAfterFinished: 86400
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: report
              image: example/report:2.4

The schedule uses standard five-field cron syntax: minute, hour, day of month, month, day of week. The table shows the symbols:

Symbol Meaning Example
* every value * * * * * runs every minute
, list 0 8,17 * * * runs at 08:00 and 17:00
- range 30 2 * * 1-5 runs at 02:30 Monday to Friday
/ step */15 * * * * runs every 15 minutes

There are also shortcuts such as @hourly, @daily and @weekly. If you restrict both the day of month and the day of week, cron runs when either matches, not when both do, which surprises many people.

By default the schedule is read in the time zone of the controller manager, which in managed clusters is usually UTC. Set timeZone with an IANA name such as Europe/Berlin to be explicit, and remember that daylight saving changes can make a local-time schedule skip or repeat an hour.

concurrencyPolicy

What if last night's run is still going when the next one is due?

Policy Behavior
Allow (default) Start the new Job anyway, so runs can overlap
Forbid Skip the new run while the old one is still active
Replace Delete the running Job and start the new one

Use Forbid for tasks that must not run twice at once, such as a cleanup that locks tables.

Missed runs and history

If the controller was down (or the CronJob was suspended) at the scheduled time, startingDeadlineSeconds says how late a run may still start. Missing the window counts as a missed run and is skipped. If a schedule has been missed more than 100 times in a row, the CronJob stops scheduling and reports an error, so a long-suspended CronJob does not fire a flood of catch-up runs.

successfulJobsHistoryLimit (default 3) and failedJobsHistoryLimit (default 1) say how many finished Jobs to keep. suspend: true pauses scheduling without deleting anything. To trigger a run by hand right now, create a Job from the CronJob's template:

Terminal
kubectl create job manual-report --from=cronjob/nightly-report

Also remember that Kubernetes promises at most roughly one Job per schedule slot, not exactly one: in rare cases a slot produces zero or two Jobs. That is another reason for idempotent tasks.

Stand-in 1: a cron expression matcher

The real CronJob controller wakes up regularly, compares the current time with the schedule, and creates a Job when they match. This Python stand-in parses the five fields and lists the next run times, so you can check a schedule before you apply it. It also follows the "either day field" rule described above.

Python Parse a cron schedule and list the next runs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the last schedule, 0 9 13 * 5. It does not mean "09:00 on Friday the 13th". It runs at 09:00 on every 13th and on every Friday.

Stand-in 2: a Job retry simulation

This stand-in plays the part of the Job controller. Each round it starts as many Pods as parallelism allows (never more than are still needed), counts successes and failures, and fails the Job once failures exceed backoffLimit. A seeded random generator decides which Pods fail, so the output is the same every run. What it leaves out: the exponential delay between retries, deadlines and node failures.

Python Simulate completions, parallelism and backoffLimit

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Change fail_rate or backoff_limit and watch when the Job gives up. Notice that with a high failure rate a small backoffLimit fails fast, which is usually what you want for a task that is simply broken.

Common mistakes

  • Using a Deployment for a task that should finish. It will restart the Pods forever.
  • Forgetting restartPolicy: Never or OnFailure. The API server rejects the Job.
  • No ttlSecondsAfterFinished and no history limits, so finished Pods pile up.
  • Tasks that are not idempotent, then surprised by a double run.
  • Writing a cron schedule in local time without timeZone, then wondering why it fires at the wrong hour.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which restartPolicy values are allowed in a Job's Pod template?

    A Job's Pods must be able to finish, so Always is rejected. Never creates a new Pod for each retry; OnFailure restarts the container inside the same Pod.
  2. 2. A Job has completions: 6 and parallelism: 4. How many Pods run at the same time in the first round, and in the last?

    The controller starts up to parallelism Pods but never more than are still needed, so the first round has 4 and the final round only the 2 that remain.
  3. 3. A CronJob's previous run is still going when the next one is due, and you do not want two at once. Which setting do you choose?

    Forbid skips the new run while an old Job is active. Replace would kill the running one, Allow would overlap, and suspend stops all scheduling.
  4. 4. What does the cron expression 0 9 13 * 5 mean in Kubernetes?

    When both the day-of-month and day-of-week fields are restricted, cron runs when either one matches.
  5. 5. Why should Job tasks be idempotent?

    A Pod can be lost after doing its work but before reporting success, and the retry repeats the work. Safe repeats avoid duplicate side effects.

Summary

  • A Job runs Pods to completion and retries failures; a CronJob creates Jobs on a cron schedule.
  • restartPolicy must be Never or OnFailure; completions and parallelism set how much runs and how fast.
  • backoffLimit limits retries, activeDeadlineSeconds limits total time, and ttlSecondsAfterFinished cleans up.
  • completionMode: Indexed gives each Pod a JOB_COMPLETION_INDEX so it knows its slice of the work.
  • CronJobs need concurrencyPolicy, a timeZone, history limits and idempotent tasks to be dependable.

Next, the chapter on networking begins with how Pods are reached by a stable address: Services.