Batch workloads, often referred to as offline computing tasks, differ fundamentally from long-running services. Unlike deployments that maintain a specific number of replicas indefinitely, batch processes execute a specific task and terminate upon completion. Managing such transient workloads with standard controllers like Deployment leads to unnecessary restarts, as the controller attempts to maintain the desired replica count even after the task finishes. To address this, Kubernetes provides specialized API objects designed for finite tasks.

Prior to version 1.4, Kubernetes lacked native support for batch operations, relying on external solutions. The introduction of the Job API object filled this gap, allowing users to define tasks that run to completion. The structure of a Job is straightforward, centering around a Pod template similar to other controllers.

Defining a Basic Job

A Job resource specifies the Pod template required to execute the task. Consider a scenario where a container needs to perform a calculation and then exit. The following manifest defines such a task:

apiVersion: batch/v1
kind: Job
metadata:
  name: calc-task
spec:
  template:
    spec:
      containers:
      - name: worker
        image: alpine:3.18
        command: ["sh", "-c", "echo 'Starting computation'; sleep 15; echo 'Computation complete'"]
      restartPolicy: Never
  backoffLimit: 3

In this configuration, the Pod uses an Alpine image to run a shell script. The script simulates work by sleeping for 15 seconds before printing a completion message. The restartPolicy is set to Never, ensuring that if the container fails, the Pod itself is not restarted indefinitely within the same instance. Instead, the Job controller manages retries at the Pod level.

Unlike Deployments, Jobs do not require a spec.selector field. Upon creation, the Job controller automatically injects a unique label (e.g., controller-uid=<unique-id>) into the Pod template. The Job object itself adopts a selector matching this label, ensuring strict ownership without user intervention. This mechanism prevents label collisions between different Job instances.

Monitoring Execution and Failures

Once applied, the Job creates a Pod that transitions through standard phases. Initially, the status is Running. Upon successful execution of the command, the Pod moves to Completed. Logs can be inspected to verify output:

kubectl logs -l job-name=calc-task

If the task fails, the restartPolicy dictates the behavior. With Never, the Job controller spawns a new Pod to replace the failed one. This retry logic is governed by backoffLimit. The default value is 6, but it can be customized. Retries occur with an exponential backoff delay (e.g., 10s, 20s, 40s). If restartPolicy is set to OnFailure, the controller attempts to restart the container within the existing Pod rather than creating a new Pod instance.

To prevent jobs from running indefinitely, the activeDeadlineSeconds field sets a hard timeout. If the duration exceeds this limit, all associated Pods are terminated with a DeadlineExceeded reason.

Parallel Execution Control

Batch processing often requires parallelism. The Job API supports this through two key parameters:

1. spec.parallelism: The maximum number of Pods running simultaneously.
2. spec.completions: The total number of successful Pods required to mark the Job as complete.

Modifying the previous example to support parallel execution:

apiVersion: batch/v1
kind: Job
metadata:
  name: parallel-calc
spec:
  parallelism: 3
  completions: 6
  template:
    spec:
      containers:
      - name: worker
        image: alpine:3.18
        command: ["sh", "-c", "echo 'Processing'; sleep 10"]
      restartPolicy: Never
  backoffLimit: 3

Here, the system ensures that no more than 3 Pods run at once, and the Job is only considered finished after 6 Pods have successfully completed. The Job status reflects this via DESIRED (matching completions) and SUCCESSFUL counters. The controller reconciles the state by creating new Pods as others finish, maintaining the parallelism limit until the completion count is reached.

Common Usage Patterns

1. External Orchestration with Templates

In many environments, an external system manages the lifecycle of Jobs. The Job manifest serves as a template where variables are substituted before submission. For instance, processing a list of files:

apiVersion: batch/v1
kind: Job
metadata:
  name: file-processor-$FILE_ID
  labels:
    batch-family: production-run
spec:
  template:
    metadata:
      labels:
        batch-family: production-run
    spec:
      containers:
      - name: processor
        image: alpine:3.18
        command: ["sh", "-c", "echo Processing $FILE_ID && sleep 5"]
      restartPolicy: Never

A script iterates over a list of IDs, replacing $FILE_ID and applying the resulting manifests. All Jobs share a common label (batch-family), allowing grouped monitoring. In this pattern, parallelism and completions typically remain at default values (1), as the external tool handles concurrency.

2. Fixed Work Queue

When the total number of tasks is known beforehand, completions defines the workload size. Pods act as consumers pulling tasks from a queue (e.g., Redis or RabbitMQ). Each Pod processes one item and exits.

apiVersion: batch/v1
kind: Job
metadata:
  name: queue-worker-fixed
spec:
  completions: 10
  parallelism: 2
  template:
    spec:
      containers:
      - name: consumer
        image: custom-worker:latest
        env:
        - name: QUEUE_HOST
          value: redis-service
        - name: QUEUE_NAME
          value: tasks-fixed
      restartPolicy: OnFailure

The application logic inside the container pops a single task, processes it, and terminates. The Job completes once 10 Pods have successfully exited.

3. Dynamic Work Queue

If the total task count is unknown, completions is omitted. Pods must continuously poll the queue until its empty.

apiVersion: batch/v1
kind: Job
metadata:
  name: queue-worker-dynamic
spec:
  parallelism: 2
  template:
    spec:
      containers:
      - name: consumer
        image: custom-worker:latest
        env:
        - name: QUEUE_HOST
          value: redis-service
        - name: QUEUE_NAME
          value: tasks-dynamic
      restartPolicy: OnFailure

The container logic involves a loop: check queue, process task, repeat until empty. Once the queue is drained, the Pod exits. The Job finishes when all active Pods terminate successfully. This pattern requires the application to handle termination conditions internally.

Scheduled Tasks with CronJob

For recurring batch operations, Kubernetes offers the CronJob resource. This object acts as a controller for Jobs, triggering them based on a Unix Cron schedule.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-backup
spec:
  schedule: "0 2 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: backup
            image: alpine:3.18
            command: ["sh", "-c", "date; echo 'Backup executed'"]
          restartPolicy: OnFailure

The schedule field uses standard Cron syntax (minute, hour, day of month, month, day of week). The example above triggers the Job daily at 2:00 AM. The jobTemplate section defines the Job specification to be created upon each trigger.

Concurrency handling is critical for scheduled tasks. The spec.concurrencyPolicy field dictates behavior when a new schedule triggers while a previous Job is still active:

• Allow (Default): Multiple Jobs run concurrently.
• Forbid: Skips the new trigger if the previous Job is active.
• Replace: Cancels the running Job and starts a new one.

To prevent resource exhaustion due to missed schedules (e.g., controller downtime), spec.startingDeadlineSeconds sets a window. If the CronJob fails to start within this seconds-wide window after the scheduled time, the execution is marked as missed. If 100 consecutive misses occur, the CronJob stops scheduling further executions.