Kubernetes Job Controller Mechanism and Source Code Analysis

Overview of JobsJob ExampleJob SpecificationPod TemplateConcurrency ConsiderationsOther Properties

Overview

Job is one of the primary native workload resources in Kubernetes, offering the simplest way to run batch tasks on Kubernetes. In scenarios like AI model training, the most basic implementation involves launching a Job to complete a single training task. Later, various custom "Job" implementations handle advanced processing, such as distributed training requiring multiple Pods with different startup parameters from a single "Job". Understanding the functionality and implementation details of Jobs is fundamental for developing custom "Job" workload types. This series will explore the features of Jobs and their underlying Job controller through several parts.

The series "Kubernetes Job Controller Mechanism and Source Code Analysis" consists of three parts:

  • "Kubernetes Job Controller Mechanism and Source Code Analysis (Part 1)" - A detailed explanation of Job usage and supported features
  • "Kubernetes Job Controller Mechanism and Source Code Analysis (Part 2)" - First part of source code analysis, covering logic from the controller entry point to all EventHandler implementations, i.e., all logic before "tuning tasks" enter the workqueue
  • "Kubernetes Job Controller Mechanism and Source Code Analysis (Part 3)" - Second part of source code analysis, covering the consumption of "tuning tasks" from the workqueue and specific tuning process implementations

What is a Job

After creating a Job, the Job controller starts one or more Pods to run until the number of successfully exited Pods reaches the specified count. A Job can be suspended, which directly deletes the running Pods; when a Job is deleted, its corresponding Pods are also deleted, consistent with other controllers.

A common use case for Jobs is to launch a Pod via the Job to ensure it runs successfully once. If the sole Pod fails for some reason, the Job will restart another Pod to continue trying to achieve the goal of "success once." Naturally, Jobs support launching multiple Pods concurrently.

Job Example

Using the official example of calculating pi:

 1apiVersion: batch/v1<br></br> 2kind: Job<br></br> 3metadata:<br></br> 4  name: pi<br></br> 5spec:<br></br> 6  template:<br></br> 7    spec:<br></br> 8      containers:<br></br> 9      - name: pi<br></br>10        image: perl<br></br>11        command: ["perl",  "-Mbignum=bpi", "-wle", "print bpi(2000)"]<br></br>12      restartPolicy: Never<br></br>13  backoffLimit: 4<br></br>

Applying this YAML file creates a Job. Here's the information:

  • Check the Job status immediately after creation
1# kubectl get job<br></br>2NAME   COMPLETIONS   DURATION   AGE<br></br>3pi     0/1           3s         3s<br></br>
  • Check the corresponding pod status
 1# kubectl get pod --selector=job-name=pi<br></br> 2NAME          READY   STATUS              RESTARTS   AGE<br></br> 3pi--1-25l8f   0/1     ContainerCreating   0          8s<br></br> 4<br></br> 5# kubectl get pod --selector=job-name=pi<br></br> 6NAME          READY   STATUS    RESTARTS   AGE<br></br> 7pi--1-25l8f   1/1     Running   0          12s<br></br> 8<br></br> 9# kubectl get pod --selector=job-name=pi<br></br>10NAME          READY   STATUS      RESTARTS   AGE<br></br>11pi--1-25l8f   0/1     Completed   0          21s<br></br>
  • Describe the job details
 1kubectl describe job pi<br></br> 2Name:             pi<br></br> 3Namespace:        default<br></br> 4Selector:         controller-uid=47c8b54a-76e9-4259-b900-750df8a88dd2<br></br> 5Labels:           controller-uid=47c8b54a-76e9-4259-b900-750df8a88dd2<br></br> 6                  job-name=pi<br></br> 7Annotations:      <none><br></br> 8Parallelism:      1<br></br> 9Completions:      1<br></br>10Completion Mode:  NonIndexed<br></br>11Start Time:       Mon, 18 Oct 2021 15:55:02 +0800<br></br>12Completed At:     Mon, 18 Oct 2021 15:55:23 +0800<br></br>13Duration:         21s<br></br>14Pods Statuses:    0 Running / 1 Succeeded / 0 Failed<br></br>15Pod Template:<br></br>16  Labels:  controller-uid=47c8b54a-76e9-4259-b900-750df8a88dd2<br></br>17          job-name=pi<br></br>18  Containers:<br></br>19   pi:<br></br>20    Image:      perl<br></br>21    Port:       <none><br></br>22    Host Port:  <none><br></br>23    Command:<br></br>24      perl<br></br>25      -Mbignum=bpi<br></br>26      -wle<br></br>27      print bpi(2000)<br></br>28    Environment:  <none><br></br>29    Mounts:       <none><br></br>30  Volumes:        <none><br></br>31Events:<br></br>32  Type    Reason            Age    From            Message<br></br>33  ----    ------            ----   ----            -------<br></br>34  Normal  SuccessfulCreate  5m59s  job-controller  Created pod: pi--1-25l8f<br></br>35  Normal  Completed         5m39s  job-controller  Job completed<br></br>

Looking at the output, we can think about it from a "source code" perspective. Observing line by line, we can see many traces of the Job controller's work, such as automatically configured labels, selectors, etc., recording the current state of the pods under the Job, duration, start and end times, etc. There are two events: one for successful pod creation and one for the completion of the Job. All these logics can be found when studying the source code later.

You can also check the result of the pi calculation via logs:

  • View the calculation result
1# kubectl logs `kubectl get pod --selector=job-name=pi | grep -v NAME | awk '{print $1}'`<br></br>23.1415926535897932384626433832795028841971693993751058209749445923078164062862089986280348253421170679821480865132823066470938446095505822317253594081284811174502841027019385211055596446229489549303819644288109756659334461284756482337867831652712019091456485669234603486104543266482133936072602491412737245870066063155881748815209209628292540917153643678925903600113305305488204665213841469519415116094330572703657595919530921861173819326117931051185480744623799627495673518857527248912279381830119491298336733624406566430860213949463952247371907021798609437027705392171762931767523846748184676694051320005681271452635608277857713427577896091736371787214684409012249534301465495853710507922796892589235420199561121290219608640344181598136297747713099605187072113499999983729780499510597317328160963185950244594553469083026425223082533446850352619311881710100031378387528865875332083814206171776691473035982534904287554687311595628638823537875937519577818577805321712268066130019278766111959092164201989380952572010654858632788659361533818279682303019520353018529689957736225994138912497217752834791315155748572424541506959508295331168617278558890750983817546374649393192550604009277016711390098488240128583616035637076601047101819429555961989467678374494482553797747268471040475346462080466842590694912933136770289891521047521620569660240580381501935112533824300355876402474964732639141992726042699227967823547816360093417216412199245863150302861829745557067498385054945885869269956909272107975093029553211653449872027559602364806654991198818347977535663698074265425278625518184175746728909777727938000816470600161452491921732172147723501414419735685481613611573525521334757418494684385233239073941433345477624168625189835694855620992192221842725502542568876717904946016534668049886272327917860857843838279679766814541009538837863609506800642251252051173929848960841284886269456042419652850222106611863067442786220391949450471237137869609563643719172874677646575739624138908658326459958133904780275901<br></br>

Job Specification

When defining a resource, besides the explicit apiVersion and kind, the rest that can be customized are metadata and spec, where metadata mainly includes "name" related configurations, while the behaviors are controlled in spec. Let's look at what properties can be configured in the spec of a Job resource.

Pod Template

The only required configuration in .spec is .spec.template, which is a pod template, like this format:

 1metadata:<br></br> 2  name: hello<br></br> 3spec:<br></br> 4  template:<br></br> 5    spec:<br></br> 6      containers:<br></br> 7      - name: hello<br></br> 8        image: busybox<br></br> 9        command: ['sh', '-c', 'echo "Hello, Kubernetes!" && sleep 3600']<br></br>10      restartPolicy: OnFailure<br></br>

Note that this entire block is a simple pod template. Using the kubectl explain job.spec.template command clearly shows the relationship between this configuration and job.spec:

 1# kubectl explain job.spec.template<br></br> 2KIND:     Job<br></br> 3VERSION:  batch/v1<br></br> 4<br></br> 5RESOURCE: template <Object><br></br> 6<br></br> 7DESCRIPTION:<br></br> 8     Describes the pod that will be created when executing a job. More info:<br></br> 9     https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/<br></br>10<br></br>11     PodTemplateSpec describes the data a pod should have when created from a<br></br>12     template<br></br>13<br></br>14FIELDS:<br></br>15   metadata    <Object><br></br>16     Standard object's metadata. More info:<br></br>17     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata<br></br>18<br></br>19   spec    <Object><br></br>20     Specification of the desired behavior of the pod. More info:<br></br>21     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status<br></br>22<br></br>

This means that job.spec.template contains the complete pod.metadata and pod.spec, i.e., all attributes that a pod can change.

There are two things to note:

  1. Do not easily set .spec.selector. As seen in the previous example, the Job controller automatically adds a selector like controller-uid=47c8b54a-76e9-4259-b900-750df8a88dd2. This selector ensures uniqueness and satisfies almost all scenario needs. If you add a selector yourself but accidentally introduce duplicate names, it may lead to unnecessary errors where the Job controller operates on pods managed by other controllers.
  2. The RestartPolicy of a Pod can only be set to Never and OnFailure, which is obvious because setting it to Always would create an infinite loop.

Concurrency Issues

From a concurrency perspective, Jobs can be divided into three categories:

  1. No Concurrency:
  2. Only one Pod is started at a time. Another new Pod is started only after the current Pod fails.
  3. When one Pod successfully completes, the entire Job ends.
  4. In this case, .spec.completions and .spec.parallelism do not need to be set, i.e., the effective default values are 1.
  5. Specify Completion Count:
  6. Set .spec.completions to a positive integer.
  7. The Job ends when the number of successfully completed Pods reaches the specified completion count.
  8. You can set .spec.completionMode=Indexed, which gives the Pod a numbered name starting from 0. Otherwise, if not specified, the Pod name is pi--1-25l8f, and even when creating multiple Pods concurrently, the naming style remains pi--1-xxxxx, with the middle number always being 1. Additionally, the .spec.completionMode=Indexed configuration adds annotations like batch.kubernetes.io/job-completion-index and environment variables like JOB_COMPLETION_INDEX=job-completion-index to the Pod.
  9. After configuring .spec.completions, you can optionally configure .spec.parallelism to control the concurrency level. For example, if completions is 10 and parallelism is 3, the Job controller will try to maintain a concurrency level of 3 to launch Pods until 10 Pods successfully complete.
  10. Work Queue:
  11. Do not specify .spec.completions, set .spec.parallelism to a non-negative integer (setting it to 0 is equivalent to suspending).
  12. Manage the work queue using methods like MQ, where each Pod works independently and determines whether the entire task is complete. If one Pod exits successfully, it indicates the entire task is complete, and no new Pods will be created, thus ending the Job.

Other Properties

In addition to the aforementioned template, completions, parallelism, completionMode, and selector, the Job.spec has five other configurable properties (with minor differences in versions, my environment is v1.22.0):

  • activeDeadlineSeconds specifies the maximum allowed duration for this Job, and the timer is paused when in a suspended state.
  • backoffLimit specifies the number of retries. After exceeding this, the Job is marked as failed, with a default value of 6, meaning the Pod can be retried 6 times after failure.
  • manualSelector enables the custom selector function, which is rarely needed and should not be configured; this is a boolean value, meaning you must set it to true to override the default behavior using a selector.
  • suspend configures the suspension of a Job, which directly deletes all running Pods and resets the Job's StartTime, pausing the ActiveDeadlineSeconds timer.
  • ttlSecondsAfterFinished requires enabling the TTLAfterFinished feature to take effect, which is in the alpha stage. Its effect is that after a Job finishes (successfully or unsuccessfully), it will be cleaned up after the specified time. Setting it to 0 means immediate cleanup, and by default, the Job remains in the environment until manually deleted after execution.

Overview of JobsJob ExampleJob SpecificationPod TemplateConcurrency ConsiderationsOther Properties

Overview

Job is one of the primary native workload resources in Kubernetes, offering the simplest way to run batch tasks on Kubernetes. In scenarios like AI model training, the most basic implementation involves launching a Job to complete a single training task. Later, various custom "Job" implementations handle advanced processing, such as distributed training requiring multiple Pods with different startup parameters from a single "Job". Understanding the functionality and implementation details of Jobs is fundamental for developing custom "Job" workload types. This series will explore the features of Jobs and their underlying Job controller through several parts.

The series "Kubernetes Job Controller Mechanism and Source Code Analysis" consists of three parts:

  • "Kubernetes Job Controller Mechanism and Source Code Analysis (Part 1)" - A detailed explanation of Job usage and supported features
  • "Kubernetes Job Controller Mechanism and Source Code Analysis (Part 2)" - First part of source code analysis, covering logic from the controller entry point to all EventHandler implemantations, i.e., all logic before "tuning tasks" enter the workqueue
  • "Kubernetes Job Controller Mechanism and Source Code Analysis (Part 3)" - Second part of source code analysis, covering the consumption of "tuning tasks" from the workqueue and specific tuning process implementasions

What is a Job

After creating a Job, the Job controller starts one or more Pods to run until the number of successfully exited Pods reaches the specified count. A Job can be suspended, which directly deletes the running Pods; when a Job is deleted, its corresponding Pods are also deleted, consistent with other controllers.

A common use case for Jobs is to launch a Pod via the Job to ensure it runs successfully once. If the sole Pod fails for some reason, the Job will restart another Pod to continue trying to achieve the goal of "success once." Naturally, Jobs support launching multiple Pods concurrently.

Job Example

Using the official example of calculating pi:

 1apiVersion: batch/v1<br></br> 2kind: Job<br></br> 3metadata:<br></br> 4  name: pi<br></br> 5spec:<br></br> 6  template:<br></br> 7    spec:<br></br> 8      containers:<br></br> 9      - name: pi<br></br>10        image: perl<br></br>11        command: ["perl",  "-Mbignum=bpi", "-wle", "print bpi(2000)"]<br></br>12      restartPolicy: Never<br></br>13  backoffLimit: 4<br></br>

Applying this YAML file creates a Job. Here's the enformation:

  • Check the Job status immediately after creation
1# kubectl get job<br></br>2NAME   COMPLETIONS   DURATION   AGE<br></br>3pi     0/1           3s         3s<br></br>
  • Check the corresponding pod status
 1# kubectl get pod --selector=job-name=pi<br></br> 2NAME          READY   STATUS              RESTARTS   AGE<br></br> 3pi--1-25l8f   0/1     ContainerCreating   0          8s<br></br> 4<br></br> 5# kubectl get pod --selector=job-name=pi<br></br> 6NAME          READY   STATUS    RESTARTS   AGE<br></br> 7pi--1-25l8f   1/1     Running   0          12s<br></br> 8<br></br> 9# kubectl get pod --selector=job-name=pi<br></br>10NAME          READY   STATUS      RESTARTS   AGE<br></br>11pi--1-25l8f   0/1     Completed   0          21s<br></br>
  • Describe the job details
 1kubectl describe job pi<br></br> 2Name:             pi<br></br> 3Namespace:        default<br></br> 4Selector:         controller-uid=47c8b54a-76e9-4259-b900-750df8a88dd2<br></br> 5Labels:           controller-uid=47c8b54a-76e9-4259-b900-750df8a88dd2<br></br> 6                  job-name=pi<br></br> 7Annotations:      <none><br></br> 8Parallelism:      1<br></br> 9Completions:      1<br></br>10Completion Mode:  NonIndexed<br></br>11Start Time:       Mon, 18 Oct 2021 15:55:02 +0800<br></br>12Completed At:     Mon, 18 Oct 2021 15:55:23 +0800<br></br>13Duration:         21s<br></br>14Pods Statuses:    0 Running / 1 Succeeded / 0 Failed<br></br>15Pod Template:<br></br>16  Labels:  controller-uid=47c8b54a-76e9-4259-b900-750df8a88dd2<br></br>17          job-name=pi<br></br>18  Containers:<br></br>19   pi:<br></br>20    Image:      perl<br></br>21    Port:       <none><br></br>22    Host Port:  <none><br></br>23    Command:<br></br>24      perl<br></br>25      -Mbignum=bpi<br></br>26      -wle<br></br>27      print bpi(2000)<br></br>28    Environment:  <none><br></br>29    Mounts:       <none><br></br>30  Volumes:        <none><br></br>31Events:<br></br>32  Type    Reason            Age    From            Message<br></br>33  ----    ------            ----   ----            -------<br></br>34  Normal  SuccessfulCreate  5m59s  job-controller  Created pod: pi--1-25l8f<br></br>35  Normal  Completed         5m39s  job-controller  Job completed<br></br>

Looking at the output, we can think about it from a "source code" perspective. Observing line by line, we can see many traces of the Job controller's work, such as automatically configured labels, selectors, etc., recording the current state of the pods under the Job, duration, start and end times, etc. There are two events: one for successful pod creation and one for the completion of the Job. All these logics can be found when studying the source code later.

You can also check the result of the pi calculation via logs:

  • View the calculation result
1# kubectl logs `kubectl get pod --selector=job-name=pi | grep -v NAME | awk '{print $1}'`<br></br>23.1415926535897932384626433832795028841971693993751058209749445923078164062862089986280348253421170679821480865132823066470938446095505822317253594081284811174502841027019385211055596446229489549303819644288109756659334461284756482337867831652712019091456485669234603486104543266482133936072602491412737245870066063155881748815209209628292540917153643678925903600113305305488204665213841469519415116094330572703657595919530921861173819326117931051185480744623799627495673518857527248912279381830119491298336733624406566430860213949463952247371907021798609437027705392171762931767523846748184676694051320005681271452635608277857713427577896091736371787214684409012249534301465495853710507922796892589235420199561121290219608640344181598136297747713099605187072113499999983729780499510597317328160963185950244594553469083026425223082533446850352619311881710100031378387528865875332083814206171776691473035982534904287554687311595628638823537875937519577818577805321712268066130019278766111959092164201989380952572010654858632788659361533818279682303019520353018529689957736225994138912497217752834791315155748572424541506959508295331168617278558890750983817546374649393192550604009277016711390098488240128583616035637076601047101819429555961989467678374494482553797747268471040475346462080466842590694912933136770289891521047521620569660240580381501935112533824300355876402474964732639141992726042699227967823547816360093417216412199245863150302861829745557067498385054945885869269956909272107975093029553211653449872027559602364806654991198818347977535663698074265425278625518184175746728909777727938000816470600161452491921732172147723501414419735685481613611573525521334757418494684385233239073941433345477624168625189835694855620992192221842725502542568876717904946016534668049886272327917860857843838279679766814541009538837863609506800642251252051173929848960841284886269456042419652850222106611863067442786220391949450471237137869609563643719172874677646575739624138908658326459958133904780275901<br></br>

Job Specification

When defining a resource, besides the explicit apiVersion and kind, the rest that can be customized are metadata and spec, where metadata mainly includes "name" related configurations, while the behaviors are controlled in spec. Let's look at what properties can be configured in the spec of a Job resource.

Pod Template

The only required configuration in .spec is .spec.template, which is a pod template, like this format:

 1metadata:<br></br> 2  name: hello<br></br> 3spec:<br></br> 4  template:<br></br> 5    spec:<br></br> 6      containers:<br></br> 7      - name: hello<br></br> 8        image: busybox<br></br> 9        command: ['sh', '-c', 'echo "Hello, Kubernetes!" && sleep 3600']<br></br>10      restartPolicy: OnFailure<br></br>

Note that this entire block is a simple pod template. Using the kubectl explain job.spec.template command clearly shows the relationship between this configuration and job.spec:

 1# kubectl explain job.spec.template<br></br> 2KIND:     Job<br></br> 3VERSION:  batch/v1<br></br> 4<br></br> 5RESOURCE: template <Object><br></br> 6<br></br> 7DESCRIPTION:<br></br> 8     Describes the pod that will be created when executing a job. More info:<br></br> 9     https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/<br></br>10<br></br>11     PodTemplateSpec describes the data a pod should have when created from a<br></br>12     template<br></br>13<br></br>14FIELDS:<br></br>15   metadata    <Object><br></br>16     Standard object's metadata. More info:<br></br>17     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata<br></br>18<br></br>19   spec    <Object><br></br>20     Specification of the desired behavior of the pod. More info:<br></br>21     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status<br></br>22<br></br>

This means that job.spec.template contains the complete pod.metadata and pod.spec, i.e., all attributes that a pod can change.

There are two things to note:

  1. Do not easily set .spec.selector. As seen in the previous example, the Job controller automatically adds a selector like controller-uid=47c8b54a-76e9-4259-b900-750df8a88dd2. This selector ensures uniqueness and satisfies almost all scenario needs. If you add a selector yourself but accidentally introduce duplicate names, it may lead to unnecessary errors where the Job controller operates on pods managed by other controllers.
  2. The RestartPolicy of a Pod can only be set to Never and OnFailure, which is obvious because setting it to Always would create an infinite loop.

Concurrency Issues

From a concurrency perspective, Jobs can be divided into three categories:

  1. No Concurrency:
  2. Only one Pod is started at a time. Another new Pod is started only after the current Pod fails.
  3. When one Pod successfully completes, the entire Job ends.
  4. In this case, .spec.completions and .spec.parallelism do not need to be set, i.e., the effective default values are 1.
  5. Specify Completion Count:
  6. Set .spec.completions to a positive integer.
  7. The Job ends when the number of successfully completed Pods reaches the specified completion count.
  8. You can set .spec.completionMode=Indexed, which gives the Pod a numbered name starting from 0. Otherwise, if not specified, the Pod name is pi--1-25l8f, and even when creating multiple Pods concurrently, the naming style remains pi--1-xxxxx, with the middle number always being 1. Additionally, the .spec.completionMode=Indexed configuration adds annotations like batch.kubernetes.io/job-completion-index and environment variables like JOB_COMPLETION_INDEX=job-completion-index to the Pod.
  9. After configuring .spec.completions, you can optionally configure .spec.parallelism to control the concurrency level. For example, if completions is 10 and parallelism is 3, the Job controller will try to maintain a concurrency level of 3 to launch Pods until 10 Pods successfully complete.
  10. Work Queue:
  11. Do not specify .spec.completions, set .spec.parallelism to a non-negative integer (setting it to 0 is equivalent to suspending).
  12. Manage the work queue using methods like MQ, where each Pod works independently and determines whether the entire task is complete. If one Pod exits successfully, it indicates the entire task is complete, and no new Pods will be created, thus ending the Job.

Other Properties

In addition to the aforementioned template, completions, parallelism, completionMode, and selector, the Job.spec has five other configurable properties (with minor differences in versions, my environment is v1.22.0):

  • activeDeadlineSeconds specifies the maximum allowed duration for this Job, and the timer is paused when in a suspended state.
  • backoffLimit specifies the number of retries. After exceeding this, the Job is marked as failed, with a default value of 6, meaning the Pod can be retried 6 times after failure.
  • manualSelector enables the custom selector function, which is rarely needed and should not be configured; this is a boolean value, meaning you must set it to true to override the default behavior using a selector.
  • suspend configures the suspension of a Job, which directly deletes all running Pods and resets the Job's StartTime, pausing the ActiveDeadlineSeconds timer.
  • ttlSecondsAfterFinished requires enabling the TTLAfterFinished feature to take effect, which is in the alpha stage. Its effect is that after a Job finishes (successfully or unsuccessfully), it will be cleaned up after the specified time. Setting it to 0 means immediate cleanup, and by default, the Job remains in the environment until manually deleted after execution.

(Reprint please retain the original link of this article https://www.danielhu.cn)

Tags: kubernetes job-controller workload job-specification pod-template

Posted on Fri, 24 Jul 2026 17:14:34 +0000 by sun14php

Hot Tags

©2026 Freaks City