Understanding Volcano PodGroup Integration

In the Volcano ecosystem, the Scheduler and Controller operate as distinct yet synchronized components. While the Scheduler focuses on placing pods based on constraints, the Controller acts as the orchestrator that translates high-level Job abstractions into Kubernetes-native resources like PodGroups and Pods. The core integration point between these layers is the use of shared annotations and resource metadata.

Scheduler Logic: Translating PodGroups and Pods

The Volcano Scheduler maps external Kubernetes resources into internal objects for performance and logic handling:

Mapping to JobInfo

When a PodGroup is created, the scheduler uses a Informer event handler to intercept the object. This object is then converted into a JobInfo wrapper internally.

func (sc *SchedulerCache) setPodGroup(pg *schedulingapi.PodGroup) error {
    jobIdentifier := getJobID(pg)
    if _, exists := sc.Jobs[jobIdentifier]; !exists {
        sc.Jobs[jobIdentifier] = schedulingapi.NewJobInfo(jobIdentifier)
    }
    sc.Jobs[jobIdentifier].SetPodGroup(pg)
    return nil
}

Mapping to TaskInfo

Similarly, Pods are intercepted via they own Informer. The scheduler creates a TaskInfo wrapper for each pod. The critical link between a TaskInfo and its parent JobInfo is established via the scheduling.k8s.io/group-name annotation, which acts as the foreign key.

func getJobID(pod *v1.Pod) JobID {
    if groupName, found := pod.Annotations[GroupAnnotationKey]; found && len(groupName) > 0 {
        return JobID(fmt.Sprintf("%s/%s", pod.Namespace, groupName))
    }
    return ""
}

Controller Logic: Orchestrating Deployment

The JobController manages the lifecycle of a Job by reconciling its state. The process follows a state machine pattern, typically starting in a Pending phase.

Initializing the PodGroup

The controller performs an initiateJob step where it defines a PodGroup corresponding to the Job's requirements. This group acts as the atomic unit for the scheduler, defining MinAvailable thresholds and resource requirements.

Generating Pods

Once the PodGroup is established, the controller iterates over the Tasks defined in the Job specification. For each task replica, it generates a Kubernetes Pod object. The essential glue is the metadata injection during pod creation:

// Injecting group linkage
pod.Annotations[GroupAnnotationKey] = generatedGroupName
pod.Annotations[JobNameKey] = job.Name
// ... further metadata injection

By injecting the PodGroup name into the annotations, the controller ensures that the Scheduler can correctly associate individual tasks with the overarching job constraints, even though the Kubernetes API lacks a direct link between these pods and the PodGroup object.

Architecture Summary

1. Controller Layer: Watches Job custom resources, creates the required PodGroup, and instantiates Pods with specific metadata annotations identifying their membership.
2. Scheduler Layer: Observes PodGroup and Pod lifecycle events, wrapping them into JobInfo and TaskInfo objects, and executes scheduling policies using these internal wrappers.
3. Synchronization: The coupling relies entirely on the scheduling.k8s.io/group-name annotation, which enables seamless interoperability with third-party operators (such as Kubeflow or Spark-on-K8s) that delegate scheduling to Volcano.