Customizing the Kubernetes scheduling logic follows a similar pattern to writing MySQL UDFs or Hive custom functions; it requires adhering to specific interfaces and extension points defined by the platform.

Background

The goal is to implement scheduling capabilities for database services running on Kubernetes. Standard K8s scheduling focuses primarily on CPU and memory, but databaes management requires specific considerations like disk availability and custom affinity rules, especially in hybrid environments where containers and physical machines coexist.

Challenges

• Resource Limitations: Native K8s resources do not account for disk space requirements effectively, which is critical for MySQL or similar storage-heavy applications.
• Strategy Mismatch: Default scheduling strategies do not align with production realities, such as mixed deployment environments and specific service-level policies.

Implementation Plan

1. Fork and modify the Kubernetes scheduler source code to implement custom logic.
2. Synchronize metadata (e.g., MySQL port capacity, server disk usage) into Node annotations using external scripts.
3. Deploy the customized scheduler and configure pods to use it via the schedulerName field.

Build and Deployment Workflow

1. Obtain the scheduler source code (standard K8s repo).
2. Compile the binary for the target architecture: ``` CGO_ENABLED=0 GOOS=linux GOARCH=amd64 make all WHAT=cmd/kube-scheduler/

3. Containerize the binary and push to the registry: ``` docker build -f Dockerfile -t registry.example.com/custom-scheduler:v1.0 . docker push registry.example.com/custom-scheduler:v1.0

4. Update the scheduler deployment in the cluster: ``` kubectl get pod -n kube-system | grep custom-scheduler kubectl delete pod custom-scheduler-xxxx -n kube-system

   
   Verify the new pods are running before proceeding.
   
   

Code Structure and Logic

The core logic resides in pkg/scheduler/algorithm. The scheduling process is split into two phases:

• Predicates (Filtering): Hard constraints. Nodes failing these checks are immediately removed from the candidate list.
• Priorities (Scoring): Soft constraints. Nodes passing the filtering phase are scored, and the highest-scoring node is selected.

Example: Disk Filtering (Predicate)

This logic filters out nodes that do not have enough disk space for specific database services (e.g., MySQL). It reads disk stats from Node annotations.

package predicates

import (
    "fmt"
    "k8s.io/api/core/v1"
    "k8s.io/kubernetes/pkg/scheduler/algorithm"
    schedulercache "k8s.io/kubernetes/pkg/scheduler/cache"
)

func CheckStorageCapacity(pod *v1.Pod, meta algorithm.PredicateMetadata, nodeInfo *schedulercache.NodeInfo) (bool, []algorithm.PredicateFailureReason, error) {
    var minFreePercentage int64 = 10
    var totalRequested int64
    var minBufferGB int64 = 250 // 250GB hard floor for small disks
    
    node := nodeInfo.Node()
    if node == nil {
        return false, nil, fmt.Errorf("node instance missing")
    }

    // Determine if the service requires special disk handling (e.g., DB services)
    serviceType := pod.Annotations["scheduler.ext/resource-pool"]
    
    currentPodDisk := getPodDiskRequest(pod)
    for _, existingPod := range nodeInfo.Pods() {
        totalRequested += getPodDiskRequest(existingPod)
    }

    freeSpace := getNodeFreeDisk(node)
    totalSpace := getNodeTotalDisk(node)

    // Calculate effective buffer
    buffer := int64(totalSpace * minFreePercentage / 100)
    if buffer < minBufferGB {
        buffer = minBufferGB
    }

    if serviceType == "mysql" || serviceType == "pika" {
        // 1. Ensure total commitments don't exceed 90% capacity
        // 2. Ensure actual free space minus new request leaves enough buffer
        if (totalRequested + currentPodDisk > totalSpace - buffer) || 
           (freeSpace - currentPodDisk < buffer) {
            return false, []algorithm.PredicateFailureReason{
                &PredicateFailureError{
                    PredicateName: "StorageCheck",
                    PredicateDesc: fmt.Sprintf("insufficient disk: requested %d, available %d", currentPodDisk, freeSpace),
                },
            }, nil
        }
    }
    return true, nil, nil
}

func init() {
    predicates := Ordering()
    predicates = append(predicates, storagePred)
    SetPredicatesOrdering(predicates)
}

Example: Memory Scoring (Priority)

This logic assigns higher scores to nodes with more available memory. It fetches live stats from the node's metrics endpoint.

package priorities

import (
    "fmt"
    "net/http"
    "time"
    "k8s.io/api/core/v1"
    schedulerapi "k8s.io/kubernetes/pkg/scheduler/api"
    schedulercache "k8s.io/kubernetes/pkg/scheduler/cache"
)

func ScoreByFreeMemory(pod *v1.Pod, meta interface{}, nodeInfo *schedulercache.NodeInfo) (schedulerapi.HostPriority, error) {
    node := nodeInfo.Node()
    if node == nil {
        return schedulerapi.HostPriority{}, fmt.Errorf("node not found")
    }

    totalMem := getNodeTotalMemory(node)
    var usedMem int64

    // Fetching live stats (caution: adds latency to scheduling)
    client := &http.Client{Timeout: 5 * time.Second}
    resp, err := client.Get("http://" + node.Status.Addresses[0].Address + ":10255/stats")
    if err == nil {
        defer resp.Body.Close()
        // Parse response to get RSS memory usage
        // usedMem = parsedValue
    }

    // Score: 10 points for empty, 0 for full
    score := 10 - (usedMem / totalMem * 10)
    
    return schedulerapi.HostPriority{
        Host:  node.Name,
        Score: int(score),
    }, nil
}

// Reduce function to normalize scores if multiple priority functions are used
var NormalizeMemoryScore = NormalizeReduce(schedulerapi.MaxPriority, false)

Registering the Policies

After writing the logic, you must register the custom functions in the factory defaults located at pkg/scheduler/algorithmprovider/defaults/defaults.go.

For Predicates (Filtering):

factory.RegisterFitPredicate(predicates.storagePred, predicates.CheckStorageCapacity)

For Priorities (Scoring):

factory.RegisterPriorityFunction2("MemoryScore", priorities.ScoreByFreeMemory, priorities.NormalizeMemoryScore, 1000000)

Note on Weight: The last parameter (1000000) is the weight. Since database scheduling requires this memory check to be the dominant factor, the weight is set significantly higher than default K8s priorities (usually 1 or 10000).

Summary of Custom Rules

Filtering Rules (Predicates)

• StorageCheck: For MySQL/Pika. Ensures (requested + new) < 90% of total and free - new > 10% buffer.
• MemDBCheck: For MySQL. Ensures free_mem - request > 10% of total.
• CPULoadCheck: Generic. Ensures free CPU > requested CPU.

Scoring Rules (Priorities)

• MemoryScore: For MySQL. Scores nodes based on the ratio of free memory to total memory.