1. Core Cnocepts

Linux kernel's task scheduling is complex, but we'll focus on how CGroup limits CPU usage in the CFS scheduler. This explanation simplifies the process by omiting nested CGroup scenarios and using CentOS 7.6 (3.10.0-957.el7) kernel code as reference.

2. Basic CGroup CPU Configuration

To create a CGroup named 'test' with CPU limits:

1. Create directory /sys/fs/cgroup/cpu/test
2. Set cpu.cfs_period_us=100000 and cpu.cfs_quota_us=10000
   • This allocates 10ms CPU time every 100ms period (10% of single core)
3. Add process IDs to cgroup.procs to enforce limits

3. CPU Limitation Mechanism

1. CFS scheduler operates on sched_entity (se) objects, which can represent either tasks or groups
2. A periodic timer allocates CPU quota (cfs_quota_us) to the group
3. After each task runs, its CPU time is deducted from the group's quota
4. When quota is exhausted, the group se is throttled (removed from runqueue)
5. Next period timer refills quota and unthrottles the group

4. Key Data Structures

struct cfs_rq {
    struct rb_root tasks_timeline;  // VRuntime-sorted task tree
    struct sched_entity *curr;      // Currently running entity
    struct task_group *tg;          // Owning task group
    int throttled;                  // Throttle status flag
    s64 runtime_remaining;          // Remaining CPU time quota
};

struct sched_entity {
    unsigned on_rq;                 // On runqueue status
    u64 vruntime;                   // CPU runtime tracking
    struct cfs_rq *cfs_rq;          // Current runqueue
    struct cfs_rq *my_q;            // Group runqueue (NULL for tasks)
};

struct cfs_bandwidth {
    ktime_t period;                 // Allocation period (cfs_period_us)
    u64 quota;                      // CPU quota (cfs_quota_us)
    u64 runtime;                    // Current quota remaining
    struct hrtimer period_timer;    // Quota refresh timer
};

5. Implemantation Flow

1. Quota Tracking:

void update_curr(struct cfs_rq *cfs_rq) {
    curr->vruntime += delta_exec;
    account_cfs_rq_runtime(cfs_rq, delta_exec);
}

void account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta) {
    cfs_rq->runtime_remaining -= delta;
    if (cfs_rq->runtime_remaining <= 0) {
        if (!assign_cfs_rq_runtime(cfs_rq)) {
            resched_curr(cfs_rq->rq);
        }
    }
}

2. Throttling:

void throttle_cfs_rq(struct cfs_rq *cfs_rq) {
    struct sched_entity *se = cfs_rq->tg->se[cpu_of(rq_of(cfs_rq))];
    dequeue_entity(se->cfs_rq, se, DEQUEUE_SLEEP);
    cfs_rq->throttled = 1;
}

3. Quota Refresh:

void sched_cfs_period_timer(struct hrtimer *timer) {
    __refill_cfs_bandwidth_runtime(cfs_b);
    distribute_cfs_runtime(cfs_b);
}

void unthrottle_cfs_rq(struct cfs_rq *cfs_rq) {
    enqueue_entity(cfs_rq, se, ENQUEUE_WAKEUP);
}