CI/CD Workflow Overview

Transitioning from traditional virtual machine-based CI/CD to a Kubernetes-native approach offers significant advantages in resource utilization and environment consistency.

Traditional Workflow:

• CI: Developers push code to GitLab -> Jenkins triggers build on a static slave -> Code scanning and compilation -> Artifacts pushed to Nexus -> Deploy to test environment.
• CD: Jenkins retrieves artifacts -> Deploy to production.

Kubernetes Workflow:

• CI: Developers push code -> Jenkins Master triggers a dynamic Pod -> Code scanning/compilation -> Docker image built and pushed to Harbor -> Deploy to K8s test environment.
• CD: Jenkins pulls specific image version -> Deploys to K8s production environment.

Understanding Jenkins Agents

Static Agents: These are persistent VMs registered to the Jenkins Master. While they offload work from the master, they suffer from resource waste (idle nodes), maintenance overhead, and scaling limitations.

Dynamic Agents (Kubernetes): Jenkins leverages the Kubernetes API to spin up worker Pods on-demand. When a build is triggered, Jenkins requests a Pod defined by a template. Once the build finishes, the Pod is destroyed, freeing up resources immediately. This model is ideal for elastic, high-efficiency environments.

Prerequisites

The following services should be deployed and operational:

• GitLab (Source Code Management)
• Jenkins (CI/CD Server)
• Harbor (Container Registry)
• SonarQube (Code Quality Analysis)
• NFS (Optional, for Maven repository caching)

Configuring Jankins for Kubernetes

To enable dynamic agents, Jenkins must communicate with the Kubernetes cluster.

1. Enable Agent Port: Navigate to Manage Jenkins -> Configure Global Security. Set a fixed TCP port for inbound agents (e.g., 50000).
2. Kubernetes Credentials: Create a Credential in Jenkins of type "Secret File" and upload your kubeconfig file (typical found at ~/.kube/config on the master node).
3. Cloud Configuration:
   • Go to Manage Jenkins -> Clouds -> New Cloud.
   • Name: k8s (or your preferred name).
   • Kubernetes URL: Retrieve via kubectl cluster-info.
   • Jenkins URL: The HTTP endpoint of your Jenkins service.
   • Jenkins Tunnel: The endpoint for agent communication (e.g., jenkins-service:50000).

Custom Agent Images

The default jenkins/inbound-agent lacks build tools. We must create custom images containing Maven, Node.js, Docker, and kubectl.

1. Maven Image

Create a Dockerfile that includes Maven and necessary configurations.

FROM maven:3.8.6-openjdk-8

# Set timezone
ENV TZ=Asia/Shanghai
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone

# Add custom settings if necessary
COPY settings.xml /usr/share/maven/conf/settings.xml

ENTRYPOINT ["cat"]

2. Docker Image

Used for building and pushing images. It requires acccess to the host's Docker daemon.

FROM docker:20.10

RUN apk add --no-cache curl

ENTRYPOINT ["cat"]

3. Kubectl Image

Used for deployments to the cluster.

FROM alpine:3.17

# Install kubectl
RUN apk add --no-cache curl bash && \
    curl -LO "https://dl.k8s.io/release/v1.23.17/bin/linux/amd64/kubectl" && \
    chmod +x kubectl && \
    mv kubectl /usr/local/bin/

ENTRYPOINT ["cat"]

Defining the Pod Template

In the Jenkins Cloud configuration, define a Pod Template. This template specifies which containers to launch for the build.

• Name: jenkins-slave
• Namespace: default
• Usage: Only build jobs with label expressions matching this node.

Add containers to the template (Maven, Docker, kubectl). Ensure the container name for the Jenkins agent is jnlp to override the default entrypoint correctly. Mount the Docker socket for the Docker container.

CI Pipeline Implementation

The Continuous Integration pipeline checks out code, analyzes it, builds it, and publishes the artifact.

1. Code Checkout

Configure DNS resolution within the cluster to reach internal GitLab. Use checkout scmGit with credentials.

2. Code Analysis (SonarQube)

Use the SonarQube scanner container. The withSonarQubeEnv wrapper injects necessary server configurations.

3. Build and Push Image

Use the Docker container to build the image. Tag it with the Git Commit ID for traceability and push to Harbor.

pipeline {
    agent {
        kubernetes {
            cloud 'k8s'
            yaml '''
                apiVersion: v1
                kind: Pod
                spec:
                  containers:
                  - name: maven
                    image: my-registry.com/dev/maven:3.8.6
                    command: ["cat"]
                    tty: true
                  - name: docker
                    image: my-registry.com/dev/docker:20.10
                    command: ["cat"]
                    tty: true
                    volumeMounts:
                    - name: docker-sock
                      mountPath: /var/run/docker.sock
                  volumes:
                  - name: docker-sock
                    hostPath:
                      path: /var/run/docker.sock
            '''
        }
    }
    environment {
        HARBOR_URL = 'harbor.example.com'
        PROJECT_NAME = 'my-app'
    }
    stages {
        stage('Checkout') {
            steps {
                checkout scmGit(
                    branches: [[name: '*/main']],
                    userRemoteConfigs: [[credentialsId: 'gitlab-creds', url: "git@gitlab.com:group/${PROJECT_NAME}.git"]]
                )
            }
        }
        stage('Build Image') {
            steps {
                container('docker') {
                    script {
                        def commitHash = sh(script: 'git rev-parse --short HEAD', returnStdout: true).trim()
                        def imageTag = "${commitHash}-${BUILD_NUMBER}"
                        def fullImage = "${HARBOR_URL}/library/${PROJECT_NAME}:${imageTag}"
                        
                        withCredentials([usernamePassword(credentialsId: 'harbor-creds', usernameVariable: 'USER', passwordVariable: 'PASS')]) {
                            sh "echo '${PASS}' | docker login ${HARBOR_URL} -u '${USER}' --password-stdin"
                        }
                        sh "docker build -t ${fullImage} ."
                        sh "docker push ${fullImage}"
                    }
                }
            }
        }
    }
}

CD Pipeline Implementation

The Continuous Deployment pipeline allows users to select specific image versions and deploy them, with options for rollback.

Dynamic Parameter Selection

Use the Active Choices Plugin to query the Harbor API and list available tags dynamically. This allows the user to choose exactly which version to deploy.

Deploying and Rollback Logic

The deployment stage uses kubectl to update the image. A subsequent stage asks for user input to determine if a rollback is needed.

pipeline {
    agent {
        kubernetes {
            // ... pod template with kubectl container ...
        }
    }
    parameters {
        // Parameters populated by Active Choices Plugin
        // Example: IMAGE_TAG selected based on Harbor list
        string(name: 'IMAGE_TAG', defaultValue: 'latest', description: 'Image tag to deploy')
    }
    stages {
        stage('Deploy') {
            steps {
                container('kubectl') {
                    withCredentials([file(credentialsId: 'kubeconfig-file', variable: 'KUBECONFIG')]) {
                        sh "kubectl set image deployment/my-app my-app=${HARBOR_URL}/library/${PROJECT_NAME}:${IMAGE_TAG}"
                    }
                }
            }
        }
        stage('Verification') {
            steps {
                script {
                    // Logic to wait for rollout or check status
                    sh "kubectl rollout status deployment/my-app"
                }
            }
        }
    }
    post {
        failure {
            script {
                // Auto-rollback on failure logic
                sh "kubectl rollout undo deployment/my-app"
            }
        }
    }
}

Notification Integration

Integrate with chat platforms like Lark (Feishu) or DingTalk using the Lark Notice Plugin.

1. Install Plugin: Upload the HPI file manually if not available in the plugin manager.
2. Configure Bot: In Jenkins system configuration, add the webhook URL and secret for the Lark bot.
3. Pipeline Stage: Add a post-build action or a stage to send a rich card message with build status.

post {
    always {
        dingTalk (
            robot: 'lark-bot-id',
            type: 'CARD',
            title: 'Build Notification',
            text: [
                "### Build Status: ${currentBuild.currentResult}",
                "- Project: ${PROJECT_NAME}",
                "- Image: ${IMAGE_TAG}"
            ]
        )
    }
}