Introduction
In recent years, container orchestration with Kubernetes has become the standard for deploying applications at scale. As organizations migrate their workloads to Kubernetes, they often face challenges in managing complex applications consistently. Kubernetes Operators provide a solution by extending Kubernetes' capabilities to automate application lifecycle management.
This guide will walk you through creating your first Kubernetes Operator using Go and Kubebuilder. We'll build a simple Operator that creates pods based on custom resource definitions.
Prerequisites
Before starting, ensure you have the following components installed and configured:
- Go (>= 1.23): Operator are typically developed in Go. Download from https://go.dev/dl/ and configure GOPATH and PATH environment variables.
- Kubernetes Cluster: A working Kubernetes environment. Options include Minikube, Kind, or any other Kubernetes distribution.
- kubectl: The Kubernetes command-line interface. Install and configure it to connect to your cluster.
- Kubebuilder (>= 3.0): A framework for building Kubernetes Operators. Install with:
cd $HOME/go/bin
curl -L -o kubebuilder "https://go.kubebuilder.io/dl/latest/$(go env GOOS)/$(go env GOARCH)"
chmod +x kubebuilder
Ensure $HOME/go/bin is in your PATH. Verify installation with kubebuilder version.
- Docker (optional): Required if you plan to build Docker images for your Operator. Understanding Kubernetes Operators
A Kubernetes Operator is a controller that extends Kubernetes API to manage applications. It uses Custom Resources (CRs) to represent applications and their configuration. Operators watch for changes to these CRs and take action to maintain the desired state of the application.
Operators are particularly useful for managing stateful applications that require complex deployement, scaling, backup, and recovery procedures beyond what Kubernetes provides out of the box.
Our First Operator: Greeting Generator
We'll create an Operator that monitors a custom resource called Greeting and generates pods that display custom greeting messages. This example demonstrates the fundamental concepts of Operator development.
Step 1: Initialize the Kubebuilder Project
Create a new directory for your project and initialize it with Kubebuilder:
mkdir greeting-operator
cd greeting-operator
kubebuilder init --domain example.com --repo github.com/username/greeting-operator
This command sets up the basic project structure with necessary directories and files for an Operator.
Step 2: Define the Custom Resource
Create a new API definition for our Greeting resource:
kubebuilder create api --group demo --version v1 --kind Greeting
Edit api/v1/greeting_types.go to define the resource specification:
package v1
import (
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
// GreetingSpec defines the desired state of Greeting
type GreetingSpec struct {
// Name of the greeting
Name string `json:"name,omitempty"`
// The message to display
Message string `json:"message,omitempty"`
// Number of replicas to create
Replicas int32 `json:"replicas,omitempty"`
}
// GreetingStatus defines the observed state of Greeting
type GreetingStatus struct {
// Number of created pods
CreatedPods int32 `json:"createdPods"`
}
//+kubebuilder:object:root=true
//+kubebuilder:subresource:status
// Greeting is the Schema for the greetings API
type Greeting struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec GreetingSpec `json:"spec,omitempty"`
Status GreetingStatus `json:"status,omitempty"`
}
//+kubebuilder:object:root=true
// GreetingList contains a list of Greeting
type GreetingList struct {
metav1.TypeMeta `json:",inline"`
metav1.ListMeta `json:"metadata,omitempty"`
Items []Greeting `json:"items"`
}
func init() {
SchemeBuilder.Register(&Greeting{}, &GreetingList{})
}
Step 3: Implement the Reconciliation Logic
Edit controllers/greeting_controller.go to implement the reconciliation logic:
package controllers
import (
"context"
"fmt"
"strconv"
appsv1 "k8s.io/api/apps/v1"
corev1 "k8s.io/api/core/v1"
apierrors "k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/log"
demov1 "github.com/username/greeting-operator/api/v1"
)
// GreetingReconciler reconciles a Greeting object
type GreetingReconciler struct {
client.Client
Scheme *runtime.Scheme
}
//+kubebuilder:rbac:groups=demo.example.com,resources=greetings,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=demo.example.com,resources=greetings/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=demo.example.com,resources=greetings/finalizers,verbs=update
//+kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=core,resources=pods,verbs=get;list;watch
// Reconcile is the main reconciliation loop
func (r *GreetingReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
logger := log.FromContext(ctx)
// Fetch the Greeting instance
greeting := &demov1.Greeting{}
err := r.Get(ctx, req.NamespacedName, greeting)
if err != nil {
if apierrors.IsNotFound(err) {
logger.Info("Greeting resource not found. Probably deleted.")
return ctrl.Result{}, nil
}
logger.Error(err, "Failed to get Greeting")
return ctrl.Result{}, err
}
// Create a Deployment for the greeting
deployment := r.createDeployment(greeting)
if err := ctrl.SetControllerReference(greeting, deployment, r.Scheme); err != nil {
logger.Error(err, "Failed to set controller reference")
return ctrl.Result{}, err
}
// Check if deployment already exists
found := &appsv1.Deployment{}
err = r.Get(ctx, client.ObjectKey{Name: deployment.Name, Namespace: deployment.Namespace}, found)
if err != nil && apierrors.IsNotFound(err) {
logger.Info("Creating new Deployment", "Deployment.Namespace", deployment.Namespace, "Deployment.Name", deployment.Name)
err = r.Create(ctx, deployment)
if err != nil {
logger.Error(err, "Failed to create Deployment", "Deployment.Namespace", deployment.Namespace, "Deployment.Name", deployment.Name)
return ctrl.Result{}, err
}
// Update status
greeting.Status.CreatedPods = greeting.Spec.Replicas
err = r.Status().Update(ctx, greeting)
if err != nil {
logger.Error(err, "Failed to update Greeting status")
return ctrl.Result{}, err
}
return ctrl.Result{Requeue: true}, nil
} else if err != nil {
logger.Error(err, "Failed to get Deployment")
return ctrl.Result{}, err
}
// Update status with current pod count
podCount := r.countPods(ctx, deployment)
greeting.Status.CreatedPods = int32(podCount)
err = r.Status().Update(ctx, greeting)
if err != nil {
logger.Error(err, "Failed to update Greeting status")
return ctrl.Result{}, err
}
logger.Info("Reconciliation complete", "Name", greeting.Name, "CreatedPods", greeting.Status.CreatedPods)
return ctrl.Result{}, nil
}
// createDeployment builds a Deployment for the Greeting
func (r *GreetingReconciler) createDeployment(greeting *demov1.Greeting) *appsv1.Deployment {
replicas := greeting.Spec.Replicas
if replicas <= 0 {
replicas = 1
}
labels := map[string]string{
"app": greeting.Name,
"greeting": "demo",
}
return &appsv1.Deployment{
ObjectMeta: metav1.ObjectMeta{
Name: greeting.Name + "-deployment",
Namespace: greeting.Namespace,
Labels: labels,
},
Spec: appsv1.DeploymentSpec{
Replicas: &replicas,
Selector: &metav1.LabelSelector{
MatchLabels: labels,
},
Template: corev1.PodTemplateSpec{
ObjectMeta: metav1.ObjectMeta{
Labels: labels,
},
Spec: corev1.PodSpec{
Containers: []corev1.Container{
{
Name: "greeting-container",
Image: "busybox",
Command: []string{"/bin/sh", "-c", fmt.Sprintf("echo '%s'; while true; do echo '%s'; sleep 30; done", greeting.Spec.Message, greeting.Spec.Message)},
},
},
},
},
},
}
}
// countPods returns the number of pods for a deployment
func (r *GreetingReconciler) countPods(ctx context.Context, deployment *appsv1.Deployment) int {
podList := &corev1.PodList{}
labelSelector := metav1.FormatLabelSelector(deployment.Spec.Selector)
listOpts := []client.ListOption{client.MatchingLabelsSelector{Selector: metav1.SetAsLabelSelector(deployment.Spec.Selector)}}
if err := r.List(ctx, podList, listOpts...); err != nil {
return 0
}
return len(podList.Items)
}
// SetupWithManager sets up the controller with the Manager
func (r *GreetingReconciler) SetupWithManager(mgr ctrl.Manager) error {
return ctrl.NewControllerManagedBy(mgr).
For(&demov1.Greeting{}).
Owns(&appsv1.Deployment{}).
Complete(r)
}
Step 4: Install the Custom Resource Definition
Run the following command to install your CRD to the Kubernetes cluster:
make install
Step 5: Run the Operator Locally
Execute this command to run your Operator in development mode:
make run
Step 6: Create a Greeting Resource
Create a file named my-greeting.yaml with the following content:
apiVersion: demo.example.com/v1
kind: Greeting
metadata:
name: hello-demo
spec:
name: hello-demo
message: "Hello from my first Kubernetes Operator!"
replicas: 2
Apply the configuration to your cluster:
kubectl apply -f my-greeting.yaml
Step 7: Verify the Result
Check that the deployment was created:
kubectl get deployment
Verify the pods are running and displaying the correct message:
kubectl logs deployment/hello-demo-deployment
You should see your greeting message being displayed in the logs.
Conclusion
You've successfully created your first Kubernetes Operator! This example demonstrates the fundamental pattern of Operator development: watching custom resources and creating Kubernetes resources to achieve the desired state.
In this implementation, our Operator creates deployments based on the Greeting custom resource. The reconciliation logic ensures the desired number of replicas are running and updates the status with the actual pod count.
As you continue exploring Operator development, you can extend this example to handle more complex scenarios such as error handling, scaling, and advanced lifecycle management.