Operational Domains and Maturity Path

Career ladder:

1. Bootstrap – manual install & run
2. Tune – optimise configs, add metrics
3. Automate – scripts & playbooks
4. Design – architect self-service platforms

Standardisation Layers

Hardware

• Asset tags: DC-RACK-POS-SERIAL
• Uniform BOM per role (CPU, RAM, SSD, NIC)
• Colour-coded rack elevations
• IPMI, dual-PSU, redundant ToR

OS

• Golden image: Ubuntu 22.04 LTS
• Kickstart answers: hostname, bond0, DNS, NTP, sysctl
• Mandatory agents: node-exporter, filebeat, salt-minion
• Baseline alerts: CPU>80 %, RAM>85 %, disk>90 %

Application

• Naming: <service>-<env>-<region>
• Ports: 8080 app, 9090 metrics, 22 ssh
• Logs: /var/log/<service>/access.log → Loki
• Config hierarchy: base/ + env/ + node/

Process

• Change ticket → Git PR → CI → Canary → Prod
• Incident runbook in Confluence
• Post-mortem within 24 h

Tooling Evolution

Reference Implementation: Git-Driven Web Deploy

1. Repository Layout

web-demo/
├── src/
├── config/
│   ├── base/
│   ├── dev/
│   └── prod/
├── deploy/
│   ├── playbook.yml
│   └── rollback.yml
└── .gitlab-ci.yml

2. Dpeloyment Node Setup

# prepare directories
sudo mkdir -p /opt/deploy/{code,config,tar,tmp}
sudo chown -R deploy:deploy /opt/deploy

# clone once
su - deploy
git clone git@git.example.com:web/web-demo.git /opt/deploy/code/web-demo

3. Ansible Playbook Snippet

- hosts: web:&{{ env }}
  serial: 25%
  vars:
    pkg_name: "web-demo-{{ git_sha[:8] }}-{{ ansible_date_time.epoch }}"
  tasks:
    - name: pull latest code
      git:
        repo: git@git.example.com:web/web-demo.git
        dest: "/opt/releases/{{ pkg_name }}"
        version: "{{ git_sha }}"

    - name: overlay environment config
      template:
        src: "config/{{ env }}/app.yml.j2"
        dest: "/opt/releases/{{ pkg_name }}/config/app.yml"

    - name: atomically switch symlink
      file:
        src: "/opt/releases/{{ pkg_name }}"
        dest: /srv/web-demo/current
        state: link
      notify: reload php-fpm

    - name: add node back to LB
      uri:
        url: "http://lb.internal/api/pool/{{ ansible_hostname }}/enable"
        method: POST
      delegate_to: localhost

  handlers:
    - name: reload php-fpm
      systemd:
        name: php8.2-fpm
        state: reloaded

4. Rollback Script

#!/usr/bin/env bash
set -euo pipefail
ENV=${1:-prod}
TO_VERSION=${2:-$(readlink /srv/web-demo/current | xargs basename)}

PREVIOUS=$(ls -1 /opt/releases | grep -B1 "$TO_VERSION" | head -n1)
[ -z "$PREVIOUS" ] && { echo "no previous release"; exit 1; }

ansible-playbook deploy/rollback.yml -e "env=$ENV version=$PREVIOUS"

5. GitLab CI/CD

stages: [build, deploy-dev, deploy-prod]

build:
  stage: build
  script:
    - npm ci && npm run build
    - tar czf web-demo-${CI_COMMIT_SHA}.tar.gz dist/
  artifacts:
    paths: [web-demo-${CI_COMMIT_SHA}.tar.gz]

deploy-dev:
  stage: deploy-dev
  script:
    - ansible-playbook deploy/playbook.yml -e "env=dev git_sha=$CI_COMMIT_SHA"
  only: [develop]

deploy-prod:
  stage: deploy-prod
  script:
    - ansible-playbook deploy/playbook.yml -e "env=prod git_sha=$CI_COMMIT_SHA"
  when: manual
  only: [main]

6. Auto-Scaling Policy (KEDA)

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: web-demo
spec:
  scaleTargetRef:
    name: web-demo
  minReplicaCount: 2
  maxReplicaCount: 20
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus:9090
      metricName: http_requests_per_second
      threshold: "1000"

Daily Workflow

1. Developer pushes to main → pipeline starts
2. Build artefact stored in Nexus
3. Canary deploy to 10 % of prod nodes
4. 5-minute SLO check (p95 latency < 200 ms)
5. Auto-promote or auto-rollback
6. Slack notification with Grafana snapshot

Checklist for Production Readiness

• Secrets in Vault, not in Git
• Blue/Green or Canary strategy
• DB migrations in separate job
• Feature flags for big changes
• On-call runbook linked in alert
• Capacity headroom ≥ 40 %