github.com/mmrajput/kubernetes-platform-production

Kubernetes Platform Engineering
Production

One Platform · Nine Layers · Five Deployment Models · Infrastructure-Agnostic

Talos Linux · CAPI · ArgoCD GitOps · Cilium

Bootstrap sequence · five phases · one time only

Phase 01 · imperative

Bare metal prep

Proxmox VE · pvecm cluster · bridge networking · raw disks for Ceph

Phase 02 · imperative

Management cluster

3 CP · 3 worker · 3 storage nodes · talosctl gen secrets · gen config · bootstrap

Phase 03 · imperative

Core platform services

cert-manager · Rook-Ceph · CNPG · GitLab CE · then ArgoCD points at GitLab · Git = source of truth from day one

Phase 04 · pivot

CAPI + GitOps pivot

clusterctl init · Talos node template · commit cluster manifests · last manual step ever

Phase 05 · gitops

Full GitOps

CAPI provisions nodes · ArgoCD deploys everything · Git is the only interface

Phases 1–3: imperative zone Phase 4: pivot · last manual step Phase 5 onward: pure GitOps · no kubectl · no helm

Control Plane

3-node HA · dedicated CP nodes · no workloads · fixed across all deployment models

infrastructure-agnostic

Layer 1 · Infrastructure · Talos Linux + CAPI

only layer that changes per environment ▾

Layer 1 is the only layer that changes between environments — and even here, only the hypervisor changes, not the OS. Talos Linux runs identically on Proxmox VMs, VMware, bare metal PXE, or any cloud VM. Role assignment (control plane vs worker) is determined purely by which machine config YAML is applied via talosctl. Every layer above has zero awareness of what is underneath.

Pilot: Proxmox VMs · single hostOn-prem: VMware or bare-metal PXEEnterprise: dedicated HW · IPMI · Sidero Metal

Node OS · Talos extensions · role assignment

Talos Linux

Immutable · API-only · no SSH
Identical image on all node types

Talos extensions

rbd · iscsi · GPU drivers
Built at factory.talos.dev pre-CAPI

extensionsrequired

Proxmox VE

pvecm cluster · multi-host
Raw disks → Ceph OSDs

Bare-metal / VMware

PXE boot · IPMI/BMC · vCenter
Same Talos ISO · zero platform change

enterprise

CAPI · node lifecycle via Git

CAPI

Cluster API · node lifecycle
Git commit = node provisioned

provisioning

Control plane nodes

controlplane.yaml · etcd · scheduler
Min 3 · dedicated · no workload pods

control-plane

Worker nodes

worker.yaml · kubelet · containerd
Pools: general · storage · kubevirt

worker

Layer 2 · High Availability · etcd · kube-vip · KubePrism · MetalLB

no single point of failure ▾

HA is a property of the cluster, not a single component. Control plane: 3 etcd members for quorum. kube-vip for the management cluster API VIP (low load). MetalLB distributing to all 3 CP nodes for production workload clusters (high API load — Sidero warning: kube-vip alone serves only one CP node at a time). KubePrism mandatory on every node — keeps workers connected to the API server even if the external load balancer fails.

Pilot: single CP — accepted riskGrowth: 3 CP nodes · 2+ physical hostsEnterprise: 3 CP · MetalLB · etcd quorum · KubePrism

Control plane HA · etcd quorum

etcd quorum

3 CP nodes · tolerates 1 failure
Quorum = 2/3 · Talos auto-manages

quorum

kube-vip

Management cluster API VIP
Low-load only — active/passive

load-balancingmgmt only

MetalLB

Workload clusters · L2/BGP
Distributes to ALL 3 CP nodes

load-balancing

KubePrism

Per-node API LB · 127.0.0.1:7445
Workers reach API if external LB fails

resiliencemandatory

Network HA · Cilium eBPF · ExternalDNS

Cilium kube-proxy replacement

eBPF per node · no central proxy
Node loss does not affect routing

networking

ExternalDNS

Auto DNS records from Services
LoadBalancer + Ingress sync

dns

Failure domain spread

CP nodes on separate physical hosts
Host failure loses 1 of 3 — survives

topology

Management plane HA · replicated services

ArgoCD HA

3 app-controller replicas · Redis HA
No single pod dependency

gitops

Vault HA (Raft)

3 Vault nodes · integrated Raft storage
Active/standby · auto-unseal

secrets

Keycloak HA

2+ replicas · Infinispan session cache
CNPG HA backend

identity

PodDisruptionBudgets

minAvailable:1 all management services
Safe node drain without service loss

resilience

Management Plane

Layers 3–8 · fixed · serves all workload clusters and deployment models · never runs tenant workloads

shared services · all clusters

Layer 3 · GitOps Bootstrap · ArgoCD · GitLab CE

one imperative step · everything else GitOps ▾

GitLab CE is installed first — before ArgoCD — so that internal Git exists from day one and ArgoCD points at it immediately with no external dependency ever. The correct Phase 03 order is: cert-manager → Rook-Ceph → CNPG → GitLab CE → push all manifests → ArgoCD. From the moment ArgoCD starts, it reads from internal GitLab. There is no temporary external repo and no migration. Reloader watches ConfigMaps and Secrets and triggers rolling restarts — the runtime complement to ArgoCD sync.

ArgoCD · source of truth · automation

ArgoCD Hub (HA)

App-of-Apps · multi-cluster · all vClusters
Drift detection · self-heal · audit trail

gitops

GitLab CE

In-cluster · CNPG + Ceph RBD
Keycloak SSO · source of truth

git

ApplicationSet

Git-dir generator per tenant
New YAML file = new cluster provisioned

automation

Reloader

ConfigMap + Secret watcher
Triggers rolling restart on change

runtime

Layer 4 · Storage · Rook-Ceph · MinIO · Velero · CNPG

distributed · no SPOF ▾

Rook-Ceph is the primary storage layer. Ceph operates via the rbd kernel module (requires Talos extension baked into factory.talos.dev image — Day 0 requirement). A single Rook operator provides block (RBD), filesystem (CephFS), and object storage (RadosGW). MinIO complements Ceph for tenant-facing object storage and offsite replication. Dedicated storage nodes recommended to isolate rebalancing performance impact.

Pilot: virtual Ceph OSDs on Proxmox VMsGrowth: dedicated OSD disks per nodeEnterprise: dedicated storage nodes · 10 GbE

Rook-Ceph · unified storage backend

Rook-Ceph operator

Full Ceph lifecycle via CRDs
MON · MGR · OSD · MDS managed

operator

Ceph RBD (block)

Default StorageClass · RWO volumes
Kernel rbd module via Talos extension

block

CephFS (filesystem)

RWX StorageClass · multi-pod access
Nextcloud file store · shared configs

filesystem

Ceph RadosGW

S3-compatible · Loki backend
Internal Velero target

object

MinIO · Velero · CNPG

MinIO (distributed)

4-node erasure coding · HA
Per-tenant buckets · CNPG WAL

object

Offsite replication

rclone → Hetzner / IONOS
EU-sovereign · BSI C5 · 3-2-1 rule

offsite

Velero

K8s state + CSI snapshots
Single instance covers all vClusters

backup

CNPG WAL archiving

Continuous PostgreSQL backup · PITR
Per-tenant MinIO bucket isolation

database

Layer 5 · Secrets + Identity · Vault · ESO · Keycloak · cert-manager

shared services ▾

Single Vault cluster and single Keycloak instance on the management plane serve all tenant clusters via Cilium Cluster Mesh. ESO runs a ClusterSecretStore per tenant, scoped to that tenant's Vault path only. Keycloak OIDC is configured directly on the Kubernetes API server — highly recommended for production by Sidero. Talos CCM enables automatic certificate rotation.

Vault HA (Raft)

Single source of truth · all clusters
Per-tenant KV paths · least-privilege

secrets

External Secrets Operator

ClusterSecretStore per tenant
ExternalSecret CRDs · Vault → K8s Secret

secrets

Keycloak HA

OIDC SSO · all clusters + portal
Keycloak group → vCluster RBAC

identity

cert-manager + Talos CCM

Webhook TLS · CNPG · ESO
Auto cert rotation via Talos CCM

tls

Layer 6 · Observability · OTel · Prometheus / VictoriaMetrics · Loki · Grafana · Hubble

all tenants · one stack ▾

Single observability stack covers all tenant clusters automatically. OpenTelemetry Collector is the unified ingestion point — workloads instrument once against the OTel SDK, collector handles routing to Prometheus/Loki/Tempo. Prometheus for small/few clusters; VictoriaMetrics for large multi-cluster deployments (Sidero recommendation). Per-tenant log labels and per-namespace Hubble flows give tenants visibility without cross-tenant leakage.

Small: Prometheus · Loki · GrafanaLarge multi-cluster: VictoriaMetrics · VictoriaLogs

Telemetry ingestion + metrics + logs

OTel Collector

Unified ingestion · traces · metrics · logs
Fan-out to Prometheus · Loki · Tempo

telemetry

Prometheus / VictoriaMetrics

kube-prometheus-stack · all nodes
VictoriaMetrics at scale (Sidero rec)

metrics

Loki / VictoriaLogs

Host DaemonSet · all cluster pods
Per-tenant labels · Ceph S3 backend

logs

Grafana

Unified dashboards · Keycloak SSO
Ceph dashboard · tenant resource view

dashboards

Network visibility + alerting + uptime

Cilium Hubble

Network flow visibility per namespace
Dropped packets · policy violations

network

Alertmanager

Namespace-based routing per tenant
Platform alerts · Slack / PagerDuty

alerting

Uptime Kuma

Endpoint availability · all clusters
SLI signal for availability SLOs

uptime

Layer 7 · Security · Kyverno · Harbor · Falco · Cilium · Headscale · kube-bench · Kubescape

BSI IT-Grundschutz ▾

Four security layers enforced at platform level — all clusters inherit automatically. Kyverno replaces Pod Security Standards: does everything PSS does plus image signature verification against Harbor, mutation policies, and generate policies (auto-create NetworkPolicy on namespace creation). Talos Ingress Firewall adds OS-level default-deny for internet-facing clusters. Headscale WireGuard replaces Cloudflare Tunnel — full EU data sovereignty, BSI IT-Grundschutz compliant.

Policy enforcement + runtime security

Kyverno

Replaces PSS · policy · mutate
Image signature verification

policy

Harbor

OCI registry · image signing
Cosign/Notary · Kyverno integration

registry

Falco

Host DaemonSet · all workloads
eBPF probe · Talos-compatible

runtime

CiliumNetworkPolicy

Default-deny per tenant namespace
Auto-applied by provisioning template

network

Edge + compliance scanners + OS firewall

Headscale WireGuard

Self-hosted · EU sovereign
BSI IT-Grundschutz · no inbound ports

edge

Talos Ingress Firewall

OS-level default-deny
Internet-facing clusters · zero-trust

os-firewall

kube-bench + Kubescape

CIS benchmark · RBAC posture
NSA/MITRE · GitLab CI scheduled

compliance

Gitleaks

Secret scan · pre-build CI gate
Blocks pipeline on detection

secret-scan

Layer 8 · CI/CD Platform · GitLab Runner · Harbor · Renovate · Karmada

image supply chain ▾

The CI security gate chain runs in strict order: Gitleaks (pre-build secret scan) → build → Trivy CVE scan (fail on Critical) → Harbor push + Cosign sign → Kyverno admission verify. Renovate watches Harbor image digests and Helm chart repos, opening MRs automatically when updates are available. Karmada adds multi-cluster workload scheduling above ArgoCD — needed only for active-active DR or capacity-aware placement across 10+ clusters.

CI pipeline · security gate order: Gitleaks → build → Trivy → Harbor → Kyverno

GitLab Runner

Kubernetes executor · ephemeral pods
Image promotion pipeline

ci/cd

Trivy CVE gate

Scan on every build · fail on Critical
Blocks push to Harbor

security

Renovate

Dependency MR bot · Helm + images
Opens MR → pipeline re-runs

automation

Karmada

Multi-cluster scheduling · PropagationPolicy
Add only for active-active DR · 10+ clusters

multi-clusterconditional

Workload Plane

Layer 9 · five deployment models · this is the only plane that changes per use case

Git is the only interface

Layer 9 · Deployment Models CAPI provisions · ArgoCD deploys · Vault delivers · Keycloak SSO · Headscale connects · all five models

One management plane manages all five deployment models. The Git directory structure is the single interface regardless of model. Add a file → platform provisions it. Remove a file → platform decommissions it. Vault, Keycloak, Observability stack, and Falco serve all five models from the management plane with zero per-model duplication.

Model 1

vCluster IDP

Self-service developer environments · shared host cluster

team dev staging testing

Virtual API server per team S / M / L resource tiers Sleep · scale-to-zero Backstage portal provisioning

clusters/tenants/*.yaml

Model 2

Mission Critical

Dedicated cluster per certified workload · IEC 62443

SCADA / OT IEC 62443 railway

Dedicated HW + own etcd Own K8s version lifecycle CAB-governed changes Certification boundary

clusters/dedicated/*.yaml

Model 3

Large Microservices

Zalando-scale · one prod cluster per domain · namespace per team

100s of services multi-team high deploy rate

Dedicated prod+nonprod pair per domain Namespace per team inside cluster Cilium L7 between namespaces AppSet per service directory

clusters/dedicated/*.yaml

Model 4

KubeEdge

Factory floors · substations · remote OT sites

factory / OT substation edge

No cluster on device EdgeCore · outbound only OPC-UA · Modbus · DeviceTwin SecureBoot + TPM

edge/sites/*.yaml

Model 5

KubeVirt

Legacy VM migration · SAP · ERP · Windows Server

legacy VM SAP / ERP migration path

VirtualMachine CRD in Git QEMU/KVM · Ceph RBD PVC Forklift import · VMware/oVirt Multus CNI on kubevirt pool

vms/workloads/*.yaml

Model 1 · vCluster IDP

clusters/tenants/*.yaml

Model 2 + 3 · dedicated clusters

clusters/dedicated/*.yaml

Model 4 · edge sites

edge/sites/*.yaml

Model 5 · legacy VMs

vms/workloads/*.yaml

Git is the only interface — Add a YAML file → platform provisions it. Remove a file → platform decommissions it. CAPI provisions nodes, ArgoCD deploys workloads, Vault delivers secrets, Keycloak provides SSO, Headscale connects edge. Same management plane serves all five models.