Homelab

A Flux GitOps repository for my personal homelab Kubernetes cluster.

GitOps · Disaster Recovery · Bootstrap

---

This repository manages the desired state of my personal homelab production cluster.

It runs Flux on K3s and uses the manifests under apps and infrastructure to declaratively manage applications, networking, certificates, storage, and observability.

GitOps

• clusters/production is the Flux bootstrap path.
• infrastructure/overlays/production contains the cluster foundation.
• apps/overlays/production contains homelab application configuration.
• apps/base/* and infrastructure/base/* hold the workload manifests.
• Secrets are encrypted with Sealed Secrets before they are committed.
• ansible/ prepares the replacement machine used for cluster migration.

Disaster Recovery

Recovery uses the normal production path. There is no separate recovery Flux installation or recovery overlay.

Before bootstrapping a replacement cluster, temporarily remove the application entry point from Flux and push that change:

mv clusters/production/apps.yaml clusters/production/apps.yaml.bak
git add clusters/production/apps.yaml clusters/production/apps.yaml.bak
git commit -m "chore: pause applications for cluster recovery"
git push

Use ansible/ to install K3s, bootstrap Cilium, and restore the Sealed Secrets private key before bootstrapping Flux from clusters/production. The bootstrap uses the same cilium Helm release and values that Flux adopts later. Wait until the infrastructure and Longhorn backup target are ready. Access Longhorn without ingress if necessary:

kubectl -n longhorn-system port-forward service/longhorn-frontend 8000:80

In the Longhorn UI, restore the latest Ready system backup created by the same Longhorn minor version. The system backup restores application volumes from their latest volume backups.

PostgreSQL is restored by CloudNativePG from Barman S3 backups, not from Longhorn volume backups. Before enabling applications, remove any CNPG data PVCs restored by an old Longhorn backup and confirm their PVs and Longhorn volumes are gone:

kubectl get pvc -A -l cnpg.io/pvcRole=PG_DATA
kubectl delete pvc -A -l cnpg.io/pvcRole=PG_DATA
kubectl get pv

Enable applications again and push the change:

mv clusters/production/apps.yaml.bak clusters/production/apps.yaml
git add clusters/production/apps.yaml clusters/production/apps.yaml.bak
git commit -m "chore: resume applications after cluster recovery"
git push

All seven CNPG manifests use bootstrap.recovery; they recreate their databases from S3 when Flux applies the application manifests. Verify CNPG recovery before allowing external traffic, then check Flux, certificates, ingress, storage, and core application data.

Bootstrap

K3s, Cilium, and Sealed Secrets key

Store the become password and Sealed Secrets TLS certificate/private key in the Ansible Vault, then let Ansible prepare everything up to the Flux boundary:

cd ansible
make vault-edit
make preflight
make check
make apply
make apply
make verify
cd ..

Do not bootstrap Flux until make verify confirms that Cilium is healthy and that the recovery key in the new cluster matches the Vault values. K3s starts with Flannel, its network-policy controller, and kube-proxy disabled; Cilium must therefore be running before Flux controllers and workloads can start.

Flux

flux bootstrap github \
  --repository=homelab \
  --branch=main \
  --path=./clusters/production \
  --owner=tinyrack-net

Flux reconciles the cilium HelmRelease in kube-system and takes over the release installed by Ansible. Multus remains the first CNI configuration and delegates the primary Pod network to Cilium while continuing to provide the wg-easy macvlan attachment.

Configuration files

• Keep a HelmRelease focused on chart lifecycle and load chart configuration from a sibling <component>.values.yaml through spec.valuesFrom.
• Generate Helm values ConfigMaps with a stable name, the reconcile.fluxcd.io/watch: Enabled label, and values.yaml as the data key.
• Keep application-native YAML, TOML, JSON, ENV, and Alloy River configuration in files named for the owning component. Use Kustomize's default name hash when a Pod directly mounts or imports a ConfigMap so changes roll the Pod.
• Disable the name hash only when the consumer requires a stable name, such as Helm values and Alloy's externally managed, dynamically reloaded ConfigMap.
• Keep credentials out of values and configuration files. Secrets remain encrypted SealedSecret manifests or references to existing Secrets.

CNI migration

The production cluster was migrated from Flannel and kube-proxy to Cilium. The one-time migration and rollback playbooks were removed after verification.

Traefik network policy ownership

• Keep only shared entrypoint, Kubernetes API, and telemetry access in the Traefik infrastructure policies.
• Put route-specific access beside its owning application or proxy in a <owner>.traefik.cilium-network-policy.yaml file.
• Create the policy in traefik-system or traefik-external-system, name it route-<owner>, and add the networking.tinyrack.net/owner label and networking.tinyrack.net/hosts annotation.
• Use toServices with the backend target port for Kubernetes Services and toCIDR with explicit ports for LAN or ExternalName backends.
• Group routes that share a backend into one owner policy. Host annotations are documentation and do not enable Cilium L7 HTTP filtering.

Sealed Secrets

kubectl create secret generic some-secret \
  --namespace some-namespace \
  --dry-run=client \
  --from-literal=SOME_SECRET_KEY=SOME_SECRET_VALUE \
  -o yaml | \
  kubeseal --cert ./tinyrack-homelab-secret-key.crt \
  > ./some.secret.yaml