GitOps with ArgoCD

GitOps with ArgoCD

git.skypalette.ai/skyadmin/gitops is the source of truth for every Kubernetes resource that isn’t bootstrap state. ArgoCD continuously reconciles main into the cluster.

Layout

gitops/
├── apps/                    # ArgoCD Application manifests (app-of-apps root)
├── base/                    # Reusable per-app manifests
│   ├── web-app/             # Includes deployment, service, ingress, servicemonitor
│   └── …
├── components/              # Kustomize Components — annotate, observability, etc.
├── overlays/                # Environment-specific patches (staging/production)
├── infrastructure/          # Cluster-level: monitoring, ingress-nginx, cert-manager…
└── docs/                    # The README of every layer

App-of-apps

The root ArgoCD Application (apps/kustomization.yaml) renders every other Application — one per third-party stack (kube-prometheus-stack, loki, alloy, vaultwarden, velero, …) plus one per homegrown app overlay. Adding a new app means a PR that drops an Application manifest under apps/templates/ or apps/infrastructure/ and its base/overlay; ArgoCD picks it up on the next sync.

SOPS + KSOPS

Secrets are committed encrypted. An age recipient is hard-coded in .sops.yaml; the corresponding key lives only in the operator’s password manager and in the running ArgoCD pod (mounted from a manually- applied Secret bootstrapped once).

The ArgoCD repo-server runs with the KSOPS plugin enabled, so during manifest render it decrypts any *.sops.yaml it encounters and emits the plaintext to the in-memory render. The plaintext never lands on disk or in ArgoCD’s UI.

# .sops.yaml
creation_rules:
  - encrypted_regex: ^(data|stringData)$
    age:
      - age16tffvlf4cd79utck57ypm68xvy2y5v6ruzlum6hxf5593s5vsctqle5zhq

Editing a secret is sops infrastructure/monitoring/alertmanager-ntfy/url-secret.sops.yaml; SOPS opens an editor on the decrypted view, re-encrypts on save.

Image Updater

argocd-image-updater polls the Forgejo registry for new tags matching a regex per Application. When it finds a new tag it commits the bump back to the gitops repo (via Forgejo basic-auth using a scoped token), which Argo then picks up. The whole loop runs without manual helm upgrade or kubectl set image.

Sync policy

Every Application has automated: { selfHeal: true, prune: true } with two consequences:

  • Drift is fixed automatically. If someone kubectl edits a live resource that Argo owns, Argo reverts within seconds.
  • Removed resources are deleted. Comment a resource out in Git and Argo deletes the live object. The exception: namespaces have prune: false so accidental file deletion can’t wipe out everything in a namespace.

Verifying state

kubectl -n argocd get applications
kubectl -n argocd get application kube-prometheus-stack \
  -o jsonpath='sync={.status.sync.status} health={.status.health.status}\n'

For a one-pass-fits-all check including PVC capacity, scrape targets, and custom alerts, see scripts/validate-observability.sh covered in Observability.

Kubernetes ClusterNetworking & TLS