Application Patterns

Application Patterns

Every homegrown app follows the same shape so monitoring, ingress, and TLS come for free.

The base/web-app contract

A web app drops into base/web-app/ and gets:

base/web-app/
├── deployment.yaml      # one container at port 8080 named `http`
├── service.yaml         # ClusterIP, port 80 → targetPort http
├── ingress.yaml         # nginx, with cert-manager annotation
├── hpa.yaml             # HorizontalPodAutoscaler
├── servicemonitor.yaml  # ← auto-scrapes /metrics on port http
└── kustomization.yaml

The base has no namespace, no image tag, no environment-specific values. Overlays patch all three. A new app inherits the metric scrape config and the TLS-terminated ingress without writing either file.

ServiceMonitor convention

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: web-app
  labels:
    release: kube-prometheus-stack    # required — operator selects on this
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: web-app
  endpoints:
    - port: http      # matches the Service's named port
      path: /metrics
      interval: 30s

Two rules that have bitten us:

  • release: kube-prometheus-stack is mandatory. The Prometheus Operator selects ServiceMonitors by this label; without it the SM is silently ignored.
  • The selector matches the Service’s labels, not its name. See the argocd selector gotcha in the Observability chapter.

components/observability

A Kustomize Component that annotates every Deployment with prometheus.io/{scrape,port,path} so legacy bare-Prometheus or future scrape-discovery agents (Alloy in prometheus.scrape mode) also pick the app up.

# Include in an overlay:
components:
  - ../../components/observability

Belt-and-suspenders next to the ServiceMonitor — the operator picks up the SM, but anything scrape-config-based reads the pod annotations.

Instrumenting an app

Use the /instrument Claude Code slash command in any app repo. It:

  1. Detects the language (Node, Python, Go, Rust).
  2. Adds the Prometheus client library.
  3. Wires up the standard metrics:
    • http_requests_total (counter; labels: method, path, status_code)
    • http_request_duration_seconds (histogram; buckets 0.01–5s)
    • app_info (gauge=1; version, commit, service)
  4. Replaces unstructured logging with structured JSON.
  5. Adds OTel auto-instrumentation with stdout exporter (so traces are already shaped for Tempo if/when it’s deployed).

The metric names are load-bearing — the generic Grafana dashboard (infrastructure/monitoring/kube-prometheus-stack/dashboards/app-template.yaml) queries those strings literally. Don’t rename them per app.

Path label cardinality

Use the route template (/users/:id), not the raw URL (/users/12345), as the path label on metrics. Raw URLs explode cardinality and force Prometheus into series-count panic mode.

Logging convention

{"timestamp":"2026-05-21T03:00:00.123Z","level":"error",
 "message":"DB connection failed","service":"user-api",
 "trace_id":"4bf92f3577b34da6a3ce929d0e0e4736"}

Required keys: timestamp (ISO-8601 UTC), level, message, service. Optional but recommended: trace_id for log-to-trace correlation once Tempo lands.

Alloy auto-detects JSON and parses it into Loki labels. The {namespace="…", app="…"} | json | level="error" LogQL pattern Just Works.

Migration off the homelab

Because every app uses Prometheus client libraries (which remote_write to AMP, GCP managed Prometheus, or Azure Monitor managed Prometheus unchanged), and structured JSON logs (which any log aggregator can ingest), moving an app to a cloud-managed observability stack is a config change — not a re-instrumentation. See docs/cloud-provider-migration.md.

Forgejo & PR-AgentDay-2 Operations