Lessons — chmonitor on homelab k3s

Two different instances (do not conflate)

Instance	Purpose	Data
Public Cloud/demo (dash.chmonitor.dev)	SaaS + limited public demo	Limited tables, intentionally constrained
Homelab self-host (private Traefik + OAuth)	Full cluster ops UI	All DBs/schemas; expected higher metrics volume

Homelab is supposed to “show more performance” — that is normal, not a bug vs public.

Image (2026-08)

• Canonical: ghcr.io/chmonitor/chmonitor:latest (imagePullPolicy: Always)
• Stale: ghcr.io/duyet/chmonitor:latest — no longer receives CI builds; redeploy pulls same old digest forever
• Helm chart repo: https://charts.chmonitor.dev (chart chmonitor); source repo moved to chmonitor/chmonitor

ClickHouse user for the dashboard

• Use a dedicated user with privilege-based read-only: GRANT SELECT, SHOW, dictGet ON *.* (or narrower), not the server readonly settings profile (readonly=1).

• chmonitor always sets max_execution_time as a session setting on the client. With readonly=1 ClickHouse returns:

  Cannot modify 'max_execution_time' setting in readonly mode
  

  → /api/healthz fails → readiness 503 → pod never Ready.

• Prefer a normal/monitoring profile + grants. Homelab pattern: user chmonitor + role with SELECT/SHOW on all DBs.

Auth (homelab)

• Behind Traefik forwardAuth + oauth2-proxy (GitHub via Dex). See tech-forwardauth-preserve-status.
• chmonitor-scoped middleware stamps a shared secret header (X-Chm-Proxy-Secret); app trusts identity headers only when secret matches (CHM_AUTH_PROVIDER=proxy or trusted depending on image).
• Unauthenticated external hit → 302 to the auth portal is expected.

Ops checklist

1. Confirm image is ghcr.io/chmonitor/chmonitor:… and pull/restart after new builds.
2. Confirm CH user can run SELECT 1 SETTINGS max_execution_time = 60.
3. Confirm /api/healthz reports host up and tables/overview APIs return full schema count.
4. Do not use public-demo connection limits as the model for homelab grants.

Why: future agents diagnose “chmonitor down” or “empty schemas” without mixing public SaaS and private cluster, and avoid the readonly-profile trap. How to apply: when redeploying or debugging homelab chmonitor, check image org + CH profile first; grant schema via SQL privileges, not readonly=1.