ArgoCD Says OutOfSync Forever. Ignore the Secret Diff and Move On?
A third-party Helm chart moved to ArgoCD now shows OutOfSync forever, resets its admin password and broke its database and agents after one sync. Defend the real fix over the shortcuts.
Kubernetes
CI/CD
SecurityAnswer key reviewed by Branko Petric
Read first: Kubernetes ConfigMaps and Secrets: Managing Configuration
Your decision
After moving a third-party Helm chart from helm install to ArgoCD, the app is permanently OutOfSync, the admin password keeps resetting, one sync left a component crash-looping on database auth and agents in another cluster failing TLS. The team proposes ignoreDifferences on all the Secrets, turning on auto-sync with selfHeal, or going back to running helm upgrade by hand. Which do you pick, or what do you do instead, and why?
The situation
Your team moved a third-party security scanner from a manually run helm install to an ArgoCD Application that points at the same upstream chart and a values file in Git. Since the move, the Application has never shown Synced: the diff always lists several Secrets (TLS certs, a CA bundle, database credentials, an htpasswd Secret). Someone added ignoreDifferences on the htpasswd Secret last week to quiet it. The app has been synced a few times since, and the admin password has stopped working twice. Yesterday an engineer pressed Sync. Afterwards the scanner's job service crash-loops with a Postgres "password authentication failed" error, and the scanner agents running in a second cluster now fail with "x509: certificate signed by unknown authority".
Context
The chart creates its CA, TLS certs and database passwords itself using Helm template functions, and its templates contain lookup calls on those Secrets. Several of the Secrets carry helm.sh/hook: pre-install,pre-upgrade annotations with no delete policy set. The chart's values file documents existingSecret options for most of these credentials and a field for supplying your own CA, all currently unset. The bundled Postgres stores data on a persistent volume that was created during the original helm install. ArgoCD sync is currently manual. In the incident channel three options are on the table: add ignoreDifferences for every Secret in the app, enable auto-sync with selfHeal so Argo keeps it converged, or remove the app from ArgoCD and go back to helm upgrade from a laptop. Other workloads in the organization get their credentials from a cloud secret manager through External Secrets Operator.