Server-Side Diff is hiding manual changes in managed resources

[langesven]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

ArgoCD with server-side diff+apply (SSD / SSA for the search terms) enabled does not behave equal to the client-side diff+apply version.
Objects that should be entirely managed by ArgoCD have diffs that are not reflected in the UI because of a manager mismatch (argocd-controller vs. argocd-server)

To Reproduce

1. Configure ArgoCD to use Server-Side Diff

   argo-cd:
     configs:
       params:
         controller.diff.server.side: true

2. Deploy an Application using Server-Side Apply

   apiVersion: argoproj.io/v1alpha1
   kind: Application
   metadata:
     name: my-app
     namespace: argocd
   spec:
     destination:
       name: in-cluster
       namespace: my-app
     project: default
     sources:
       - helm:
           releaseName: my-app
         path: charts/my-app
         repoURL: https://github.com/org/chart.git
         targetRevision: HEAD
     syncPolicy:
       automated:
         enabled: false
         prune: true
         selfHeal: true
       retry:
         backoff:
           duration: 5s
           factor: 2
           maxDuration: 3m0s
         limit: 3
       syncOptions:
         - ServerSideApply=true

3. Disable Auto-Sync after everything is deployed

4. Manually edit a resource using the ArgoCD UI

5. Observe that "Diff" for the resource is empty, application does not show as "OutOfSync"

6. Compare "Live Manifest" and "Desired Manifest" with a diff utility of your choosing - notice that there is a diff

From my real test, I've been using a Karpenter NodePool here, but I've been running into issues with all kinds of resources before so this shouldn't make any relevant difference.
I added .spec.limits.memory: 2000Gi manually by editing the NodePool resource using the ArgoCD UI (click on resource, in live manifest hit Edit, add 2 lines, Save)

The Diff tab in the resource detail page shows as empty. A manual diff of the "Desired Manifest" vs. "Live Manifest" looks like this

6a7,10
>     karpenter.sh/nodepool-hash: '15659376333646248543'
>     karpenter.sh/nodepool-hash-version: v3
>   creationTimestamp: '2026-03-11T17:44:47Z'
>   generation: 5
16a21,22
>   resourceVersion: '2017280494'
>   uid: 676498de-5c85-46e5-90ab-36830c66bba4
18a25,26
>     budgets:
>       - nodes: 10%
20a29,30
>   limits:
>     memory: 2000Gi
54a65,93
> status:
>   conditions:
>     - lastTransitionTime: '2026-03-11T17:44:47Z'
>       message: ''
>       observedGeneration: 5
>       reason: ValidationSucceeded
>       status: 'True'
>       type: ValidationSucceeded
>     - lastTransitionTime: '2026-03-11T17:44:56Z'
>       message: ''
>       observedGeneration: 5
>       reason: NodeClassReady
>       status: 'True'
>       type: NodeClassReady
>     - lastTransitionTime: '2026-04-08T14:20:09Z'
>       message: ''
>       observedGeneration: 5
>       reason: Ready
>       status: 'True'
>       type: Ready
>   resources:
>     cpu: '8'
>     ephemeral-storage: 102334Mi
>     hugepages-1Gi: '0'
>     hugepages-2Mi: '0'
>     memory: 16077376Ki
>     nodes: '1'
>     pods: '58'
>     vpc.amazonaws.com/pod-eni: '38'

I believe this to be due to a manager mismatch. The changes that ArgoCD auto-syncs are owned by argocd-controller but the changes I made manually are owned by `argocd-server.

  managedFields:
    - apiVersion: karpenter.sh/v1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            f:argocd.argoproj.io/tracking-id: {}
          f:labels:
            f:app.kubernetes.io/instance: {}
            f:app.kubernetes.io/managed-by: {}
            f:app.kubernetes.io/name: {}
            f:app.kubernetes.io/version: {}
            f:helm.sh/chart: {}
        f:spec:
          f:disruption:
            f:consolidateAfter: {}
            f:consolidationPolicy: {}
          f:template:
            f:spec:
              f:expireAfter: {}
              f:nodeClassRef:
                f:group: {}
                f:kind: {}
                f:name: {}
              f:requirements: {}
              f:taints: {}
      manager: argocd-controller
      operation: Apply
      time: '2026-04-08T10:48:28Z'

and

    - apiVersion: karpenter.sh/v1
      fieldsType: FieldsV1
      fieldsV1:
        f:spec:
          f:limits:
            .: {}
            f:memory: {}
      manager: argocd-server
      operation: Update
      time: '2026-04-08T14:20:09Z'

Expected behavior

Doing these same steps with client-side apply/diff correctly (in my opinion anyway) shows an "OutOfSync" state here because changes that are not in Git state are live.

With server-side apply + server-side diff enabled any number of manual changes could be live and I have no way of finding out. For what it's worth, the same is also happening when using client-side apply + server-side diff - so this appears to be largely server-side diff related, not so much apply.

It's not unusual that people might disable auto-sync during an incident, manually patch in changes to resolve the situation, and would then expect once they re-enable auto-sync that their manual changes would be reverted.
But this is simply not happening and things that I believe to be part of git state (or even better, not part of git state anymore) might not be applied or not be removed, leaving me in a "somewhat git managed" state 😅

In my particular situation here I've removed the .spec.limits field that was previously managed by ArgoCD. But this particular field has first been adjusted manually via the Edit flow, before it made its way into Git state. I assume that's how it ended up staying around even though it's now no longer part of the Git state.

I strongly suspect open issues like #16092 are suffering from the same behavior.

Screenshots

Version

argocd: v3.2.6+65b0293
  BuildDate: 2026-01-22T19:37:41Z
  GitCommit: 65b029342d656c03c57f0d0e14433438750c8f5d
  GitTreeState: clean
  GoVersion: go1.25.5
  Compiler: gc
  Platform: linux/arm64

Logs

Paste any relevant application logs here.

[ppapapetrou76]

Hey @langesven - I'm trying to reproduce your case and I have to this point
I followed your example and indeed when I change something to a resource the diff shows empty , but the app is showing as "Progressing" but it never completes the state change

However I cannot get the manual diff that you are describing .... Manual diff shows nothing again ( with both ServerSideDiff enabled and disabled )

Can you please share the exact diff command you are using .

Also when I remove ServerSIdeApply I have exactly the same behavior - app stuck in "progressing"

So overall I experience a poor behavior but I'm not sure if it's the same behavior you are describing 😄