fix clusterpolicy fluctuation when nvidiadriver upgrade is happening

[rahulait]

Description

Fixes ClusterPolicy status fluctuation during NVIDIADriver rolling upgrades.

When an NVIDIADriver upgrade is in progress (nodes transitioning through cordon/drain/restart/validation states), ClusterPolicy now explicitly reports NotReady with a clear reason, instead of oscillating between ready/not-ready as individual node states change.

The approach:

• After reconciling all operand states, check if any NVIDIADriver-owned node has an incomplete upgrade state (pending, in-progress, or failed).
• If so, hold overallStatus = NotReady and append descriptive reasons to the condition message.
• Refactor the not-ready message into a structured "; "-joined format that combines operand states and upgrade reasons.

Checklist

• No secrets, sensitive information, or unrelated changes
• Lint checks passing (make lint)
• Generated assets in-sync (make validate-generated-assets)
• Go mod artifacts in-sync (make validate-modules)
• Test cases are added for new code paths

Testing

Tested this change on a 3 node cluster.

Status when the upgrade starts:

== Node owner and upgrade labels ==
NODE            GPU   OWNER    UPGRADE_STATE         CLUSTERPOLICY  NVIDIADRIVER
a1u1g-mil-0369  true  default  upgrade-required      notReady       notReady
ipp2-1815       true  default  pod-restart-required  notReady       notReady
ipp2-2152       true  default  upgrade-required      notReady       notReady

k get clusterpolicy cluster-policy -o yaml | yq .status
conditions:
  - lastTransitionTime: "2026-07-24T18:05:33Z"
    message: ""
    reason: Error
    status: "False"
    type: Ready
  - lastTransitionTime: "2026-07-24T18:05:33Z"
    message: 'ClusterPolicy is not ready; states not ready: [state-operator-validation state-dcgm-exporter]; NVIDIADriver upgrade has not completed; one or more NVIDIADriver-owned Nodes are marked pending, in-progress, or failed'
    reason: OperandNotReady
    status: "True"
    type: Error
namespace: gpu-operator
state: notReady

Status when all operands are ready but driver upgrade is still in progress:

== Node owner and upgrade labels ==
NODE            GPU   OWNER    UPGRADE_STATE         CLUSTERPOLICY  NVIDIADRIVER
a1u1g-mil-0369  true  default  upgrade-required      notReady       notReady
ipp2-1815       true  default  upgrade-done          notReady       notReady
ipp2-2152       true  default  upgrade-required      notReady       notReady

k get clusterpolicy cluster-policy -o yaml | yq .status
conditions:
  - lastTransitionTime: "2026-07-24T18:05:33Z"
    message: ""
    reason: Error
    status: "False"
    type: Ready
  - lastTransitionTime: "2026-07-24T18:05:33Z"
    message: ClusterPolicy is not ready; NVIDIADriver upgrade has not completed; one or more NVIDIADriver-owned Nodes are marked pending, in-progress, or failed
    reason: OperandNotReady
    status: "True"
    type: Error
namespace: gpu-operator
state: notReady

Status when the upgrade finishes:

== Node owner and upgrade labels ==
NODE            GPU   OWNER    UPGRADE_STATE  CLUSTERPOLICY  NVIDIADRIVER
a1u1g-mil-0369  true  default  upgrade-done   ready          ready
ipp2-1815       true  default  upgrade-done   ready          ready
ipp2-2152       true  default  upgrade-done   ready          ready

k get clusterpolicy cluster-policy -o yaml | yq .status
conditions:
  - lastTransitionTime: "2026-07-24T18:20:17Z"
    message: ClusterPolicy is ready as all resources have been successfully reconciled
    reason: Reconciled
    status: "True"
    type: Ready
  - lastTransitionTime: "2026-07-24T18:20:17Z"
    message: ""
    reason: Ready
    status: "False"
    type: Error
namespace: gpu-operator
state: ready