Skip to content

fix: stop a failed Helm install/upgrade from silently uninstalling the release #1912

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
gianlucam76 merged 1 commit into from
Aug 12, 2026
Merged

fix: stop a failed Helm install/upgrade from silently uninstalling the release #1912

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions api/v1beta1/spec.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -298,6 +298,19 @@ type HelmInstallOptions struct {
// +kubebuilder:default:=false
// +optional
TakeOwnership bool `json:"takeOwnership,omitempty"`

// RecoverAfterConsecutiveFailures is the number of consecutive install failures for this
// chart after which Sveltos uninstalls any existing release under this name before
// retrying, to clear potentially stale Helm release history that would otherwise keep
// blocking every subsequent install attempt. This only ever runs when there is no
// currently deployed release to protect: a release that is deployed, or mid-upgrade, or
// failed while already existing, is always retried through helm upgrade instead, never
// through this. It only applies to a release that was never successfully installed, or
// was already cleanly uninstalled.
// Default to 5
// +kubebuilder:default:=5
// +optional
RecoverAfterConsecutiveFailures int `json:"recoverAfterConsecutiveFailures,omitempty"`
}

type HelmUpgradeOptions struct {
Expand Down
13 changes: 13 additions & 0 deletions config/crd/bases/config.projectsveltos.io_clusterprofiles.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -318,6 +318,19 @@ spec:
disable hooks on install
Default to false
type: boolean
recoverAfterConsecutiveFailures:
default: 5
description: |-
RecoverAfterConsecutiveFailures is the number of consecutive install failures for this
chart after which Sveltos uninstalls any existing release under this name before
retrying, to clear potentially stale Helm release history that would otherwise keep
blocking every subsequent install attempt. This only ever runs when there is no
currently deployed release to protect: a release that is deployed, or mid-upgrade, or
failed while already existing, is always retried through helm upgrade instead, never
through this. It only applies to a release that was never successfully installed, or
was already cleanly uninstalled.
Default to 5
type: integer
replace:
default: true
description: Replaces if set indicates to replace an
Expand Down
13 changes: 13 additions & 0 deletions config/crd/bases/config.projectsveltos.io_clusterpromotions.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -218,6 +218,19 @@ spec:
disable hooks on install
Default to false
type: boolean
recoverAfterConsecutiveFailures:
default: 5
description: |-
RecoverAfterConsecutiveFailures is the number of consecutive install failures for this
chart after which Sveltos uninstalls any existing release under this name before
retrying, to clear potentially stale Helm release history that would otherwise keep
blocking every subsequent install attempt. This only ever runs when there is no
currently deployed release to protect: a release that is deployed, or mid-upgrade, or
failed while already existing, is always retried through helm upgrade instead, never
through this. It only applies to a release that was never successfully installed, or
was already cleanly uninstalled.
Default to 5
type: integer
replace:
default: true
description: Replaces if set indicates to replace
Expand Down
13 changes: 13 additions & 0 deletions config/crd/bases/config.projectsveltos.io_clustersummaries.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -355,6 +355,19 @@ spec:
disable hooks on install
Default to false
type: boolean
recoverAfterConsecutiveFailures:
default: 5
description: |-
RecoverAfterConsecutiveFailures is the number of consecutive install failures for this
chart after which Sveltos uninstalls any existing release under this name before
retrying, to clear potentially stale Helm release history that would otherwise keep
blocking every subsequent install attempt. This only ever runs when there is no
currently deployed release to protect: a release that is deployed, or mid-upgrade, or
failed while already existing, is always retried through helm upgrade instead, never
through this. It only applies to a release that was never successfully installed, or
was already cleanly uninstalled.
Default to 5
type: integer
replace:
default: true
description: Replaces if set indicates to replace
Expand Down
13 changes: 13 additions & 0 deletions config/crd/bases/config.projectsveltos.io_profiles.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -318,6 +318,19 @@ spec:
disable hooks on install
Default to false
type: boolean
recoverAfterConsecutiveFailures:
default: 5
description: |-
RecoverAfterConsecutiveFailures is the number of consecutive install failures for this
chart after which Sveltos uninstalls any existing release under this name before
retrying, to clear potentially stale Helm release history that would otherwise keep
blocking every subsequent install attempt. This only ever runs when there is no
currently deployed release to protect: a release that is deployed, or mid-upgrade, or
failed while already existing, is always retried through helm upgrade instead, never
through this. It only applies to a release that was never successfully installed, or
was already cleanly uninstalled.
Default to 5
type: integer
replace:
default: true
description: Replaces if set indicates to replace an
Expand Down
56 changes: 30 additions & 26 deletions controllers/handlers_helm.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,11 +100,12 @@ var (
)

const (
notInstalledMessage = "Not installed yet and action is uninstall"
defaultMaxHistory = 2
defaultDeletionPropagation = "background"
defaultChartVersion = "0.1.0"
conditionStatusTrue = "True"
notInstalledMessage = "Not installed yet and action is uninstall"
defaultMaxHistory = 2
defaultRecoverAfterConsecutiveFailures = 5
defaultDeletionPropagation = "background"
defaultChartVersion = "0.1.0"
conditionStatusTrue = "True"
)

type registryClientOptions struct {
Expand Down Expand Up @@ -1590,11 +1591,11 @@ func handleInstall(ctx context.Context, dCtx *deploymentContext,

logger.V(logs.LogDebug).Info("install helm release")

maxHistory := uint(getMaxHistoryValue(currentChart.Options))
recoverAfter := getRecoverAfterConsecutiveFailuresValue(currentChart.Options)

if !isPullMode {
if fs := getFeatureSummaryForFeatureID(dCtx.clusterSummary, libsveltosv1beta1.FeatureHelm); fs != nil {
if fs.ConsecutiveFailures%maxHistory == 0 && fs.FailureMessage != nil {
if fs.ConsecutiveFailures%recoverAfter == 0 && fs.FailureMessage != nil {
err := doUninstallRelease(ctx, dCtx.clusterSummary, currentChart, kubeconfig, registryOptions, logger)
if err != nil {
// Ignore release not found error
Expand Down Expand Up @@ -2801,33 +2802,25 @@ func recoverRelease(ctx context.Context, clusterSummary *configv1beta1.ClusterSu
requestedChart.ReleaseNamespace, kubeconfig, registryOptions, requestedChart, logger)
}

// shouldInstall returns true if action is not uninstall and either there
// is no installed or version, or version is same requested by customer but status is
// not yet deployed
// shouldInstall returns true if action is not uninstall and there is no release to manage
// the lifecycle of: either none was ever created, or the last one was cleanly uninstalled.
// Any other state (deployed at a different version, failed, pending-*, superseded, ...) is
// an existing release that shouldUpgrade must handle instead. In particular, a release whose
// last upgrade/install attempt failed still has a release record (Helm keeps history for
// failed attempts too, stamped with the version that attempt tried to reach) — that must not
// be mistaken for "nothing to do here but install", since it goes through handleInstall's
// own uninstall-on-repeated-failure recovery, which is destructive and must stay reserved for
// the case where there is genuinely nothing deployed to lose.
func shouldInstall(currentRelease *releaseInfo, requestedChart *configv1beta1.HelmChart) bool {
if requestedChart.HelmChartAction == configv1beta1.HelmChartActionUninstall {
return false
}

if currentRelease != nil &&
currentRelease.Status == releasecommon.StatusUninstalled.String() {

if currentRelease == nil {
return true
}

if currentRelease != nil &&
currentRelease.ChartVersion != requestedChart.ChartVersion {

return false
}

if currentRelease != nil &&
currentRelease.Status == releasecommon.StatusDeployed.String() {

return false
}

return true
return currentRelease.Status == releasecommon.StatusUninstalled.String()
}

// shouldUpgrade returns true if action is not uninstall and current installed chart is different
Expand Down Expand Up @@ -4291,6 +4284,17 @@ func getMaxHistoryValue(options *configv1beta1.HelmOptions) int {
return defaultMaxHistory
}

// getRecoverAfterConsecutiveFailuresValue returns the number of consecutive install failures
// handleInstall tolerates, for a release it is not currently protecting (see shouldInstall),
// before uninstalling any stale release record under that name and retrying.
func getRecoverAfterConsecutiveFailuresValue(options *configv1beta1.HelmOptions) uint {
if options != nil && options.InstallOptions.RecoverAfterConsecutiveFailures > 0 {
return uint(options.InstallOptions.RecoverAfterConsecutiveFailures)
}

return defaultRecoverAfterConsecutiveFailures
}

func getCleanupOnFailValue(options *configv1beta1.HelmOptions) bool {
if options != nil {
return options.UpgradeOptions.CleanupOnFail
Expand Down
29 changes: 29 additions & 0 deletions controllers/handlers_helm_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,35 @@ var _ = Describe("HandlersHelm", func() {
Expect(controllers.ShouldInstall(nil, requestChart)).To(BeFalse())
})

It("shouldInstall returns false for a failed release even when its recorded version matches the request", func() {
// A failed upgrade/install attempt still leaves a release record behind, stamped
// with the version that attempt tried (and failed) to reach. That must not be
// mistaken for "nothing to do here but install" - it has to be retried as an
// upgrade instead, never routed through handleInstall's uninstall-on-repeated-
// failure recovery.
currentRelease := &controllers.ReleaseInfo{
Status: releasecommon.StatusFailed.String(),
ChartVersion: testChartVersion253,
}
requestChart := &configv1beta1.HelmChart{
ChartVersion: testChartVersion253,
HelmChartAction: configv1beta1.HelmChartActionInstall,
}
Expect(controllers.ShouldInstall(currentRelease, requestChart)).To(BeFalse())
})

It("shouldInstall returns true when the current release was cleanly uninstalled", func() {
currentRelease := &controllers.ReleaseInfo{
Status: releasecommon.StatusUninstalled.String(),
ChartVersion: testChartVersion253,
}
requestChart := &configv1beta1.HelmChart{
ChartVersion: testChartVersion253,
HelmChartAction: configv1beta1.HelmChartActionInstall,
}
Expect(controllers.ShouldInstall(currentRelease, requestChart)).To(BeTrue())
})

It("shouldUninstall returns false when there is no current release installed", func() {
requestChart := &configv1beta1.HelmChart{
ChartVersion: testChartVersion253,
Expand Down
13 changes: 13 additions & 0 deletions lib/crd/clusterprofiles.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -337,6 +337,19 @@ spec:
disable hooks on install
Default to false
type: boolean
recoverAfterConsecutiveFailures:
default: 5
description: |-
RecoverAfterConsecutiveFailures is the number of consecutive install failures for this
chart after which Sveltos uninstalls any existing release under this name before
retrying, to clear potentially stale Helm release history that would otherwise keep
blocking every subsequent install attempt. This only ever runs when there is no
currently deployed release to protect: a release that is deployed, or mid-upgrade, or
failed while already existing, is always retried through helm upgrade instead, never
through this. It only applies to a release that was never successfully installed, or
was already cleanly uninstalled.
Default to 5
type: integer
replace:
default: true
description: Replaces if set indicates to replace an
Expand Down
13 changes: 13 additions & 0 deletions lib/crd/clusterpromotions.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -237,6 +237,19 @@ spec:
disable hooks on install
Default to false
type: boolean
recoverAfterConsecutiveFailures:
default: 5
description: |-
RecoverAfterConsecutiveFailures is the number of consecutive install failures for this
chart after which Sveltos uninstalls any existing release under this name before
retrying, to clear potentially stale Helm release history that would otherwise keep
blocking every subsequent install attempt. This only ever runs when there is no
currently deployed release to protect: a release that is deployed, or mid-upgrade, or
failed while already existing, is always retried through helm upgrade instead, never
through this. It only applies to a release that was never successfully installed, or
was already cleanly uninstalled.
Default to 5
type: integer
replace:
default: true
description: Replaces if set indicates to replace
Expand Down
13 changes: 13 additions & 0 deletions lib/crd/clustersummaries.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -374,6 +374,19 @@ spec:
disable hooks on install
Default to false
type: boolean
recoverAfterConsecutiveFailures:
default: 5
description: |-
RecoverAfterConsecutiveFailures is the number of consecutive install failures for this
chart after which Sveltos uninstalls any existing release under this name before
retrying, to clear potentially stale Helm release history that would otherwise keep
blocking every subsequent install attempt. This only ever runs when there is no
currently deployed release to protect: a release that is deployed, or mid-upgrade, or
failed while already existing, is always retried through helm upgrade instead, never
through this. It only applies to a release that was never successfully installed, or
was already cleanly uninstalled.
Default to 5
type: integer
replace:
default: true
description: Replaces if set indicates to replace
Expand Down
13 changes: 13 additions & 0 deletions lib/crd/profiles.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -337,6 +337,19 @@ spec:
disable hooks on install
Default to false
type: boolean
recoverAfterConsecutiveFailures:
default: 5
description: |-
RecoverAfterConsecutiveFailures is the number of consecutive install failures for this
chart after which Sveltos uninstalls any existing release under this name before
retrying, to clear potentially stale Helm release history that would otherwise keep
blocking every subsequent install attempt. This only ever runs when there is no
currently deployed release to protect: a release that is deployed, or mid-upgrade, or
failed while already existing, is always retried through helm upgrade instead, never
through this. It only applies to a release that was never successfully installed, or
was already cleanly uninstalled.
Default to 5
type: integer
replace:
default: true
description: Replaces if set indicates to replace an
Expand Down
Loading