Improve backups/jobs reliability and observability

[Kajot-dev]

CHANGE DESCRIPTION

This MR adds:

• Degraded condition to the PerconaPgUpgrade resource and error handling on the path where it was not handled previously (found when debugging K8SPG-1058), so the user can see if there is some kind of error preventing the upgrade from progressing
• ScheduledBackupDegraded condition on the PerconaPGCluster - which allows observability when backup/job creation failed for some reason
• Job creation timeout and backup completion timeout (backup completion default timeout gated > 3.1.0)
• E2E Tests (altough I did not test the tests since I do not have kuttl)

---

Problem:
This MR adresses general backup reliability and jobs observability such as:

• Backups could get stuck indefinitely in Running phase
• If quota was insufficient backup could get stuck indefinietely in Starting Phase (k8s error, hiccup)
• There was no single place to check if Scheduled Backups are healthy
• There was no way other than operator logs to view the error on PerconaPGUpgrade

CHECKLIST

Jira

• Is the Jira ticket created and referenced properly?
• Does the Jira ticket have the proper statuses for documentation (Needs Doc) and QA (Needs QA)?
• Does the Jira ticket link to the proper milestone (Fix Version field)?

Tests

• Is an E2E test/test case added for the new feature/change?
• Are unit tests added where appropriate?

Config/Logging/Testability

• Are all needed new/changed options added to default YAML files?
• Are all needed new/changed options added to the Helm Chart?
• Did we add proper logging messages for operator actions?
• Did we ensure compatibility with the previous version or cluster upgrade process?
• Does the change support oldest and newest supported PG version?
• Does the change support oldest and newest supported Kubernetes version?

[Kajot-dev]

Hmm, I think I need to fix this

[egegunes]

I think the general intent is good.

But according to tests operator panics on cluster deletion, @Kajot-dev could you please check?

[Kajot-dev]

I think the general intent is good.

But according to tests operator panics on cluster deletion, @Kajot-dev could you please check?

Sorry, should be fixed now, I forgot a guard

[Kajot-dev]

retrieving API resource for kuttl.dev/v1beta1, Kind=TestAssert failed: the server could not find the requested resource - I do'nt think I understand

[egegunes]

retrieving API resource for kuttl.dev/v1beta1, Kind=TestAssert failed: the server could not find the requested resource - I do'nt think I understand

Files that use TestAssert, should be named like 20-assert.yaml but you have files like 20-patch-assert.yaml. I think that's the problem. @Kajot-dev

[JNKPercona]

Test Name	Result	Time
backup-enable-disable	passed	00:13:15
builtin-extensions	passed	00:06:51
custom-extensions	passed	00:15:12
cert-manager-tls	passed	00:10:49
cert-management-policy	passed	00:05:07
custom-envs	passed	00:20:24
custom-tls	passed	00:07:03
database-init-sql	passed	00:03:39
demand-backup	failure	00:27:28
demand-backup-offline-snapshot	passed	00:13:42
dynamic-configuration	passed	00:03:22
extra-volumes	passed	00:04:16
finalizers	passed	00:03:58
init-deploy	passed	00:03:10
huge-pages	passed	00:03:11
logical-replica	passed	00:11:34
major-upgrade-14-to-15	passed	00:10:35
major-upgrade-15-to-16	passed	00:11:44
major-upgrade-16-to-17	passed	00:11:12
major-upgrade-17-to-18	passed	00:14:04
major-upgrade-18-to-19	passed	00:10:08
ldap	passed	00:06:00
ldap-tls	passed	00:05:51
logcollection	passed	00:07:18
monitoring	passed	00:11:15
one-pod	passed	00:06:11
repo-host-autogrow	passed	00:01:55
operator-self-healing	failure	01:30:11
pgbouncer	passed	00:08:41
pg-tde	passed	00:15:51
pg-tde-wal-encrypt	failure	00:08:47
pitr	passed	00:13:44
scaling	passed	00:06:23
scheduled-backup	failure	00:37:37
self-healing	failure	00:07:23
sidecars	passed	00:03:23
standby-pgbackrest	passed	00:19:59
standby-streaming	passed	00:16:51
start-from-backup	passed	00:12:35
start-from-volumes	passed	00:04:15
tablespaces	passed	00:07:59
telemetry-transfer	passed	00:04:42
upgrade-consistency	passed	00:07:31
upgrade-minor	passed	00:08:21
users	passed	00:05:34
migration-from-crunchy-standby	passed	00:20:02
migration-from-crunchy-pv	passed	00:09:12
migration-from-crunchy-backup-restore	failure	00:25:31

Summary	Value
Tests Run	48/48
Job Duration	02:33:54
Total Test Time	09:44:10

commit: 9e49866
image: perconalab/percona-postgresql-operator:PR-1776-9e4986606