Add CTST coverage for Days=0 lifecycle expiration by delthas · Pull Request #2461 · scality/Zenko

delthas · 2026-07-08T09:41:05Z

What does this PR do, and why do we need it?

Adds end-to-end (CTST) coverage for S3 lifecycle Expiration rules with Days=0 ("delete all matching objects as soon as backbeat runs"), and bumps the component images that ship the feature.

Bumps solution/deps.yaml: backbeat 9.5.0-preview.1 → 9.5.0-preview.3 (BB-791), cloudserver 9.4.0-preview.2 → 9.4.0-preview.4 (CLDSRV-928, arsenal 8.5.6), pensieve-api 1.10.1 → 1.11.0-preview.1 (PSVAPI-128).
Adds features/lifecycleExpiration.feature with two @PreMerge scenarios using the direct S3 PutBucketLifecycleConfiguration API:
- non-versioned bucket + Expiration Days=0 → all objects deleted;
- versioned bucket + Expiration Days=0 + NoncurrentVersionExpiration NoncurrentDays=0 → all versions and delete markers removed.
Factors the shared lifecycle-config PUT-with-retry logic out of the transition workflow helper into putBucketLifecycleConfigurationWithRetry, and adds an addExpirationWorkflow helper.

Days=0 makes objects immediately eligible (backbeat lists with no BeforeDate), so the test needs no time-progression or one-day-earlier tricks and is not tagged @Flaky.

The backbeat index-skip heuristic (BB-779) is intentionally out of scope — it is not yet merged and is covered by backbeat unit tests, not e2e.

Which issue does this PR fix?

ZENKO-5314

Special notes for your reviewers:

The new scenarios follow tests/functional/ctst/HOW_TO_WRITE_TESTS.md: unique per-scenario buckets, external-state-only assertions (via ListObjectVersions), poll-based assertions (idempotent after waits), automatic teardown via the global After hook, and no @Flaky.

Issue: ZENKO-5314

bert-e · 2026-07-08T09:41:09Z

Hello delthas,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request `TBA`
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

bert-e · 2026-07-08T09:42:37Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

delthas · 2026-07-13T15:16:58Z

CI failing due to cloudserver bump introducing unrelated issue, tracked in https://github.com/scality/cli-testing/pull/98. Please review regardless! :) (The introduced tests pass.)

DarkIsDude

To be honest it's clear 🙏. Thanks !

delthas · 2026-07-17T12:07:00Z

The check-workflows failure on this PR is unrelated to its changes: softprops/action-gh-release v3.0.2 (published 2026-07-13, pulled in automatically via the floating @v3 tag in release.yaml) changed its GitHub API call sequence, which breaks the tests/workflows/release.spec.ts mocks on every PR based on development/2.14+. Tracked in ZENKO-5317, fix in #2464.

…ration backbeat 9.5.0-preview.1 -> 9.5.0-preview.4 (BB-791; arsenal >=8.5.6 so the v1 LifecycleTask path also honors Days=0) cloudserver 9.4.0-preview.2 -> 9.4.0-preview.4 (CLDSRV-928, arsenal 8.5.6) pensieve-api 1.10.1 -> 1.11.0-preview.1 (PSVAPI-128) Issue: ZENKO-5314

Add an end-to-end test using the S3 PutBucketLifecycleConfiguration API: a Scenario Outline over non-versioned, versioned and suspended buckets that sets a whole-bucket Days=0 expiration (current version, or current and noncurrent versions) and asserts the bucket empties. Factor the lifecycle-config PUT-with-retry logic shared with the transition workflow helper into putBucketLifecycleConfigurationWithRetry, and add an addExpirationWorkflow helper. Days=0 makes objects immediately eligible, so the test needs no time-progression or one-day-earlier tricks. Issue: ZENKO-5314

Picks up the makeAuthV4Request x-amz-content-sha256 fix (CTST-64) so the quota SigV4 admin routes work against cloudserver >= 9.4.0-preview.3, unblocking the CTST Quotas suite. Issue: ZENKO-5314

delthas · 2026-07-20T16:12:20Z

/approve

bert-e · 2026-07-20T18:02:56Z

Build failed

The build for commit did not succeed in branch improvement/ZENKO-5314/days-0-expiration-ctst

The following options are set: approve

makeUpdateRequest signed requests without their payload, so aws4 stamped x-amz-content-sha256 with the empty-body hash while the real body was sent on the wire. cloudserver >= 9.4.0-preview.3 (CLDSRV-932) validates that header against the actual body on non-streaming routes and rejects the mismatch with 400, which made every RestoreObject IAM-policy test fail (400 instead of the expected 403) and silently turned other body-carrying calls into 400s masked by loose assertions. Pass the body to aws4.sign so the signature covers the actual payload. Issue: ZENKO-5314

delthas · 2026-07-21T09:07:10Z

CI investigation summary (run 29758508050, all 8 attempts failed): the three deterministic failures have two distinct root causes, both introduced by the deps bumps in this PR.

test_iam_policies (end2end-2-shards-http) — cloudserver >= 9.4.0-preview.3 (CLDSRV-932, scality/cloudserver#6208) now validates x-amz-content-sha256 against the actual request body. The mocha helper makeUpdateRequest never passed the body to aws4.sign, so it signed the empty-body hash while sending the real payload → all 10 RestoreObject IAM-policy tests failed 400 !== 403 (rejected before policy evaluation). Same class of bug as the cli-testing quota helper fixed in v1.3.1. Fixed by 423f8c9 on this branch.

PRA nominal case (end2end-pra) + Kafka Cleaner (ctst-end2end-sharded) — backbeat >= 9.5.0-preview.2 (BB-740) ships lifecycle alert threshold defaults like 10*60 that render as {{ 10*60 | humanizeDuration }} in alert annotations — an invalid Go template (unexpected "*" in operand). prometheus-operator rejects the rule group and aborts the entire Prometheus config sync, so the PRA federation scrape config and the kafka-cleaner ServiceMonitor never reach Prometheus. kafka-cleaner actually works fine (broker logs show DeleteRecords every minute) — its metrics are just never scraped; platform alerting is also silently dead for the whole run. Tracked in BB-824, fix in scality/backbeat#2778. This PR will need a backbeat bump to 9.5.0-preview.7 once released — until then end2end-pra and the Kafka Cleaner scenario will keep failing deterministically, retries won't help.

Picks up the BB-824 fix: lifecycle alert threshold defaults are now literal seconds instead of arithmetic expressions, which rendered invalid Go-template annotations and made prometheus-operator abort the whole Prometheus config sync (deterministic end2end-pra and Kafka Cleaner CTST failures on backbeat >= 9.5.0-preview.2). Issue: ZENKO-5314

bert-e · 2026-07-21T11:55:27Z

Build failed

The build for commit did not succeed in branch improvement/ZENKO-5314/days-0-expiration-ctst

The following options are set: approve

bert-e · 2026-07-21T14:02:15Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/2.16

The following branches have NOT changed:

development/2.10
development/2.11
development/2.12
development/2.13
development/2.14
development/2.15
development/2.5
development/2.6
development/2.7
development/2.8
development/2.9

This pull request did not target the following hotfix branch(es) so they
were left untouched:

hotfix/2.13.5

Please check the status of the associated issue ZENKO-5314.

Goodbye delthas.

The following options are set: approve

scality deleted a comment from bert-e Jul 8, 2026

delthas force-pushed the improvement/ZENKO-5314/days-0-expiration-ctst branch 2 times, most recently from 9061fa1 to ac6b65e Compare July 10, 2026 14:50

delthas marked this pull request as ready for review July 13, 2026 15:17

delthas requested review from a team, DarkIsDude and francoisferrand and removed request for DarkIsDude July 13, 2026 15:17

DarkIsDude approved these changes Jul 15, 2026

View reviewed changes

Comment thread tests/functional/ctst/features/lifecycleExpiration.feature Outdated

francoisferrand approved these changes Jul 16, 2026

View reviewed changes

Comment thread tests/functional/ctst/features/lifecycleExpiration.feature Outdated

delthas force-pushed the improvement/ZENKO-5314/days-0-expiration-ctst branch 2 times, most recently from 2998a3c to 49b0bfc Compare July 17, 2026 08:06

delthas added 3 commits July 20, 2026 18:11

Bump cli-testing to v1.3.1

9083c90

Picks up the makeAuthV4Request x-amz-content-sha256 fix (CTST-64) so the quota SigV4 admin routes work against cloudserver >= 9.4.0-preview.3, unblocking the CTST Quotas suite. Issue: ZENKO-5314

delthas force-pushed the improvement/ZENKO-5314/days-0-expiration-ctst branch from 49b0bfc to 9083c90 Compare July 20, 2026 16:11

delthas mentioned this pull request Jul 21, 2026

Use literal seconds for lifecycle alert threshold defaults scality/backbeat#2778

Merged

This was referenced Jul 21, 2026

Add CRR cascaded tests #2447

Merged

Add Cucumber tests for multi-destination CRR #2451

Merged

bert-e merged commit 142be09 into development/2.16 Jul 21, 2026
37 of 38 checks passed

bert-e deleted the improvement/ZENKO-5314/days-0-expiration-ctst branch July 21, 2026 14:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add CTST coverage for Days=0 lifecycle expiration - #2461

Add CTST coverage for Days=0 lifecycle expiration#2461
bert-e merged 5 commits into
development/2.16from
improvement/ZENKO-5314/days-0-expiration-ctst

delthas commented Jul 8, 2026

Uh oh!

bert-e commented Jul 8, 2026

Uh oh!

bert-e commented Jul 8, 2026

Uh oh!

delthas commented Jul 13, 2026

Uh oh!

DarkIsDude left a comment

Uh oh!

Uh oh!

Uh oh!

delthas commented Jul 17, 2026 •

edited by atlassian Bot

Loading

Uh oh!

delthas commented Jul 20, 2026

Uh oh!

bert-e commented Jul 20, 2026

Uh oh!

delthas commented Jul 21, 2026 •

edited by atlassian Bot

Loading

Uh oh!

bert-e commented Jul 21, 2026

Uh oh!

bert-e commented Jul 21, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

delthas commented Jul 8, 2026

Uh oh!

bert-e commented Jul 8, 2026

Hello delthas,

Uh oh!

bert-e commented Jul 8, 2026

Waiting for approval

Uh oh!

delthas commented Jul 13, 2026

Uh oh!

DarkIsDude left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

delthas commented Jul 17, 2026 • edited by atlassian Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

delthas commented Jul 20, 2026

Uh oh!

bert-e commented Jul 20, 2026

Build failed

Uh oh!

delthas commented Jul 21, 2026 • edited by atlassian Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bert-e commented Jul 21, 2026

Build failed

Uh oh!

bert-e commented Jul 21, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

delthas commented Jul 17, 2026 •

edited by atlassian Bot

Loading

delthas commented Jul 21, 2026 •

edited by atlassian Bot

Loading