Skip to content

Capture pull request labels before archiving a review stack - #1501

Open
c-gerke wants to merge 1 commit into
Shopify:mainfrom
c-gerke:capture-labels-before-archiving
Open

Capture pull request labels before archiving a review stack#1501
c-gerke wants to merge 1 commit into
Shopify:mainfrom
c-gerke:capture-labels-before-archiving

Conversation

@c-gerke

@c-gerke c-gerke commented Sep 9, 2026

Copy link
Copy Markdown

Problem

LabelCapturingHandler is the only handler that writes pull_request.labels on a labeled or
unlabeled event, and it skips stacks that are already archived — labeled_active_stack? and
unlabeled_active_stack? both require !stack.archived?. Webhooks.default_handlers registers
it after LabeledHandler and UnlabeledHandler.

So on a label event that archives the stack, the capture never runs:

  1. The lifecycle handler sees archive? and calls stack.archive!.
  2. LabelCapturingHandler runs next, finds the stack archived, and skips the capture.

The persisted list keeps the label that caused the teardown until some later event happens to
rewrite it. Both provisioning behaviors are affected, in opposite directions:

Behavior Event that archives Result
allow_with_label provisioning label removed the archived stack still carries the label whose removal archived it
prevent_with_label blocking label added the label that archived the stack is never stored at all

Any other label is unaffected under either behavior: nothing archives, the stack stays active,
and the capture runs normally. The defect is specific to the label named by
provisioning_label_name.

The column is not only informational — ReviewStack#env turns each stored label into a
LABEL_NAME=true environment variable for the stack.

Approach

Capture the payload's labels in LabeledHandler#handle and UnlabeledHandler#handle,
immediately before the archive, so an archived stack's labels describe the event that archived
it.

Direction Why not
Register LabelCapturingHandler before the lifecycle handlers The capture would run while the stack is still archived (or absent), so the unarchive and creation paths would lose the capture they have today.
Let the capture run for archived stacks too An archived stack's labels would become a running mirror of the pull request instead of a record of the event that archived it.
Capture in ReviewStackAdapter#archive! One call site instead of two, and its blank/archived guards are already there — but ClosedHandler archives through the same method, so every pull request close would take on a label write and a pull_request update hook.
Capture in the lifecycle handlers Chosen. Only the archive path is missing a capture, so only the archive path changes.

The capture is guarded on the stack being present and not already archived, which keeps an
archived stack's labels a snapshot rather than a running mirror, and keeps repeat deliveries
from rewriting them.

capture_labels is duplicated across the two handlers, matching how archive?, unarchive?,
stack, repository, and pull_request_label_names are already duplicated between them. Glad
to extract a shared module instead if you would rather have one.

Scope

  • Unarchive and creation are untouched. LabelCapturingHandler still captures for a stack
    that a label event brings back into service, and ReviewStackAdapter#create! still stores the
    payload's labels on a newly created stack.
  • No error handling around the new call. It is the same update!(labels:) that
    LabelCapturingHandler#capture_labels already performs on every label event against an active
    stack, so it carries the same exposure. Rescuing here and nowhere else seemed worse than
    matching the existing behavior.
  • One extra pull_request update hook on the archive path, emitted before the stack is
    archived.
  • Not a backfill. Stacks archived before this change keep their stale labels.
  • Not the last word on the column. EditedHandler and AssignedHandler call
    update(github_pull_request:) with no archived check, and PullRequest#github_pull_request=
    assigns labels, so an edited or assigned event on an archived pull request overwrites
    what was captured. That does not reintroduce this bug — the overwrite writes current labels,
    which no longer carry the removed provisioning label — so it is left alone, but the capture
    should not be read as permanent.

Verification

Eight cases added to the existing handler tests, four in each file. They use each file's local
create_stack / create_archived_stack / configure_provisioning_behavior helpers and the
existing pull_request_labeled / pull_request_unlabeled payload fixtures; no new fixtures were
needed.

  • unlabeled_handler_test.rb — the allow_with_label case, the empty-list case where the last
    label is removed, no capture when the stack is already archived, no capture when unarchiving,
    and one case that dispatches through Shipit::Webhooks.for_event("pull_request") so the whole
    registered chain runs in its real order rather than one handler in isolation.
  • labeled_handler_test.rb — the prevent_with_label case, no capture when the stack is already
    archived, no capture when unarchiving.

Confirmed failing without the fix. Reverting just the two handle changes fails 4 of the 8:
the allow_with_label case, the prevent_with_label case, the empty-list case, and the
full-chain case. The other 4 assert unchanged behavior and stay green either way, which is what
they are there for.

Full suite locally, sqlite: 1253 runs, 3842 assertions, 0 failures, 0 errors. Rubocop clean on
the four changed Ruby files.

`LabelCapturingHandler` is the only handler that writes
`pull_request.labels` on a `labeled` or `unlabeled` event, and it skips
stacks that are already archived. `Webhooks.default_handlers` registers
it after `LabeledHandler` and `UnlabeledHandler`, so on the one label
event that matters most the capture never runs:

1. The lifecycle handler sees `archive?` and archives the stack.
2. `LabelCapturingHandler` runs next, finds the stack archived, and
   skips the capture.

The persisted list therefore keeps the label that caused the teardown
until some later event happens to rewrite it. Both provisioning
behaviors are affected, in opposite directions: under
`allow_with_label` an archived stack still carries the provisioning
label whose removal archived it, and under `prevent_with_label` the
blocking label that archived the stack is never stored at all.

The column is not only informational. `ReviewStack#env` turns each
stored label into a `LABEL_NAME=true` environment variable for the
stack.

Capture the payload's labels in the lifecycle handlers, immediately
before the archive, so an archived stack's labels describe the event
that archived it.

The capture is deliberately narrow. A stack that is already archived is
left alone, keeping its labels a snapshot of the event that archived it
rather than a running mirror of the pull request. The unarchive and
creation paths are untouched and keep capturing through
`LabelCapturingHandler` and `ReviewStackAdapter#create!` as before.
@c-gerke
c-gerke marked this pull request as ready for review September 9, 2026 16:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant