Add release validator and regression tracking by alisawavezen12 · Pull Request #590 · codex-team/hawk.workers

alisawavezen12 · 2026-09-17T14:58:20Z

Summary

Introduce release-based detection of likely fixed events and track regressions when resolved events occur again.

Changes

Add a scheduled release-validator worker.
Check unchecked releases between 24 hours and 30 days old.
Mark events with the first eligible resolvedInRelease.
Store release names directly on new repetitions.
Track regressionInRelease for occurrences in the resolved or newer releases.
Support repeated resolution and regression cycles.
Add tests for validation, grouping, release maps, and regression scenarios.

codex-assistant · 2026-09-17T14:58:57Z

Please add a PR description 🙂

Respect the reviewers — a description helps others understand the changes and review them faster. Keep it short, clear, and to the point. It also serves as documentation for future reference.

The PR was moved to Draft until a description is added.

codex-assistant · 2026-09-18T10:26:59Z

Please add a PR description 🙂

Respect the reviewers — a description helps others understand the changes and review them faster. Keep it short, clear, and to the point. It also serves as documentation for future reference.

The PR was moved to Draft until a description is added.

codex-assistant · 2026-09-18T15:07:35Z

Please add a PR description 🙂

Respect the reviewers — a description helps others understand the changes and review them faster. Keep it short, clear, and to the point. It also serves as documentation for future reference.

The PR was moved to Draft until a description is added.

codex-assistant · 2026-09-18T15:23:28Z

Please add a PR description 🙂

Respect the reviewers — a description helps others understand the changes and review them faster. Keep it short, clear, and to the point. It also serves as documentation for future reference.

The PR was moved to Draft until a description is added.

codex-assistant · 2026-09-18T15:25:01Z

Please add a PR description 🙂

Respect the reviewers — a description helps others understand the changes and review them faster. Keep it short, clear, and to the point. It also serves as documentation for future reference.

The PR was moved to Draft until a description is added.

codex-assistant · 2026-09-18T15:25:53Z

Thanks for adding a description — the PR is now marked as Ready for Review.

neSpecc · 2026-09-19T11:38:29Z

        return this.saveRepetition(task.projectId, newRepetition);
      });

+      if (task.payload.release && existedEvent.resolvedInRelease) {


we need to check if task.payload.release is newer than existedEvent.resolvedInRelease

we do this later in the markRegression function.

then we need to rename markRegression to checkForRegression

Copilot

Copilot review overview

🟡 Changes recommended

Release history compatibility, durable regression reconciliation, deployment wiring, and bounded database processing remain unresolved.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 3 · 1

Open (4)

Release validator worker is not deployed or scheduled · New
Regression recording can leave resolved events unreconciled · New
Legacy repetitions are skipped during release validation · New
Unbounded event queries can exceed limits and exhaust memory · New

What changed in this PR

Implements release-based event resolution and regression tracking across the validator and grouper workers.

Changes:

Adds release validation logic and worker scaffolding.
Records repetition releases and regression cycles.
Adds supporting tests and updates shared types.

File	Description
yarn.lock	Updates Hawk types dependency.
workers/release-validator/tests/validate-releases.test.ts	Tests release validation scenarios.
workers/release-validator/tests/utils/group-releases-by-project.test.ts	Tests release grouping.
workers/release-validator/tests/utils/build-event-release-map.test.ts	Tests event-release mapping.
workers/release-validator/tests/index.test.ts	Tests worker queue configuration.
workers/release-validator/src/validate-releases.ts	Implements release validation.
workers/release-validator/src/utils/group-releases-by-project.ts	Groups releases by project.
workers/release-validator/src/utils/build-event-release-map.ts	Maps events to releases.
workers/release-validator/src/index.ts	Defines the validator worker.
workers/release-validator/README.md	Documents worker behavior.
workers/release-validator/package.json	Declares worker metadata.
workers/grouper/tests/index.test.ts	Tests repetition and regression behavior.
workers/grouper/src/mark-regression.ts	Implements regression updates.
workers/grouper/src/index.ts	Records releases and invokes regression tracking.
package.json	Adds scripts and updates types.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

neSpecc · 2026-09-23T22:12:21Z

+}
+
+/**
+ * Mark a resolved event as regressed in the resolved or a newer release.


Suggested change

* Mark a resolved event as regressed in the resolved or a newer release.

* Mark a resolved event as regressed if ...

neSpecc · 2026-09-23T22:15:05Z

@@ -0,0 +1,67 @@
+import { Db, ObjectID } from 'mongodb';
+
+interface ReleaseRecord {


Suggested change

interface ReleaseRecord {

type ReleaseRecordPart = Pick<ReleaseDBScheme, '_id' | 'projectId' | 'release'>

neSpecc · 2026-09-23T22:19:08Z

+  const hasRegressionForCurrentCycle = previousRegressionRelease &&
+    previousRegressionRelease._id.toHexString() >= resolvedReleaseId;
+
+  if (!isResolvedOrNewerRelease || hasRegressionForCurrentCycle) {


documentation needed

neSpecc · 2026-09-23T22:24:21Z

+      expect((await eventsCollection.findOne({})).regressionInRelease).toBe('release-c');
+    });
+
+    test('Should mark a resolved event as regressed in the resolved release', async () => {


Suggested change

test('Should mark a resolved event as regressed in the resolved release', async () => {

test('Should mark as regressed ff we later encounter this event with a release that is considered a resolving release.', async () => {

neSpecc · 2026-09-23T22:25:11Z

+      expect(savedRepetition.release).toBeUndefined();
+    });
+
+    test('Should mark a resolved event as regressed in a newer repetition release', async () => {


lets wrap them into describe('Regression marking, ...`

neSpecc · 2026-09-23T22:57:25Z

+  releasesByName: Map<string, ReleaseDBScheme>
+): Promise<void> {
+  const eventGroupHashes = events.map(event => event.groupHash);
+  const repetitions = await repetitionsCollection.find({


this seems to be deadly big. You're getting all millions of repetitions.
You need to retrieve only distinct "release" variations of each event.

neSpecc · 2026-09-23T23:00:08Z

+     * For a regressed event, continue validation from the regression release
+     * instead of its original release. This enables repeated resolve cycles.


Docs seems unlcear to me. What is "repeated resolve cycles"?

neSpecc · 2026-09-23T23:00:18Z

+        return projectRelease._id.toHexString() > releaseId && releasesWithEvent.has(projectRelease.release);
+      });
+
+      if (!isNewerThanLastOccurrence || occurredInRelease || occurredInNewerRelease) {


doc is missing

neSpecc · 2026-09-23T23:00:56Z

+        continue;
+      }
+
+      /**


describe the logic first, then technical notes

neSpecc · 2026-09-23T23:01:33Z

redundant test

Reversean · 2026-09-30T05:39:04Z

+     * Skip events whose original or latest occurrence release is missing from
+     * the project release history.
+     */
+    if (!lastOccurrenceRelease) {


payload.release is the release where the event first occurred. For an event older than MAX_DAYS_NUMBER, the archiver has already removed that release record while the event itself is still alive. So it's skipped here on every run and never gets resolved. A removed release is always older than any candidate, so could we treat a missing one as such instead of skipping?

Fixed. If the original release was archived, I now use the event timestamp instead. I also use MAX_DAYS_NUMBER as the maximum release age.

Reversean · 2026-09-30T05:39:05Z

+  const releasesByProject = groupReleasesByProject(releasesToCheck);
+
+  for (const [projectId, projectReleases] of releasesByProject) {
+    await validateProject(db, projectId, projectReleases);


If it throws for one project, all remaining projects are skipped. Better log and continue.

Reversean · 2026-09-30T05:39:05Z

+  const repetitions = await repetitionsCollection.aggregate<Pick<RepetitionDBScheme, 'groupHash' | 'release'>>([
+    {
+      $match: {
+        groupHash: { $in: eventGroupHashes },


Maybe we should add { groupHash: 1, release: 1 } index to avoid excessive data reading when we need distinct releases only?

This needs a migration for existing projects and an API change for new projects. The current code works with the existing groupHash index, I think we can move this optimization to next pr

Reversean · 2026-09-30T05:39:05Z

        return this.saveRepetition(task.projectId, newRepetition);
      });

+      if (task.payload.release && existedEvent.resolvedInRelease) {


This issues releases query for every repetition of resolved event, even after regression is already recorded, so a noisy regressed event pays an extra job per repetition.

Do we need the previous cycle's regression release for anything?

Reversean · 2026-09-30T13:32:56Z

+    const releasesWithEvent = eventReleases.get(event.groupHash) || new Set<string>();
+
+    for (const release of releasesToCheck) {
+      const releaseId = release._id.toHexString();
+      const isNewerThanLastOccurrence = releaseId > lastOccurrenceRelease._id.toHexString();
+      const occurredInRelease = releasesWithEvent.has(release.release);
+
+      /**
+       * A repetition in any later release, including one younger than 24 hours,
+       * proves that this candidate did not fix the event.
+       */
+      const occurredInNewerRelease = allProjectReleases.some(projectRelease => {
+        return projectRelease._id.toHexString() > releaseId && releasesWithEvent.has(projectRelease.release);
+      });
+
+      /**
+       * A candidate resolves the event only if it was deployed after the latest
+       * occurrence and the event appears neither in that candidate nor in any
+       * newer release. Otherwise, continue with the next candidate.
+       */
+      if (!isNewerThanLastOccurrence || occurredInRelease || occurredInNewerRelease) {
+        continue;
+      }


allProjectReleases is rescanned for every candidate of every event. A candidate resolves the event only if it's newer than every release the event occurred in, so that bound can be computed once per event.

Use MAX_DAYS_NUMBER to define the candidate release window and fall back to event timestamps when original release records are archived.

Add release validator worker

8f58256

Track event regressions by release

7d752cb

chore

8ac4dab

Support repeated resolution and regression cycles

d8164f7

alisawavezen12 changed the title ~~Add release validator worker~~ Add release validator and regression tracking Sep 18, 2026

codex-assistant Bot marked this pull request as ready for review September 18, 2026 15:25

Merge branch 'master' into feat/release-validator-worker

609adbc

neSpecc reviewed Sep 19, 2026

View reviewed changes

Comment thread workers/release-validator/src/utils/build-event-release-map.ts Outdated

Comment thread workers/release-validator/src/validate-releases.ts Outdated

Comment thread workers/release-validator/src/validate-releases.ts Outdated

neSpecc requested a balanced review from Copilot September 19, 2026 11:43

Copilot started reviewing on behalf of neSpecc September 19, 2026 11:44 View session

Copilot AI reviewed Sep 19, 2026

View reviewed changes

Comment thread package.json

Comment thread workers/grouper/src/index.ts

Comment thread workers/release-validator/src/utils/build-event-release-map.ts

Comment thread workers/release-validator/src/validate-releases.ts Outdated

alisawavezen12 added 2 commits September 20, 2026 12:29

validate Events Batch

6455530

Add release validator to Docker Compose

e77e4a3

neSpecc reviewed Sep 23, 2026

View reviewed changes

alisawavezen12 added 4 commits September 25, 2026 07:16

chore

60df837

chore

1ee7b01

chore

aa79701

chore

ecca00d

Reversean requested changes Sep 30, 2026

View reviewed changes

alisawavezen12 added 4 commits October 4, 2026 16:19

Align release validation with retention policy

ef64cf1

Use MAX_DAYS_NUMBER to define the candidate release window and fall back to event timestamps when original release records are archived.

Isolate release validation failures by project

15817a5

Clear regression when resolving event again

e2760e8

Track latest event occurrence before validation

a82acd3

alisawavezen12 added 3 commits October 4, 2026 17:33

tests fix

59ef3e4

Merge branch 'master' into feat/release-validator-worker

34e2cd7

chore

8b4e8d0

	* Mark a resolved event as regressed in the resolved or a newer release.
	* Mark a resolved event as regressed if ...

		@@ -0,0 +1,67 @@
		import { Db, ObjectID } from 'mongodb';

		interface ReleaseRecord {

	interface ReleaseRecord {
	type ReleaseRecordPart = Pick<ReleaseDBScheme, '_id' \| 'projectId' \| 'release'>

	test('Should mark a resolved event as regressed in the resolved release', async () => {
	test('Should mark as regressed ff we later encounter this event with a release that is considered a resolving release.', async () => {

		* For a regressed event, continue validation from the regression release
		* instead of its original release. This enables repeated resolve cycles.

Conversation

alisawavezen12 commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Uh oh!

codex-assistant Bot commented Sep 17, 2026

Please add a PR description 🙂

Uh oh!

codex-assistant Bot commented Sep 18, 2026

Please add a PR description 🙂

Uh oh!

codex-assistant Bot commented Sep 18, 2026

Please add a PR description 🙂

Uh oh!

codex-assistant Bot commented Sep 18, 2026

Please add a PR description 🙂

Uh oh!

codex-assistant Bot commented Sep 18, 2026

Please add a PR description 🙂

Uh oh!

codex-assistant Bot commented Sep 18, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alisawavezen12 commented Sep 17, 2026 •

edited

Loading