Skip to content

Feat stop all tests on error - #1343

Open
sonic16x wants to merge 3 commits into
masterfrom
stop-tests-on-error
Open

sonic16x wants to merge 3 commits into
masterfrom
stop-tests-on-error

Conversation

@sonic16x

@sonic16x sonic16x commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Adds run-wide failure limits via system.maxFailures and immediate termination on errors matching system.patternsOnReject.

interface ErrorPattern {
    name: string; // Unique, non-empty name within the array.
    pattern: RegExp | ((errorText: string) => boolean);
}

// Strings are regular expression sources; strings and RegExp remain supported.
type ErrorPatterns = Array<string | RegExp | ErrorPattern>;

type MaxFailuresConfig =
    | number // Count any failed attempt. Default: Infinity (no limit).
    | {
          limit: number;
          patterns: ErrorPatterns;
      };

// Relevant SystemConfig fields; both are optional in the input config.
interface SystemConfig {
    maxFailures: MaxFailuresConfig;
    patternsOnReject: ErrorPatterns; // Default: []; stop on the first match.
}

A finite limit must be a positive safe integer. The counter is shared across all tests, retries and browsers and resets for each run. Each failed attempt counts once, even when several patterns match. In the object form, unmatched errors do not count and an empty patterns array matches nothing. patternsOnReject takes priority over maxFailures; matching it also continues to mark the browser session as broken. Stopping the run cancels the remaining tests and exits with a non-zero code. There is no enabled flag or per-pattern limit.

Shared setup for the examples:

import type { ConfigInput, ErrorPatterns } from "testplane";

const browsers = {
    chrome: { desiredCapabilities: { browserName: "chrome" } },
};

const myPatternsTemplate: ErrorPatterns = [
    { name: "quota", pattern: /Too Many Requests/i },
    {
        name: "service-unavailable",
        pattern: text => text.includes("Service unavailable"),
    },
];

Stop a CI run after any 20 failed attempts, including assertion failures and timeouts:

const countAllErrors = {
    browsers,
    system: { maxFailures: 20 },
} satisfies ConfigInput;

Stop after 20 infrastructure errors in total, such as 12 quota errors and 8 unavailable-service errors. Other failures do not affect this counter:

const countInfrastructureErrors = {
    browsers,
    system: {
        maxFailures: { limit: 20, patterns: myPatternsTemplate },
    },
} satisfies ConfigInput;

Reuse the same template to stop immediately on the first infrastructure error:

const stopImmediately = {
    browsers,
    system: { patternsOnReject: myPatternsTemplate },
} satisfies ConfigInput;

Stop immediately if access is denied, otherwise allow up to 20 failed attempts of any kind:

const stopOnAccessDeniedOrFailureLimit = {
    browsers,
    system: {
        patternsOnReject: [
            { name: "access-denied", pattern: /Access denied|Forbidden/i },
        ],
        maxFailures: 20,
    },
} satisfies ConfigInput;

@pkg-pr-new

pkg-pr-new Bot commented Oct 12, 2026 •

Copy link
Copy Markdown

Open in StackBlitz

npm i https://pkg.pr.new/gemini-testing/testplane@1343

commit: 352ebe8

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown

✅ Testplane browser-env run succeed

Report

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown

✅ Testplane E2E run succeed

Report

const plugin = path.join(directory, "events.js");
await fs.writeFile(
plugin,
`module.exports = testplane => {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks hard to read & maintain. If we want some sort of integration test here, why not create the needed files in a separate fixture folder and copy them as needed. I think it would be simpler & cleaner.

Comment thread src/config/options.js Outdated

exitOnError: section({
enabled: options.boolean("exitOnError.enabled"),
filters: option({

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am a bit skeptical of this shape.

First, the object is 100% reduntant here and the wrong shape to use. If we choose to preserve the overall idea of this PR, I'd use something like this:

stopOnErrors: {
    rules: [
        { filter: /Service unavailable/, limit: 2 },
        { filter: text => text.includes('Something'), limit: 20 },
    ],
}

We could support passing exitOnErrors: false to make it easy to disable via env vars for example. Much simpler & cleaner IMO.


However, I would think more about the API shape... We already have patterns on reject. With a very simple shape: regexp array. And I think they strongly correlate. So what about this:

system: {
    patternsOnReject: myPatternsTemplate,
    maxFailures: 20, // Stops after any 20 errors
    maxFailures: { // OR, in object shape like this
        limit: 20,
        patterns: myPatternsTemplate, // <-- Easy reuse of the existing patterns
    }
}

This shape is even easier to work with & lets reuse existing patterns on reject collection that users already have! We can even consider defaulting to those patterns, but this might be unexpected for some users.

Let's discuss.

@sonic16x
sonic16x force-pushed the stop-tests-on-error branch from 6053333 to 8ef1395 Compare October 20, 2026 07:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants