Repository navigation
Conversation
commit: |
✅ Testplane browser-env run succeed
|
✅ Testplane E2E run succeed
|
| const plugin = path.join(directory, "events.js"); | ||
| await fs.writeFile( | ||
| plugin, | ||
| `module.exports = testplane => { |
There was a problem hiding this comment.
This looks hard to read & maintain. If we want some sort of integration test here, why not create the needed files in a separate fixture folder and copy them as needed. I think it would be simpler & cleaner.
|
|
||
| exitOnError: section({ | ||
| enabled: options.boolean("exitOnError.enabled"), | ||
| filters: option({ |
There was a problem hiding this comment.
I am a bit skeptical of this shape.
First, the object is 100% reduntant here and the wrong shape to use. If we choose to preserve the overall idea of this PR, I'd use something like this:
stopOnErrors: {
rules: [
{ filter: /Service unavailable/, limit: 2 },
{ filter: text => text.includes('Something'), limit: 20 },
],
}
We could support passing exitOnErrors: false to make it easy to disable via env vars for example. Much simpler & cleaner IMO.
However, I would think more about the API shape... We already have patterns on reject. With a very simple shape: regexp array. And I think they strongly correlate. So what about this:
system: {
patternsOnReject: myPatternsTemplate,
maxFailures: 20, // Stops after any 20 errors
maxFailures: { // OR, in object shape like this
limit: 20,
patterns: myPatternsTemplate, // <-- Easy reuse of the existing patterns
}
}
This shape is even easier to work with & lets reuse existing patterns on reject collection that users already have! We can even consider defaulting to those patterns, but this might be unexpected for some users.
Let's discuss.
6053333 to
8ef1395
Compare
Adds run-wide failure limits via
system.maxFailuresand immediate termination on errors matchingsystem.patternsOnReject.A finite limit must be a positive safe integer. The counter is shared across all tests, retries and browsers and resets for each run. Each failed attempt counts once, even when several patterns match. In the object form, unmatched errors do not count and an empty
patternsarray matches nothing.patternsOnRejecttakes priority overmaxFailures; matching it also continues to mark the browser session as broken. Stopping the run cancels the remaining tests and exits with a non-zero code. There is noenabledflag or per-pattern limit.Shared setup for the examples:
Stop a CI run after any 20 failed attempts, including assertion failures and timeouts:
Stop after 20 infrastructure errors in total, such as 12 quota errors and 8 unavailable-service errors. Other failures do not affect this counter:
Reuse the same template to stop immediately on the first infrastructure error:
Stop immediately if access is denied, otherwise allow up to 20 failed attempts of any kind: