From 68761767e70dc6d852dc5b72d50c64d70507f5c0 Mon Sep 17 00:00:00 2001 From: Marcos Sevilla Date: Wed, 16 Sep 2026 13:25:48 +0200 Subject: [PATCH 1/3] docs: document test optimization annotation preservation --- AGENTS.md | 2 +- CONTRIBUTING.md | 4 ++-- bricks/test_optimizer/CONTRIBUTING.md | 10 ++++++--- bricks/test_optimizer/README.md | 31 ++++++++++++++++++++++++--- site/docs/commands/test.md | 25 +++++++++++++++++++-- site/docs/configuration.md | 3 +-- 6 files changed, 62 insertions(+), 13 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 95c8acabe..ff6b6bd8d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -28,7 +28,7 @@ Very Good CLI is a Dart command-line tool by Very Good Ventures for generating s ```bash # Install dependencies -dart pub get && cd bricks/test_optimizer && dart pub get && cd ../../ +dart pub get && cd bricks/test_optimizer/hooks && dart pub get && cd ../../../ # Run unit tests (excludes expensive pull-request-only and e2e tests) flutter test -x pull-request-only -x e2e diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 9c707172a..e679cf58c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -42,7 +42,7 @@ To develop for Very Good CLI you will need to become familiar with Very Good Ven very_good packages get -r # Or get project dependencies manually -dart pub get && cd bricks/test_optimizer && dart pub get && cd ../../ +dart pub get && cd bricks/test_optimizer/hooks && dart pub get && cd ../../../ ``` 3. Run all Very Good CLI tests: @@ -82,7 +82,7 @@ Where `` is an appropriate name describing your change. very_good packages get -r # Or get project dependencies manually -dart pub get && cd bricks/test_optimizer && dart pub get && cd ../../ +dart pub get && cd bricks/test_optimizer/hooks && dart pub get && cd ../../../ ``` 3. Ensure you have a meaningful [semantic][conventional_commits_link] commit message. diff --git a/bricks/test_optimizer/CONTRIBUTING.md b/bricks/test_optimizer/CONTRIBUTING.md index 2eb355008..cbee2fe76 100644 --- a/bricks/test_optimizer/CONTRIBUTING.md +++ b/bricks/test_optimizer/CONTRIBUTING.md @@ -33,9 +33,7 @@ dart pub get dart test # 💻 Run `very_good test` end to end tests (from e2e/) -dart test test/src/commands/test/async_main_test.dart && -dart test test/src/commands/test/no_project_test.dart && -dart test test/src/commands/test/spaced_golden_file_name.dart +dart test test/commands/test --run-skipped ``` If not all test passed out of the box please submit an [issue](https://github.com/VeryGoodOpenSource/very_good_cli/issues/new/choose) so it can get fixed. @@ -65,3 +63,9 @@ dart pub global activate --source path . ``` 💡 **Note**: After changing the test optimizer brick, make sure to always generate a new test optimizer bundle and commit this as part of your pull request. + +### Cost of the `analyzer` dependency + +Adding `analyzer` takes the hook's kernel compile from ~1.4s to ~3.4s and its +compiled size from ~26MB to ~51MB, paid once per cold `.dart_tool/mason`. +Resolution is unchanged, since `test` already pulled `analyzer` in. diff --git a/bricks/test_optimizer/README.md b/bricks/test_optimizer/README.md index 3454d7531..c3e3a561f 100644 --- a/bricks/test_optimizer/README.md +++ b/bricks/test_optimizer/README.md @@ -18,15 +18,40 @@ The above command will generate a `.test_optimizer.dart` in the `test` directory // GENERATED CODE - DO NOT MODIFY BY HAND // Consider adding this file to your .gitignore. +import 'package:test/test.dart'; + import 'app/view/app_test.dart' as _a; import 'counter/cubit/counter_cubit_test.dart' as _b; import 'counter/view/counter_page_test.dart' as _c; void main() { - group('app_view_app_test_dart', () { _a.main(); }); - group('counter_cubit_counter_cubit_test_dart', () { _b.main(); }); - group('counter_view_counter_page_test_dart', () { _c.main(); }); + group('app/view/app_test.dart', () { _a.main(); }); + group('counter/cubit/counter_cubit_test.dart', () { _b.main(); }); + group('counter/view/counter_page_test.dart', () { _c.main(); }, tags: ['slow']); } ``` +## Annotations + +`package:test` reads `@Skip`, `@Tags`, `@Timeout`, `@TestOn`, `@OnPlatform` and +`@Retry` only from the metadata of a file's first directive, which a bundled +file no longer has. The pre-gen hook parses each file and forwards those +annotations to its `group` call, as `tags` is forwarded above, so a bundled file +behaves the way it does under plain `dart test`. + +Arguments are forwarded as source text into a file that imports only +`package:test` and `dart:core`, so an annotation naming anything else, such as +`@Timeout(kSlowSuite)`, is left out with a warning instead of breaking the +bundle. `@TestOn` cannot save a file that imports something platform-specific, +since every optimized file is imported regardless of platform. + +Tag a file with `skip_very_good_optimization` to keep it out of the bundle. The +tag needs a directive such as `library;` beneath it, the only place +`package:test` reads metadata from: + +```dart +@Tags(['skip_very_good_optimization']) +library; +``` + [1]: https://github.com/felangel/mason diff --git a/site/docs/commands/test.md b/site/docs/commands/test.md index bf18a8a0c..a46c7a052 100644 --- a/site/docs/commands/test.md +++ b/site/docs/commands/test.md @@ -100,12 +100,33 @@ Golden tests are tests that compare the output of a test to a "golden" file. If :::info For an example on specifying a custom [`GoldenFileComparator`](https://api.flutter.dev/flutter/flutter_test/GoldenFileComparator-class.html) that accepts a certain amount of difference (toleration threshold), refer to the [`goldenFileComparator` Flutter documentation](https://api.flutter.dev/flutter/flutter_test/goldenFileComparator.html). ::: +### File-level annotations + +`package:test` reads `@Skip`, `@Tags`, `@Timeout`, `@TestOn`, `@OnPlatform` and `@Retry` only from the metadata of a file's first directive. The optimizer forwards them to the group that wraps each bundled file, so an optimized run behaves the way the same suite does under plain `dart test` or `flutter test`. + +```dart +@Tags(['integration']) +@Timeout(Duration(minutes: 5)) +library; + +import 'package:test/test.dart'; +``` + +The file above keeps its five minute timeout inside the bundle, is skipped when `dart_test.yaml` marks `integration` as `skip:`, and is left out by `--exclude-tags integration`. + +Arguments are forwarded as source text into a file that imports only `package:test` and `dart:core`, so an annotation naming anything else, such as `@Timeout(kSlowSuite)`, is left out with a warning instead of breaking the bundle. Inline the value to keep the annotation. + +`@TestOn` cannot save a file that imports something platform-specific, such as `dart:html`, since every optimized file is imported regardless of platform. Keep those files out with the `skip_very_good_optimization` tag or `--exclude-optimization`. + ### Skip optimization for specific tests By default, all tests run with optimizations enabled; use the `--no-optimization` flag to disable globally, or add the `skip_very_good_optimization` tag to specific test files to disable them individually. -```js +The tag needs a directive such as `library;` beneath it, the only place `package:test` reads metadata from. + +```dart @Tags(['skip_very_good_optimization']) +library; import 'package:test/test.dart'; ``` @@ -126,7 +147,7 @@ test: Globs are matched against each test file's path relative to the package root, and they match everything nested underneath what they name, so `test/integration` excludes every test in that directory. See [`optimization`](../configuration.md#optimization) for the glob syntax details. -Excluded files are left out of the optimized bundle but still run, each as its own test suite. Because they run as their own suites, their file-level `@Tags` are honored again, so combining `--exclude-optimization test/integration` with `--exclude-tags integration` skips a suite annotated with `@Tags(['integration'])`. +Excluded files are left out of the optimized bundle but still run, each as its own test suite, in their own isolate. Exclude a file when it cannot share an isolate with the rest of the suite, for example because it mutates global state or installs its own bindings. File-level annotations do not need an exclusion, as the bundle forwards them. Prefer narrow globs. Every excluded file becomes its own suite with its own VM startup, so excluding hundreds of files gives up most of what the optimization buys you. diff --git a/site/docs/configuration.md b/site/docs/configuration.md index f421bee2c..9105e5167 100644 --- a/site/docs/configuration.md +++ b/site/docs/configuration.md @@ -135,8 +135,7 @@ so `test/**/serial_*_test.dart` matches both `test/serial_a_test.dart` and `test/nested/serial_a_test.dart`. Matching is case-sensitive on every platform. -Excluded files still run, each as its own test suite, which also means their -file-level `@Tags` are honored again. See +Excluded files still run, each as its own test suite, in their own isolate. See [Skip optimization for specific tests](commands/test.md#skip-optimization-for-specific-tests). With `--recursive`, the closest `very_good.yaml` applies to every package in the From 4b95e1fdf245b29481264775f6ca9e68808ff807 Mon Sep 17 00:00:00 2001 From: "unicoderbot[bot]" <269805761+unicoderbot[bot]@users.noreply.github.com> Date: Wed, 16 Sep 2026 12:48:22 +0000 Subject: [PATCH 2/3] fix: address review feedback on PR #1736 Co-authored-by: marcossevilla --- lib/src/cli/templates/test_optimizer_bundle.dart | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/src/cli/templates/test_optimizer_bundle.dart b/lib/src/cli/templates/test_optimizer_bundle.dart index 899b874d3..0cf3785c1 100644 --- a/lib/src/cli/templates/test_optimizer_bundle.dart +++ b/lib/src/cli/templates/test_optimizer_bundle.dart @@ -61,7 +61,7 @@ final testOptimizerBundle = MasonBundle.fromJson({ "publish_to": "none", "readme": { "path": "README.md", - "data": "IyB0ZXN0X29wdGltaXplcgoKWyFbUG93ZXJlZCBieSBNYXNvbl0oaHR0cHM6Ly9pbWcuc2hpZWxkcy5pby9lbmRwb2ludD91cmw9aHR0cHMlM0ElMkYlMkZ0aW55dXJsLmNvbSUyRm1hc29uLWJhZGdlKV0oaHR0cHM6Ly9naXRodWIuY29tL2ZlbGFuZ2VsL21hc29uKQoKQSBicmljayB0aGF0IGdlbmVyYXRlcyBhIHNpbmdsZSBlbnRyeXBvaW50IGZvciBEYXJ0IHRlc3RzLgoKX0dlbmVyYXRlZCBieSBbbWFzb25dWzFdIPCfp7FfCgojIyBHZXR0aW5nIFN0YXJ0ZWQg8J+agAoKYGBgc2gKbWFzb24gbWFrZSB0ZXN0X29wdGltaXplciAtLXBhY2thZ2Utcm9vdCAuL3BhdGgvdG8vcGFja2FnZSAtLW9uLWNvbmZsaWN0IG92ZXJ3cml0ZQpgYGAKClRoZSBhYm92ZSBjb21tYW5kIHdpbGwgZ2VuZXJhdGUgYSBgLnRlc3Rfb3B0aW1pemVyLmRhcnRgIGluIHRoZSBgdGVzdGAgZGlyZWN0b3J5IHRoYXQgaW1wb3J0cyBhbmQgZXhlY3V0ZXMgYWxsIHRlc3RzCgpgYGBkYXJ0Ci8vIEdFTkVSQVRFRCBDT0RFIC0gRE8gTk9UIE1PRElGWSBCWSBIQU5ECi8vIENvbnNpZGVyIGFkZGluZyB0aGlzIGZpbGUgdG8geW91ciAuZ2l0aWdub3JlLgoKaW1wb3J0ICdhcHAvdmlldy9hcHBfdGVzdC5kYXJ0JyBhcyBfYTsKaW1wb3J0ICdjb3VudGVyL2N1Yml0L2NvdW50ZXJfY3ViaXRfdGVzdC5kYXJ0JyBhcyBfYjsKaW1wb3J0ICdjb3VudGVyL3ZpZXcvY291bnRlcl9wYWdlX3Rlc3QuZGFydCcgYXMgX2M7Cgp2b2lkIG1haW4oKSB7CiAgZ3JvdXAoJ2FwcF92aWV3X2FwcF90ZXN0X2RhcnQnLCAoKSB7IF9hLm1haW4oKTsgfSk7CiAgZ3JvdXAoJ2NvdW50ZXJfY3ViaXRfY291bnRlcl9jdWJpdF90ZXN0X2RhcnQnLCAoKSB7IF9iLm1haW4oKTsgfSk7CiAgZ3JvdXAoJ2NvdW50ZXJfdmlld19jb3VudGVyX3BhZ2VfdGVzdF9kYXJ0JywgKCkgeyBfYy5tYWluKCk7IH0pOwp9CmBgYAoKWzFdOiBodHRwczovL2dpdGh1Yi5jb20vZmVsYW5nZWwvbWFzb24K", + "data": "IyB0ZXN0X29wdGltaXplcgoKWyFbUG93ZXJlZCBieSBNYXNvbl0oaHR0cHM6Ly9pbWcuc2hpZWxkcy5pby9lbmRwb2ludD91cmw9aHR0cHMlM0ElMkYlMkZ0aW55dXJsLmNvbSUyRm1hc29uLWJhZGdlKV0oaHR0cHM6Ly9naXRodWIuY29tL2ZlbGFuZ2VsL21hc29uKQoKQSBicmljayB0aGF0IGdlbmVyYXRlcyBhIHNpbmdsZSBlbnRyeXBvaW50IGZvciBEYXJ0IHRlc3RzLgoKX0dlbmVyYXRlZCBieSBbbWFzb25dWzFdIPCfp7FfCgojIyBHZXR0aW5nIFN0YXJ0ZWQg8J+agAoKYGBgc2gKbWFzb24gbWFrZSB0ZXN0X29wdGltaXplciAtLXBhY2thZ2Utcm9vdCAuL3BhdGgvdG8vcGFja2FnZSAtLW9uLWNvbmZsaWN0IG92ZXJ3cml0ZQpgYGAKClRoZSBhYm92ZSBjb21tYW5kIHdpbGwgZ2VuZXJhdGUgYSBgLnRlc3Rfb3B0aW1pemVyLmRhcnRgIGluIHRoZSBgdGVzdGAgZGlyZWN0b3J5IHRoYXQgaW1wb3J0cyBhbmQgZXhlY3V0ZXMgYWxsIHRlc3RzCgpgYGBkYXJ0Ci8vIEdFTkVSQVRFRCBDT0RFIC0gRE8gTk9UIE1PRElGWSBCWSBIQU5ECi8vIENvbnNpZGVyIGFkZGluZyB0aGlzIGZpbGUgdG8geW91ciAuZ2l0aWdub3JlLgoKaW1wb3J0ICdwYWNrYWdlOnRlc3QvdGVzdC5kYXJ0JzsKCmltcG9ydCAnYXBwL3ZpZXcvYXBwX3Rlc3QuZGFydCcgYXMgX2E7CmltcG9ydCAnY291bnRlci9jdWJpdC9jb3VudGVyX2N1Yml0X3Rlc3QuZGFydCcgYXMgX2I7CmltcG9ydCAnY291bnRlci92aWV3L2NvdW50ZXJfcGFnZV90ZXN0LmRhcnQnIGFzIF9jOwoKdm9pZCBtYWluKCkgewogIGdyb3VwKCdhcHAvdmlldy9hcHBfdGVzdC5kYXJ0JywgKCkgeyBfYS5tYWluKCk7IH0pOwogIGdyb3VwKCdjb3VudGVyL2N1Yml0L2NvdW50ZXJfY3ViaXRfdGVzdC5kYXJ0JywgKCkgeyBfYi5tYWluKCk7IH0pOwogIGdyb3VwKCdjb3VudGVyL3ZpZXcvY291bnRlcl9wYWdlX3Rlc3QuZGFydCcsICgpIHsgX2MubWFpbigpOyB9LCB0YWdzOiBbJ3Nsb3cnXSk7Cn0KYGBgCgojIyBBbm5vdGF0aW9ucwoKYHBhY2thZ2U6dGVzdGAgcmVhZHMgYEBTa2lwYCwgYEBUYWdzYCwgYEBUaW1lb3V0YCwgYEBUZXN0T25gLCBgQE9uUGxhdGZvcm1gIGFuZApgQFJldHJ5YCBvbmx5IGZyb20gdGhlIG1ldGFkYXRhIG9mIGEgZmlsZSdzIGZpcnN0IGRpcmVjdGl2ZSwgd2hpY2ggYSBidW5kbGVkCmZpbGUgbm8gbG9uZ2VyIGhhcy4gVGhlIHByZS1nZW4gaG9vayBwYXJzZXMgZWFjaCBmaWxlIGFuZCBmb3J3YXJkcyB0aG9zZQphbm5vdGF0aW9ucyB0byBpdHMgYGdyb3VwYCBjYWxsLCBhcyBgdGFnc2AgaXMgZm9yd2FyZGVkIGFib3ZlLCBzbyBhIGJ1bmRsZWQgZmlsZQpiZWhhdmVzIHRoZSB3YXkgaXQgZG9lcyB1bmRlciBwbGFpbiBgZGFydCB0ZXN0YC4KCkFyZ3VtZW50cyBhcmUgZm9yd2FyZGVkIGFzIHNvdXJjZSB0ZXh0IGludG8gYSBmaWxlIHRoYXQgaW1wb3J0cyBvbmx5CmBwYWNrYWdlOnRlc3RgIGFuZCBgZGFydDpjb3JlYCwgc28gYW4gYW5ub3RhdGlvbiBuYW1pbmcgYW55dGhpbmcgZWxzZSwgc3VjaCBhcwpgQFRpbWVvdXQoa1Nsb3dTdWl0ZSlgLCBpcyBsZWZ0IG91dCB3aXRoIGEgd2FybmluZyBpbnN0ZWFkIG9mIGJyZWFraW5nIHRoZQpidW5kbGUuIGBAVGVzdE9uYCBjYW5ub3Qgc2F2ZSBhIGZpbGUgdGhhdCBpbXBvcnRzIHNvbWV0aGluZyBwbGF0Zm9ybS1zcGVjaWZpYywKc2luY2UgZXZlcnkgb3B0aW1pemVkIGZpbGUgaXMgaW1wb3J0ZWQgcmVnYXJkbGVzcyBvZiBwbGF0Zm9ybS4KClRhZyBhIGZpbGUgd2l0aCBgc2tpcF92ZXJ5X2dvb2Rfb3B0aW1pemF0aW9uYCB0byBrZWVwIGl0IG91dCBvZiB0aGUgYnVuZGxlLiBUaGUKdGFnIG5lZWRzIGEgZGlyZWN0aXZlIHN1Y2ggYXMgYGxpYnJhcnk7YCBiZW5lYXRoIGl0LCB0aGUgb25seSBwbGFjZQpgcGFja2FnZTp0ZXN0YCByZWFkcyBtZXRhZGF0YSBmcm9tOgoKYGBgZGFydApAVGFncyhbJ3NraXBfdmVyeV9nb29kX29wdGltaXphdGlvbiddKQpsaWJyYXJ5OwpgYGAKClsxXTogaHR0cHM6Ly9naXRodWIuY29tL2ZlbGFuZ2VsL21hc29uCg==", "type": "text", }, "vars": { From 7f6288ad556760288879b9b9cad6d1cf32104e08 Mon Sep 17 00:00:00 2001 From: Marcos Sevilla <31174242+marcossevilla@users.noreply.github.com> Date: Thu, 17 Sep 2026 11:46:31 +0200 Subject: [PATCH 3/3] Update bricks/test_optimizer/CONTRIBUTING.md --- bricks/test_optimizer/CONTRIBUTING.md | 6 ------ 1 file changed, 6 deletions(-) diff --git a/bricks/test_optimizer/CONTRIBUTING.md b/bricks/test_optimizer/CONTRIBUTING.md index cbee2fe76..09533ab8b 100644 --- a/bricks/test_optimizer/CONTRIBUTING.md +++ b/bricks/test_optimizer/CONTRIBUTING.md @@ -63,9 +63,3 @@ dart pub global activate --source path . ``` 💡 **Note**: After changing the test optimizer brick, make sure to always generate a new test optimizer bundle and commit this as part of your pull request. - -### Cost of the `analyzer` dependency - -Adding `analyzer` takes the hook's kernel compile from ~1.4s to ~3.4s and its -compiled size from ~26MB to ~51MB, paid once per cold `.dart_tool/mason`. -Resolution is unchanged, since `test` already pulled `analyzer` in.