diff --git a/AGENTS.md b/AGENTS.md index 95c8acabe..ff6b6bd8d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -28,7 +28,7 @@ Very Good CLI is a Dart command-line tool by Very Good Ventures for generating s ```bash # Install dependencies -dart pub get && cd bricks/test_optimizer && dart pub get && cd ../../ +dart pub get && cd bricks/test_optimizer/hooks && dart pub get && cd ../../../ # Run unit tests (excludes expensive pull-request-only and e2e tests) flutter test -x pull-request-only -x e2e diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 9c707172a..e679cf58c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -42,7 +42,7 @@ To develop for Very Good CLI you will need to become familiar with Very Good Ven very_good packages get -r # Or get project dependencies manually -dart pub get && cd bricks/test_optimizer && dart pub get && cd ../../ +dart pub get && cd bricks/test_optimizer/hooks && dart pub get && cd ../../../ ``` 3. Run all Very Good CLI tests: @@ -82,7 +82,7 @@ Where `` is an appropriate name describing your change. very_good packages get -r # Or get project dependencies manually -dart pub get && cd bricks/test_optimizer && dart pub get && cd ../../ +dart pub get && cd bricks/test_optimizer/hooks && dart pub get && cd ../../../ ``` 3. Ensure you have a meaningful [semantic][conventional_commits_link] commit message. diff --git a/bricks/test_optimizer/CONTRIBUTING.md b/bricks/test_optimizer/CONTRIBUTING.md index 2eb355008..09533ab8b 100644 --- a/bricks/test_optimizer/CONTRIBUTING.md +++ b/bricks/test_optimizer/CONTRIBUTING.md @@ -33,9 +33,7 @@ dart pub get dart test # 💻 Run `very_good test` end to end tests (from e2e/) -dart test test/src/commands/test/async_main_test.dart && -dart test test/src/commands/test/no_project_test.dart && -dart test test/src/commands/test/spaced_golden_file_name.dart +dart test test/commands/test --run-skipped ``` If not all test passed out of the box please submit an [issue](https://github.com/VeryGoodOpenSource/very_good_cli/issues/new/choose) so it can get fixed. diff --git a/bricks/test_optimizer/README.md b/bricks/test_optimizer/README.md index 3454d7531..c3e3a561f 100644 --- a/bricks/test_optimizer/README.md +++ b/bricks/test_optimizer/README.md @@ -18,15 +18,40 @@ The above command will generate a `.test_optimizer.dart` in the `test` directory // GENERATED CODE - DO NOT MODIFY BY HAND // Consider adding this file to your .gitignore. +import 'package:test/test.dart'; + import 'app/view/app_test.dart' as _a; import 'counter/cubit/counter_cubit_test.dart' as _b; import 'counter/view/counter_page_test.dart' as _c; void main() { - group('app_view_app_test_dart', () { _a.main(); }); - group('counter_cubit_counter_cubit_test_dart', () { _b.main(); }); - group('counter_view_counter_page_test_dart', () { _c.main(); }); + group('app/view/app_test.dart', () { _a.main(); }); + group('counter/cubit/counter_cubit_test.dart', () { _b.main(); }); + group('counter/view/counter_page_test.dart', () { _c.main(); }, tags: ['slow']); } ``` +## Annotations + +`package:test` reads `@Skip`, `@Tags`, `@Timeout`, `@TestOn`, `@OnPlatform` and +`@Retry` only from the metadata of a file's first directive, which a bundled +file no longer has. The pre-gen hook parses each file and forwards those +annotations to its `group` call, as `tags` is forwarded above, so a bundled file +behaves the way it does under plain `dart test`. + +Arguments are forwarded as source text into a file that imports only +`package:test` and `dart:core`, so an annotation naming anything else, such as +`@Timeout(kSlowSuite)`, is left out with a warning instead of breaking the +bundle. `@TestOn` cannot save a file that imports something platform-specific, +since every optimized file is imported regardless of platform. + +Tag a file with `skip_very_good_optimization` to keep it out of the bundle. The +tag needs a directive such as `library;` beneath it, the only place +`package:test` reads metadata from: + +```dart +@Tags(['skip_very_good_optimization']) +library; +``` + [1]: https://github.com/felangel/mason diff --git a/lib/src/cli/templates/test_optimizer_bundle.dart b/lib/src/cli/templates/test_optimizer_bundle.dart index 899b874d3..0cf3785c1 100644 --- a/lib/src/cli/templates/test_optimizer_bundle.dart +++ b/lib/src/cli/templates/test_optimizer_bundle.dart @@ -61,7 +61,7 @@ final testOptimizerBundle = MasonBundle.fromJson({ "publish_to": "none", "readme": { "path": "README.md", - "data": "IyB0ZXN0X29wdGltaXplcgoKWyFbUG93ZXJlZCBieSBNYXNvbl0oaHR0cHM6Ly9pbWcuc2hpZWxkcy5pby9lbmRwb2ludD91cmw9aHR0cHMlM0ElMkYlMkZ0aW55dXJsLmNvbSUyRm1hc29uLWJhZGdlKV0oaHR0cHM6Ly9naXRodWIuY29tL2ZlbGFuZ2VsL21hc29uKQoKQSBicmljayB0aGF0IGdlbmVyYXRlcyBhIHNpbmdsZSBlbnRyeXBvaW50IGZvciBEYXJ0IHRlc3RzLgoKX0dlbmVyYXRlZCBieSBbbWFzb25dWzFdIPCfp7FfCgojIyBHZXR0aW5nIFN0YXJ0ZWQg8J+agAoKYGBgc2gKbWFzb24gbWFrZSB0ZXN0X29wdGltaXplciAtLXBhY2thZ2Utcm9vdCAuL3BhdGgvdG8vcGFja2FnZSAtLW9uLWNvbmZsaWN0IG92ZXJ3cml0ZQpgYGAKClRoZSBhYm92ZSBjb21tYW5kIHdpbGwgZ2VuZXJhdGUgYSBgLnRlc3Rfb3B0aW1pemVyLmRhcnRgIGluIHRoZSBgdGVzdGAgZGlyZWN0b3J5IHRoYXQgaW1wb3J0cyBhbmQgZXhlY3V0ZXMgYWxsIHRlc3RzCgpgYGBkYXJ0Ci8vIEdFTkVSQVRFRCBDT0RFIC0gRE8gTk9UIE1PRElGWSBCWSBIQU5ECi8vIENvbnNpZGVyIGFkZGluZyB0aGlzIGZpbGUgdG8geW91ciAuZ2l0aWdub3JlLgoKaW1wb3J0ICdhcHAvdmlldy9hcHBfdGVzdC5kYXJ0JyBhcyBfYTsKaW1wb3J0ICdjb3VudGVyL2N1Yml0L2NvdW50ZXJfY3ViaXRfdGVzdC5kYXJ0JyBhcyBfYjsKaW1wb3J0ICdjb3VudGVyL3ZpZXcvY291bnRlcl9wYWdlX3Rlc3QuZGFydCcgYXMgX2M7Cgp2b2lkIG1haW4oKSB7CiAgZ3JvdXAoJ2FwcF92aWV3X2FwcF90ZXN0X2RhcnQnLCAoKSB7IF9hLm1haW4oKTsgfSk7CiAgZ3JvdXAoJ2NvdW50ZXJfY3ViaXRfY291bnRlcl9jdWJpdF90ZXN0X2RhcnQnLCAoKSB7IF9iLm1haW4oKTsgfSk7CiAgZ3JvdXAoJ2NvdW50ZXJfdmlld19jb3VudGVyX3BhZ2VfdGVzdF9kYXJ0JywgKCkgeyBfYy5tYWluKCk7IH0pOwp9CmBgYAoKWzFdOiBodHRwczovL2dpdGh1Yi5jb20vZmVsYW5nZWwvbWFzb24K", + "data": "IyB0ZXN0X29wdGltaXplcgoKWyFbUG93ZXJlZCBieSBNYXNvbl0oaHR0cHM6Ly9pbWcuc2hpZWxkcy5pby9lbmRwb2ludD91cmw9aHR0cHMlM0ElMkYlMkZ0aW55dXJsLmNvbSUyRm1hc29uLWJhZGdlKV0oaHR0cHM6Ly9naXRodWIuY29tL2ZlbGFuZ2VsL21hc29uKQoKQSBicmljayB0aGF0IGdlbmVyYXRlcyBhIHNpbmdsZSBlbnRyeXBvaW50IGZvciBEYXJ0IHRlc3RzLgoKX0dlbmVyYXRlZCBieSBbbWFzb25dWzFdIPCfp7FfCgojIyBHZXR0aW5nIFN0YXJ0ZWQg8J+agAoKYGBgc2gKbWFzb24gbWFrZSB0ZXN0X29wdGltaXplciAtLXBhY2thZ2Utcm9vdCAuL3BhdGgvdG8vcGFja2FnZSAtLW9uLWNvbmZsaWN0IG92ZXJ3cml0ZQpgYGAKClRoZSBhYm92ZSBjb21tYW5kIHdpbGwgZ2VuZXJhdGUgYSBgLnRlc3Rfb3B0aW1pemVyLmRhcnRgIGluIHRoZSBgdGVzdGAgZGlyZWN0b3J5IHRoYXQgaW1wb3J0cyBhbmQgZXhlY3V0ZXMgYWxsIHRlc3RzCgpgYGBkYXJ0Ci8vIEdFTkVSQVRFRCBDT0RFIC0gRE8gTk9UIE1PRElGWSBCWSBIQU5ECi8vIENvbnNpZGVyIGFkZGluZyB0aGlzIGZpbGUgdG8geW91ciAuZ2l0aWdub3JlLgoKaW1wb3J0ICdwYWNrYWdlOnRlc3QvdGVzdC5kYXJ0JzsKCmltcG9ydCAnYXBwL3ZpZXcvYXBwX3Rlc3QuZGFydCcgYXMgX2E7CmltcG9ydCAnY291bnRlci9jdWJpdC9jb3VudGVyX2N1Yml0X3Rlc3QuZGFydCcgYXMgX2I7CmltcG9ydCAnY291bnRlci92aWV3L2NvdW50ZXJfcGFnZV90ZXN0LmRhcnQnIGFzIF9jOwoKdm9pZCBtYWluKCkgewogIGdyb3VwKCdhcHAvdmlldy9hcHBfdGVzdC5kYXJ0JywgKCkgeyBfYS5tYWluKCk7IH0pOwogIGdyb3VwKCdjb3VudGVyL2N1Yml0L2NvdW50ZXJfY3ViaXRfdGVzdC5kYXJ0JywgKCkgeyBfYi5tYWluKCk7IH0pOwogIGdyb3VwKCdjb3VudGVyL3ZpZXcvY291bnRlcl9wYWdlX3Rlc3QuZGFydCcsICgpIHsgX2MubWFpbigpOyB9LCB0YWdzOiBbJ3Nsb3cnXSk7Cn0KYGBgCgojIyBBbm5vdGF0aW9ucwoKYHBhY2thZ2U6dGVzdGAgcmVhZHMgYEBTa2lwYCwgYEBUYWdzYCwgYEBUaW1lb3V0YCwgYEBUZXN0T25gLCBgQE9uUGxhdGZvcm1gIGFuZApgQFJldHJ5YCBvbmx5IGZyb20gdGhlIG1ldGFkYXRhIG9mIGEgZmlsZSdzIGZpcnN0IGRpcmVjdGl2ZSwgd2hpY2ggYSBidW5kbGVkCmZpbGUgbm8gbG9uZ2VyIGhhcy4gVGhlIHByZS1nZW4gaG9vayBwYXJzZXMgZWFjaCBmaWxlIGFuZCBmb3J3YXJkcyB0aG9zZQphbm5vdGF0aW9ucyB0byBpdHMgYGdyb3VwYCBjYWxsLCBhcyBgdGFnc2AgaXMgZm9yd2FyZGVkIGFib3ZlLCBzbyBhIGJ1bmRsZWQgZmlsZQpiZWhhdmVzIHRoZSB3YXkgaXQgZG9lcyB1bmRlciBwbGFpbiBgZGFydCB0ZXN0YC4KCkFyZ3VtZW50cyBhcmUgZm9yd2FyZGVkIGFzIHNvdXJjZSB0ZXh0IGludG8gYSBmaWxlIHRoYXQgaW1wb3J0cyBvbmx5CmBwYWNrYWdlOnRlc3RgIGFuZCBgZGFydDpjb3JlYCwgc28gYW4gYW5ub3RhdGlvbiBuYW1pbmcgYW55dGhpbmcgZWxzZSwgc3VjaCBhcwpgQFRpbWVvdXQoa1Nsb3dTdWl0ZSlgLCBpcyBsZWZ0IG91dCB3aXRoIGEgd2FybmluZyBpbnN0ZWFkIG9mIGJyZWFraW5nIHRoZQpidW5kbGUuIGBAVGVzdE9uYCBjYW5ub3Qgc2F2ZSBhIGZpbGUgdGhhdCBpbXBvcnRzIHNvbWV0aGluZyBwbGF0Zm9ybS1zcGVjaWZpYywKc2luY2UgZXZlcnkgb3B0aW1pemVkIGZpbGUgaXMgaW1wb3J0ZWQgcmVnYXJkbGVzcyBvZiBwbGF0Zm9ybS4KClRhZyBhIGZpbGUgd2l0aCBgc2tpcF92ZXJ5X2dvb2Rfb3B0aW1pemF0aW9uYCB0byBrZWVwIGl0IG91dCBvZiB0aGUgYnVuZGxlLiBUaGUKdGFnIG5lZWRzIGEgZGlyZWN0aXZlIHN1Y2ggYXMgYGxpYnJhcnk7YCBiZW5lYXRoIGl0LCB0aGUgb25seSBwbGFjZQpgcGFja2FnZTp0ZXN0YCByZWFkcyBtZXRhZGF0YSBmcm9tOgoKYGBgZGFydApAVGFncyhbJ3NraXBfdmVyeV9nb29kX29wdGltaXphdGlvbiddKQpsaWJyYXJ5OwpgYGAKClsxXTogaHR0cHM6Ly9naXRodWIuY29tL2ZlbGFuZ2VsL21hc29uCg==", "type": "text", }, "vars": { diff --git a/site/docs/commands/test.md b/site/docs/commands/test.md index bf18a8a0c..a46c7a052 100644 --- a/site/docs/commands/test.md +++ b/site/docs/commands/test.md @@ -100,12 +100,33 @@ Golden tests are tests that compare the output of a test to a "golden" file. If :::info For an example on specifying a custom [`GoldenFileComparator`](https://api.flutter.dev/flutter/flutter_test/GoldenFileComparator-class.html) that accepts a certain amount of difference (toleration threshold), refer to the [`goldenFileComparator` Flutter documentation](https://api.flutter.dev/flutter/flutter_test/goldenFileComparator.html). ::: +### File-level annotations + +`package:test` reads `@Skip`, `@Tags`, `@Timeout`, `@TestOn`, `@OnPlatform` and `@Retry` only from the metadata of a file's first directive. The optimizer forwards them to the group that wraps each bundled file, so an optimized run behaves the way the same suite does under plain `dart test` or `flutter test`. + +```dart +@Tags(['integration']) +@Timeout(Duration(minutes: 5)) +library; + +import 'package:test/test.dart'; +``` + +The file above keeps its five minute timeout inside the bundle, is skipped when `dart_test.yaml` marks `integration` as `skip:`, and is left out by `--exclude-tags integration`. + +Arguments are forwarded as source text into a file that imports only `package:test` and `dart:core`, so an annotation naming anything else, such as `@Timeout(kSlowSuite)`, is left out with a warning instead of breaking the bundle. Inline the value to keep the annotation. + +`@TestOn` cannot save a file that imports something platform-specific, such as `dart:html`, since every optimized file is imported regardless of platform. Keep those files out with the `skip_very_good_optimization` tag or `--exclude-optimization`. + ### Skip optimization for specific tests By default, all tests run with optimizations enabled; use the `--no-optimization` flag to disable globally, or add the `skip_very_good_optimization` tag to specific test files to disable them individually. -```js +The tag needs a directive such as `library;` beneath it, the only place `package:test` reads metadata from. + +```dart @Tags(['skip_very_good_optimization']) +library; import 'package:test/test.dart'; ``` @@ -126,7 +147,7 @@ test: Globs are matched against each test file's path relative to the package root, and they match everything nested underneath what they name, so `test/integration` excludes every test in that directory. See [`optimization`](../configuration.md#optimization) for the glob syntax details. -Excluded files are left out of the optimized bundle but still run, each as its own test suite. Because they run as their own suites, their file-level `@Tags` are honored again, so combining `--exclude-optimization test/integration` with `--exclude-tags integration` skips a suite annotated with `@Tags(['integration'])`. +Excluded files are left out of the optimized bundle but still run, each as its own test suite, in their own isolate. Exclude a file when it cannot share an isolate with the rest of the suite, for example because it mutates global state or installs its own bindings. File-level annotations do not need an exclusion, as the bundle forwards them. Prefer narrow globs. Every excluded file becomes its own suite with its own VM startup, so excluding hundreds of files gives up most of what the optimization buys you. diff --git a/site/docs/configuration.md b/site/docs/configuration.md index f421bee2c..9105e5167 100644 --- a/site/docs/configuration.md +++ b/site/docs/configuration.md @@ -135,8 +135,7 @@ so `test/**/serial_*_test.dart` matches both `test/serial_a_test.dart` and `test/nested/serial_a_test.dart`. Matching is case-sensitive on every platform. -Excluded files still run, each as its own test suite, which also means their -file-level `@Tags` are honored again. See +Excluded files still run, each as its own test suite, in their own isolate. See [Skip optimization for specific tests](commands/test.md#skip-optimization-for-specific-tests). With `--recursive`, the closest `very_good.yaml` applies to every package in the