Skip to content

fix: test optimizer drops library-level annotations (@Skip, @Tags, @Timeout, ...) #1723

Closed

Description

@peter-trost

Description

The test optimizer imports every test file and wraps its main() in a plain
group('<path>', () { ... }). Annotations that package:test only reads at the
library level (@Skip, @Tags, @Timeout, @TestOn, @OnPlatform) are
never carried over, so under very_good test / very_good dart test:

  • a file marked @Skip(...) runs;
  • a file marked @Tags([...]) runs even when dart_test.yaml marks that tag as
    skip:, and -x <tag> / --exclude-tags has no effect on it;
  • a file-level @Timeout(...) is ignored and the default 30 seconds applies.

very_good test is the default, optimized path, so a project that relies on
these annotations behaves differently from plain dart test / flutter test
without any warning. The only documented escape hatch is opting the whole file
out of the optimizer with skip_very_good_optimization.

This repository is itself affected: test/ensure_build_test.dart is tagged
pull-request-only (skipped by dart_test.yaml) and carries a 5 minute
@Timeout, yet very_good test -x pull-request-only still runs it with the
default timeout.

Steps To Reproduce

Five files in an empty directory:

# pubspec.yaml
name: optimizer_annotations_repro
environment:
  sdk: ^3.13.0
dev_dependencies:
  test: ^1.31.2
# dart_test.yaml
tags:
  slow:
    skip: "Only run in CI"
// test/fast_test.dart
import 'package:test/test.dart';

void main() {
  test('fast', () => expect(1, 1));
}
// test/slow_test.dart
@Tags(['slow'])

import 'package:test/test.dart';

void main() {
  test('slow', () => expect(1, 1));
}
// test/skipped_test.dart
@Skip('not ready')

import 'package:test/test.dart';

void main() {
  test('skipped', () => fail('must never run'));
}
// test/timeout_test.dart
@Timeout(Duration(milliseconds: 100))

import 'package:test/test.dart';

void main() {
  test('slow to finish', () => Future<void>.delayed(const Duration(seconds: 1)));
}

Then:

dart pub get
dart test            # 1 passed, 2 skipped, 1 failed (the timeout)
dart test -x slow    # same, minus the `slow` suite
very_good dart test  # 3 passed, 1 failed: `slow` and the timed-out test pass, the @Skip file fails

Observed with very_good_cli 1.5.0 on Dart 3.13.2 (macOS):

$ dart test -r expanded
00:00 +0: test/slow_test.dart: (suite)
  Skip: Only run in CI
00:00 +0 ~1: test/skipped_test.dart: (suite)
  Skip: not ready
00:00 +1 ~2 -1: test/timeout_test.dart: slow to finish [E]
  TimeoutException after 0:00:00.100000: Test timed out after 0.1 seconds.
00:00 +1 ~2 -1: Some tests failed.
Failing tests:
  test/timeout_test.dart: slow to finish

$ very_good dart test
00:01 +1: slow to finish            # @Timeout dropped: 1s test passes under the default 30s
00:01 +2: slow                      # @Tags dropped: runs despite dart_test.yaml (and despite -x slow)
skipped test/skipped_test.dart (FAILED)   # @Skip dropped: file ran and hit fail()
00:01 +3 -1: fast
00:01 +3 -1: Some tests failed.
 - test/skipped_test.dart
 	- [FAILED] skipped

Expected Behavior

very_good dart test matches dart test: the @Skip file is skipped, the
slow tag is honoured (skipped by dart_test.yaml, excluded by -x slow),
and the file-level @Timeout fails the 1 second test. Failing that, the
optimizer should at least surface a warning when it drops an annotation, and
the docs should state the limitation next to skip_very_good_optimization.

Activity

  1. moved this from Needs Triage to In Progress in VGV Open Source 🦄 🧙🌟on Sep 9, 2026
  2. marcossevilla commented on Sep 16, 2026

    @marcossevilla
    Member

    hey @peter-trost thanks for opening this issue!

    I opened #1735 to address this, you mind taking a look and testing locally to see if this solves the issue? thanks again 👍

  3. peter-trost commented on Sep 16, 2026

    @peter-trost
    ContributorAuthor

    @marcossevilla #1735 fixes the issue for me ✅ Just tested it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions