Add -testQuiet to report only failing tests#1204
Closed
Donach wants to merge 1 commit into
Closed
Conversation
Running a project with hundreds of test functions prints several lines per passing test, which buries the failures in CI. -testQuiet (short form -tq) buffers the output of the running test, discards it when the test passes and prints it in full when the test fails. The summary is always printed. Also repairs the interpreter output redirection this relies on, which never reached the tests' own output: - ProgramStateIO shadowed ProgramState.outStream and overrode the accessors without forwarding to the native providers. - ReflectionNativeProvider.setOutStream was empty, so OutputProvider kept writing to its default System.err. - The PrintStream wrapping the redirect had no autoflush, so its buffer was never emptied. As a result, output printed by tests during -runtests now goes to the stream RunTests prints to instead of System.err, as the redirection intended.
Contributor
Author
|
Closing in favour of #1205. The What is worth keeping are the three bugs I ran into while testing the flag, all of which make |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
On a project with hundreds of
@testfunctions,wurst -runtestsprints two or morelines per passing test plus everything the tests themselves print. In CI that buries the
handful of lines that actually matter, even though the summary at the end already reports
what passed.
-compactOutput(#1188) shortens the output, but it does so by dropping detail forfailures too: no assertion message, no stack trace, no output from the failing test.
Change
Adds
-testQuiet(short form-tq), which reports only what is wrong:message, the stack trace and, for timeouts, the stack frames.
Composes with
-compactOutput; neither flag changes the behavior of the other.Fixes to the interpreter output redirection
RunTests.redirectInterpreterOutputwas dead code, so output the tests themselves printednever reached
RunTestsand went straight toSystem.err. Three separate reasons, allfixed here, because quiet mode can only suppress output that it actually receives:
ProgramStateIOshadowedProgramState.outStreamwith its own field and overrodegetOutStream/setOutStreamwithout forwarding to the native providers. Removed; thebase class already implements both and does the forwarding.
ReflectionNativeProvider.setOutStreamwas an empty method, soOutputProvider(whichimplements
println,BJDebugMsg,DisplayTextToPlayer, ...) kept writing toits default
System.err. It now forwards to theOutputProviderit owns.PrintStreamwrapping the redirect had no autoflush, so anything short of 8 KBstayed in its buffer forever. Now autoflush with an explicit UTF-8 charset.
The visible consequence is that output printed by tests during
-runtestsnow goes to thestream
RunTestsprints to (stdout on the CLI) instead ofSystem.err, which is what theredirect was written to do. It also makes
-compactOutputsuppress that output, asintended.
Tests
RunTestsQuietTestscompiles a package with one passing and one failing test and assertson the produced output: quiet mode hides the passing test and its output, keeps the failing
test's output and assertion message, and keeps the summary; the default mode still reports
passing tests. Flag parsing is covered for both
-testQuietand-tq.