Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions HISTORY.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,22 @@ Changes:
and the lookup of other project metadata as constants are gone.
* Correct the README, which named a `progressbar` reporter; it is
`progress`.
* Reporters no longer raise for a test that didn't pass and says little
about why. Only an exception's message is required by TAP-Y/J, but the
`markdown` and `html` reporters needed its class or backtrace, and
every reporter needed the exception section itself, which a failed test
from plain TAP doesn't have.
* The `tap` reporter prints a line for a test that was omitted
(`ok N - label # SKIP reason`) or is still to be done
(`not ok N - label # TODO reason`), so the lines match the plan (#14).
Without a count at the start of the stream, it prints the plan at the end.
* The `outline` reporter lists omitted and to-do tests, with the reason.
* Reading plain TAP: a `TAP version` line is accepted, as is a stream that
starts with a test line or a comment; `# SKIP` and `# TODO` directives
become `omit` and `todo`; a test line needn't have a number; and the
final counts include errors, omissions and to-dos.
* The `outline` and `markdown` reporters no longer raise for tests that
belong to no case, as in plain TAP.

## 0.4.5 / 2013-11-20

Expand Down
98 changes: 98 additions & 0 deletions demo/01_reporters/sparse_streams.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
# Sparse Streams

A test that didn't pass is meant to carry an `exception` section, but only
its `message` is required, and not every producer gives even that. Rubytest
gives no exception class, and a backtrace only for errors. Plain TAP gives
nothing at all for a failed test. Every reporter has to cope.

require 'stringio'

run = lambda do |format, text, parser = Tapout::YamlParser|
stdout, stderr = $stdout, $stderr
$stdout = StringIO.new(out = String.new)
$stderr = StringIO.new(String.new)
begin
parser.new(:format => format).consume(StringIO.new(text))
ensure
$stdout, $stderr = stdout, stderr
end
out
end

reporters = %w[dot progress outline pretty breakdown runtime tap turn markdown html]

Here a failure has a message and a location but no class or backtrace, an
error has a message alone, and another failure has no exception section.

sparse = <<-END.gsub(/^ /, '')
---
type: suite
count: 6
rev: 5
---
type: case
level: 0
label: Sparse
---
type: test
status: pass
label: passes
---
type: test
status: fail
label: fails
exception:
file: test/sparse.rb
line: 9
message: wrong value
---
type: test
status: error
label: errors
exception:
message: boom
---
type: test
status: fail
label: fails without saying why
---
type: test
status: todo
label: is pending
exception:
message: later
---
type: test
status: omit
label: is omitted
---
type: final
counts:
total: 6
pass: 1
fail: 2
error: 1
omit: 1
todo: 1
...
END

reporters.each do |format|
run.call(format, sparse).empty?.assert == false
end

The same goes for tests that belong to no case, as in a stream converted
from plain TAP.

plain = <<-END.gsub(/^ /, '')
TAP version 13
1..4
ok 1 - passes
not ok 2 - fails
ok 3 - is skipped # SKIP no network
not ok 4 - is pending # TODO soon
END

reporters.each do |format|
run.call(format, plain, Tapout::PerlParser).empty?.assert == false
end
34 changes: 34 additions & 0 deletions demo/02_adapters/perl_adapter.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,3 +64,37 @@ and five passing tests.

entries.size.assert == 8


## Version line and directives

A stream may open with a version line, and a test line may end with a
`SKIP` or `TODO` directive. Given a legacy TAP stream:

TAP version 13
1..5
ok 1 - passes
not ok 2 - fails
ok 3 - is skipped # SKIP no network
not ok 4 - is pending # TODO soon
ok

The version line is ignored. A skipped test becomes an `omit` and a test
still to be done a `todo`, each with its reason as the message. A test
without a number takes the next one.

entries = Tapout::PerlAdapter.new(StringIO.new(@tap)).to_a

tests = entries.select{ |e| e['type'] == 'test' }

tests.map{ |e| e['status'] }.assert == %w[pass fail omit todo pass]
tests.map{ |e| e['label'] }.assert == ['passes', 'fails', 'is skipped', 'is pending', '']
tests.map{ |e| e['index'] }.assert == [1, 2, 3, 4, 5]

tests[2]['exception']['message'].assert == 'no network'
tests[3]['exception']['message'].assert == 'soon'

The final entry counts each kind.

entries.last['counts'].assert == {
'total' => 5, 'pass' => 2, 'fail' => 1, 'error' => 0, 'omit' => 1, 'todo' => 1
}
60 changes: 60 additions & 0 deletions demo/09_issues/tap_reporter_plan.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
## TAP Reporter

### Issue #14 - tap reporter with skipped tests reports wrong number of tests

The tap reporter printed the plan, say `1..4`, and then a line for each
test that passed, failed or errored, but none for a test that was omitted
or still to be done. A stream with such tests came out short of its plan.

See https://github.com/rubyworks/tapout/issues/14.

require 'stringio'

tapy = <<-END.gsub(/^ /, '')
---
type: suite
count: 4
---
type: test
status: pass
label: one
---
type: test
status: omit
label: two
exception:
message: no network
---
type: test
status: todo
label: three
exception:
message: soon
---
type: test
status: pass
label: four
---
type: final
counts: {total: 4, pass: 2, fail: 0, error: 0, omit: 1, todo: 1}
...
END

stdout = $stdout
$stdout = StringIO.new(out = String.new)
begin
Tapout::YamlParser.new(:format => 'tap').consume(StringIO.new(tapy))
ensure
$stdout = stdout
end

Every test now has its line, with TAP's directives for the two that
weren't run to a verdict.

out.lines.map(&:chomp).assert == [
'1..4',
'ok 1 - one',
'ok 2 - two # SKIP no network',
'not ok 3 - three # TODO soon',
'ok 4 - four'
]
71 changes: 55 additions & 16 deletions lib/tapout/adapters/perl.rb
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
require 'tapout/version'
require 'yaml'

module Tapout

Expand All @@ -21,6 +22,7 @@ def reset
@state = nil
@entries = []
@cache = []
@count = nil
end

# We need to keep an internal list of all entries
Expand Down Expand Up @@ -55,9 +57,11 @@ def <<(line)
parse(@cache)
@state = :footer
finish
when /^TAP version/i
return
when /^\d+\.\.\d+/
#parse(@cache) if @state != :plan
return if @head
parse(@cache)
@head = true
@state = :plan
@cache << line
Expand Down Expand Up @@ -86,19 +90,17 @@ def parse(cache)
when :plan
line = cache[0]
md = /^(\d+)\.\.(\d+)\s*$/.match(line)
count = md[2].to_i - md[1].to_i + 1
entry = {'count'=>count, 'type'=>'suite', 'rev'=>REVISION}
md = /^(\d+)\.\.(\d+)/.match(line)
@count = md[2].to_i - md[1].to_i + 1
entry = {'count'=>@count, 'type'=>'suite', 'rev'=>REVISION}
when :ok
line = cache[0]
md = /^ok\s+(\d+)\s*\-?\s*(.*?)($|#)/.match(line)
entry = {'type'=>'test','status'=>'pass','index'=>md[1].to_i, 'label'=>md[2]}
number, label, directive, reason = parse_test_line(cache[0])
entry = {'type'=>'test', 'status'=>'pass', 'index'=>number, 'label'=>label}
apply_directive(entry, directive, reason)
when :not_ok
line = cache[0]
yaml = cache[1..-2].join('')
yaml = unindent(yaml)
data = YAML.load(yaml)
md = /^not ok\s+(\d+)\s*\-?\s*(.*?)($|#)/.match(line)
entry = convert_not_ok(md[1], md[2], data)
number, label, directive, reason = parse_test_line(cache[0])
entry = convert_not_ok(number, label, diagnostics(cache[1..-1]))
apply_directive(entry, directive, reason)
when :comment
desc = cache.map{ |c| c.sub(/^\#\s{0,1}/, '') }.join("\n")
entry = {'type'=>'note', 'text'=>desc.rstrip}
Expand Down Expand Up @@ -129,6 +131,39 @@ def finish

private

# Split a test line into its number, description, and the directive
# (`skip` or `todo`) and reason that may follow a `#`. The number is
# optional in TAP; a test without one is numbered in order.
def parse_test_line(line)
md = /^(?:not )?ok\b\s*(\d+)?\s*-?\s*(.*)$/.match(line.chomp)
number = md[1] ? md[1].to_i : @entries.count{ |e| e['type'] == 'test' } + 1
label, directive, reason = md[2], nil, nil
if dm = /(\A|\s)#\s*(skip\w*|todo)\b\s*(.*)\z/i.match(label)
label = dm.pre_match
directive = dm[2].downcase.start_with?('skip') ? 'skip' : 'todo'
reason = dm[3]
end
return number, label.strip, directive, reason
end

# In TAP a skipped test is reported `ok`, and one still to be done
# either way. TAP-Y calls these `omit` and `todo`.
def apply_directive(entry, directive, reason)
return unless directive
entry['status'] = (directive == 'skip' ? 'omit' : 'todo')
entry['exception'] ||= {}
entry['exception']['message'] = reason unless reason.to_s.empty?
end

# The YAML block that may follow a test line, as a Hash, or nil.
def diagnostics(lines)
lines = lines.reject{ |l| l =~ /^\s*(---|\.\.\.)\s*$/ }
data = YAML.load(unindent(lines.join('')))
Hash === data ? data : nil
rescue Psych::SyntaxError
nil
end

#
# TODO: Any way to distinguish fail vs error?
def convert_not_ok(number, label, data)
Expand All @@ -141,7 +176,7 @@ def convert_not_ok(number, label, data)
entry['exception'] = {}
entry['exception']['file'] = data['file']
entry['exception']['line'] = data['line']
entry['exception']['message'] = data['description']
entry['exception']['message'] = data['description'] || data['message']

entry['expected'] = data['wanted']
entry['returned'] = data['found']
Expand All @@ -157,10 +192,14 @@ def make_footer

entry = {}
entry['type'] = 'final'
tests = @entries.count{ |e| e['type'] == 'test' }
entry['counts'] = {
'total' => @count,
'pass' => (groups['pass'] || []).size,
'fail' => (groups['fail'] || []).size
'total' => @count || tests,
'pass' => (groups['pass'] || []).size,
'fail' => (groups['fail'] || []).size,
'error' => (groups['error'] || []).size,
'omit' => (groups['omit'] || []).size,
'todo' => (groups['todo'] || []).size
}
entry
end
Expand Down
2 changes: 1 addition & 1 deletion lib/tapout/cli.rb
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ def self.cli(*argv)
stdin = Curmudgeon.new($stdin)

case stdin.line1
when /^\d/
when /^\d/, /^TAP version/i, /^(not )?ok\b/, /^\#/
type = :perl
when /^\-/
type = :yaml
Expand Down
15 changes: 15 additions & 0 deletions lib/tapout/reporters/abstract.rb
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,7 @@ def initialize

@case_stack = []
@source = {}
@start_time = Time.now
@exit_code = 0 # assume passing
end

Expand Down Expand Up @@ -158,6 +159,7 @@ def handle(entry)
when 'note'
note(entry)
when 'test'
complete_exception(entry)
start_test(entry)
case entry['status']
when 'pass'
Expand All @@ -182,6 +184,18 @@ def handle(entry)
end
end

# A test that didn't pass is supposed to say why, in an `exception`
# section with at least a message. Not every producer gives one, or
# all of it: plain TAP has none, and others leave out the class or
# the backtrace. Fill in what reporters rely on.
def complete_exception(entry)
return if entry['status'] == 'pass' && !entry['exception']
exception = entry['exception']
exception = entry['exception'] = {} unless Hash === exception
exception['message'] = exception['message'].to_s
exception['class'] = exception['class'].to_s
end

# Get the exit code.
def exit_code
@exit_code
Expand Down Expand Up @@ -282,6 +296,7 @@ def backtrace(test)

# Clean the backtrace of any "boring" reference.
def clean_backtrace(backtrace)
backtrace = Array(backtrace)
if ENV['debug']
trace = backtrace
else
Expand Down
Loading
Loading