diff --git a/HISTORY.md b/HISTORY.md index 2f3e04e..5de822b 100644 --- a/HISTORY.md +++ b/HISTORY.md @@ -19,6 +19,22 @@ Changes: and the lookup of other project metadata as constants are gone. * Correct the README, which named a `progressbar` reporter; it is `progress`. +* Reporters no longer raise for a test that didn't pass and says little + about why. Only an exception's message is required by TAP-Y/J, but the + `markdown` and `html` reporters needed its class or backtrace, and + every reporter needed the exception section itself, which a failed test + from plain TAP doesn't have. +* The `tap` reporter prints a line for a test that was omitted + (`ok N - label # SKIP reason`) or is still to be done + (`not ok N - label # TODO reason`), so the lines match the plan (#14). + Without a count at the start of the stream, it prints the plan at the end. +* The `outline` reporter lists omitted and to-do tests, with the reason. +* Reading plain TAP: a `TAP version` line is accepted, as is a stream that + starts with a test line or a comment; `# SKIP` and `# TODO` directives + become `omit` and `todo`; a test line needn't have a number; and the + final counts include errors, omissions and to-dos. +* The `outline` and `markdown` reporters no longer raise for tests that + belong to no case, as in plain TAP. ## 0.4.5 / 2013-11-20 diff --git a/demo/01_reporters/sparse_streams.md b/demo/01_reporters/sparse_streams.md new file mode 100644 index 0000000..d8a6804 --- /dev/null +++ b/demo/01_reporters/sparse_streams.md @@ -0,0 +1,98 @@ +# Sparse Streams + +A test that didn't pass is meant to carry an `exception` section, but only +its `message` is required, and not every producer gives even that. Rubytest +gives no exception class, and a backtrace only for errors. Plain TAP gives +nothing at all for a failed test. Every reporter has to cope. + + require 'stringio' + + run = lambda do |format, text, parser = Tapout::YamlParser| + stdout, stderr = $stdout, $stderr + $stdout = StringIO.new(out = String.new) + $stderr = StringIO.new(String.new) + begin + parser.new(:format => format).consume(StringIO.new(text)) + ensure + $stdout, $stderr = stdout, stderr + end + out + end + + reporters = %w[dot progress outline pretty breakdown runtime tap turn markdown html] + +Here a failure has a message and a location but no class or backtrace, an +error has a message alone, and another failure has no exception section. + + sparse = <<-END.gsub(/^ /, '') + --- + type: suite + count: 6 + rev: 5 + --- + type: case + level: 0 + label: Sparse + --- + type: test + status: pass + label: passes + --- + type: test + status: fail + label: fails + exception: + file: test/sparse.rb + line: 9 + message: wrong value + --- + type: test + status: error + label: errors + exception: + message: boom + --- + type: test + status: fail + label: fails without saying why + --- + type: test + status: todo + label: is pending + exception: + message: later + --- + type: test + status: omit + label: is omitted + --- + type: final + counts: + total: 6 + pass: 1 + fail: 2 + error: 1 + omit: 1 + todo: 1 + ... + END + + reporters.each do |format| + run.call(format, sparse).empty?.assert == false + end + +The same goes for tests that belong to no case, as in a stream converted +from plain TAP. + + plain = <<-END.gsub(/^ /, '') + TAP version 13 + 1..4 + ok 1 - passes + not ok 2 - fails + ok 3 - is skipped # SKIP no network + not ok 4 - is pending # TODO soon + END + + reporters.each do |format| + run.call(format, plain, Tapout::PerlParser).empty?.assert == false + end diff --git a/demo/02_adapters/perl_adapter.md b/demo/02_adapters/perl_adapter.md index 86d1531..d20270a 100644 --- a/demo/02_adapters/perl_adapter.md +++ b/demo/02_adapters/perl_adapter.md @@ -64,3 +64,37 @@ and five passing tests. entries.size.assert == 8 + +## Version line and directives + +A stream may open with a version line, and a test line may end with a +`SKIP` or `TODO` directive. Given a legacy TAP stream: + + TAP version 13 + 1..5 + ok 1 - passes + not ok 2 - fails + ok 3 - is skipped # SKIP no network + not ok 4 - is pending # TODO soon + ok + +The version line is ignored. A skipped test becomes an `omit` and a test +still to be done a `todo`, each with its reason as the message. A test +without a number takes the next one. + + entries = Tapout::PerlAdapter.new(StringIO.new(@tap)).to_a + + tests = entries.select{ |e| e['type'] == 'test' } + + tests.map{ |e| e['status'] }.assert == %w[pass fail omit todo pass] + tests.map{ |e| e['label'] }.assert == ['passes', 'fails', 'is skipped', 'is pending', ''] + tests.map{ |e| e['index'] }.assert == [1, 2, 3, 4, 5] + + tests[2]['exception']['message'].assert == 'no network' + tests[3]['exception']['message'].assert == 'soon' + +The final entry counts each kind. + + entries.last['counts'].assert == { + 'total' => 5, 'pass' => 2, 'fail' => 1, 'error' => 0, 'omit' => 1, 'todo' => 1 + } diff --git a/demo/09_issues/tap_reporter_plan.md b/demo/09_issues/tap_reporter_plan.md new file mode 100644 index 0000000..24226f0 --- /dev/null +++ b/demo/09_issues/tap_reporter_plan.md @@ -0,0 +1,60 @@ +## TAP Reporter + +### Issue #14 - tap reporter with skipped tests reports wrong number of tests + +The tap reporter printed the plan, say `1..4`, and then a line for each +test that passed, failed or errored, but none for a test that was omitted +or still to be done. A stream with such tests came out short of its plan. + +See https://github.com/rubyworks/tapout/issues/14. + + require 'stringio' + + tapy = <<-END.gsub(/^ /, '') + --- + type: suite + count: 4 + --- + type: test + status: pass + label: one + --- + type: test + status: omit + label: two + exception: + message: no network + --- + type: test + status: todo + label: three + exception: + message: soon + --- + type: test + status: pass + label: four + --- + type: final + counts: {total: 4, pass: 2, fail: 0, error: 0, omit: 1, todo: 1} + ... + END + + stdout = $stdout + $stdout = StringIO.new(out = String.new) + begin + Tapout::YamlParser.new(:format => 'tap').consume(StringIO.new(tapy)) + ensure + $stdout = stdout + end + +Every test now has its line, with TAP's directives for the two that +weren't run to a verdict. + + out.lines.map(&:chomp).assert == [ + '1..4', + 'ok 1 - one', + 'ok 2 - two # SKIP no network', + 'not ok 3 - three # TODO soon', + 'ok 4 - four' + ] diff --git a/lib/tapout/adapters/perl.rb b/lib/tapout/adapters/perl.rb index d7ea9ab..f3c53cf 100644 --- a/lib/tapout/adapters/perl.rb +++ b/lib/tapout/adapters/perl.rb @@ -1,4 +1,5 @@ require 'tapout/version' +require 'yaml' module Tapout @@ -21,6 +22,7 @@ def reset @state = nil @entries = [] @cache = [] + @count = nil end # We need to keep an internal list of all entries @@ -55,9 +57,11 @@ def <<(line) parse(@cache) @state = :footer finish + when /^TAP version/i + return when /^\d+\.\.\d+/ - #parse(@cache) if @state != :plan return if @head + parse(@cache) @head = true @state = :plan @cache << line @@ -86,19 +90,17 @@ def parse(cache) when :plan line = cache[0] md = /^(\d+)\.\.(\d+)\s*$/.match(line) - count = md[2].to_i - md[1].to_i + 1 - entry = {'count'=>count, 'type'=>'suite', 'rev'=>REVISION} + md = /^(\d+)\.\.(\d+)/.match(line) + @count = md[2].to_i - md[1].to_i + 1 + entry = {'count'=>@count, 'type'=>'suite', 'rev'=>REVISION} when :ok - line = cache[0] - md = /^ok\s+(\d+)\s*\-?\s*(.*?)($|#)/.match(line) - entry = {'type'=>'test','status'=>'pass','index'=>md[1].to_i, 'label'=>md[2]} + number, label, directive, reason = parse_test_line(cache[0]) + entry = {'type'=>'test', 'status'=>'pass', 'index'=>number, 'label'=>label} + apply_directive(entry, directive, reason) when :not_ok - line = cache[0] - yaml = cache[1..-2].join('') - yaml = unindent(yaml) - data = YAML.load(yaml) - md = /^not ok\s+(\d+)\s*\-?\s*(.*?)($|#)/.match(line) - entry = convert_not_ok(md[1], md[2], data) + number, label, directive, reason = parse_test_line(cache[0]) + entry = convert_not_ok(number, label, diagnostics(cache[1..-1])) + apply_directive(entry, directive, reason) when :comment desc = cache.map{ |c| c.sub(/^\#\s{0,1}/, '') }.join("\n") entry = {'type'=>'note', 'text'=>desc.rstrip} @@ -129,6 +131,39 @@ def finish private + # Split a test line into its number, description, and the directive + # (`skip` or `todo`) and reason that may follow a `#`. The number is + # optional in TAP; a test without one is numbered in order. + def parse_test_line(line) + md = /^(?:not )?ok\b\s*(\d+)?\s*-?\s*(.*)$/.match(line.chomp) + number = md[1] ? md[1].to_i : @entries.count{ |e| e['type'] == 'test' } + 1 + label, directive, reason = md[2], nil, nil + if dm = /(\A|\s)#\s*(skip\w*|todo)\b\s*(.*)\z/i.match(label) + label = dm.pre_match + directive = dm[2].downcase.start_with?('skip') ? 'skip' : 'todo' + reason = dm[3] + end + return number, label.strip, directive, reason + end + + # In TAP a skipped test is reported `ok`, and one still to be done + # either way. TAP-Y calls these `omit` and `todo`. + def apply_directive(entry, directive, reason) + return unless directive + entry['status'] = (directive == 'skip' ? 'omit' : 'todo') + entry['exception'] ||= {} + entry['exception']['message'] = reason unless reason.to_s.empty? + end + + # The YAML block that may follow a test line, as a Hash, or nil. + def diagnostics(lines) + lines = lines.reject{ |l| l =~ /^\s*(---|\.\.\.)\s*$/ } + data = YAML.load(unindent(lines.join(''))) + Hash === data ? data : nil + rescue Psych::SyntaxError + nil + end + # # TODO: Any way to distinguish fail vs error? def convert_not_ok(number, label, data) @@ -141,7 +176,7 @@ def convert_not_ok(number, label, data) entry['exception'] = {} entry['exception']['file'] = data['file'] entry['exception']['line'] = data['line'] - entry['exception']['message'] = data['description'] + entry['exception']['message'] = data['description'] || data['message'] entry['expected'] = data['wanted'] entry['returned'] = data['found'] @@ -157,10 +192,14 @@ def make_footer entry = {} entry['type'] = 'final' + tests = @entries.count{ |e| e['type'] == 'test' } entry['counts'] = { - 'total' => @count, - 'pass' => (groups['pass'] || []).size, - 'fail' => (groups['fail'] || []).size + 'total' => @count || tests, + 'pass' => (groups['pass'] || []).size, + 'fail' => (groups['fail'] || []).size, + 'error' => (groups['error'] || []).size, + 'omit' => (groups['omit'] || []).size, + 'todo' => (groups['todo'] || []).size } entry end diff --git a/lib/tapout/cli.rb b/lib/tapout/cli.rb index 7283bb6..4ae99fd 100644 --- a/lib/tapout/cli.rb +++ b/lib/tapout/cli.rb @@ -61,7 +61,7 @@ def self.cli(*argv) stdin = Curmudgeon.new($stdin) case stdin.line1 - when /^\d/ + when /^\d/, /^TAP version/i, /^(not )?ok\b/, /^\#/ type = :perl when /^\-/ type = :yaml diff --git a/lib/tapout/reporters/abstract.rb b/lib/tapout/reporters/abstract.rb index ef2a8cb..04e3f38 100644 --- a/lib/tapout/reporters/abstract.rb +++ b/lib/tapout/reporters/abstract.rb @@ -69,6 +69,7 @@ def initialize @case_stack = [] @source = {} + @start_time = Time.now @exit_code = 0 # assume passing end @@ -158,6 +159,7 @@ def handle(entry) when 'note' note(entry) when 'test' + complete_exception(entry) start_test(entry) case entry['status'] when 'pass' @@ -182,6 +184,18 @@ def handle(entry) end end + # A test that didn't pass is supposed to say why, in an `exception` + # section with at least a message. Not every producer gives one, or + # all of it: plain TAP has none, and others leave out the class or + # the backtrace. Fill in what reporters rely on. + def complete_exception(entry) + return if entry['status'] == 'pass' && !entry['exception'] + exception = entry['exception'] + exception = entry['exception'] = {} unless Hash === exception + exception['message'] = exception['message'].to_s + exception['class'] = exception['class'].to_s + end + # Get the exit code. def exit_code @exit_code @@ -282,6 +296,7 @@ def backtrace(test) # Clean the backtrace of any "boring" reference. def clean_backtrace(backtrace) + backtrace = Array(backtrace) if ENV['debug'] trace = backtrace else diff --git a/lib/tapout/reporters/markdown_reporter.rb b/lib/tapout/reporters/markdown_reporter.rb index 2bec518..4d1dd3a 100644 --- a/lib/tapout/reporters/markdown_reporter.rb +++ b/lib/tapout/reporters/markdown_reporter.rb @@ -9,6 +9,7 @@ class Markdown < Abstract # def start_suite(entry) @start_time = Time.now + @_level = 0 # for tests outside of any case end # @@ -38,8 +39,10 @@ def fail(entry) $stdout.puts "\n**MESSAGE**\n\n" $stdout.puts entry['exception']['message'].tabto(4) - $stdout.puts "\n**TYPE**\n\n" - $stdout.puts entry['exception']['class'].tabto(4) + unless entry['exception']['class'].empty? + $stdout.puts "\n**TYPE**\n\n" + $stdout.puts entry['exception']['class'].tabto(4) + end #$stdout.puts " " + ok.caller #clean_backtrace(exception.backtrace)[0] @@ -68,8 +71,10 @@ def error(entry) $stdout.puts "\n**MESSAGE**\n\n" $stdout.puts entry['exception']['message'].tabto(4) - $stdout.puts "\n**TYPE**\n\n" - $stdout.puts entry['exception']['class'].tabto(4) + unless entry['exception']['class'].empty? + $stdout.puts "\n**TYPE**\n\n" + $stdout.puts entry['exception']['class'].tabto(4) + end #$stdout.puts " " + ok.caller #clean_backtrace(exception.backtrace)[0..2].join(" \n") diff --git a/lib/tapout/reporters/outline_reporter.rb b/lib/tapout/reporters/outline_reporter.rb index bc9f9d7..c1d2ab6 100644 --- a/lib/tapout/reporters/outline_reporter.rb +++ b/lib/tapout/reporters/outline_reporter.rb @@ -11,6 +11,8 @@ class Outline < Abstract # def start_suite(entry) @_case_count = [0] + @_test_count = 0 + @_level = -1 # tests outside of any case start at the margin @start_time = Time.now end @@ -68,6 +70,18 @@ def error(entry) printout(entry, 'ERROR', config.error) end + def todo(entry) + super(entry) + + remark(entry, 'TODO', config.todo) + end + + def omit(entry) + super(entry) + + remark(entry, 'OMIT', config.omit) + end + # def finish_suite(entry) time, rate, avg = time_tally(entry) @@ -81,6 +95,17 @@ def finish_suite(entry) private + # One line for a test that wasn't run to a verdict, with the reason. + def remark(entry, type, ansi) + reason = entry['reason'] || entry['exception']['message'] + reason = reason.to_s.lines.first.to_s.strip + + print(' ' * level_tab) + line = "#{test_count}. " + entry['label'].ansi(*ansi) + " #{type}" + line += ": #{reason}" unless reason.empty? + puts line + end + # def printout(entry, type, ansi) counter = "#{test_count}. " diff --git a/lib/tapout/reporters/tap_reporter.rb b/lib/tapout/reporters/tap_reporter.rb index 31e0f6a..106c223 100644 --- a/lib/tapout/reporters/tap_reporter.rb +++ b/lib/tapout/reporters/tap_reporter.rb @@ -11,7 +11,13 @@ def start_suite(entry) @i = 0 #n = 0 #suite.concerns.each{ |f| f.concerns.each { |s| n += s.ok.size } } - puts "1..#{entry['count']}" + @plan = entry['count'] + puts "1..#{@plan}" if @plan + end + + # Without a count at the start, the plan goes at the end. + def finish_suite(entry) + puts "1..#{@i}" unless @plan end # @@ -64,7 +70,7 @@ def error(entry) body = [] body << "ERROR #{x['file']}:#{x['line']}" #clean_backtrace(exception.backtrace)[0..2].join(" \n") - body << "#{x['class']}: #{x['message']}" + body << [x['class'], x['message']].reject{ |s| s.to_s.empty? }.join(': ') #body << "#{x['message']}" body << "" body << code_snippet(entry) @@ -75,12 +81,30 @@ def error(entry) puts body end - # - #def pending(ok, exception) - # puts "not ok #{@i} - #{unit.description}" - # puts " PENDING" - # puts " #{exception.backtrace[1]}" - #end + # A test still to be done is `not ok` with a TODO directive, which + # TAP consumers don't count as a failure. + def todo(entry) + super(entry) + + @i += 1 + puts "not ok #{@i} - #{entry['label']} # TODO #{directive_reason(entry)}".rstrip + end + + # An omitted test is `ok` with a SKIP directive. + def omit(entry) + super(entry) + + @i += 1 + puts "ok #{@i} - #{entry['label']} # SKIP #{directive_reason(entry)}".rstrip + end + + private + + # The reason for a directive, on one line. + def directive_reason(entry) + reason = entry['reason'] || (entry['exception'] || {})['message'] + reason.to_s.lines.first.to_s.strip + end end end