Both Samba 3 and Samba 4 are now using the “subunit” protocol inside their testsuite (aka “make test”). subunit is a streaming protocol used to report test results that is aimed at being simple to generate and parse as well as being human readable.
A very simple subunit stream might look like this:
1 2 3 4 5 6 7 8 9 | test: samba4.tests.util.strlist.check_list_make creating list... list created! success: samba4.tests.util.strlist.check_list_make test: samba4.tests.util.strlist.check_list_make_shell creating list... xfail: samba4.tests.util.strlist.check_list_make_shell [ returned NT_STATUS_NOT_IMPLEMENTED ] |
For those that are familiar with the TAP protocol used by Perl, it is similar to that, although it has a couple of features that TAP does not have. For example, it can report timestamps (useful for determining test duration) and has more flexible progress reporting.
Subunit is particularly useful for projects that use multiple programming languages as it allows a single tool to be used for test visualization or analysis rather than one per language. All that’s required per-language is a test runner that can spit out subunit streams.
selftest.pl, the main engine behind Samba’s test suite, has been using subunit internally since its creation a couple of years ago. Most other test tools we use can also report subunit, in particular our Python tests, blackbox tests, Perl tests (using tap2subunit) and smbtorture.
“make test” never displays raw subunit results, it always formats them using our format-subunit script. Samba 4’s “make test” stores the raw subunit output in st/subunit.
I’m attending SNIA SDC at the moment and a couple of people here have asked me about the tools I use to display and analyse test results. They are:
subunit
The subunit project contains a bunch of convenience tools for working with subunit. Other than libraries for parsing/generating subunit for several languages it contains tools for manipulating and analysing subunit streams, including:
- subunit-ls: List all tests in a subunit stream, optionally including their run times (I used this for the test duration summary I sent to the Samba mailing list earlier)
- tap2subunit: convert a TAP stream to a Subunit stream
- subunit-stats: Print statistics for a subunit stream (how many successful tests, failed tests, skipped tests, etc)
- subunit-filter: E.g. remove test result or output from a stream
- subunit-diff: Compare two subunit streams and see what tests have started failing or are no longer failing
- subunit2pyunit: Format a subunit stream using Python’s standard unit test test result formatter
We’re including the subunit tree in the Samba git tree at lib/subunit.
tribunal
Tribunal is a GTK+ viewer for subunit streams. It allows for easy browsing of test results. Tribunal is still a bit rough around the edges, although it should already be useful.
Example usage:
1 2 | $ make test $ tribunal-subunit st/subunit |
testrepository
Test Repository provides a database of test results which fits into developers work flow and keeps track of useful information like what tests are failing, or which failures have the same backtrace.
In particular Test Repository can re-run only the tests that failed in the previous test run:
1 2 3 4 5 6 7 | $ testr init # Run the full testsuite (1 hour goes by) $ testr run # Run those tests from the testsuite that failed in the previous run # (this would be a lot shorter usually, depending on how many tests were # failing) $ testr run --failing |
testrepository is also still in its early days, but can potentially be very useful, e.g. when comparing old test runs on the buildfarm.