[schooltool] RFC: test.py --list-modules

[Marius Gedminas]

The Zope 3 test runner can filter tests by package, module, test name and
layer. Sometimes your expectations do not match reality and either the
test you want is not being run, or more tests than you wanted are being run.
When you want to debug filter patterns it helps if you can see what
tests would be run without wasting time actually running them.

I have implemented a --list-modules option in a branch. It causes the
test runner to apply package and module name filters, and then print the
list of Python module names that would be imported. It is very fast
because it doesn't actually import the modules. It is not very accurate
because it does not pay attention to test name patterns or layer filtering.

Would this option be useful to anybody?

You can find the code here:

http://svn.zope.org/zope.testing/branches/list-modules/?rev=71982&view=rev

I am also thinking about two other options:

* --list-files -- essentially the same as --list-modules, but outputs
file names instead of module names.

* --list-tests -- apply package and module filters, import the
modules, apply test name and layer filters, then list just the names
of tests.

Would these be useful to anybody?

SchoolTool's custom test runner used to have --list-tests and
--list-files. Now that SchoolTool has finally switched to the Zope 3
test runner, I wonder if anyone working on SchoolTool would miss those
(hi, Ignas ;-).

Marius Gedminas
--
TCP_SeqNum - The 32-bit Sequence Number, encoded as an ASCII string
representing the hex value of the Sequence number. This field
MUST be sent as lower case because it is not urgent.
-- RFC 3093
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://lists.schooltool.org/pipermail/schooltool/attachments/20070112/7d813549/attachment.pgp

[Marius Gedminas]

Post by Marius Gedminas
I have implemented a --list-modules option in a branch. It causes the
test runner to apply package and module name filters, and then print the
list of Python module names that would be imported. It is very fast
because it doesn't actually import the modules. It is not very accurate
because it does not pay attention to test name patterns or layer filtering.
Would this option be useful to anybody?

If its output were identical to the tests that were actually going to be
selected, then I think it'd be worth including.

That would be the proposed --list-tests option, wouldn't it?

Or do you want a list of modules that have at least one test remaining
after all the filtering (including --layer, --level and --test) is done?

OTOH, being "not very
accurate" would make the feature as implemented a bit of an attractive
nuisance.

I phrased it badly. s/It is not very accurate/It may not match user
expectations/.

As implemented in that branch, --list-modules is the list of test
modules that will be imported. That is accurate. My doubts lie in
whether this is what the user wants.

A long time ago, the Zope 3 test runner would only import unit or
functional test modules if it saw the corresponding option on the
command line. I was used to that. I think that is why I thought
seeing unit tests in 'test.py -f --list-modules' might be confusing.

Marius Gedminas

[Ignas Mikalajunas]

Post by Marius Gedminas
* --list-files -- essentially the same as --list-modules, but outputs
file names instead of module names.
* --list-tests -- apply package and module filters, import the
modules, apply test name and layer filters, then list just the names
of tests.
Would these be useful to anybody?

Yes a lot, I am using them in 2 cases, when I want to be sure that
only 1-2 tests will be run before even trying to run them (with some
functional tests running a lot of time I don't want to run more tests
than I need).

And with Zope3 testrunner's idea about the amount of doctests being
run --list-tests would be the only way to find out how many real tests
are being executes as at the moment I can't tell if my doctest has 11
"tests" in it or am I running some other test that matches the
pattern.

Post by Marius Gedminas
SchoolTool's custom test runner used to have --list-tests and
--list-files. Now that SchoolTool has finally switched to the Zope 3
test runner, I wonder if anyone working on SchoolTool would miss those
(hi, Ignas ;-).

Yes especially on long winter evenings while running functional tests
and wondering how many functional tests are there in this functional
test layer I so long for the good old --list-files ;)

Ignas

[Marius Gedminas]

Post by Marius Gedminas
The Zope 3 test runner can filter tests by package, module, test name and
layer. Sometimes your expectations do not match reality and either the
test you want is not being run, or more tests than you wanted are being run.
When you want to debug filter patterns it helps if you can see what
tests would be run without wasting time actually running them.
I have implemented a --list-modules option in a branch. It causes the
test runner to apply package and module name filters, and then print the
list of Python module names that would be imported. It is very fast
because it doesn't actually import the modules.

I have now implemented --list-tests in a different branch. That option
pays attention to all of -t, -m, -s, -u, -f, --layer, --level options.

http://svn.zope.org/zope.testing/branches/list-tests/?rev=72034&view=rev

I intend to merge this one, unless somebody suggests a cleaner way of
implementing it.

I do not think I care enough about --list-modules or --list-files to push
those to completion.

Marius Gedminas

[Marius Gedminas]

Post by Marius Gedminas
I have now implemented --list-tests in a different branch. That option
pays attention to all of -t, -m, -s, -u, -f, --layer, --level options.
http://svn.zope.org/zope.testing/branches/list-tests/?rev=72034&view=rev
I intend to merge this one, unless somebody suggests a cleaner way of
implementing it.

Merged.

Marius Gedminas
--
"Nobody will ever need more than 640k RAM!"
-- Bill Gates, 1981
"Windows 95 needs at least 8 MB RAM."
-- Bill Gates, 1996
"Nobody will ever need Windows 95."
-- logical conclusion
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://lists.schooltool.org/pipermail/schooltool/attachments/20070120/b7264ca3/attachment.pgp