Guidelines for "langtools" tests
The "langtools" tests in the test/langtools directory
of the main jdk/jdk repository are the most important set
of freely available tests for the products in the langtools family.
And the test base is growing fast: in OpenJDK 6, there were 1408
tests; in OpenJDK 7, that number rose 36% to 1924 tests. In
addition, many of the newer tests include hundreds or even
thousands of individual test cases.
With that in mind, it is important to have some guidelines for
the tests that appear here. The following reflect the current
practices for the langtools tests in the main OpenJDK repository.
Other teams may choose to have different guidelines.
1. The Golden RuleAll the tests1,2 should pass, all the
time3, on all platforms4, in all modes5,6.- except tests with @ignore, which itself is strongly discouraged
- not just javac tests
- at the time of checkin, and after
- including Windows and Mac OS/X
- jtreg modes: agentvm and othervm
- including with assertions enabled for
langtools packages
2. Write testsMost code changes require corresponding tests to exercise the
new or updated functionality. The exceptions are listed in
The OpenJDK Developers' Guide,
under "Change Planning and Guidelines:
Fixing a Bug", step
6.
3. Legal stuffAll the tests are Open Source, using GPL2. (Note:
not
GPL2 with the "Classpath" exception.) Most test files should
include the appropriate header. The primary exception is "
golden file tests".
4. No binary filesSome binary files currently exist: they need to be removed
eventually. To provide a test with a class file that has particular
characteristics, consider providing a source file, compiling it,
and then mutating the resulting class file as necessary. In
isolated cases, it may be appropriate to store an extended series
of bytes as data within a Java source file, provided that such data
is well commented.
5. No new shell scriptsShell scripts are hard to maintain across all platforms, and
are slower to execute. Write Java code instead. In many cases, in
older versions of the langtools tests, shell scripts were used to
write golden file tests. jtreg now provides improved support for
such tests, so shell tests should no longer be necessary for that
reason. Also, see the following for utilities to simplify writing
shell-like functionality in Java:
6. No unnecessary use of Runtime.execCreating child processes is (relatively) slow, so avoid use of
Runtime.exec unless it is important to the execution
of the test. Tests that exec java should normally honor/propogate
test VM options, so that options like
-Xbootclasspath/p: and options to run jcov continue to
work as expected in the child JVM.
7. No manual testsi.e. no use of /manual in test descriptions. Some
tests may have a mode in which they can be run directly: that is
different, and is OK.
8. No external dependenciesNo external dependencies on anything except the version of JDK
being tested, and jtreg. Some tests refer to the langtools source
code, and that is OK. jtreg itself may provide access to other
facilities, like JUnit.
9. Tests should be fastGenerally, tests should take under a minute or so to run. Tests
that take a long time need to be justified in terms of need and
test cases covered. The current standard deviation of test
execution time is under 3 seconds. That means that tests taking
longer than 30 seconds are already 10 standard deviations from the
mean (i.e. very atypical.) Slow tests that need to override the
default timeout should generally specify a value that is double the
expected execution time on a typical developer platform.
10. Golden file
testsGolden file tests (tests whose success depends on generating
output to be matched a reference, or golden, file) are occasionally
necessary. If the reference file contains references to line
numbers in a source file, then that source file does not need a
copyright block. Instead, it should contain the marker string
"/nodynamiccopyright/" as an explicit indication that
no copyright or other legal header should be added, because to do
so would break the test. If the source file contains a jtreg test
description, the marker string should normally appear after
"@test", on the same line.
11. Negative compiler testsNegative compiler tests (tests using
@compile/fail) should always use a golden file. This
is to protect against a false-positive test result caused by javac
crashing. The preferred directive for negative compiler tests is:@compile/fail/ref=
FILE.out -XDrawDiagnostics ...The use of -XDrawDiagnostics isolates the test from
changes in the resource file, and removes other elements, like
directory names, that are specific to the instance of the test
run.
12. JDK, and -Xbootclasspath/p:Tests should all work as expected when testing a full JDK
build. They should also work when testing an incremental build,
using options such as -Xbootclasspath/p:.
13. OrganizationThe tests in langtools/test/tools/javac need some
attention and TLC, when it comes to their organization. While that
happens, no new tests should be added directly into the
javac directory: instead, they should be added to a
suitable sub-directory. By default, the hierarchy under the
javac directory should mirror the package hierarchy
under com.sun.tools.javac. For example, tests for
com.sun.tools.javac.
file.JavacFileManager
should be placed in
langtools/test/tools/javac/
file.
Additional sub-directories may be created as needed to group the
files of a test or to group related tests. Try and come up with
helpful names for tests before you fall back on using the bug
number.
14. Whitespace and styleTests should follow the standard white space rules and coding
conventions as is the case for any OpenJDK source code. Code should
not contain tabs, and indentation should be in multiples of 4
spaces. Lines should be terminated with Unix-style newlines
('\n'), and not have extraneous spaces before the
newline character. It may be appropriate for some tests to test a
tool's ability to handle source code that does not conform to these
guidelines: in such cases, it is suggested and recommended that
such code should be generated on the fly, as part of the test. New
tests should use the standard Java coding conventions generally
encouraged by modern IDEs. Modifications to existing tests should
use the prevailing style of the code being modified. If a test
warrants breaking these guidelines, the reason should be clearly
documented in the test.
15. Disabling/ignoring
testsSometimes a test may fail, and it may be desirable to avoid
running the test until the underlying cause can be addressed. Such
tests should be marked with @ignore followed by the
bug number and description of the reason why the test has been so
marked. If a test contains multiple test cases within it, a test
may provide a way of identifying test cases that should not be run,
such as an @ignore annotation on a test method. In
such situations, the line should end with a bug number and
description of why the test case has been so marked. The general
intent is that it should be possible to write scripts that search
for files containing lines that match the regular expression
(?i)@ignore.*([0-9]{7}): +(.*) in order to report
on tests or test cases that have been disabled.