Test of TestNG 1

Please post comments on this tutorial at http://forums.opensymphony.com/thread.jspa?threadID=28866&tstart=0 so it may be improved. This is only the first part of the tutorial. I will complete new sections as I get the time.

You can email me at cquezel[at]dev[dot]java[dot]net.

Introduction

This tutorial is designed to teach TestNG (http://testng.org) using the command line, the TestNGAntTask Ant task and the Eclipse plugin. This tutorial is based on:

1) TestNG version 4.7

2) Ant version 1.6.5

3) “TestNG Eclipse support” plugin version 4.7.0.0.

4) Eclipse 3.1.1

The Eclipse plugin sections may be skipped, by users who do not use Eclipse, without loss of continuity. The same is true with the Ant sections.

Getting Help

Questions on TestNG usage should be posted to the user’s forum: http://forums.opensymphony.com/forum.jspa?forumID=14.

There are three TestNG related forums: TestNG users, TestNG developers and TestNG cvs. See http://forums.opensymphony.com/category.jspa?categoryID=9.

Project source home

TestNG: https://testng.dev.java.net/

“TestNG Eclipse support” plugin: https://testng-eclipse.dev.java.net/

Installing the Eclipse plugin

Using the Eclipse software update:

1) Select “Help” -> “Software Updates” -> ”Find and Install…”

2) Select “Search for new features to install” then “Next”

3) Select “New Remote Site …”

4) Type in a convenient name (TestNG for example) and http://beust.com/eclipse in the URL.

5) Make sure the checkbox associated to TestNG is checked then select “Finish”.

6) Follow the Eclipse instructions

Installing TestNG

Download TestNG distribution from http://testng.org/doc/download.html. The distribution contains sources and libraries necessary to execute TestNG. The distribution does not contain all elements necessary to build TestNG (see Project source home above).

@TODO what is missing?

TestNG 4.7 comes with two implementations: A JDK1.4 version which uses JavaDoc type annotations (testng-4.7-jdk14.jar) and a JDK5+ version which uses either Java type annotations or JavaDoc type annotations (testng-4.7-jdk15.jar).

Setting up your test environment

To use the command line tools you must add one of the two TestNG JARs to your class path.

1) Use testng-4.7-jdk14.jar if you use a 1.4 JRE.

2) Use testng-4.7-jdk15.jar if you use a 5.0+ JRE.

3) You may use a testng-4.7-jdk14.jar with a 5.0+ JRE but Java type annotations will be ignored. This configuration will not be discussed in this tutorial as it offers no additional value.

Note that the TestNG java sources in the distribution are divided into three directories: jdk14, jdk15 and main (common to both JDK 1.4 and JDK 5+). Many sources in the jdk14 and the jdk15 directory have the same name. If you plan on adding the TestNG sources to your source path (for debugging) make sure you select the correct directories.

Note that TestNG sources for the JDK 1.4 are written using JDK 5+ features such as generics but compiled to target 1.4. This compiler extension offered by Sun’s compiler may not be available in your compiler (Eclipse for example).

TestNG uses three third party open source projects:

The backport-util-concurrent http://dcl.mathcs.emory.edu/util/backport-util-concurrent/ (used only by version 1.4)

The bean-shell http://www.beanshell.org/

QDox 1.5 http://qdox.codehaus.org/

The source to these projects is not included in the distribution. Be aware that the classes of these third party tools are placed directly in the testng-4.7-jdkXX.jar files (i.e. check you classpath if you directly or indirectly use these third party open source projects). Using a more recent version of QDox (later than 1.5) may change TestNG’s behavior (i.e. don’t do it).

Tutorial configuration

In this tutorial, all file names using JavaDoc type annotations are suffixed with “JD”. The files using JDK5+ type annotations use “50” suffix. All binaries compiled with JDK1.4 are suffixed with “14” while those compiled with JDK5+ use the “15” suffix. For example, the classes to be tested are in the “tobetested14.jar” or the “tobetested15.jar” library.

In an Ant tasks, JAR file names are often hidden in build.properties file to make the Ant build file independent of JAR file names. Because this tutorial uses two versions of TestNG, we don’t do this to explicitly show which version of the JAR file are used.

An Introduction Example Using Java Type Annotations

This example illustrates how to launch TestNG using the command line. The Introduction50 class uses Java type annotations so testng-4.7-jdk15.jar with a JRE 5+ must be used. The IntroductionJD class, in the next section, uses JavaDoc type annotations.

Introduction50.java:

package org.testng.tutorial.test;

import org.testng.Assert;

import org.testng.annotations.Test;

import org.testng.tutorial.ToBeTested;

/**

* <code>Introduction50</code> is an introductory example to

* illustrates the most basic features of TestNG using JDK5+ type

* annotations.

public class Introduction50

{

/**

* A simple test method.

@Test

public void theTestMethod50()

{

Assert.assertEquals(ToBeTested.return1(), 1);

}

To launch this test using the command line, execute:

java -classpath testng-4.7-jdk15.jar;usetotest15.jar;tobetested15.jar org.testng.TestNG -testclass org.testng.tutorial.test.Introduction50

The following is displayed in the console:

===============================================

Suite for Command line test

Total tests run: 1, Failures: 0, Skips: 0

===============================================

Comments:

The @Test annotation is used to identify test methods. Unlike JUnit, no special naming convention or superclass is required.

In this example, a test suite named “Suite for Command line test” is created. The suite contains one test, named “Command line test”, made up of one test method: theTestMethod().

A “test-output” directory is created with a detailed html report of the test suite. The index.html is the entry point to this report. The report is made up of an xml file for each test suite. The test suite result’s xml file format is identical to JUnit’s format so tools like <junitreport> Ant Task may be used. Each time you run a test suite, the results of the previous execution are overwritten.

When test methods fail, it is possible to relaunch only the test methods that failed. TestNG generates a “testng-failures.xml” test suite made up of the test methods that failed.

@TODO what is the difference between “testng-failures.xml” and “testng-failed.xml”?

If you are familiar with JUnit, take note that the parameter convention, in the various assertion methods of the org.testng.Assert class, is opposite of that of the org.junit.Assert class. In TestNG the “actual” result comes before the “expected” result. If you are porting code from JUnit, TestNG provides the org.testng.AssertJUnit class for compatibility. Note that the org.testng.Assert class offers more functionality than its JUnit counterpart. TestNG offers a JUnit compatibility mode which will be discussed later.

The command line is not the preferred method of launching TestNG. The TestNG Ant task offers a richer set of options. The TestNG Ant task is described below.

An Introduction Example Using JavaDoc Type Annotations

This example is identical to Introduction50 except that:

1) It uses JavaDoc type annotation.

2) It changes the default “test-output” folder to IntroductionJD.

3) It uses more detailed logging.

4) It uses a xml test suite file to replace the -testclass command line parameter.

IntroductionJD.java:

package org.testng.tutorial.test;

import org.testng.Assert;

import org.testng.tutorial.ToBeTested;

/**

* <code>IntroductionJD</code> is an introductory example to

* illustrates the most basic features of TestNG using JavaDoc type

* annotations.

public class IntroductionJD

{

/**

* A simple test method.

* @testng.test

public void theTestMethodJD()

{

Assert.assertEquals(ToBeTested.return1(), 1);

}

The xml file shown below replaces the -testclass argument in the previous example. For now, this is no great simplification. As we will see shortly this method is more flexible:

IntroductionJD.xml:

<?xml version="1.0" encoding="ISO-8859-1"?>

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">

</classes>

</test>

</suite>

To launch this test using the command line using testng-4.7-jdk14.jar, execute:

java -classpath testng-4.7-jdk14.jar;usetotest14.jar;tobetested14.jar org.testng.TestNG –d IntroductionJD –log 3 –sourcedir src IntroductionJD.xml

Comments:

Note that JavaDoc type annotations are case sensitive and that the case and format slightly differs from the corresponding Java type annotations. Java type annotations are all defined using the “@testng.” prefix.

testng-4.7-jdk14.jar uses JavaDoc type annotation only so the “annotations="javadoc"” property in IntroductionJD.xml is not used in this case. Setting it explicitly actually makes IntroductionJD.xml portable to testng-4.7-jdk15.jar.

When using JavaDoc type annotations, the Java source files corresponding to the various class files must be available. TestNG parses the source files to interpret the JavaDoc type annotations at run time. The –sourcedir elements do not have to specify the root of a package hierarchy. For example, the following sourcedir is valid:

–sourcedir d:/src/org/testng;d:/src/org/xml/sax

If your sourcedir contains invalid directories, TestNG will silently ignore them.

The current DTD for TestNG suite is available at: http://testng.org/testng-1.0.dtd (HTML version).

The test suite file must be the last element on the command line.

The verbose property allows for more detailed logging. The possible values are 0 to 10 where 10 is the most detailed level. This value defaults to 1. Logging occurs at different levels in TestNG. When logging test specific information, the test verbose property is used. If not specified, the suite verbose property is used. If not specified the default value (1) is used. The –log option overrides the suite verbose property. The –log option and the verbose properties do not affect the html test report.

If you are familiar with JUnit, note that JUnit does not offer the possibility to use annotations using JDK14. This can be very limiting for projects that are not ready to move to JDK15. Users in this situation, who do not migrate to TestNG, are constrained to use pre JUnit 4.0 features (and suffer).

An Introduction Example Using JavaDoc Type Annotation part II

To repeat the previous test with testng-4.7-jdk15.jar, make sure IntroductionJD.xml indicates that you wish to use JavaDoc annotations for this test because testng-4.7-jdk15.jar defaults to Java type annotation. To launch this test using the command line using testng-4.7-jdk15.jar, execute:

java -classpath testng-4.7-jdk15.jar; usetotestXX.jar;tobetestedXX.jar org.testng.TestNG -d IntroductionJD -log 3 –sourcedir src IntroductionJD.xml

Comments:

The “annotations="javadoc"” property specified in the suite overrides the default target type and the -target command line parameter. This is essential when mixed annotation types are used by many tests in a suite. This topic will be covered later.

Using testng-4.7-jdk15.jar, we could have omitted the “annotations="javadoc"” in IntroductionJD.xml if we had explicitly set the “–target 1.4” option on the command line. For example:

java -classpath testng-4.7-jdk15.jar; usetotestXX.jar;tobetestedXX.jar org.testng.TestNG -d IntroductionJD -log 3 –sourcedir src –target 1.4 IntroductionJD.xml

The “-target” option is used to set the default annotations property in test suites when an explicit value is not specified. The “-target 1.4” is used to configure testng-4.7-jdk15.jar to default to “javadoc” type annotations. The -target option defaults to 1.5 when using testng-4.7-jdk15.jar.

Running a Class Test Using Eclipse’s “Run As” Menu Item

In Eclipse’s package explorer view, if you select a test file (Introduction50.java or IntroductionJD.java for example), the contextualized menu (right click on file) will display the “TestNG test” options for the “run as” and “debug as” menu items. When a test file is open in an editor window, the contextualized menu (right click on file) will display the “TestNG/run as test” and “TestNG/debug as test” options. The TestNG plugin scans the target file and automatically detects any TestNG annotations.

Essentially, the Eclipse plugin automatically creates the following test suites for these test classes:

For IntroductionJD.java

<?xml version="1.0" encoding="ISO-8859-1"?>

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">

<test verbose="2" name="org.testng.tutorial.test.IntroductionJD"

annotations="javadoc"

</classes>

</test>

</suite>

For Introduction50.java

<?xml version="1.0" encoding="ISO-8859-1"?>

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">

<test verbose="2" name="org.testng.tutorial.test.Introduction50"

annotations="jdk50"

</classes>

</test>

</suite>

The TestNG version used (testng-4.7-jdk14.jar or testng-4.7-jdk15.jar) to run the tests suite depends on the runtime JRE. The runtime JRE is the JRE specified in “Windows/Preferences …/Java/Installed JREs”.

Comments:

The Eclipse plugin uses the project’s source folders and all referenced project’s source folders to setup TestNG’s sourcedir.

If the default options do not satisfy your needs, you may define an Eclipse launch configuration. This is discussed later.

When test methods fail, it is possible to relaunch only the test methods that failed. The Eclipse plugin has a “Run Failed Test” button.

Unlike the JUnit pluggin, the “run as/TestNG test” and “debug as/TestNG test” commands are not available in Eclipse’s navigator view.

@TODO see this again

Unlike the JUnit pluggin, the TestNG pluggin does not create a launch configuration for every test launched using the Eclipse UI. The plugin creates and reuses a shared launch configuration named “TestNG context suite” for each test. The configuration is automatically updated each time a test is launched using the Eclipse UI. The testNG plugin also generates temporary suite files in the project’s root directory. These temporary files are never read so don’t bother editing them to change a default behavior.

Errors using the Eclipse plugin

If you run a test using the Eclipse plugin and a TestNG error occurs, TestNG throws an exception. Unfortunately, the plugin swallows these exceptions. If you are having problems running a test suite, try the command line or the Ant Task to see the Exception. Looking at “eclipse/workspace/.metadata/.log” file for runtime details may also help.

Running a Test Suite Using Eclipse’s “Run As” Menu Item

In Eclipse’s package explorer or navigator view, if you select a test suite (IntroductionJD.xml for example), the contextualized menu (right click on file) will display the “TestNG suite” options for the “run as” and “debug as” menu items.

Running a Class Test Using an Eclipse Launch Configuration

1) From the “Run” menu, select “Run…” or “Debug…”.

2) Select “TestNG” in the configuration tree and select “New”.

3) In the “Run…” fieldset, select the “Class” radio button and enter the class name (for example: org.testng.tutorial.test.IntroductionJD). Note that the “Browse …” button will automatically display all test classes in the project.

4) In the “Runtime” fieldset, select the compliance level (for example: 1.5 for Introduction50 and 1.4 or 1.5 for IntroductionJD). The Runtime compliance level is not the –target command line option. The Runtime compliance level actually selects the TestNG JAR version to be used (testng-jdkXX.jar) by the plugin.

5) Enter the various fields common to all launch configurations.

Running a Test Suite Using an Eclipse Launch Configuration

The steps to do this are identical to the “Running a Class Test Using an Eclipse Launch Configuration” above except for step 3 where:

In the “Run…” fieldset, select the “Suite” radio button and enter the suite name (for example: IntroductionJD.xml). Note that the “Browse …” button will automatically display all TestNG suites in the project.

Improving TestNG Speed

When using JavaDoc type annotations, you may override the –sourcedir option by including the “testng-sourcedir-override.properties” resource on the classpath. The content of this property file is:

# A semi-colon separated list of directories to scan for TestNG JavaDoc

# annotations. When testng-sourcedir-override.properties resource is

# found in the classpath, the sourcedir specified here overrides the

# TestNG -sourcedir command line argument. Note that the source path

# does not have to be the root of a package hierarchy.

sourcedir=D:/dev/java/maintenance1.4/src/test/javax/xml/transform/sax

Some tools, such as the TestsNG Eclipse plugin, automatically uses the project’s Java Build Path Source Folders and all dependant project Java Build Path Source Folders. If you are using JavaDoc style notation, the scanning of files may take a long time. By using “testng-sourcedir-override.properties” you may shorten this process.

*The feature presented above has been implemented after TestNG version 4.7. This feature has been committed in the CVS repository.

Introduction Using Java Type Annotation and Ant

This example shows how to configure an Ant task to accomplish what was done using the command line version of TestNG in the previous sections.

introduction50_ant.xml:

<?xml version="1.0" encoding="UTF-8"?>

<description>testng tutorial</description>

</classpath>

</testng>

</target>

</project>

To launch this test using Ant, execute:

java -classpath ant.jar;ant-launcher.jar -Dant.home=apache-ant-1.6.5 org.apache.tools.ant.launch.Launcher -buildfile Introduction50_ant.xml

Comments:

This section’s goal is to show how to define use the TestNG Ant task. For more information on Ant see http://ant.apache.org/manual.

The classpath element specifies the location of classes being tested and classes performing the test.

The xmlfilesetdir element specifies which suites to run.

There are various attributes and nested elements for the TestNG Ant task. See the reference documentation of the TestNG Ant at http://testng.org/doc/ant.html.

Introduction Using JavaDoc Type Annotation and Ant

This example is similar to the previous example except for the following changes:

1) The classpath references "testng-4.7-jdk15.jar"

2) The includes references the "introductionJD.xml" test suite.

3) The sourcedir path-like structure is specified.

introductionJD_ant.xml:

<?xml version="1.0" encoding="UTF-8"?>

<description>testng tutorial</description>

</classpath>

</sourcedir>

</testng>

</target>

</project>

To launch this test using Ant, execute:

java -classpath ant.jar;ant-launcher.jar -Dant.home=apache-ant-1.6.5 org.apache.tools.ant.launch.Launcher -buildfile introductionJD_ant.xml

Running Both Introduction Examples in One Suite

This is simply a merge of introduction50.xml and of introductionJD.xml suites.

introductions.xml:

<?xml version="1.0" encoding="ISO-8859-1"?>

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">

</classes>

</test>

</classes>

</test>

</suite>

To launch this test using the command line, execute:

java -classpath testng-4.7-jdk15.jar;usetotest15.jar;tobetested15.jar org.testng.TestNG -sourcedir src introductions.xml

Comments:

Note that contrary to the command line version of this test, using introductions.xml allows explicitly specifying a different annotation type for each test.

Running Both Introduction Examples within One Suite and Ant

The Ant script for this case is simply a merge of the previous Ant scripts.

introductions_ant.xml:

<?xml version="1.0" encoding="UTF-8"?>

<description>testng tutorial</description>

</classpath>

</sourcedir>

</testng>

</target>

</project>

To launch this test using Ant, execute:

java -classpath ant.jar;ant-launcher.jar -Dant.home=apache-ant-1.6.5 org.apache.tools.ant.launch.Launcher -buildfile introductions_ant.xml

Running Both Introductions from the Command Line

If we try to execute the following command,

java -classpath testng-4.7-jdk15.jar;usetotest15.jar;tobetested15.jar org.testng.TestNG -testclass org.testng.tutorial.test.Introduction50 org.testng.tutorial.test.IntroductionJD

TestNG defaults to –target 1.5 and basically, TestNG generates the following suite internally to run the test classes.

<?xml version="1.0" encoding="ISO-8859-1"?>

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">

</classes>

</test>

</suite>

In this case, only the Java type annotations will be interpreted (IntroductionJD will be ignored). If we explicitly specify –target 1.4 option, TestNG generates the following suite internally to run the test classes.

<?xml version="1.0" encoding="ISO-8859-1"?>

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">

</classes>

</test>

</suite>

In this case, only the JavaDoc type annotations will be interpreted (Introduction50 will be ignored).

Comments:

The –target command line option basically sets the default annotation type to be used in test suites when none is explicitly specified.

Mixing JavaDoc and Java Type Annotations in the Same File

It is “not” possible to mix both types of annotations in the same file. Classes with mixed type annotations will be ignored if the test they are part specifies a “javadoc” type annotation. JavaDoc type annotations will be ignored for these classes if the test’s annotation type is “jdk15”.

Specifying packages in TestNG Suite

A TestNG suite is made up of tests. Tests are made up of classes. Up to now, we have specified classes explicitly. Classes may also be specified using a package name. For example:

<?xml version="1.0" encoding="ISO-8859-1"?>

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">

</packages>

</test>

</classes>

</test>

</suite>

In this case, classes of the specified packages will be scanned for annotations and included in the test. In this specific example, all classes of packages "org.testng.tutorial.test" and "org.testng.tutorial.test2" having Java type annotations will be added to the "packages" test. If classes "org.testng.tutorial.test.Example1", "org.testng.tutorial.test.Example2" and "org.testng.tutorial.test2.Example3" contain JavaDoc style annotations, they will be added to the "classes" test.

Comments:

If the specified classes or packages do not exist, an exception is thrown.

Classes not directly in the specified packages are ignored. For example, if package "org.testng.tutorial.test.sub" existed, its classes would not be included by the "<package name=”org.testng.tutorial.test"/> clause.

Filtering Methods in TestNG Suite Classes

When a <class> element is specified, it is possible to explicitly filter out test methods using the <methods> element. For the following examples, we will use the ThreeLoggingMethodsXX class. Here is this class:

ThreeLoggingMethodsJD.java:

package org.testng.tutorial.test;

/**

* <code>ThreeLoggingMethodsJD</code> is an example class that contains

* three test methods and logging statement to show test flow.

public class ThreeLoggingMethodsJD

{

static

{

// Visual information for the user

System.out.println("Loading ThreeLoggingMethodsJD.");

}

/**

* Constructs a <code>ThreeLoggingMethodsJD</code>. The constructor

* is specified only for the visual information it provides.

public ThreeLoggingMethodsJD()

{

// Visual information for the user

System.out.println(

"Constructing ThreeLoggingMethodsJD.");

}

/**

* Logs the test method name to the System.out

* @testng.test

public void theFirstTestMethodJD()

{

System.out.println(

"Calling ThreeLoggingMethodsJD.theFirstTestMethodJD.");

}

/**

* Logs the test method name to the System.out

* @testng.test

public void theSecondTestMethodJD()

{

System.out.println(

"Calling ThreeLoggingMethodsJD.theSecondTestMethodJD.");

}

/**

* Logs the test method name to the System.out

* @testng.test

public void theThirdTestMethodJD()

{

System.out.println(

"Calling ThreeLoggingMethodsJD.theThirdTestMethodJD.");

}

Here is a test suite illustrating filtering:

filteringJD.xml:

<?xml version="1.0" encoding="ISO-8859-1"?>

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">

</methods>

</class>

</classes>

</test>

</methods>

</class>

</classes>

</test>

</methods>

</class>

</classes>

</test>

</suite>

The method name is a regular expression. The “Include test” filters out all methods that do not match the include regular expression ("theF.*TestMethodJD"). The “Exclude test” filters out all methods that do not match the exclude regular expression ("theF.*TestMethodJD"). The “Include Exclude test” is the union of both previous tests.

The result of this suite execution is:

Loading ThreeLoggingMethodsJD.

Constructing ThreeLoggingMethodsJD.

Calling ThreeLoggingMethodsJD.theFirstTestMethodJD.

PASSED: theFirstTestMethodJD

===============================================

Include test

Tests run: 1, Failures: 0, Skips: 0

===============================================

Calling ThreeLoggingMethodsJD.theThirdTestMethodJD.

Calling ThreeLoggingMethodsJD.theSecondTestMethodJD.

PASSED: theThirdTestMethodJD

PASSED: theSecondTestMethodJD

===============================================

Exclude test

Tests run: 2, Failures: 0, Skips: 0

===============================================

Include Exclude test

Tests run: 0, Failures: 0, Skips: 0

===============================================

Include Exclude Example suite

Total tests run: 3, Failures: 0, Skips: 0

===============================================

Comments:

As shown in the “Include Exclude test”, exclusion preempts inclusion.

The include element does not allow inclusion of methods that do not have test annotations.

The test classes are instantiated once per test (not once per test method as JUnit).

The Ant Task Revisited

@TODO Describe the most useful features of the TestNG Ant task here.

Test Annotation Attributes and @ExpectedException

The following example introduces five attributes of the Test annotation and the @ExpectedException annotation.

The five Test annotation attributes covered here are:

1) alwaysRun

2) dependsOnMethods

3) description

4) enabled and

5) timeOut

Other Test attributes will be discussed later. The TestAttributes50.java file is used to show these:

TestAttributes50.java:

package org.testng.tutorial.test;

import org.testng.annotations.ExpectedExceptions;

import org.testng.annotations.Test;

/**

* <code>TestAttributes50</code> illustrates simple Test attributes.

public class TestAttributes50

{

/**

* Illustrates a throwing method.

@Test(description="throwingMethod description")

public void throwingMethod()

{

System.out.println("TestAttributes50.throwingMethod.");

throw new RuntimeException(

"See what happens when method throws.");

}

/**

* Illustrates an expecting throwing method.

@Test(description="expectingThrowingMethod description")

@ExpectedExceptions({RuntimeException.class})

public void expectingThrowingMethod()

{

System.out.println(

"TestAttributes50.exepectingThowingMethod.");

throw new RuntimeException(

"See what happens when method throws.");

}

/**

* Illustrates an expected exception that is not thrown.

@Test(description="nonThrowingMethod description")

@ExpectedExceptions({RuntimeException.class})

public void nonThrowingMethod()

{

System.out.println("TestAttributes50.nonThrowingMethod.");

}

/**

* Illustrates dependency on method 1 that fails.

@Test(description="dependsOnThrowingMethod description",

dependsOnMethods={"throwingMethod", "expectingThrowingMethod"})

public void dependsOnThrowingMethod()

{

System.out.println("TestAttributes50.dependsOnThowingMethod.");

}

/**

* Show various Test attributes.

* @throws InterruptedException

@Test(

description="Show various Test attributes",

timeOut=1000,

dependsOnMethods={"throwingMethod"},

alwaysRun=true

)

public void testTimeout() throws InterruptedException

{

Thread.sleep(1500);

// Note that thread will be interrupted and the

// following message will never be displayed.

System.out.println("TestAttributes50.testTimeout.");

}

/**

* disabledMethod

@Test(enabled = false, description="disabledMethod description")

public void disabledMethod()

{

System.out.println("TestAttributes50.disabledMethod.");

}

The description attribute is self explanatory.

The dependsOnMethods attribute specifies the methods that must be executed before the annotated method is executed. In this example, the throwingMethod() method will be executed before the dependsOnThrowingMethod() and testTimeout() methods. The expectingThrowingMethod() method will be executed before the dependsOnThrowingMethod() method.

When a test method depends on another, by default, the dependant method is skipped if the method it depends on fails. In this example, the dependsOnThrowingMethod() method will be skipped if the throwingMethod() or the expectingThrowingMethod() methods fail. The default behavior may be overridden by setting the “alwaysRun” attribute to true. For example, even though the testTimeout() method depends on the throwingMethod() method, the method will not be skipped if the throwingMethod() method fails.

The testTimeout() method fails because the tests takes longer to execute than the specified “timeout” attribute value. When a “timeout” attribute is specified and the timeout is exceeded, TestNG calls the Thread.interrupt() method on the thread executing the test method. This is why, in the previous example, the "TestAttributes50.testTimeout." string is never displayed. In this case the message is “java.lang.InterruptedException didn't finish within the time-out 1000”.

The disabled test method is not called because of the “enabled = false” attribute.

The @ExpectedExceptions specifies the exceptions that are expected to be thrown by the test method. If none of the expected exceptions are thrown, the test fails. In this example, the nonThrowingMethod() method fails because it does not throw a RuntimeException but the expectingThrowingMethod() method passes because it throws the expected RuntimeException.

All the Test attributes are described here:

http://testng.org/javadocs/org/testng/annotations/Test.html

The console output for the previous TestAttributes50.java is:

TestAttributes50.exepectingThowingMethod.

TestAttributes50.throwingMethod.

TestAttributes50.nonThrowingMethod.

PASSED: expectingThrowingMethod

expectingThrowingMethod description

FAILED: throwingMethod

throwingMethod description

java.lang.RuntimeException: See what happens when method throws.

at org.testng.tutorial.test.TestAttributes50.throwingMethod(TestAttributes50.java:19)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:585)

at org.testng.internal.MethodHelper.invokeMethod(MethodHelper.java:552)

at org.testng.internal.Invoker.invokeMethod(Invoker.java:407)

at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:778)

at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:105)

at org.testng.TestRunner.privateRun(TestRunner.java:682)

at org.testng.TestRunner.run(TestRunner.java:566)

at org.testng.SuiteRunner.privateRun(SuiteRunner.java:220)

at org.testng.SuiteRunner.run(SuiteRunner.java:146)

at org.testng.TestNG.createAndRunSuiteRunners(TestNG.java:736)

at org.testng.TestNG.runSuitesLocally(TestNG.java:699)

at org.testng.TestNG.run(TestNG.java:467)

at org.testng.TestNG.privateMain(TestNG.java:826)

at org.testng.TestNG.main(TestNG.java:768)

FAILED: testTimeout

Show the timeout attribute

org.testng.internal.thread.ThreadTimeoutException: Method public void org.testng.tutorial.test.TestAttributes50.testTimeout() throws java.lang.InterruptedEx

ception didn't finish within the time-out 1000

at org.testng.internal.Invoker.invokeMethod(Invoker.java:435)

at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:778)

at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:105)

at org.testng.TestRunner.privateRun(TestRunner.java:682)

at org.testng.TestRunner.run(TestRunner.java:566)

at org.testng.SuiteRunner.privateRun(SuiteRunner.java:220)

at org.testng.SuiteRunner.run(SuiteRunner.java:146)

at org.testng.TestNG.createAndRunSuiteRunners(TestNG.java:736)

at org.testng.TestNG.runSuitesLocally(TestNG.java:699)

at org.testng.TestNG.run(TestNG.java:467)

at org.testng.TestNG.privateMain(TestNG.java:826)

at org.testng.TestNG.main(TestNG.java:768)

FAILED: nonThrowingMethod

nonThrowingMethod description

org.testng.TestException:

Expected an exception in test method org.testng.tutorial.test.TestAttributes50.nonThrowingMethod()

at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:813)

at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:105)

at org.testng.TestRunner.privateRun(TestRunner.java:682)

at org.testng.TestRunner.run(TestRunner.java:566)

at org.testng.SuiteRunner.privateRun(SuiteRunner.java:220)

at org.testng.SuiteRunner.run(SuiteRunner.java:146)

at org.testng.TestNG.createAndRunSuiteRunners(TestNG.java:736)

at org.testng.TestNG.runSuitesLocally(TestNG.java:699)

at org.testng.TestNG.run(TestNG.java:467)

at org.testng.TestNG.privateMain(TestNG.java:826)

at org.testng.TestNG.main(TestNG.java:768)

SKIPPED: dependsOnThowingMethod

dependsOnThowingMethod description

===============================================

TestAttributes50 test

Tests run: 5, Failures: 3, Skips: 1

===============================================

TestAttributes suite

Total tests run: 5, Failures: 3, Skips: 1

===============================================

Comments:

If no method matches a dependant method name, an Exception is thrown.

The method name cannot be a regular expression.

The corresponding JavaDoc annotation for the testTimeout() method is:

/**

* @testng.test description="Show the timeout attribute"

* timeOut="1000"

* dependsOnMethods="throwingMethod"

* alwaysRun="true"

For the dependsOnMethods attribute (notice the space separated values):

/**

* @testng.test description="dependsOnThrowingMethod description"

* dependsOnMethods="throwingMethod expectingThrowingMethod"

For the ExpectedException annotation:

/**

* @testng.expected-exceptions value = "java.lang.RuntimeException"

If a cycle occurs in the dependsOnMethods graph, an Exception will be thrown.

Test Annotation on Classes

The @Test annotation can be used on class declaration. Specifying this annotation on a class is like annotating all class methods. The “groups” and “enabled” attributes may be specified at the class level. Other attributes are ignored.

The “groups” attribute value specified at the class @Test annotation will apply to all methods. The “groups” attribute value specified at the method level @Test annotation will be added to those specified at the class level. The “enabled” attribute value specified in the class level @Test annotation will be overridden by the one specified at the method level.

Configuration Methods: Setup Code for Classes and Methods

The @Test (or @testng.test) annotations presented above are use to describe test methods. Another annotation, @Configuration (or @testng.configuration), is used to describe the configuration of test methods and tests.

The following example shows how to use configurations to execute methods:

1) Before and after test methods.

2) Before and after the test class.

We first define a class that will act as a super class to setup the methods and the tests.

BeforeMethodClass50.java

package org.testng.tutorial.test;

import org.testng.annotations.Configuration;

/**

* <code>BeforeMethodClass50</code> illustrates the Configuration

* before/after TestClass/TestMethod

* <ul>

* <li>Demonstrates Configuration(beforeTestClass = true)</li>

* <li>Demonstrates Configuration(afterTestClass = true)</li>

* <li>Demonstrates Configuration(beforeTestMethod = true)</li>

* <li>Demonstrates Configuration(afterTestMethod = true)</li>

* </ul>

public class BeforeMethodClass50

{

static

{

// Visual information for the user

System.out.println("Loading BeforeMethodClass50.");

}

/**

* Constructs a <code>BeforeMethodClass50</code>. The constructor

* is specified only for the visual information it provides.

public BeforeMethodClass50()

{

// Visual information for the user

System.out.println(

"Constructing BeforeMethodClass50.");

}

/**

* Demonstrates Configuration(beforeTestClass = true)

@Configuration(beforeTestClass = true)

public static void beforeTestClass() {

System.out.println(

"Calling BeforeMethodClass50.beforeTestClass.");

}

/**

* Demonstrates Configuration(afterTestClass = true)

@Configuration(afterTestClass = true)

public void afterTestClass() {

System.out.println(

"Calling BeforeMethodClass50.afterTestClass.");

}

/**

* Demonstrates Configuration(beforeTestMethod = true)

@Configuration(beforeTestMethod = true)

public void beforeTestMethod() {

System.out.println(

"Calling BeforeMethodClass50.beforeTestMethod.");

}

/**

* Demonstrates Configuration(afterTestMethod = true)

@Configuration(afterTestMethod = true)

public void afterTestMethod() {

System.out.println(

"Calling BeforeMethodClass50.afterTestMethod.");

}

We then extend from this class

package org.testng.tutorial.test;

import org.testng.annotations.Test;

/**

* <code>ThreeLoggingMethodsBeforeMethodClass50</code> is identical to

* <code>ThreeLoggingMethods50</code> except it extends <code>

* BeforeMethodClass50</code>.

public class ThreeLoggingMethodsBeforeMethodClass50

extends BeforeMethodClass50

{

// . . .

// Same as ThreeLoggingMethods above except for messages

}

Running the test yields the following output on the console:

Loading BeforeMethodClassJD.

Loading ThreeLoggingMethodsBeforeMethodClassJD.

Constructing BeforeMethodClassJD.

Constructing ThreeLoggingMethodsBeforeMethodClassJD.

Calling BeforeMethodClassJD.beforeTestClass.

Calling BeforeMethodClassJD.beforeTestMethod.

Calling ThreeLoggingMethodsBeforeMethodClassJD.theThirdTestMethodJD.

Calling BeforeMethodClassJD.afterTestMethod.

Calling BeforeMethodClassJD.beforeTestMethod.

Calling ThreeLoggingMethodsBeforeMethodClassJD.theSecondTestMethodJD.

Calling BeforeMethodClassJD.afterTestMethod.

Calling BeforeMethodClassJD.beforeTestMethod.

Calling ThreeLoggingMethodsBeforeMethodClassJD.theFirstTestMethodJD.

Calling BeforeMethodClassJD.afterTestMethod.

Calling BeforeMethodClassJD.afterTestClass.

===============================================

Before After Method and Test JD test

Tests run: 3, Failures: 0, Skips: 0

===============================================

Before After Method and Test suite

Total tests run: 3, Failures: 0, Skips: 0

===============================================

The corresponding JavaDoc annotations are:

@testng.configuration beforeTestClass = "true"

@testng.configuration afterTestClass = "true"

@testng.configuration beforeTestMethod = "true"

@testng.configuration afterTestMethod = "true"

Comments:

If you are familiar with JUnit versions running on JDK 1.4, you will notice that having a setup method for test classes is a missing functionality in JUnit.

If a test suite contains a test with two classes having the same configuration superclass that sets up a “before/afterTestClass” method, the testClass configuration methods are called twice (as expected). For example, if ThreeLoggingMethodsBeforeMethodClass250 has superclass BeforeMethodClass50 and is added to the test, the output would be:

Loading BeforeMethodClass50.

Loading ThreeLoggingMethodsBeforeMethodClass50.

Constructing BeforeMethodClass50.

Constructing ThreeLoggingMethodsBeforeMethodClass50.

Loading ThreeLoggingMethodsBeforeMethodClass250.

Constructing BeforeMethodClass50.

Constructing ThreeLoggingMethodsBeforeMethodClass250.

Calling BeforeMethodClass50.beforeTestClass.

Calling BeforeMethodClass50.beforeTestMethod.

Calling ThreeLoggingMethodsBeforeMethodClass250.theThirdTestMethod50.

Calling BeforeMethodClass50.afterTestMethod.

Calling BeforeMethodClass50.beforeTestMethod.

Calling ThreeLoggingMethodsBeforeMethodClass250theFirstTestMethod50.

Calling BeforeMethodClass50.afterTestMethod.

Calling BeforeMethodClass50.beforeTestMethod.

Calling ThreeLoggingMethodsBeforeMethodClass250.theSecondTestMethod50.

Calling BeforeMethodClass50.afterTestMethod.

Calling BeforeMethodClass50.afterTestClass.

Calling BeforeMethodClass50.beforeTestClass.

Calling BeforeMethodClass50.beforeTestMethod.

Calling ThreeLoggingMethodsBeforeMethodClass50.theSecondTestMethod50.

Calling BeforeMethodClass50.afterTestMethod.

Calling BeforeMethodClass50.beforeTestMethod.

Calling ThreeLoggingMethodsBeforeMethodClass50.theThirdTestMethod50.

Calling BeforeMethodClass50.afterTestMethod.

Calling BeforeMethodClass50.beforeTestMethod.

Calling ThreeLoggingMethodsBeforeMethodClass50theFirstTestMethod50.

Calling BeforeMethodClass50.afterTestMethod.

Calling BeforeMethodClass50.afterTestClass.

===============================================

Before After Method and Test JD test

Tests run: 6, Failures: 0, Skips: 0

===============================================

Before After Method and Test suite

Total tests run: 6, Failures: 0, Skips: 0

===============================================

Configuration Methods: Setup Code for Tests and Suites

It is possible to have code execute before and/or after a Test or a Suite. In this example we will define a stand-alone class (BeforeTestAndSuiteJD) to hold these configuration methods and use it as a superclass for two test classes (InheritsBeforeTestAndSuiteJD and Inherits2BeforeTestAndSuiteJD).

BeforeTestAndSuiteJD.java:

package org.testng.tutorial.test;

/**

* <code>BeforeTestAndSuiteJD</code> illustrates the Configuration

* before/after Test/Suite.

* <ul>

* <li>Demonstrates *testng.configuration beforeTest = "true"</li>

* <li>Demonstrates *testng.configuration afterTest = "true"</li>

* <li>Demonstrates *testng.configuration beforeSuite = "true"</li>

* <li>Demonstrates *testng.configuration afterSuite = "true"</li>

* </ul>

public class BeforeTestAndSuiteJD

{

static

{

// Visual information for the user

System.out.println("Loading BeforeTestAndSuiteJD.");

}

/**

* Constructs a <code>BeforeTestAndSuiteJD</code>. The constructor

* is specified only for the visual information it provides.

public BeforeTestAndSuiteJD()

{

// Visual information for the user

System.out.println("Constructing BeforeTestAndSuiteJD.");

}

/**

* Demonstrates Configuration(beforeTest = true)

* @testng.configuration beforeTest = "true"

public static void beforeTest() {

System.out.println("Calling BeforeTestAndSuiteJD.beforeTest.");

}

/**

* Demonstrates Configuration(afterTest = true)

* @testng.configuration afterTest = "true"

public void afterTest() {

System.out.println("Calling BeforeTestAndSuiteJD.afterTest.");

}

/**

* Demonstrates Configuration(beforeSuite = true)

* @testng.configuration beforeSuite = "true"

public void beforeSuite() {

System.out.println(

"Calling BeforeTestAndSuiteJD.beforeSuite.");

}

/**

* Demonstrates Configuration(afterSuite = true)

* @testng.configuration afterSuite = "true"

public void afterSuite() {

System.out.println("Calling BeforeTestAndSuiteJD.afterSuite.");

}

We then define two test classes extending from this class ( InheritsBeforeTestAndSuiteJD and Inherits2BeforeTestAndSuiteJD):

InheritsBeforeTestAndSuiteJD.java :

/**

* <code>InheritsBeforeTestAndSuiteJD</code> illustrates inheriting

* before/after Test/Suite Configuration methods.

* <ul>

* <li>Demonstrates *testng.configuration beforeTestClass = "true"</li>

* <li>Demonstrates *testng.configuration afterTestClass = "true"</li>

* <li>Demonstrates *testng.configuration beforeTestMethod ="true"</li>

* <li>Demonstrates *testng.configuration afterTestMethod = "true"</li>

* </ul>

public class InheritsBeforeTestAndSuiteJD extends BeforeTestAndSuiteJD

{

static

{

// Visual information for the user

System.out.println("Loading InheritsBeforeTestAndSuiteJD.");

}

/**

* Constructs a <code>InheritsBeforeTestAndSuiteJD</code>. The

* constructor is specified only for the visual information it

* provides.

public InheritsBeforeTestAndSuiteJD()

{

// Visual information for the user

System.out.println(

"Constructing InheritsBeforeTestAndSuiteJD.");

}

// before/after TestMethod/TestClass here

// . . .

// Three test methods here

// . . .

}

Running the following suite:

beforeAfterSuiteJD.xml:

<?xml version="1.0" encoding="ISO-8859-1"?>

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">

</classes>

</test>

</suite>

Yields:

Loading BeforeTestAndSuiteJD.

Loading InheritsBeforeTestAndSuiteJD.

Constructing BeforeTestAndSuiteJD.

Constructing InheritsBeforeTestAndSuiteJD.

Loading Inherits2BeforeTestAndSuiteJD.

Constructing BeforeTestAndSuiteJD.

Constructing Inherits2BeforeTestAndSuiteJD.

Calling BeforeTestAndSuiteJD.beforeSuite.

Calling BeforeTestAndSuiteJD.beforeTest.

// . . . Calls to Inherits2BeforeTestAndSuiteJD test methods

// . . . Calls to InheritsBeforeTestAndSuiteJD test methods

Calling BeforeTestAndSuiteJD.afterTest.

Calling BeforeTestAndSuiteJD.afterSuite.

Comments:

Both test classes use the same superclass so, as expected, the before/after Test/Suite methods are only called once. Having a common superclass perform the configuration allows the test classes to be executed individually (using Eclipse plugin for example) or grouped with other test classes sharing the same configuration (without performing setup for each test class).

*The feature presented above does not work correctly with TestNG version 4.7 and earlier. When using inheritance, the before/after suite method is called twice in the previous example. This has been fixed and committed in the CVS repository.

The corresponding JDK5 annotations are:

@Configuration(beforeTest = true)

@Configuration(afterTest = true)

@Configuration(beforeSuite = true)

@Configuration(afterSuite = true)

The following illustrates a test suite with two tests. Running this suite:

beforeAfterTest50.xml:

<?xml version="1.0" encoding="ISO-8859-1"?>

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">

</classes>

</test>

</classes>

</test>

</suite>

Yields:

Loading BeforeTestAndSuite50.

Loading InheritsBeforeTestAndSuite50.

Constructing BeforeTestAndSuite50.

Constructing InheritsBeforeTestAndSuite50.

Loading Inherits2BeforeTestAndSuite50.

Constructing BeforeTestAndSuite50.

Constructing Inherits2BeforeTestAndSuite50.

Calling BeforeTestAndSuite50.beforeSuite.

Calling BeforeTestAndSuite50.beforeTest.

// . . . Calls to Inherits2BeforeTestAndSuite50 test methods

Calling BeforeTestAndSuite50.afterTest.

Calling BeforeTestAndSuite50.beforeTest.

// . . . Calls to InheritsBeforeTestAndSuite50 test methods

Calling BeforeTestAndSuite50.afterTest.

Calling BeforeTestAndSuite50.afterSuite.

*The feature presented above does not work correctly with TestNG version 4.7 and earlier. When using inheritance, the before/after test method is called twice in the previous example. This has been fixed and committed in the CVS repository.

Test and Suite Configuration using a non superclass

In the previous example, the test and suite configuration methods were placed in the superclass. If the stand-alone configuration class is explicitly added to the suite, inheritance is not needed. In this example, the test methods and the test and suite setup are in different classes:

<?xml version="1.0" encoding="ISO-8859-1"?>

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">

</classes>

</test>

</classes>

</test>

</suite>

This setup allows for the same test methods to be executed in a context that is setup outside the testing class. Unfortunately, the test class is not a stand alone test and a suite must be defined to run it. (i.e. it cannot be run as a “Run As TestNG Test” using the Eclipse plugin)

Running Both JavaDoc and JDK5 Test and Suite Configuration

As a final example, here is a mix of both JavaDoc and JDK5 annotations mixed in the same suite:

<?xml version="1.0" encoding="ISO-8859-1"?>

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">

</classes>

</test>

</classes>

</test>

</suite>

Loading BeforeTestAndSuite50.

Loading InheritsBeforeTestAndSuite50.

Constructing BeforeTestAndSuite50.

Constructing InheritsBeforeTestAndSuite50.

Loading Inherits2BeforeTestAndSuite50.

Constructing BeforeTestAndSuite50.

Constructing Inherits2BeforeTestAndSuite50.

Loading BeforeTestAndSuiteJD.

Loading Inherits2BeforeTestAndSuiteJD.

Constructing BeforeTestAndSuiteJD.

Constructing Inherits2BeforeTestAndSuiteJD.

Loading InheritsBeforeTestAndSuiteJD.

Constructing BeforeTestAndSuiteJD.

Constructing InheritsBeforeTestAndSuiteJD.

Calling BeforeTestAndSuite50.beforeSuite.

Calling BeforeTestAndSuiteJD.beforeSuite.

Calling BeforeTestAndSuite50.beforeTest.

Calling Inherits2BeforeTestAndSuite50.theThirdTestMethod.

Calling Inherits2BeforeTestAndSuite50.theSecondTestMethod.

Calling Inherits2BeforeTestAndSuite50.theFirstTestMethod.

Calling InheritsBeforeTestAndSuite50.theSecondTestMethod.

Calling InheritsBeforeTestAndSuite50.theFirstTestMethod.

Calling InheritsBeforeTestAndSuite50.theThirdTestMethod.

Calling BeforeTestAndSuite50.afterTest.

Calling BeforeTestAndSuiteJD.beforeTest.

Calling Inherits2BeforeTestAndSuiteJD.beforeTestClass.

Calling Inherits2BeforeTestAndSuiteJD.beforeTestMethod.

Calling Inherits2BeforeTestAndSuiteJD.theFirstTestMethodJD.

Calling Inherits2BeforeTestAndSuiteJD.afterTestMethod.

Calling Inherits2BeforeTestAndSuiteJD.beforeTestMethod.

Calling Inherits2BeforeTestAndSuiteJD.theSecondTestMethodJD.

Calling Inherits2BeforeTestAndSuiteJD.afterTestMethod.

Calling Inherits2BeforeTestAndSuiteJD.beforeTestMethod.

Calling Inherits2BeforeTestAndSuiteJD.theThirdTestMethodJD.

Calling Inherits2BeforeTestAndSuiteJD.afterTestMethod.

Calling Inherits2BeforeTestAndSuiteJD.afterTestClass.

Calling InheritsBeforeTestAndSuiteJD.beforeTestClass.

Calling InheritsBeforeTestAndSuiteJD.beforeTestMethod.

Calling InheritsBeforeTestAndSuiteJD.theThirdTestMethod.

Calling InheritsBeforeTestAndSuiteJD.afterTestMethod.

Calling InheritsBeforeTestAndSuiteJD.beforeTestMethod.

Calling InheritsBeforeTestAndSuiteJD.theSecondTestMethod.

Calling InheritsBeforeTestAndSuiteJD.afterTestMethod.

Calling InheritsBeforeTestAndSuiteJD.beforeTestMethod.

Calling InheritsBeforeTestAndSuiteJD.theFirstTestMethod.

Calling InheritsBeforeTestAndSuiteJD.afterTestMethod.

Calling InheritsBeforeTestAndSuiteJD.afterTestClass.

Calling BeforeTestAndSuiteJD.afterTest.

Calling BeforeTestAndSuite50.afterSuite.

Calling BeforeTestAndSuiteJD.afterSuite.

To be continued …