2011-04-01

Testing a Plug-in - Where to put the tests

This post is part of my "Testing a Plug-in" series.

Say you have decided to test your plug-ins, the first question of cause is, where exactly to put the tests. As it comes, there are a number of possibilities each with its own advantages and disadvantages.

In the following, I assume we are using JUnit 4 as this does not put any requirements on the super-classes. Otherwise, with JUnit 3, our options are a little more limited.

The tests can be put inline in the application plug-ins
  • Pros:
    • This is very easy to manage and understand. Also it is very easy to refactor the code, as you do not have to remember to have any specific test plug-ins or fragments in your workspace when you refactor your code - though it is always a good idea!
    • Your tests will have access to everything - you can even put the test in the same class as the methods to be tested.
  • Cons:
    • The tests will be in the final product, though it is possible to ge the Java compiler to leave out the test methods based on a special annotation. If you remove them from the plug-in during the release phase of the project, you have the additional problem, that what you test is not exactly your release...
    • If your tests requires any specific extensions in plugin.xml, then these will be harder to remove from the plug-in during compilation. This can be needed if you deckare any additional extension points in your target plug-ins and need to test these.
The tests can be put into a separate plug-ins
  • Pros:
    • The tests are separate from the product, so the cons of the previous option are not present here.
    • The test plug-in will have the same access to the exported Java packages as any "ordinary" consumer plug-in.
  • Cons:
    • Because of OSGi and the access rules it impose on bundles, you have only access to the exported Java packages. Of cause you can use x-friends to allow a specific visibility for specific package and friendly bundles, but who wants to do that if it can be avoided.
The tests can be put in an attached fragments
  • Pros:
    • The tests have access to everything as a fragment effectively "runs" inside the host plug-in.
    • Like the previous option, the tests are separate from the product.
    • You can even test new extension points and other other stuff that requires additions to plugin.xml...
  • Cons:
    • A fragment is a slightly different "thing" than a plug-in, and many developers shy away from them for this reason.
    • Some build tools have problems running tests that are put inside fragments - though these problems seems to be gone in the 
I usually prefer a mix of fragments and plug-ins for my tests. The fragments are used for all tests of the internals of the target plug-in and the plug-ins are used for the needed black-box tests for the same target plug-ins.

One area where the plug-in approach is needed, is in order to test that the needed interfaces from the extension points are indeed exported and fully available. One problem you might otherwise see, is that super-classes are internal to the target plug-in which can give some very hard to understand error messages.

5 comments:

David Carver said...

If using maven/tycho you can still do option one and not include the tests in the final product. Put the tests in a src/test/java folder.

Anonymous said...

@David Yep. But AFAIK you cannot have the same Java package in multiple source folders. So you cannot access the package protected fields of the view in this case. Or... am I wrong here?

Thomas Hallgren said...

I think you can have the same package in multiple source folders and even if you couldn't, you can always use an:

exclude.<library> = **/*Test.class

or similar in your build.properties. But we don't do that. Instead, our approach is to use separate test bundles. We want to keep production code and test code separate for several reasons:

Aside from the pros that you've listed, one prominent reason is that since we never publish the test, we have much greater freedom when it comes to licensing. Nothing stops us from using GPL'ed software in the test bundles.

Our tests might introduce new bundle dependencies and they may also have a bundle activator (not possible for a fragment). The activator receives a context which makes it easy to find test data delivered as part of the bundle and other OSGi locations related to the current execution.

The tests form a bundle hierarchy of their own which makes it very convenient to use the standard PDE mechanisms to pull things together. The top of the test hierarchy gets automatic access to other test bundles and can create tests suites that includes them.

I don't see access restrictions as a problem really. If needed, why not just export the package as internal and let the test override the "discouraged usage" warning. Alternatively, use a friends declaration as you suggest.

Anonymous said...

@Thomas I agree completely with you, expect that I prefer to avoid x-friend and company....

When using fragments it is correct you don't have a activator. But I usually have a dependency on a separate common test utility bundle, which does have an activator. I'll introduce this test bundle in the next blog entry.

Unknown said...

Our structure related to test-plugins depends on the type of the tests.

If they are unit test (probably the Tonny's internal test), are in the same plugin of the logic, but with different folder and output (src/test/java). They aren't part of the product.

If they are Integration test, that are not related to only one plugin, are placed in a separate test plugin.

With this structure we have separate code coverage reports.

Rgds,
Juan.