Note: As of 2023-04-03, this hasn't been updated since 2013...

Overview

In this tutorial, you will learn how to write automated tests for a particular Tiki site. For example, maybe you are a Tiki consultant who has created a highly customized Tiki site for one of your customers, and you want to make sure that this customer's critical Use Cases will not be broken by eventual upgrades. Or, maybe you are a member of the Tiki Testing community and you want to write tests against one of the tiki.org sites (or rather, copies of them) to make sure that certain generic feature configurations will not be broken by changes in trunk.

In the tutorial, we will be writing a test against doc.tiki.org, but the procedure would be exactly the same if you were writing a test against a customer's site.

The specific test that we will write will check that if you try to login with a bad user name, then Tiki will reject it.

Components used for creating the tests

To create and run the tests, we will be using a number of building blocks. Sections below will provide more details on how to install, configure and use those basic components, but for now, here is a brief description of each of them.

Selenium 2 (WebDriver)

Selenium is a framework that allows you to programmatically control a browser and make it do the steps of a test scenario. There are two versions of it: Selenium 1 (also known as Selenium Remote Control or Selenium RC), and Selenium 2 (also known as Selenium WebDriver). For Tiki testing, we use Selenium 2 (WebDriver), because it is more modern, and more robust to variations between browsers.

SeleniumBuilder

The easiest way to quickly write a test is to record it using a browser plugin. For Tiki testing, we use SeleniumBuilder. Note that there is another popular recorder called Selenium IDE, but it is only compatible with Selenium 1, so we can't use it.

Java and JUnit

Ideally, our tests would be saved as PHP scripts. Unfortunately, as of this writing (May 2013), PHP support is abysmal in both Selenium WebDriver and SeleniumBuilder. Maybe the situation will eventually get better, but for now, we decided to go with saving the scripts as Java/JUnit classes. This introduces yet another language in the Tiki ecosystem, but fortunately, this will be limited to the tests and will not pollute the actual production code.

SauceLabs

SauceLabs is a cloud service that allows you to test web sites using any version of any browser, running in any version of any OS. For Tiki testing, we will be using that service so we can test that Tiki works well with different browser-OS combinations.

Optional (but very useful): Firebug and FireFinder

Although not absolutely necessary, those two browser plugins can help you tremendously, when you need to write locators for elements that your tests want to manipulate or inspect. Firebug allows you to point at an element in the browser and see its HTML implementation. FireFinder allows you to right click on an element, and see a list of locators that can be used to access it.

Recording a test with SeleniumBuilder

The easiest way to create a draft for a test, is to record it with SeleniumBuilder. This section provides details on how to install and use it.

Install SeleniumBuilder

To install SeleniumBuilder:

• Go to https://saucelabs.com/builder
• Click on Download
• Authorize installation of the plugin

Create a clone of the site to be tested

Before you start recording tests on your site with SeleniumBuilder, you might want to create a clone of that site. That's because you don't want your tests to screw up an actual site that is being used in production. The procedure for creating such a clone is out of the scope of the present tutorial. In any case, in the present tutorial we will be writing our tests against the actual tiki.org site, because it would be too hard to create a clone of it, and because the test we will create will not write to the site's database anyway.

Think about the test you are about to record

Before you jump in and start recording, take a moment to think about what you want to test, and the steps you will be taking through the Tiki UI to implement that test.

The reason for this is that you can only record a complete test in one go. If you need to make adjustments to the test after that initial recording, you will have to make them manually, which tends to be much harder and slower than recording. So it's a good idea to try and get as much of the test right in the initial recording.

In our case, what we want to do is to login using a bad user name and password, and assert that: a) there is an error message and b) we are NOT logged in.

Record the test

You are now ready to record a test against doc.tiki.org. The test will check that the site does not allow logging in with non-valid credentials.

• In Firefox: Tool > Launch Selenium Builder

• In Start recording at, enter http://doc.tiki.org/
• Hit the Selenium 2 button
• This will open the URL in a browser tab, and that tab will be highlighted, signalling that this is where you are recording the test. The SeleniumBuilder window will now look like this:

• As you can see, there is already an instruction in the list, which says "get http://doc.tiki.org/Documentation".
• Next, you want to carry out the steps in the test
  • Click on the Login button on doc.tiki.org.
  • Type 'badusername' and 'badpassword' into the login dialog, and hit the Log in button.
• At that point, your SeleniumBuilder window looks something like this:

• The next step is to add an assertion that checks that error message "XMLRPC Error: 101 - Invalid username or password" appears on the screen.
  • Normally, you should be able to click on the Record a verification button and then highlight the text of the error, but that doesn't seem to always work, and it doesn't in this case.
  • So instead, we will add the verification manually...
  • Put the mouse over Step 5 (clickElement name:login).
  • You will see a menu appear on the left.

• Click on new step below

• You will see a new step that says "clickElement id:". This is the default type of step. You will have to change it to be an assertion.
• Put the mouse over that new step, and click on edit type.
• In the lefthand side, click on Assertion, then on the righthand side, click on assertTextPresent, then on OK.
• Now you need to change the text that the assertion is checking for. Put mouse over Step 6, which now says: "assertTextPresent text", and click on edit text. Enter "XMLRPC Error: 101 - Invalid username or password" and hit OK.
• We also want to make sure that we are NOT logged in.
  • The easiest way to do this is to make sure that the Login button is still present (if we were logged in, the button would now say Logout
  • Just click on the Login button.
  • There is now a Step 7 that says "clickElement link text: Log in >>"
• This is probably a good time to save the test.
  • File > Save > Selenium Builder
  • Save the test as DocTikiBadLoginGestRejected.json

Run and debug the test

Next, you will use SeleniumBuilder to run the test you just recorded and debug it.

• Run > Run test locally
• You will see that the first step turns green fairly rapidly.
• However the test seems to get stuck on the Step 2, "clickElement link text: Log in >>".
• In fact, if you wait long enough, you will see that this step turns red with error message

Unable to locate element: {"method":"link text", "selector":"Log in >>"}

• This is a fairly common snag when recording tests. Basically, when you clicked on the Login button during recording, SeleniumBuilder decided that the best locator (i.e. a method for specifying which button we want to click on), was to use the link text. But as it turns out, Selenium is unable to find that button using that locator (for some reason or another).
• The way to fix this is to change the accessor for that step.
  • Put the mouse over step 2, then click on edit locator.
  • Under Suggested alternatives:, you will find a list of other locators that could have been used to locate the Login button. Click on the one that says css selector:a.login_link. Then click on OK.
  • While you are at it, you might as well change the locator for Step 7, which checks that the Login button is still there after the failed login attempt.
• Run the script again locally (hit the Clear Results button first).
• This time, the script runs to completion, and all the steps are highlighted in green.
• Congratulations! You have just created and run your very first test!
• Make sure you save the modified version as a Selenium Builder (json) test, as above.

Package and run the test as Java/JUnit

Save the test as a Java/JUnit format

In the previous section, you saved the test as a Selenium Builder Test. This is a json format that can only be loaded and run from inside SeleniumBuilder. But we need more flexibility than that. For example, we want to be able to run the tests on saucelabs.

For that reason, you will have to convert the script to a Java/JUnit format (as mentioned above, it is not practical as of this writing to save the tests in a PHP format).

• Open the script in SeleniumBuilder
  • Tools > Launch Selenium Builder > Open a script or suite
  • Browse your way to the script DocTikiBadLoginGetsRejected.json and click on it.
• Save the script in Java/JUnit format
  • File > Save > Java/JUnit
  • Name the file DocTikiBadLoginGestRejected.java

Rewrite all verification code into assertions

If you look in the Java code that was generated by SeleniumBuilder, you will something along the lines of:

if (!wd.findElement(By.tagName("html")).getText().contains("XMLRPC Error: 101 - Invalid username or password")) {
            System.out.println("verifyTextPresent failed");
        }

This, is the code that verifies that there is an error message for the bad credentials. As it turns out, it's very badly written JUnit code. Indeed, this statement does not raise an exception, so when you run the test, JUnit won't even know that the test failed!!!

So you should rewrite this code (and probably all verification code) to use one of the assert* methods of JUnit. In the present case, it would be:

assertTrue("The server should have refused the connection but it didn't.",
                     wd.findElement(By.tagName("html")).getText().contains("XMLRPC Error: 101 - Invalid username or password"));

Document the Java/JUnit file

At this point, it's a good idea to put comments in the Java file that was generated. Do it NOW while you remember what you were trying to test. If you look at the code in file DocTikiBadLoginGestRejected.java, you will see a test method that looks like this:

@Test
    public void DocTikiBadLoginGetsRejected() {
        wd.get("http://doc.tiki.org/Documentation");
        wd.findElement(By.cssSelector("a.login_link")).click();
        wd.findElement(By.id("login-user_tiki-login")).click();
        wd.findElement(By.id("login-user_tiki-login")).clear();
        wd.findElement(By.id("login-user_tiki-login")).sendKeys("badusername");
        wd.findElement(By.id("login-pass_tiki-login")).click();
        wd.findElement(By.id("login-pass_tiki-login")).clear();
        wd.findElement(By.id("login-pass_tiki-login")).sendKeys("badpassword");
        wd.findElement(By.name("login")).click();
        assertTrue(wd.findElement(By.tagName("html")).getText().contains("XMLRPC Error: 101 - Invalid username or password"));
        wd.findElement(By.cssSelector("a.login_link")).click();
    }

If you, or even worse, someone else, looks at this method in 6 months time, will they be able to figure out what you were trying to test exactly? Possibly not. This is a very common way that teams fail with this kind of testing. At first, everything is honky dory, but eventually, they end up with hundreds of tests, some of which are failing, and nobody can tell what the tests were trying to test in the first place.

So take a minute or two to document the intent of this test. For example:

@Test
    public void DocTikiBadLoginGetsRejected() {

        //
        // Enter a user and password that do not exist
        //
        wd.get("http://doc.tiki.org/Documentation");
        wd.findElement(By.cssSelector("a.login_link")).click();
        wd.findElement(By.id("login-user_tiki-login")).click();
        wd.findElement(By.id("login-user_tiki-login")).clear();
        wd.findElement(By.id("login-user_tiki-login")).sendKeys("badusername");
        wd.findElement(By.id("login-pass_tiki-login")).click();
        wd.findElement(By.id("login-pass_tiki-login")).clear();
        wd.findElement(By.id("login-pass_tiki-login")).sendKeys("badpassword");
        wd.findElement(By.name("login")).click();

        //
        // Make sure that the login request was rejected
        //
        assertTrue("The server should have refused the connection but it didn't.",
                    wd.findElement(By.tagName("html")).getText().contains("XMLRPC Error: 101 - Invalid username or password"));
        
        //
        // Make sure we are not logged in. The easiest way to do this it to check that the
        // Login button is still available.
        //
        wd.findElement(By.cssSelector("a.login_link")).click();
    }

Notice how the test is now documented in three different ways.

• The long and very explicit name of the test method DocTikiBadLoginGetsRejected provides a pretty good idea of what it is trying to test.
• There are comments that explain what each section of the test do.
• Notice also the informative error message that was added as the first argument to assertTrue(). That way, if that step fail, saucelab will provide that message instead of the more generic "failed assertion".

Get a saucelab account

Parts of the instructions that follow assume that you have a SauceLabs account (saucelabs.com). If you don't have one, now is the time to register. The account is free for the first month or so.

If you are not interested in running tests with SauceLabs, you should still be able to follow the instructions below (with modifications), and run your tests locally.

Note that Tiki.org has access to an account that SauceLabs provides us for free, because we are an OpenSource project. If you are writing generic tests that could be modified and run by any member of the Tiki community, then you might be able to use that account. Just ask Pascal St-Jean or Alain Désilets.

Install JUnit

To run the Java/JUnit test you saved above, you will need to install JUnit (we're assuming that you already have Java installed).

Because we ultimately want to run the tests with saucelabs, we will be using the installation instructions that they provide on this page:

https://saucelabs.com/java

but using different values for the various properties of the Maven project.

• Create a directory TikiTestsTutorial, and go to it.
• Execute this command:

mvn archetype:generate -DarchetypeRepository=http://repository-saucelabs.forge.cloudbees.com/release -DarchetypeGroupId=com.saucelabs -DarchetypeArtifactId=quickstart-webdriver-junit -DarchetypeVersion=1.0.17 -DsauceUserName=yourSauceID -DsauceAccessKey=yourSauceKey

• Where yourSauceID and sauceAccessKey can be found by logging into your saucelabs account. If you don't have a sauce account, simply omit the -DsauceUserName and -DsauceAccessKey arguments.
• The command will ask you to specify a number of properties. Answer as follows:
  • groupId: Answer with 'org.tiki'.
  • artifactId: Answer with 'tikitests'.
  • version: Just hit Enter.
  • package: Just hit Enter (will default to the value you specified for groupId).
  • sauceAccessKey and sauceUserName: If you specified these values with the -D switch at the command line, just hit Enter (will default to those values). If you do not have a sauce account, then enter any non-empty string for either value.
• You should now have a directory TikiTestsTutorial/tikitests
• If you installed using some sauce credentials, you can then check that the installation worked by running the sample tests (if you don't have a sauce account, just skip this test).

cd tikitests
mvn -Dtest=* test

• Login to your saucelab account and you should see that there is a list of tests, some of which are running, some of which are completed.
• After a minute or two the tests should have completed, and the command line will report that N tests were run, with 0 error and failures (exact number of tests may depend on the version of the tutorial you installed).
• Note that the instructions contained in this section end up installing several sample tests under TikiTestsTutorial/tikitests/src/test/java/org/tiki. Once you are 100% sure that everything is working fine, and that you are able to create your own tests and add them there, you may delete the sample tests.

Run the JUnit test locally

The next step is to run the tests locally. Again, instructions for doing this are on this page:

https://saucelabs.com/java
More specifically:

• Move the file DocTikiBadLoginGestRejected.java to directory TikiTestsTutorial/tikitests/src/test/java/org/tiki.
• In a terminal window, go to the TikiTestsTutorial/tikitests directory and issue the following command:

mvn -Dtest=* test

The first time you run a test this way, it may take some time, as the program will upload a bunch of java binaries to the saucelab server.

Note that the mvn command will run all the test cases that can be found anywhere under the test directory. If you only want to run a subset of the the tests, just do:

mvn -Dtest=TestCaseName#testMethodName test

This commnand will only run the one test method testMethodName from the one test case class called TestCaseName. You can even use wildcards in the names of test cases and test methods, as per this page:

http://maven.apache.org/surefire/maven-surefire-plugin/examples/single-test.html

mvn -Dtest=TestCaseName test

mvn -Dtest=TestCaseName#testMethodName test

Running the tests on SauceLabs

The previous section showed you how to run tests locally on your machine. It's probably a good idea to do that first, because it will make it easier to debug your test. But eventually, you will want to run your tests on saucelabs.com, because it allows you to test on any combination of OS-Browser.

To do this, you will have to embed your test method inside a different Java class, which is designed specifically for running tests on saucelabs.

• Recreate the new DocTikiBadLoginGetsRejected class, using TemplateSauceTestClass.java as a model
  • Make a backup of file DocTikiBadLoginGetsRejected.java as DocTikiBadLoginGetsRejected.java.bak.
  • Download the following code template TemplateSauceTestClass.java, and save it as TikiTestsTutorial/tikitests/src/test/java/org/tiki/DocTikiBadLoginGetsRejected.java .
  • Open DocTikiBadLoginGetsRejected.java and rename the class from TemplateSauceTestClass to DocTikiBadLoginGetsRejected.
  • If that wasn't done previously, download the TikiRootWebDriverTest.java file, and put it in the same directory as the tests you created.

• Next, you must specify your Saucelabs credentials if that is not done already
  • Open the file ~/.sauce-ondemand, and enter your credentials in the following format:

username=yourSauceUserName
key=yourSauceKey

• You can find your sauce key by logging into saucelabs.com and going to https://saucelabs.com/account/. It should be in the lower left corner.

Once that is done, you can run your test on saucelabs, using the same command as before

mvn -Dtest=DocTikiBadLoginGetsRejected test

This time though, the command won't open a browser window locally on your machine. Instead, it will upload the tests to saucelabs and run them from there. You can see the test being run on sauce, by logging into your saucelabs account.

If there were any errors, you will see them printed out on the command line. The failed tests will also appear on your saucelabs.com account, with a "failed" label. If you need more visual clues as to what happened, you can get a video of what saucelabs "saw" in the browser, when it ran the test. Alternatively, you can also re-run the test locally (see next section).

Switching between running tests locally or on sauce

In the previous step, we showed you how to rewrite your test so you could run it on sauce.

But sometimes when you are debugging a test, it's easier to see it running locally, instead of remotely on sauce. Fortunately, you can easily switch from one mode to the other by modifying one line in the class TikiRootWebDriverTest:

private boolean runTestsOnSauceLab = false;

Testing using different OS-Browser combinations

By default, when testing on sauce, the TikiRootWebDriverTest class uses firefox on unix. If you want to test on a different OS-Browser combination, simply edit the TikiRootWebDriverTest class and change the values of the following attributes:

• browserName: firefox, chrome or safari.
• browserVersion: admissible values depend on the browser specified above.
• platformName: UNIX, XP, MAC.

Maintaining your tests

OK, so now you know how to record a test, save it as a Java/JUnit test case, and run it on saucelabs or locally.

Eventually, you will have dozens, maybe hundreds of such tests, and it may take several hours to run the whole thing. At that point, you will face certain challenges in terms of keeping the tests organized. Fortunately, you have the power of the JUnit framework at your disposal.

Below, we will illustrate some best practices that you may want to use in maintaining your test suite.

Keep similar tests together

Say you have a bunch of tests that exercise different scenarios related to the login/logout procedure. It would be nice to keep those tests together in a same place. That way, when you want to add another login/logout test, you know where to put it. Also, if you want to check that login/logout works, you can execute only those tests without having to wait hours for the whole suite to execute.

You can do this with JUnit, simply by putting all the login/logout tests in a same TestCase class. Let us illustrate this by showing you how you could create a second test and add it to the same TestCase class as the one that you generated earlier in the tutorial.

The second test will check that if you provide a valid user and login, then the server will accept the request and log you in. Record this second test using SeleniumBuilder as described previously in this tutorial. At the end of the recording session, you should have something like this:

where someValidUsername and someValidPassword will correspond to values that are valid credentials on the clone of the site you are testing.

Run the test locally inside of SeleniumBuilder to make sure it's OK, and then, save it as a Java/JUnit class DocTikiGoodLoginGetsAccepted.java.

At this point, you have two tests, each located in their own class file. To keep those two tests together, merge the two classes into one.

• Refactor the class DocTikiBadLoginGetsRejected to rename it to DocTikiLoginLogout
  • Rename file DocTikiBadLoginGestRejected.java to DocTikiLoginLogout.java.
  • Open that file, and replace this line:

public class DocTikiBadLoginGetsRejected

public class DocTikiLoginLogout

• Then move the test method from the DocTikiGoodLoginGetsAccepted class to the DocTikiLoginLogout
  • Open file DocTikiGoodLoginGetsAccepted.java.
  • Copy the implementation of method DocTikiGoodLoginGetsAccepted (including the Test tag above it) to the file DocTikiLoginLogout.java.
  • Save DocTikiLoginLogout.java
  • Delete the original DocTikiGoodLoginGetsAccepted.java file.

That's it. Now you have a single class DocTikiLoginLogout that implements the two tests we currently have for login/logout.

You can now run all of the login/logout tests by doing

mvn -Dtest=DocTikiLoginLogout test

Avoid code duplication between tests

If you look at the code of the two tests, you will see that they use a very similar looking snippet of code for trying to login:

wd.findElement(By.cssSelector("a.login_link")).click();
         wd.findElement(By.id("login-user_tiki-login")).click();
         wd.findElement(By.id("login-user_tiki-login")).clear();
         wd.findElement(By.id("login-user_tiki-login")).sendKeys("badusername");
         wd.findElement(By.id("login-pass_tiki-login")).click();
         wd.findElement(By.id("login-pass_tiki-login")).clear();
         wd.findElement(By.id("login-pass_tiki-login")).sendKeys("badpassword");
         wd.findElement(By.name("login")).click();

This kind of code redundancy is not a good thing. For example, say the ID of the username field changes, then you will have to change it in potentially dozens of tests.

It's much better to refactor this into a method

public void loginAsUser(String userid, String password) 
{
    wd.findElement(By.cssSelector("a.login_link")).click();
    wd.findElement(By.id("login-user_tiki-login")).click();
    wd.findElement(By.id("login-user_tiki-login")).clear();
    wd.findElement(By.id("login-user_tiki-login")).sendKeys(userid);
    wd.findElement(By.id("login-pass_tiki-login")).click();
    wd.findElement(By.id("login-pass_tiki-login")).clear();
    wd.findElement(By.id("login-pass_tiki-login")).sendKeys(password);
    wd.findElement(By.name("login")).click();
}

Then, the test method DocTikiBadLoginGetsRejected becomes:

@Test
    public void DocTikiBadLoginGetsRejected() {

         wd.get("https://doc.tiki.org/Documentation");

         //
         // Enter a user and password that do not exist
         //
         loginAsUser("badusername", "badpassword");

         //
         // Make sure that the login request was rejected
         //
         assertTrue("The server should have refused the connection but it didn't.",
                     wd.findElement(By.tagName("html")).getText().contains("XMLRPC Error: 101 - Invalid username or password"));
         
         //
         // Make sure we are not logged in. The easiest way to do this it to check that the
         // Login button is still available.
         //
         wd.findElement(By.cssSelector("a.login_link")).click();
    }

Similarly, we can call loginAsUser() in method DocTikiGoodLoginGetsAccepted.

Make your tests independant of each other

Avoid side effects between tests. In other words, avoid situations where a test will only succeed if certain other tests were run before it (or on the contrary, if certain tests were NOT run before it). When you have side effects it becomes difficult to run only a certain subset of tests, or to run tests in parallel. This in turn makes your test suite much less useful, because you essentially have to run all of it to be sure of success.

The best way to garantee test independance would be to use a setup method that restores the Tiki DB to a known starting state. At the moment, we don't have a facility for doing this, but we plan to implement it soon.

So for now, this means you will have to be more careful when writing tests. Different tricks you can use are:

• When entering data into the DB, use randomly generated data. For example, if a test creates a new page, have the name of that page be random. That way, you are sure that the page cannot have already been created by a previous test, and you are sure that no other test will later on operate on the page you created.
• If a test needs to operate on data that can't be generated randomly, then:
  • Have the test start by making sure that this data is in a state that is correct for the test. For example, if the test needs to delete a page whose name is not random, start by making sure that the page exists, and if it doesn't then create it.
  • Have the test cleanup that data when it's done. For example, if it created a page, have the test make sure that it deletes it afterwards.
  • Note that both measures may be needed. For example, if a test creates a page then deletes it, and another test needs to start in a state where that page doesn't exist, then the second test should not assume that the first test deleted the page. That's because the first test may have died before it had a chance to cleanup after itself.

Centralize helper methods

The method loginAsUser() method we created will likely be useful to test classes besides DocTikiLoginLogout. For example, say we have some tests that check that only logged in users can edit wiki pages. Those tests will need to login too. So it's a good idea to put the loginAsUser() method somewhere where it can be accessed by any test that needs it.

The best way to do that is to create a class LoginLogoutHelpers and put the method there. The class will have a wd attribute of class WebDriver (which will be an argument of the constructor). Whenever a test wants to login, logout, switch user, etc..., it creates an instance of LoginLogoutHelpers, passing it its own instance of wd, and then users that helper instance to invoke loginAsUser().

PENDING TODOs

• Is it possible to install JUnit using maven, WITHOUT ACTUALLY SPECIFYING A SAUCE ID???