Background
Building a Selenium testing infrastructure for Tiki sounds simple in principle. But in practice, there is a whole eco-system of components built around Selenium, and choosing the right combination can be confusing.
This page describes the various choices that were made for Tiki, and why.
Selenium (RC) or 2 (WebDriver)?
There are two versions of Selenium.
Selenium 1 (= RC = Remote Control)
Selenium 1 uses a proxy to inject javascript into the pages, which allows it to control that script. The advantage of this approach is that, in principle, it is supposed to be browser-neutral. But in practice, the injected javascript doesn't always behave as expected in all browsers. Also, in some circumstances, some browsers may treat some of the javascript as a malicious injection attack.
Selenium 2 (= WebDriver)
Selenium 2 uses a WebDriver approach, i.e., it uses controls the browser through a driver that is built specifically for it. The advantage of this approach is that it works more consistently across browsers, because each driver is optimised for the browser it controls.
The API for the two flavours of Selenium are completely different, so you HAVE to choose which of the two you will write your tests against. They are both relatively clean, but the Selenium 2 API is a bit more modular than the Selenium 1 API.
Overall, Selenium 2 seems to be the better option because it's more reliable, more modular, and this is the direction that the Selenium community is moving to.
Record or write tests?
There are basically two ways to create tests:
- Record through the browser
- Write them in a programming language
The first approach is more pallatable to non-programmers, and this is important because we aim to have a number of non-programming power users who write thouasands of tests.
The problem with recorded tests is that they tend to be less modular, less flexible and harder to maintain.
For example, say you have several tests that need to start from a logged in state. All of those tests may thus have to start by the steps for logging in. In contrast, if you write the test in a programming language, you can encapsulate the login sequence in a method ensure_logged_in() and any test that needs to login can invoke it.
Actually, the situation is even worse for the recorded tests. Indeed, the approach we described above (each test starts by logging in) assumes that the test starts from a logged out state. But if one test starts by logging in, and that test is followed by another test that needs to login, then the second test will end up failing because you can't log in when you are already logged in. In contrast, if you are writing the test in a programming language, you can implement the ensure_logged_in() method in such a way that it first verifies if the user is already logged in (for example by inspecting the state of the login box), and logs in if and only if that's not the case.
Now, it could be that we could deal with this kind of situation by implementing some Selenium user_extensions. These are javascript functions which can be exposed as actions to the Selenium recorder. The problem with that is that the number of common "actions" that users do in Tiki is very large, which means we could end up with dozens if not hundreds of custom actions being exposed to the Selenium Record (ex: login(), edit_page(), search(), etc...). Also, it's not clear at the moment just how easy or hard it is to write this kind of user_extension.
One possibility is to use a hybrid approach where non-programmers will record tests, but those will only serve as a starting point. Later on, a programmer will convert that test to a PHP test.
Record with Selenium ID or SeleniumBuilder?
Now, say you have decided that we do want to allow non-programmers to create tests by recording them. You then have to choose between two recorders:
Selenium IDE
This is the recorder for Selenium 1. It's very nice and full featured.
- The UI is more intuitive than SeleniumBuilder
- There are some things you can do with IDE that you can't do with Builder. For example:
- You can right click on a element and from there, click on a type of assertion or verification that is appropriate for that type of element.
But it has two drawbacks.
- Its tests can only be run under Selenium 1.
- The PHP export doesn't run out of the box (you have to install an extension, which I wasn't able to get working).
Selenium Builder
This is the recorder for Selenium 2. It's not as intuitive and nice as IDE, but it has the following advantages:
- You can save the recorded tests in either Selenium 1 or Selenium 2 format.
- The PHP export runs out of the box.
Which format to use for the test scripts?
In order to run the recorded tests in anything other than SeleniumBuilder, we need to save in as a script in a given programming language (ex: Java, PHP), and using a given testing framework (ex: jUnit, phpUnit, TestNG).
Ideally, we would want to save the tests as PHP scripts, to avoid adding yet another language to the Tiki ecosystem.
Unfortunately, PHP support is very poor both for SeleniumBuilder and for Selenium 2 (WebDriver).
It seems there are at least three options available, judging from the different PHP files I have seen that implement Selenium tests.
Looking at the different PHP files that come with the SauceLabs tutorial, it looks like they are using PhpUnit on top of WebDriver. I believe this format is called phpunit-selenium (see: https://github.com/sebastianbergmann/phpunit-selenium). This is probably the format we want, because it's very clean, and uses WebDriver which is more robust thatn RC.
Unfortunately, this is not the format you end up with when you record a Selenium 2 test with Selenium Builder, and save it as PHP. Instead, you end up with a non-OO, non-modular, non-phpunit script, that doesn't even wrap the steps of the test in a method or even function. So it will probably be a maintenance nightmare. Note however that the script at least seems to use the WebDriver API. I believe this format is called php-webdriver (see: https://github.com/facebook/php-webdriver). We probably don't want to use this format, because it will be a maintenance nightmare.
There is however another way to save a Selenium Builder test to PHP. This options becomes available if you convert the script to Selenium 1 (File > Convert to other version...). At that point, if you do File > Save, you see two PHP options: PHP and PHP - Selenium RC. The PHP option corresponds to the "ugly" php-webdriver format I described earlier, but the PHP - Selenium RC option generates a more modular PhpUnit test. Unfortunately, that format uses the Selenium 1 (i.e. RC) API, not the Selenium 2. So we probabably don't want to use it either. Let's call this format phpunit-selenium-rc.
Bottom line is, if we want our tests to be saved in a clean, modular format that also uses the more robust WebDriver API, we currently can't record them, and have to write them as we would write any PHP program. This will be OK for programmers, but not for non-programmers.
If we want to support non-programmers, we can
- Petition for the addition of phpunit-selenium format in SeleniumBuilder
- Teach non-programmers how to write phpunit-selenium tests
- Support two ways of creating tests. Programmers would write the tests directly in PHP, using the more robust webdriver API, and non-programmers would record tests and save them in the less robuts phpunit-selenium-rc format.
- Implement an adapter that can translate from phpunit-selenium-rc to phpunit-selenium (i.e. the webdriver based phpunit extension).
- That way, non-programmers could record their tests with SeleniumBuilder and save them in the phpunit-selenium-rc format, but when they ran, they would be converted on the fly into webdriver calls.
- Maybe such a mapping has already been implemented by someone?
- Note: it's not clear that all RC calls CAN be converted to webdriver calls...
- Anoter option is save the test in a language that has good support, say Java, and then use that script as a model for writing the PHP script.
But even if we were able to find a practical way of transforming a recorded script into a clean PHP script, we would run into another problem. Indeed, none of the existing PHP bindings for Selenium 2 seem to be complete.
So... for now, I think we need to go with another language than PHP. As far as I can tell, the language that has the best support is Java (not surprising since Selenium itself is written in Java).
This is obviously not optimal, because it introduces yet another language in the Tiki ecosystem. But at least, that language will be limited to the Selenium testing parts, and it won't be part of the application itself.
Run the tests on SauceLabs or Jenkins?
Assuming you have decided how you will create the test and in which format. Now you need to have a way to run the tests on a regular basis, and have errors be reported.
There are many options there, but we looked at two: SauceLabs and Jenkins.
SauceLabs is a commercial service that you can user to run your Selenium tests. You have to pay for it, but they have free accounts for Open Source projects. It seems to have lots of bells and whistles including:
- Ability to run tests on any combination of Browser + OS
- Capturing a video of the tests being run
- Ability to run several tests in parallel
They have very nice tutorials and I was able to get a test running very easily.
Jenkins is the Open Source software that Tiki has been using for some years to run the Tiki unit tests. At present, we don't know much about the features of Jenkins as it pertains to Selenium.
Alain's notes
Trying to record scripts and run them as JUnit tests
Support for Java seems much better than support for PHP, both from the point of view of support for the WebDriver API (PHP support is still not completely implemented) and support for exporting recorded tests to that language. So I am trying to see how well that works.
- Following instructions on this page: https://saucelabs.com/docs/ondemand/getting-started/env/java/se2/mac
- Actually, it looks like info on this page is more up to date and complete: https://saucelabs.com/java
- First step is to install Maven
- Followed instructions on this page: http://maven.apache.org/download.cgi, EXCEPT THAT I didn't set the M2_HOME variable and I didn't include it in the path (as per this page).
- Actually, the info
- Next step is to run this command
mvn archetype:generate \ -DarchetypeRepository=http://repository-saucelabs.forge.cloudbees.com/release \ -DarchetypeGroupId=com.saucelabs \ -DarchetypeArtifactId=quickstart-webdriver-junit \ -DarchetypeVersion=1.0.7 \ -DsauceUserName=alain_desilets \ -DsauceAccessKey=0b05a331-10ef-4876-b7ce-afe666382696
- But when I ran it, it kept asking me for things like the groupId. I just entered the values that are specified in the above command. Note that it asked me for a package name (which is not a property specified in the above command line), and I just answered with Return.
- Next step is to run the tests. You have to cd to the quickstart-webdriver-junit directory, and do 'mvn test'.
- The first time you do this, it will take some time, as the program will upload a bunch of java binaries to the saucelab server.
- To run a single test method, do something like this (as per this page http://maven.apache.org/surefire/maven-surefire-plugin/examples/single-test.html):
mvn -Dtest=TestCircle#mytest test
Pending questions
- Since PHP export in Selenium Builder is bad, would it be viable to export the tests as Java or Python instead and run them as such?
- How hard is it to write a php-selenium (i.e. PHP with WebDrive), starting from a model that is a test recorded with Selenium Builder and exported as Java?
- Time myself doing it two ways: (a) Writing the php-selenium test by hand versus (b) record it, save it as Java, and translate the Java to PHP by hand.
- How hard would it be to manage a long list of php-webdriver scripts (i.e. a scripts that doesn't use phpunit, but do use WebDriver)?
- How hard would it be to rewrite them as phpUnit tests?
- How hard would it be to extract a method out of one of those tests, for use in other tests as well (ex: ensureLoggedIn()).
- How to install saucelabs in a different place than the tutorial uses?
- How soon can we expect SeleniumBuilder to support saving tests as phpunit-selenium (i.e. phpunit with webdriver) format?
- Waiting for an answer here: https://groups.google.com/forum/#!searchin/se-builder/phpunit/se-builder/usa4kkgsccE/OGisdlv1QeEJ
- How can you run a PhpUnit Selenium 1 script?
- They don't seem to run when you try to use them in the way that the SauceLabs tutorial suggests.
- Can you run the tests on Jenkins? If so, what is that like?
- With either recorders, can you run a test, pause it at a particular location, and then insert a new command or assertion there?
- Need to get better at using Selenium Builder.
- http://saucelabs.com/docs/builder
- Some of the things on the above page don't seem to work. For example, there is no URL box, and when I click on Record a verification, I don't see an element highlighter.
- Maybe SeleniumBuilder only works with Chrome (the title of the Selenium window says chrome://seleniumbuilder - Selenium Builder)?
- How to use Sauce Connect to allow SauceLabs to test apps that reside on localhost or behind a firewall?