Loading...
 

Design Choices for the Selenium Testing Infrastructure

Background

Building a Selenium testing infrastructure for Tiki sounds simple in principle. But in practice, there is a whole eco-system of components built around Selenium, and choosing the right combination can be confusing.

This page describes the various choices that were made for Tiki, and why.

Selenium (RC) or 2 (WebDriver)?

There are two versions of Selenium.

Selenium 1 (= RC = Remote Control)
Selenium 1 uses a proxy to inject javascript into the pages, which allows it to control that script. The advantage of this approach is that, in principle, it is supposed to be browser-neutral. But in practice, the injected javascript doesn't always behave as expected in all browsers. Also, in some circumstances, some browsers may treat some of the javascript as a malicious injection attack.

Selenium 2 (= WebDriver)
Selenium 2 uses a WebDriver approach, i.e., it uses controls the browser through a driver that is built specifically for it. The advantage of this approach is that it works more consistently across browsers, because each driver is optimised for the browser it controls.

The API for the two flavours of Selenium are completely different, so you HAVE to choose which of the two you will write your tests against. They are both relatively clean, but the Selenium 2 API is a bit more modular than the Selenium 1 API.

Overall, Selenium 2 seems to be the better option because it's more reliable, more modular, and this is the direction that the Selenium community is moving to.

Record or write tests?

There are basically two ways to create tests:

  • Record through the browser
  • Write them in a programming language


The first approach is more pallatable to non-programmers, and this is important because we aim to have a number of non-programming power users who write thouasands of tests.

The problem with recorded tests is that they tend to be less modular, less flexible and harder to maintain.

For example, say you have several tests that need to start from a logged in state. All of those tests may thus have to start by the steps for logging in. In contrast, if you write the test in a programming language, you can encapsulate the login sequence in a method ensure_logged_in() and any test that needs to login can invoke it.

Actually, the situation is even worse for the recorded tests. Indeed, the approach we described above (each test starts by logging in) assumes that the test starts from a logged out state. But if one test starts by logging in, and that test is followed by another test that needs to login, then the second test will end up failing because you can't log in when you are already logged in. In contrast, if you are writing the test in a programming language, you can implement the ensure_logged_in() method in such a way that it first verifies if the user is already logged in (for example by inspecting the state of the login box), and logs in if and only if that's not the case.

Now, it could be that we could deal with this kind of situation by implementing some Selenium user_extensions. These are javascript functions which can be exposed as actions to the Selenium recorder. The problem with that is that the number of common "actions" that users do in Tiki is very large, which means we could end up with dozens if not hundreds of custom actions being exposed to the Selenium Record (ex: login(), edit_page(), search(), etc...). Also, it's not clear at the moment just how easy or hard it is to write this kind of user_extension.

One possibility is to use a hybrid approach where non-programmers will record tests, but those will only serve as a starting point. Later on, a programmer will convert that test to a PHP test.

Record with Selenium ID or SeleniumBuilder?

Now, say you have decided that we do want to allow non-programmers to create tests by recording them. You then have to choose between two recorders:

Selenium IDE
This is the recorder for Selenium 1. It's very nice and full featured.

  • The UI is more intuitive than SeleniumBuilder
  • There are some things you can do with IDE that you can't do with Builder. For example:
    • You can right click on a element and from there, click on a type of assertion or verification that is appropriate for that type of element.

But it has two drawbacks.

  • Its tests can only be run under Selenium 1.
  • The PHP export doesn't run out of the box (you have to install an extension, which I wasn't able to get working).


Selenium Builder
This is the recorder for Selenium 2. It's not as intuitive and nice as IDE, but it has the following advantages:

  • You can save the recorded tests in either Selenium 1 or Selenium 2 format.
  • The PHP export runs out of the box.

Which format to use for the test scripts?

In order to run the recorded tests in anything other than SeleniumBuilder, we need to save in as a script in a given programming language (ex: Java, PHP), and using a given testing framework (ex: jUnit, phpUnit, TestNG).

Ideally, we would want to save the tests as PHP scripts, to avoid adding yet another language to the Tiki ecosystem.

Unfortunately, PHP support is very poor both for SeleniumBuilder and for Selenium 2 (WebDriver).

It seems there are at least three options available, judging from the different PHP files I have seen that implement Selenium tests.

Looking at the different PHP files that come with the SauceLabs tutorial, it looks like they are using PhpUnit on top of WebDriver. I believe this format is called phpunit-selenium (see: https://github.com/sebastianbergmann/phpunit-selenium). This is probably the format we want, because it's very clean, and uses WebDriver which is more robust thatn RC.

Unfortunately, this is not the format you end up with when you record a Selenium 2 test with Selenium Builder, and save it as PHP. Instead, you end up with a non-OO, non-modular, non-phpunit script, that doesn't even wrap the steps of the test in a method or even function. So it will probably be a maintenance nightmare. Note however that the script at least seems to use the WebDriver API. I believe this format is called php-webdriver (see: https://github.com/facebook/php-webdriver). We probably don't want to use this format, because it will be a maintenance nightmare.

There is however another way to save a Selenium Builder test to PHP. This options becomes available if you convert the script to Selenium 1 (File > Convert to other version...). At that point, if you do File > Save, you see two PHP options: PHP and PHP - Selenium RC. The PHP option corresponds to the "ugly" php-webdriver format I described earlier, but the PHP - Selenium RC option generates a more modular PhpUnit test. Unfortunately, that format uses the Selenium 1 (i.e. RC) API, not the Selenium 2. So we probabably don't want to use it either. Let's call this format phpunit-selenium-rc.

Bottom line is, if we want our tests to be saved in a clean, modular format that also uses the more robust WebDriver API, we currently can't record them, and have to write them as we would write any PHP program. This will be OK for programmers, but not for non-programmers.

If we want to support non-programmers, we can

  • Petition for the addition of phpunit-selenium format in SeleniumBuilder
  • Teach non-programmers how to write phpunit-selenium tests
  • Support two ways of creating tests. Programmers would write the tests directly in PHP, using the more robust webdriver API, and non-programmers would record tests and save them in the less robuts phpunit-selenium-rc format.
  • Implement an adapter that can translate from phpunit-selenium-rc to phpunit-selenium (i.e. the webdriver based phpunit extension).
    • That way, non-programmers could record their tests with SeleniumBuilder and save them in the phpunit-selenium-rc format, but when they ran, they would be converted on the fly into webdriver calls.
    • Maybe such a mapping has already been implemented by someone?
    • Note: it's not clear that all RC calls CAN be converted to webdriver calls...
  • Anoter option is save the test in a language that has good support, say Java, and then use that script as a model for writing the PHP script.


But even if we were able to find a practical way of transforming a recorded script into a clean PHP script, we would run into another problem. Indeed, none of the existing PHP bindings for Selenium 2 seem to be complete.

So... for now, I think we need to go with another language than PHP. As far as I can tell, the language that has the best support is Java (not surprising since Selenium itself is written in Java).

This is obviously not optimal, because it introduces yet another language in the Tiki ecosystem. But at least, that language will be limited to the Selenium testing parts, and it won't be part of the application itself.

Run the tests on SauceLabs or Jenkins?

Assuming you have decided how you will create the test and in which format. Now you need to have a way to run the tests on a regular basis, and have errors be reported.

There are many options there, but we looked at two: SauceLabs and Jenkins.

SauceLabs is a commercial service that you can user to run your Selenium tests. You have to pay for it, but they have free accounts for Open Source projects. It seems to have lots of bells and whistles including:

  • Ability to run tests on any combination of Browser + OS
  • Capturing a video of the tests being run
  • Ability to run several tests in parallel

They have very nice tutorials and I was able to get a test running very easily.

Jenkins is the Open Source software that Tiki has been using for some years to run the Tiki unit tests. At present, we don't know much about the features of Jenkins as it pertains to Selenium.

Alain's notes

Trying to record scripts and run them as JUnit tests

Support for Java seems much better than support for PHP, both from the point of view of support for the WebDriver API (PHP support is still not completely implemented) and support for exporting recorded tests to that language. So I am trying to see how well that works.

Copy to clipboard
mvn archetype:generate \ -DarchetypeRepository=http://repository-saucelabs.forge.cloudbees.com/release \ -DarchetypeGroupId=com.saucelabs \ -DarchetypeArtifactId=quickstart-webdriver-junit \ -DarchetypeVersion=1.0.7 \ -DsauceUserName=alain_desilets \ -DsauceAccessKey=0b05a331-10ef-4876-b7ce-afe666382696
    • But when I ran it, it kept asking me for things like the groupId. I just entered the values that are specified in the above command. Note that it asked me for a package name (which is not a property specified in the above command line), and I just answered with Return.
  • Next step is to run the tests. You have to cd to the quickstart-webdriver-junit directory, and do 'mvn test'.
    • The first time you do this, it will take some time, as the program will upload a bunch of java binaries to the saucelab server.
  • To run a single test method, do something like this (as per this page http://maven.apache.org/surefire/maven-surefire-plugin/examples/single-test.html):
Copy to clipboard
mvn -Dtest=TestCircle#mytest test

Pending questions

  • Since PHP export in Selenium Builder is bad, would it be viable to export the tests as Java or Python instead and run them as such?
  • How hard is it to write a php-selenium (i.e. PHP with WebDrive), starting from a model that is a test recorded with Selenium Builder and exported as Java?
    • Time myself doing it two ways: (a) Writing the php-selenium test by hand versus (b) record it, save it as Java, and translate the Java to PHP by hand.
  • How hard would it be to manage a long list of php-webdriver scripts (i.e. a scripts that doesn't use phpunit, but do use WebDriver)?
    • How hard would it be to rewrite them as phpUnit tests?
    • How hard would it be to extract a method out of one of those tests, for use in other tests as well (ex: ensureLoggedIn()).
  • How to install saucelabs in a different place than the tutorial uses?
  • How soon can we expect SeleniumBuilder to support saving tests as phpunit-selenium (i.e. phpunit with webdriver) format?
  • How can you run a PhpUnit Selenium 1 script?
    • They don't seem to run when you try to use them in the way that the SauceLabs tutorial suggests.
  • Can you run the tests on Jenkins? If so, what is that like?
  • With either recorders, can you run a test, pause it at a particular location, and then insert a new command or assertion there?
  • Need to get better at using Selenium Builder.
    • http://saucelabs.com/docs/builder
    • Some of the things on the above page don't seem to work. For example, there is no URL box, and when I click on Record a verification, I don't see an element highlighter.
    • Maybe SeleniumBuilder only works with Chrome (the title of the Selenium window says chrome://seleniumbuilder - Selenium Builder)?
  • How to use Sauce Connect to allow SauceLabs to test apps that reside on localhost or behind a firewall?

Keywords

The following is a list of keywords that should serve as hubs for navigation within the Tiki development and should correspond to documentation keywords.

Each feature in Tiki has a wiki page which regroups all the bugs, requests for enhancements, etc. It is somewhat a form of wiki-based project management. You can also express your interest in a feature by adding it to your profile. You can also try out the Dynamic filter.

Accessibility (WAI & 508)
Accounting
Administration
Ajax
Articles & Submissions
Backlinks
Banner
Batch
BigBlueButton audio/video/chat/screensharing
Blog
Bookmark
Browser Compatibility
Calendar
Category
Chat
Comment
Communication Center
Consistency
Contacts Address book
Contact us
Content template
Contribution
Cookie
Copyright
Credits
Custom Home (and Group Home Page)
Database MySQL - MyISAM
Database MySQL - InnoDB
Date and Time
Debugger Console
Diagram
Directory (of hyperlinks)
Documentation link from Tiki to doc.tiki.org (Help System)
Docs
DogFood
Draw -superseded by Diagram
Dynamic Content
Preferences
Dynamic Variable
External Authentication
FAQ
Featured links
Feeds (RSS)
File Gallery
Forum
Friendship Network (Community)
Gantt
Group
Groupmail
Help
History
Hotword
HTML Page
i18n (Multilingual, l10n, Babelfish)
Image Gallery
Import-Export
Install
Integrator
Interoperability
Inter-User Messages
InterTiki
jQuery
Kaltura video management
Kanban
Karma
Live Support
Logs (system & action)
Lost edit protection
Mail-in
Map
Menu
Meta Tag
Missing features
Visual Mapping
Mobile
Mods
Modules
MultiTiki
MyTiki
Newsletter
Notepad
OS independence (Non-Linux, Windows/IIS, Mac, BSD)
Organic Groups (Self-managed Teams)
Packages
Payment
PDF
Performance Speed / Load / Compression / Cache
Permission
Poll
Profiles
Quiz
Rating
Realname
Report
Revision Approval
Scheduler
Score
Search engine optimization (SEO)
Search
Security
Semantic links
Share
Shopping Cart
Shoutbox
Site Identity
Slideshow
Smarty Template
Social Networking
Spam protection (Anti-bot CATPCHA)
Spellcheck
Spreadsheet
Staging and Approval
Stats
Survey
Syntax Highlighter (Codemirror)
Tablesorter
Tags
Task
Tell a Friend
Terms and Conditions
Theme
TikiTests
Federated Timesheets
Token Access
Toolbar (Quicktags)
Tours
Trackers
TRIM
User Administration
User Files
User Menu
Watch
Webmail and Groupmail
WebServices
Wiki History, page rename, etc
Wiki plugins extends basic syntax
Wiki syntax text area, parser, etc
Wiki structure (book and table of content)
Workspace and perspectives
WYSIWTSN
WYSIWYCA
WYSIWYG
XMLRPC
XMPP




Useful Tools