The Importance of Continuous Testing in the Cloud

If you worked in an ice cream shop, wouldn’t you feel obligated to periodically sample the product to ensure quality? Do you want the mechanic to make sure your car is actually fixed before returning it to you (and giving you a bill)? Quality assurance requires continuous testing. When you buy something, you want to know that it works. This is even more important in the highly integrated, ever-changing digital world where an alteration in one system can inadvertently impact the functionality of another.

Standing on the shoulders of giants, in this case JIRA, means that we get to build on a well-established, highly reputed, first class product. However, it also means that Atlassian can and does make changes that impact the performance of our add-on. Updates and innovations to JIRA can be made without notice and without our having any say in the matter. We employ automated testing with Ghost Inspector as a proactive step to ensure that unanticipated changes in the systems we integrate with do not cause interruptions or failures for our users. And when things do break, we find out about it (and can address it) ahead of our customers.

How it Works

Ghost Inspector essentially runs selenium scripts from the cloud. However, it is not just a Sauce Labs (a fantastic service for spinning up multiple browser tests of the same Selenium script) clone. Ghost Inspector allows you to write and/or record your test scripts in the cloud and then run them on a specified schedule. In our case, we run all our tests every fours hours so we get a confirmations six times a day that all of our critical features are operational. Ghost Inspector also lets you record test sessions from within Chrome. This is a convenient way for anyone on your team, not just developers, to test of a new feature, or record how a bug is triggered so that you know when it's been fixed. Each Ghost Inspector test is stored within a suite, and you can schedule individual tests, or an entire suite to run on a defined schedule. 

Screenshots taken at the end of every test run make it easy to compare to see what’s changed so we can identify and address failures. The tests are stored within suites in Ghost Inspector. We have three suites; one for server one for cloud and one for development. Using suite variables means that we only need to make one change in the development suite to test new features and updates of our product. Notifications are piped through to HipChat so we always know if one of our production tests fails. 

Other Uses

We use Ghost Inspector for dog feeding, in that we set up our demo accounts using our own production UI.  A repeating test installs all of our templates into five or six different projects in a demo JIRA cloud instance. The test creates the relevant forms, request types and request type groups. This process of creating a new, fully populated demo with all of our Process Templates used to take one or two hours. Now it happens in about 10 minutes of firing off the test. 

Ghost Inspector is also great for setting up a test instance. Demo environments typically require pre-populated data.  So how do we know that the UI actually functions? Using Ghost Inspector ensures that all data can be entered through the UI. This means our developers spend a lot less time setting up instances for testing.

Limitations

We have found some limitations. The browser tests generated by the Chrome extension are not always perfect and it pays to review the tests, particular the CSS hooks.  Our JIRA add-on sits within an iFrame in the JIRA instance which caused some issues. However, the Ghost Inspector support team was excellent and gave us an easy workaround.

There were also some challenges because our add-on is available on both Server and Cloud. We can’t use the same tests for both, as the CSS hooks are slightly different. However, we can still use Ghost Inspector, as we have made our test JIRA Sserver accessible over the net and we’ve found updating our tests from the cloud version to the servers version to be a pretty straight forward affair.

All in all, ThinkTilt has found Ghost Inspector to be a very useful tool in helping us ensure the consistency and quality of our product. As more and more systems integrate with each other, continuous UI testing will become increasingly important. At the end of the day it'is the users' / customers' experience that really matters.