QA Automation: Analyzing test results with Selenium Reports

So I’ve built my automation framework, developed my tests and ran them.

What now? How do I analyze the results?

To me, there are two kinds of results which can be derived from a certain test; an “Assertion Error” or a “False Alarm”.

Assertion Errors are not False Alarms because they are exactly what you’re testing; if you’re testing if an element is displayed in a certain page, and the element isn’t, then Bob’s your Uncle, you found a bug.

So what exactly is a False Alarm?

A False Alarm is basically any Exception you receive, along the way, before you get to your Assertion.

However, sometimes False Alarms are not really False Alarms, they might actually lead to a specific bug, i.e: “if your test clicks on a link to validate that in the new page you get after clicking the link, there are 5 checkboxes, but your test threw an Exception because it couldn’t find the link which leads to the page you needed to validate.”

So now we have a dilemma; the “False Alarm” which we received in the example, is actually a bug and not a “False Alarm”. We can conclude we had a hole in our test, which led to a False Alarm which was actually supposed to be an Assertion Error, but since we “assumed” that the link is there, we were led to a “False Alarm” which turned out to be a bug.

When writing your tests, always try to cover all situations in a proper assertion message instead of getting an Exception. This process usually takes a long time since you will have to verify every single line of code you write, so don’t overdo it, but try to cover all potential bugs while writing your tests.

For example – the process of selecting a dropdown. Verify that the dropdown is found, click the dropdown, verify that the dropdown opened, verify that the dropdown contains values, verify it contains the value you want to select, click the selectable value you would like, validate that the dropdown closed, and verify the new selected value is the one you chose.

When writing tests, always try to maintain the highest success rate possible, but as sometimes some packets get lost along the way, etc. it is very hard predicting every single exception which occurs.

Either way, I will try to drill down and explain some of the common exceptions so that it gives you an idea about how you can analyze the reports you get.

1. org.openqa.selenium.NoSuchElementException – this means that the driver tried locating an element and couldn’t find it (this happens when (1) an element doesn’t load properly when a package doesn’t get sent back properly, (2) or simply when you’re not in the right page, (3) or the time the driver waited to locate that element isn’t enough.

With (1) and (3) you can usually see in the screenshot that the element simply isn’t there.

With (2) it is easy to validate if you are on the right page, again using the screenshot, to see which element exactly the driver couldn’t locate, you just need to read the stacktrace and find out which was the last call that triggered that exception.

2. java.lang.NullPointerException – this means a command was invoked from an Object which is pointing to null.

For example: we tried to log in with the right credentials, and that command returns an Object for the returned page (WelcomeScreen).

If for some reason, you were redirected to a different page other than the welcome screen, then when trying to access one of the methods which the welcome screen has, it will throw a NullPointerException.

3. org.openqa.selenium.WebDriverException: unknown error: Element is not clickable at point – this means that we tried clicking on an element, but it wasn’t ready (loaded properly) or there was a popup on top of it which prevented the driver from being able to click on it, the element isn’t visible, or it is simply an element which you can’t click on.

4. org.openqa.selenium.StaleElementReferenceException – each time we load a page and then try to locate an element in a WebElement Object and after that, go to another page and then get back to the same page, the element we located previously becomes “stale”, which means, the page DOM got refreshed and doesn’t point to the same element we previously saved. In order to deal with that element we need to relocate it again.

A less common cause is where a JS library has deleted an element and replaced it with one with the same ID or attributes. In this case, although the replacement elements may look identical they are different; the driver has no way to determine that the replacements are actually what’s expected. This usually happens on news websites, since the page gets refreshed by itself at a fixed interval of time, because there is a need to always update the page with new articles.

All of the Exceptions above are considered False Alarms, and after every single test run, the developers always look at the reports and try to modify their tests to minimize False Alarms as much as possible.

Any failed test you see with an AssertionError is a RED FLAG and a potential bug.

Written by Amir Najjar of Galil Software