Selectors / locators - Deep dive
Jan 4, 2024, 2:21 PM: ~ 8 min read
Siblings, parents, ancestors, madness
- Imagine the web page like a city with adjacent buildings of different heights. The street is <html> tag. The buildings are <head>,<body> etc.
- Now imagine you are an apartment. So apartment == element.
- Sibling elements are the same floor apartments.
- The parent element is the flat itself. Because it holds you inside it.
- The street is your ancestor because it holds your parent, the flat. The ancestor can be as far away as you want it to be.
- Relative to you, the child elements are the rooms. Because they are inside you.
- These names apply for both CSS and XPath. They are concepts.
- Sorry, there is no madness, but the official XPath documentation be found at MDN.
Thank you for reading the TL;DR section. From here on we will continue with how it is implemented on WEB vs CSS vs PDF.
Traps for QA:
From the browser's perspective everything is displayed based on a property called z-index which overwrites the visibility discovered by nesting. So the browser stacks everything up based on nesting and then overwrites some of it based on the specified z-index. If you're curious how different websites look download a z-index visualizer for your favorite browser and give it a go. Another factor that overwrites nesting are position fixed and absolute which result in elements floating over other elements or in us clicking on "imaginary" elements. Then there's the "where" magic happens. There are 3 possible places where the "true" elements exist based on the frontend framework used. There's HTML, DOM and Shadow DOM.
HTML, DOM and Shadow DOM:
Q: What do they have to do with selectors?
A: A lot.
- HTML: what you see in the structure of the page is what you select and interact with. Easy.
- DOM: some logic will happen in the JavaScript layer and at the end the HTML will be populated with new elements. That's why sometimes in selenium there are stale element exceptions although everything looks the same. The solution is to stop operating on the old element and select again.
- Shadow DOM: the most hidden and abstract of them all because it allows the JavaScript to link a calculated HTML structure to an element. Link, not replace in HTML. These are the worst when comes to automation.
Conclusion:
- If possible, unless you really have to, for web, avoid writing your own frontend automation tools from scratch because at some point you will spend time debugging #2 and #3. When you will debug #3 you will likely end up in situations where things don't make sense at all and you have no clue what selectors to extract. See google spreadsheet for example. Try to inspect element and select a cell.
- If your application under test uses a frontend framework based on Shadow DOM be very careful when selecting your ui test automation framework.
How do I actually extract those selectors?
- Browser: try using the inspector tool. Search something along the lines of "view page source xxxx" where xxxx is your browser of choice. As soon as you get to see the source code of the page you can proceed.
- PDF: things get a little complicated because although it has a tree structure, same as HTML, the elements inside the tree could be images with text drawn inside. That structure depends a lot on the software that generates the .PDF and not on the software that reads it. Try with an PDF structure explorer and see what you get. Use the documentation to generate an XPath like selector for the required element.
- Indentify the element that you need and understand what makes it unique.
- Google search "how to select element by xxxx" where xxxx represents what makes it unique. Improve your search using the keywords from TL;DR because now you already now what it means to be relative to something else.
- If the element you are trying to target disappears (such as a hover, or a conditional visibility) your life got A LOT harder. See if the automation tool offers an extension for your browser or an tool which can help you capture selectors with a shortcut key. This will allow you to move the mouse over the elusive element and with a press of a key extract what you need. Some browsers offer under right click an option to copy the path. Some pages block the default behavior of right click.
There will be another blog post in the future specific to the bigger browsers out there. With their own particularities. This post is about learning the thinking process and how to unstuck ourselves from every situation.
Conclusion:
- #4 is particularly annoying when building your own tool from scratch. If you are unlucky to test an application like google spreadsheet or any other where the default mouse behavior is prevented don't even think about writing your automation tools for ui testing.
- For .PDF if the document includes images talk to the stakeholders and see if it can't be generated differently because if they change it to a better structure the readers used by your consumers will be able to parse that structure and offer a lot of "magic" based on text comprehension. For example in case of airplane tickets the departure and arrival dates. Collaboration with the team is key.
How do browsers use selectors and the magical select by id unicorn
This question has 3 totally different answers because there are 3 different mechanisms in place:
- When the page is downloaded for the first time the browser searches in the HTML document for references about .css and .js files. It requests them from the server. While waiting for them it goes through the HTML already present and makes references between which element has which CSS classes applied and which styles. Then, as soon as the CSS is received it goes again over that indexing and applies the classes from left to right. The class to the right always overwrites the CSS properties of the class to the left. With the exception of !important. That's why the very, very annoying effect of elements moving around when the page is not fully loaded. The browser very rarely uses IDs. Those come in handy when applying the JavaScript on top of the HTML.
- When we test a chosen selector in the browser's console the browser may or may not use the same algorithm to return us the result. Most of the times it will pass the selector string to JQuery or another JavaScript implementation that will fetch the result for us.
- When we use the same selector in our automation suite, the initial string may or may not reach the browser based on the browser communication library used (such as selenium).
Debugging:
So there's a lot of abstraction (or magic) happening and when something goes wrong apply the following thinking process:
- It looks good. So #1 worked.
- Does it work in the console? So the browser is ok with it.
- Does it work in the test? If not then either run a JavaScript to perform your action or change your selector in order to work with your testing tool.
So, what about the ID?
Usually is advertised as a good way of having a robust selector and especially on the copy paste youtube videos and blogs you will encounter that is the best way to do things. However, as detailed in the "selectors usage" post when possible include in the selector the attributes that have to be tested and the businss critical parents' structure. So yes, sprinkle a bit but don't rely on it exclusively if there are other areas that can be covered.
Conclusion:
- Just because you've watched a tutorial about "automation with selenium and xxxx" doesn't mean it is a financially sound idea to make a proof of concept for your boss. In 1-3 months time you will need a colleague to help you out and the combined price of you 2 will be greater than a subscription to most automated software testing tools. Also you will need to run the tests. That will add extra cost.
- Understand the layers of the task at hand and when you encounter problems go from top to bottom or bottom to top and search for granular answers.
Browser specific selectors
Yes, it used to be very problematic. Now less so because at the time of writing almost everyone is built on some sort of Chromium. But yes, it can happen and yes is true.
Credits:
- Initial text: gabriel@qality.tech