{"id":9581,"date":"2020-02-28T10:10:22","date_gmt":"2020-02-28T04:40:22","guid":{"rendered":"http:\/\/ivyproschool.com\/blog\/?p=9581"},"modified":"2021-06-11T12:08:21","modified_gmt":"2021-06-11T06:38:21","slug":"how-to-use-xpath-for-web-scraping-with-selenium","status":"publish","type":"post","link":"https:\/\/ivyproschool.com\/blog\/how-to-use-xpath-for-web-scraping-with-selenium\/","title":{"rendered":"How to use XPath for Web Scraping with Selenium"},"content":{"rendered":"


\n\"XPath<\/a><\/p>\n

Web scraping is a commonly used technology existing for a couple of decades now. We use programming languages like Python with libraries namely Beautiful Soup, Selenium, Scrapy, etc. or R with Rvest package for web scraping. Other web scraping tools are also available. A vivid explanation of how we do web scraping using Python is provided in the article named Web Scraping using Beautiful Soup<\/a>.\u00a0 Further to the above, in this article, we are going to understand XPath and how it can be used to navigate through the HTML document for Web Scraping with Selenium to reach the target nodes using Python.<\/p>\n

Our Target Website –<\/h2>\n

Let us consider the webpage of a very popular Indian daily English newspaper Telegraph https:\/\/www.telegraphindia.com\/<\/a><\/p>\n

What Library are we using –<\/h2>\n

We are using the robust Selenium web driver<\/a> to navigate through the Telegraph website using XPath.<\/p>\n

What is XPath –<\/h2>\n

The full form of XPath is XML Path language. It is a very popular Web Technology and W3C standard. We can access precise information in an XML\/HTML document using these concise and powerful XPath statements.<\/p>\n

Understanding the structure of XPath –<\/h2>\n

First of all, let us see how can we find XPath of any element. We are using Mozilla Firefox as our browser. Our target element is the “Opinion” section on the web page. The steps to select the XPath for this element is as follows –<\/p>\n

    \n
  1. Right-click on the ‘opinion’ section on the web page<\/li>\n
  2. select inspect element<\/li>\n
  3. The HTML code for this element will be highlighted in blue<\/li>\n
  4. Right-click on the blue portion again<\/li>\n
  5. Click on copy and select the XPath option<\/li>\n
  6. Paste this on a text file. The path shows\u00a0 \/html\/body\/section\/div[1]\/div[2]\/div[2]\/div\/a<\/li>\n<\/ol>\n

    \"XPath<\/p>\n

    The path shown above is called the absolute path<\/strong> that is the detailed path across the nodes to reach the target node. The absolute path starts with a ‘\/’ which means starting the search from the root element. Another way to represent is by using a relative path<\/strong> in which we construct a shorter path to the target node. The absolute path for the ‘opinion’ is \/\/section\/div[1]\/div[2]\/div[2]\/div\/a. Here we have replaced the ‘\/html\/body’ tags with ‘\/\/’ to shorten the path.<\/p>\n

    There are other ways to represent an XPath using the Selenium selector. Let us take a look at them one by one.<\/p>\n

    Tag \u2013 Attribute \u2013 Value Trio<\/h3>\n

    Syntax:<\/strong> \/\/tag[@attribute=\u2019value\u2018]<\/p>\n

    \"Fourlines\"<\/p>\n

    If you take a look at the screenshot above, we aim at identifying all the elements ‘a’ with attribute class = ‘frontLine’. The XPath command for this element will be as follows.<\/p>\n

    [php] \/\/a[@class=’fourLine’] [\/php]<\/p>\n

    The command means we are looking for attribute class with value ‘fourLines’ in the element ‘a’<\/p>\n

    Contains<\/h3>\n

    Syntax:<\/strong> \/\/tag[contains(@attribute, ‘value’)]<\/p>\n

    Another XPath Selenium\u00a0<\/strong>locator which is very useful in identifying attributes is ‘contains’. Let us see how to use this command with an example.<\/p>\n

    \"XPath<\/p>\n

    In the above screenshot, we want to locate all the tag ‘ul’ in which its attribute class contains ‘headImage’ in its value. The XPath command will be as follows. (We are going to use this scenario at other places in the article as well)<\/p>\n

    [php]\u00a0 \/\/ul[contains(@class,’headImage’)]\u00a0 [\/php]<\/p>\n

    Starts With<\/h3>\n

    Syntax:<\/strong> \/\/tag[starts-with(@attribute, ‘value’)]<\/p>\n

    This XPath Selenium locator is very useful when the attribute value is too long or changes dynamically. Using the same scenario as in the above-mentioned point, we can create an XPath comprising of ‘starts-with’ as follows.<\/p>\n

    [php]\u00a0 \/\/ul[starts-with(@class,’listing’)]\u00a0 [\/php]<\/p>\n

    Chained Declarations<\/h3>\n

    Syntax:<\/strong> \/\/tag[XPath Statement-1]\/\/tag[XPath Statement-2]<\/p>\n

    Multiple XPath can be chained with \/\/ (double slash) to gain more accuracy and penetration through nodes and attributes. Going back to the same scenario, we want to identify all the links (href) associated with the ul element. The XPath command will be as follows.<\/p>\n

    [php]\u00a0 \/\/ul[@class =’listing-withImage headImage’]\/\/a[@href]\u00a0 [\/php]<\/p>\n

    Operator Or<\/h3>\n

    Syntax:<\/strong> \/\/tag[XPath Statement-1\u00a0or\u00a0XPath Statement-2]<\/p>\n\n\n\n\n\n\n\n
    A<\/strong><\/td>\nB<\/strong><\/td>\nResult<\/strong><\/td>\n<\/tr>\n
    False<\/td>\nFalse<\/td>\nNo Element<\/td>\n<\/tr>\n
    True<\/td>\nFalse<\/td>\nReturns A<\/td>\n<\/tr>\n
    False<\/td>\nTrue<\/td>\nReturns B<\/td>\n<\/tr>\n
    True<\/td>\nTrue<\/td>\nReturns Both<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

     <\/p>\n

    Another flexibility provided by the XPath Selenium locator is that we can give two XPath statements with an ‘or’ operator. The output follows the above tabular result. An example command to identify attribute class with text value “fourLine” and “threeLine” is as follows.<\/p>\n

    [php]\u00a0 \/\/a[@class=’fourLine’ or @class=’threeLine’]\u00a0 [\/php]<\/p>\n

    Operator And<\/h3>\n

    Syntax:<\/strong> \/\/tag[XPath Statement-1 and XPath Statement-2]<\/p>\n\n\n\n\n\n\n\n
    A<\/strong><\/td>\nB<\/strong><\/td>\nResult<\/strong><\/td>\n<\/tr>\n
    False<\/td>\nFalse<\/td>\nNo Element<\/td>\n<\/tr>\n
    True<\/td>\nFalse<\/td>\nNo Element<\/td>\n<\/tr>\n
    False<\/td>\nTrue<\/td>\nNo Element<\/td>\n<\/tr>\n
    True<\/td>\nTrue<\/td>\nReturns Both<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

     <\/p>\n

    The various output based on the boolean value of the 2 XPaths is provided in the table above. An example command prepared for this website is as follows.<\/p>\n

    [php]\u00a0 \/\/a[@class=’threeLine’ and contains(@href,’\/india\/’)]\u00a0 [\/php]<\/p>\n

    With this, we are trying to identify all the ‘a’\u00a0 elements that contain an attribute named class with value ‘threeLine’ and another attribute href which contains a text ‘\/india\/’.<\/p>\n

    \"XPath<\/p>\n

    Numerical Predicate<\/h3>\n

    Syntax:<\/strong> \/\/tag[position()=1]<\/p>\n

    We can locate nodes based on numerical functions like ‘position<\/strong>‘ provided as an example in the above syntax. For the Telegraph website, we can use any tag and find the position desired (provided that position of the tag exists). For example, for the ul tag that we saw in a previous example, we can point to all the ul tag wherever it has the first position in succession of its relative parent div tag in the tree structure of the HTML doc.<\/p>\n

    [php]\u00a0 \/\/div\/ul[position()=1]\u00a0 [\/php]<\/p>\n

    We can use another function “last<\/strong>” and identify all the ‘ul’ tags positioned last in the relative path.<\/p>\n

    [php]\u00a0 \/\/ul[last()]\u00a0 [\/php]<\/p>\n

    ‘Count<\/strong>‘ is a very useful predicate that can be used to count the number of attributes a tag contains. For example, if we want to identify the element ‘a’ in all the div tags which have a total count of attributes more than 5, we can use the below command.<\/p>\n

    [php]\u00a0 \/\/div[count(.\/@*)>5]\/\/a\u00a0 [\/php]<\/p>\n

    Node Relations –<\/h2>\n

    Another interesting way to create XPath is by using Node Relations that follows the family tree analogy. It is represented as node1\/<\/span>relation<\/span><\/i>::node2.<\/span> In other words, any node essentially maintains several relations with other nodes, for example, ancestor, child, descendant, parent, sibling, etc. We are going to mention commands for some of these relations.<\/p>\n

    [php]\u00a0 \/\/div[count(.\/@*)>5]\/\/child::ul\u00a0 [\/php]<\/p>\n

    This command means we are trying to point out all the ‘ul’ nodes which are a child to the ‘div’ tags having a count of attributes more than 5.<\/p>\n

    [php]\u00a0 \/\/div[count(.\/@*)>5]\/\/ancestor::div\u00a0 [\/php]<\/p>\n

    The above command points to all the div tag which is the ancestor to the ‘div’ tags having a count of attributes more than 5.<\/p>\n

    [php]\u00a0 \/\/div\/\/preceding::li\u00a0 [\/php]<\/p>\n

    Here we are going to locate all the ‘list’ tag which is preceding to the div tag.<\/p>\n

    What is Selenium –<\/h2>\n

    Selenium is a Web Browser Automation Tool. With the use of Selenium, we can browse a website just as a human would. We can click buttons, automate logins, give search text inputs and perform automation of several testing tasks as well.<\/p>\n

    Selenium XPath Commands –<\/h2>\n

    We have discussed so many different ways to prepare the XPath command pointing to specific nodes and attributes. However, it is incomplete without understanding what command to use to extract all the elements the XPath points to. Selenium package provides us with commands to extract a single element or multiple elements in the form of a list from the XPath location. To name a few, commands are<\/p>\n