Get contents of entire page using Selenium

In this article, we will discuss ways to get the contents of the entire page using Selenium. There can broadly be two methods for the same. Let’s discuss them in detail.

Method 1:

For extracting the visible text from the entire page, we can use the find_element_by_* methods which help us find or locate the elements on the page. Then, We will use the text method which helps to retrieve the text from a specific web element.

Approach

• Import module
• Instantiate driver
• Get content of the page
• Display contents scraped
• Close driver

Syntax:

driver.find_element_by_class_xpath(“/html/body”).text

To find or locate multiple elements on a page:

• find_element_by_link_text
• find_element_by_partial_link_text
• find_element_by_xpath
• find_element_by_tag_name
• find_element_by_class_name
• find_element_by_css_selector
• find_element_by_id
• find_element_by_name

We can use these above methods for finding or locating elements on a entire page. Most commonly used method is find_element_by_xpath which helps us to easily locate any elements. We will use appropriate methods as per our requirement.

Program:

Output:

Method 2:

There is one another method available for achieving our desired output. This one line will retrieve the entire text of the web page. Once we get the extracted data, with the help of file system, we will store the result inside the result.html file.

Approach:

• Import module
• Instantiate webdriver
• Get contents from the URL
• Open a file
• Save contents to a file
• Close file
• Close driver

Syntax:

driver.page_source

Program:

Output:

Click here to download the output file of above program.