How To Check If Further `scroll Down` Is Not Possible Using Selenium

Am using Selenium + python to scrap a page which has infinite scroll (basically scroll till max first 500 results are shown) Using below code, am able to scroll to bottom of the pa

Solution 1:

I'm using Selenium with Chrome, not Firefox, but the following worked for me:

1. capture page height before scrolling down;
2. scroll down using key down;
3. capture page height after scrolling down;
4. if page height was same before and after scrolling, stop scrolling

My code looks like this:

import time
from selenium import webdriver
from selenium.webdriver import Chrome
from selenium.webdriver.common.keys import Keys

driver = webdriver.Chrome()
driver.get("www.yourTargetURL.com")

reached_page_end = False
last_height = driver.execute_script("return document.body.scrollHeight")

whilenot reached_page_end:
      driver.find_element_by_xpath('//body').send_keys(Keys.END)   
      time.sleep(2)
      new_height = driver.execute_script("return document.body.scrollHeight")
      if last_height == new_height:
            reached_page_end = Trueelse:
            last_height = new_height

driver.quit()

Solution 2:

You can check document.body.scrollTop by before and after each scroll attempt if there is no data to fetch then this value will stay the same

distanceToTop = driver.execute_script("return document.body.scrollTop);")