83796

Loop Dynamic Ajax Based Dropdown values using python selenium and grab data to mysql database

Question:

I am able to grab data if i apply static value of each dropdown but now i am trying to fetch dynamic values loop through each value of dropdown and store result to mysqldb.

<blockquote>

<strong>Issue :</strong> I am unable to loop each elements of drop-down and store result to database, if i pass static value for each three dropdown then i am able to do.

</blockquote>

<a href="https://i.stack.imgur.com/6bA1N.png" rel="nofollow"><img alt="enter image description here" class="b-lazy" data-src="https://i.stack.imgur.com/6bA1N.png" data-original="https://i.stack.imgur.com/6bA1N.png" src="https://etrip.eimg.top/images/2019/05/07/timg.gif" /></a>

<strong>Working Code for Static values</strong>

import time from selenium import webdriver import requests from bs4 import BeautifulSoup import MySQLdb url = "http://xlnindia.gov.in/frm_G_Cold_S_Query.aspx" browser = webdriver.Chrome() browser.get(url) #=========State==================== state = browser.find_element_by_id("ddlState") state.send_keys("GJ") #=========District==================== district = browser.find_element_by_id("ddldistrict") district.send_keys("AD2") #=========Taluka==================== category = browser.find_element_by_id("ddltaluka") category.send_keys("AMB") button = browser.find_element_by_id("btnSearch") button.click() # Open database connection db = MySQLdb.connect(host="localhost", # your host, usually localhost user="root", # your username passwd="", # your password db="test") # name of the data base cursor=db.cursor() time.sleep(10) browser.save_screenshot(browser.title + ".JPEG") html = browser.page_source soup = BeautifulSoup(html, "html.parser") table = soup.findChildren('table')[3] for row in table.findAll("tr")[1:]: cells = row.findAll("td") name = cells[0].find(text=True) city = cells[1].find(text=True) licence = cells[2].find(text=True) owner = cells[3].find(text=True) cursor.execute ("INSERT INTO distributors (name, city, licence, owner) VALUES (%s, %s, %s, %s);", (name, city, licence, owner)) #print(table.prettify()) #print(html) db.commit() db.close() browser.close() browser.quit()

<strong>Trying to pass dynamic values by looping each elements of dropdown</strong>

import time from selenium import webdriver import requests from bs4 import BeautifulSoup import MySQLdb url = "http://xlnindia.gov.in/frm_G_Cold_S_Query.aspx" browser = webdriver.Chrome() browser.get(url) #======================================================= #Array of Stats Dropdown states_array = [] el = browser.find_element_by_id('ddlState') for option in el.find_elements_by_tag_name('option'): states_array.append(option.get_attribute("value")) #print(states_array) #======================================================= #======================================================= #Array of Districts Dropdown district_array = [] el = browser.find_element_by_id('ddldistrict') for option in el.find_elements_by_tag_name('option')[1:]: district_array.append(option.get_attribute("value")) #print(district_array) #======================================================= state = browser.find_element_by_id("ddlState") district = browser.find_element_by_id("ddldistrict") category = browser.find_element_by_id("ddltaluka") # Open database connection db = MySQLdb.connect(host="localhost", # your host, usually localhost user="root", # your username passwd="", # your password db="test") # name of the data base cursor=db.cursor() taluka_array = [] for i in range(len(states_array)): state.send_keys(states_array[i]) for j in range(len(district_array)): district.send_keys(district_array[j]) district.click() el = browser.find_element_by_id('ddltaluka') for option in el.find_elements_by_tag_name('option')[1:]: taluka_array.append(option.get_attribute("value")) print(taluka_array) for k in range(len(taluka_array)): category.send_keys(taluka_array[k]) button = browser.find_element_by_id("btnSearch") button.click() html = browser.page_source soup = BeautifulSoup(html, "html.parser") table = soup.findChildren('table')[3] for row in table.findAll("tr")[1:]: cells = row.findAll("td") name = cells[0].find(text=True) city = cells[1].find(text=True) licence = cells[2].find(text=True) owner = cells[3].find(text=True) cursor.execute ("INSERT INTO distributors (name, city, licence, owner) VALUES (%s, %s, %s, %s);", (name, city, licence, owner)) db.commit() db.close()

Answer1:

First find the td tag to filter out states then get the values of option tags. Here you get all the states. For each state go to the url of that state and find districts. Districts are in the 2nd td tag so use find_next_sibling() and again get the values of the option tags. So you find all the districts too for each state. And rest of the things you already did.

import time from bs4 import BeautifulSoup from selenium import webdriver url = "http://xlnindia.gov.in/frm_G_Cold_S_Query.aspx" browser = webdriver.Chrome() browser.get(url) time.sleep(5) html = browser.page_source soup = BeautifulSoup(html, "lxml") states = [ x["value"] for x in soup.find("td", bgcolor="#ffe0c0").find_all("option") ] for state in states: browser.get(url + "?ST=" + state) time.sleep(5) html = browser.page_source soup = BeautifulSoup(html, "lxml") districts = [ x["value"] for x in soup.find("td", bgcolor="#ffe0c0").find_next_sibling().find_all("option") ] districts = districts[1:] for dist in districts: browser.get(url + "?ST=" + state) district = browser.find_element_by_id("ddldistrict") district.send_keys(dist) button = browser.find_element_by_id("btnSearch") button.click() browser.close() browser.quit()

Recommend

  • scope identity with linq
  • ContentFile not saved in Django model FileField
  • Hibernate HQL NULL comparison pattern
  • Application.match in VBA
  • How to split a string to rows in SQL without function
  • Points in Polygon, multiple polygons
  • How to convert one column's all values to separate column name in DataFrame of Pandas
  • The relationship could not be changed because one or more of the foreign-key properties is non-nulla
  • Give default value to Select Field in React js
  • ASP.NET, javascript: silly annoying apostrophe problem
  • How to use XMLReader to parse multiple, identically named attributes of XML element/sub-element
  • how to scroll with selenium
  • Clicking on link through Google with Selenium web driver
  • Symfony2 custom form
  • Unable to upload file using python selenium webdriver on http://www.dropzonejs.com
  • Join 2 tables where two sets of numbers overlap within the joining columns
  • NHibernate join and projection properties
  • How to implement 'category' based newsletter
  • select query and count based on condition
  • Logging Facebook using selenium
  • .htaccess rewriteRule conflict
  • Display name for nested complex type
  • How instagram or direct not change shape in icon launcher to circle in android oreo
  • APK 0 (zero) Device compatibility
  • MySQL table structure question
  • Wordpress Create Category AJAX Response
  • Mule ESB connecting to RabbitMQ
  • MapView: Could not find class A referenced from method B
  • Update transparancy of jFreeChart in real time with JSlider
  • Categories and SubCategories
  • AngularJS- How to pass data from one controller to another on ng-click()?
  • Custom font for MKAnnotationView Callout
  • Java thread for JDBC
  • PHPUnit: Expected status code 200 but received 419 with Laravel
  • WooCommerce - related products by tags and categories
  • How to resolve “ Multiplicity is not valid in Role” error?
  • Scrolling News Ticker Jquery - Issues
  • Using extern @class in order to add a category?
  • “A GKScore must specify a leaderboard.”
  • KeystoneJS: Relationships in Admin UI not updating