34614

can't get xpath to click on pop up in selenium

Question:

I'm trying to extract a few simple CIK codes from the SEC. If you run the code below, you will get a pop about a "survey". if you do it manually, you won't see it. it bombs my code. but since it is in selenium, I can't inspect it with chropath to get the xpath to click on the "NO". And I can't recreate the pop up in a normal browser. What do i do?

from selenium.webdriver.support import expected_conditions as EC from selenium.webdriver.support.wait import WebDriverWait from selenium.webdriver.common.by import By from selenium import webdriver from selenium.webdriver.common.keys import Keys import time from random import randint path ='C:\\Users\\Jason\\Google Drive\\python\\chromedriver.exe' ticker='alrm' main='https://www.sec.gov/search/search.htm' driver=webdriver.Chrome(path) tickers=['AAL','AAN','AAOI','AAPL'] # starts the process def get_CIK(ticker): driver.get(main) stock_code = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.ID, "cik"))) stock_code.click() stock_code.send_keys(ticker) driver.find_element_by_xpath("//input[@value='Find Companies']").click() # click on search buttom link = driver.find_element_by_xpath("/html[1]/body[1]/div[4]/div[1]/div[3]/span[1]/a[1]").get_attribute("href") # get link cik= link[link.index('CIK=')+4:link.index("owner")-1] # extract cik print cik for i in tickers: get_CIK(i)

Answer1:

What you occasionally see is the <em>"Foresee popup"</em> which would generally show up randomly.

There are 5 generic approaches I can think of:

<ul><li><strong>set a specific cookie</strong> that would disable foresee popup "pretending" you've dismissed it already. Which cookie to set exactly is an open question at the moment. There is this relevant thread about it as well: <a href="https://stackoverflow.com/questions/49517986/handle-random-foresee-popup-using-python-and-selenium" rel="nofollow">Handle random ForeSee popup using Python and Selenium</a></li> <li>

<strong>check for the presence of the "popup"</strong> all the way during the interaction with the web-site. The popup is not a classic "alert" in selenium sense and is <em>just an "overlay"</em> which has this HTML representation:

<div class="__acs " aria-labelledby="fsrHeading"> <div class="acsModalBackdrop acsAbandonButton" data-isbackdrop="true"> <div class="acsFocusFirst acsClassicInvite" tabindex="1" id="acsMainInvite" role="dialog" aria-labelledby="fsrHeading"> <div class="acsClassicInner" role="document"> <div class="acsLogoSpacer"><img src="//gateway.foresee.com/sites/sec-gov/production/trigger/sitelogo.png" class="acsSiteLogo" title="" alt=""> <img src="https://static.foresee.com/logos/foresee/150_67.png" class="acsNoDisplay" title="ForeSee" alt="ForeSee"> <div title="ForeSee" alt="ForeSee" class="acsVendorLogoSVG"></div> ...

You can then, for example, check for the presence of the "Decline" button and click it if it's present:

<a href="#" tabindex="2" class="acsInviteButton acsDeclineButton" title="" role="button"></a> </li> <li>

you can also <strong>"block" foresee JS scripts</strong> to be loaded by, for example, using a browsermobproxy where you would block all traffic from "foresee.com". Or, on a similar trajectory - you can start selenium with an ad blocker which would/should block "foresee" from out-of-the-box

</li> <li>

or, you can <strong>override the showInvite() function</strong> of the "foresee" global object:

driver.execute_script("window.FSR.showInvite = function () {};")

Note that you would need to call this script every time right after you navigate to a new page.

</li> <li>

after a bit of a reverse engineering I found that "foresee" JS object has this global config which contains lots of interesting information including a <strong>device blacklist</strong>:

device_blacklist: ["HTC_Rezound", "blackberry"]

You can then <a href="https://stackoverflow.com/a/29966769/771848" rel="nofollow">override the browser's user-agent</a> and pretend to be coming from a, say, <a href="http://www.useragentstring.com/pages/useragentstring.php?name=BlackBerry" rel="nofollow">Blackberry device</a>:

BLACKBERRY_UA = "Mozilla/5.0 (BlackBerry; U; BlackBerry 9900; en) AppleWebKit/534.11+ (KHTML, like Gecko) Version/7.1.0.346 Mobile Safari/534.11+" opts = Options() opts.add_argument("user-agent={0}".format(BLACKBERRY_UA)) driver = webdriver.Chrome(chrome_options=opts) </li> </ul><hr />

The second option is though technically more challenging and more error-prone, and would slow things down as you would constantly check for this popup to be present. Well, at least until you dismiss it.

The fourth option is quite promising but I have not fully tested it.

The last option, regardless of how crazy it sounds, actually works for me.

Recommend

  • Can not convert 13 digit UNIX timestamp in python
  • Appium Android UI testing - how to verify the style attribute of an element?
  • Stale element exception python
  • SELENIUM: how to check if a particular text is present in a particular attribute of HTML tag
  • Grails eager fetch doesn't retrieve all data
  • Sql indexes vs full table scan
  • Class implementation in a header file == bad style? [duplicate]
  • How to extract a number from a string [duplicate]
  • Google Calendar Api is not showing event list
  • Get all existing pointers to an object
  • How to get a list with description of all dba packages
  • text-align justify, cannot override
  • Angular - routerLinkActive and queryParams handling
  • Is it possible to get the word under the mouse cursor in a ``?
  • Access user's phone number on iOS 7
  • Simulate click Geckofx vb,net
  • Alternative To body {overflow:scroll;} That Will Prevent Page Jostling/Wriggling?
  • How do I pass the string value parameter of the selected list item from an auto-populated dropdown l
  • Play WS (2.2.1): post/put large request
  • Jquery UI tool tip close icon
  • FileReader+canvas image loading problem
  • PHPUnit_Framework_TestCase class is not available. Fix… - Makegood , Eclipse
  • Insert into database using onclick function
  • Deselecting radio buttons while keeping the View Model in synch
  • Why HTML5 Canvas with a larger size stretch a drawn line?
  • HTML download movie download link
  • Sending data from AppleScript to FileMaker records
  • Web-crawler for facebook in python
  • Rearranging Cells in UITableView Bug & Saving Changes
  • align graphs with different xlab
  • Warning: Can't call setState (or forceUpdate) on an unmounted component
  • Return words with double consecutive letters
  • KeystoneJS: Relationships in Admin UI not updating
  • AngularJs get employee from factory
  • Proper way to use connect-multiparty with express.js?
  • Load html files in TinyMce
  • Reading document lines to the user (python)
  • How to Embed XSL into XML
  • Conditional In-Line CSS for IE and Others?
  • Python/Django TangoWithDjango Models and Databases