21744

Can i use selenium with Scrapy without actual browser opening with python

I want to do some web crawling with scrapy and python. I have found few code examples from internet where they use selenium with scrapy.

I don't know much about selenium but only knows that it automates some web tasks. and browser actually opens and do stuff. but i don't want the actual browser to open but i want everything to happen from command line.

Can i do that in selenium and scrapy

Answer1:

You can use selenium with PyVirtualDisplay, at least on linux.

from pyvirtualdisplay import Display from selenium import webdriver display = Display(visible=0, size=(1024, 768)) display.start() browser = webdriver.Chrome()

Answer2:

Updated: PhantomJS is abandoned, and you can use headless browsers directly now, such like Firefox and Chrome!

<hr>

Use PhantomJS instead.

You can do browser = webdriver.PhantomJS() in selenium v2.32.0.

Recommend

  • MSBuild to run unit tests
  • problem with nginx + uwsgi in vhost mode, running django app
  • How to fix: The return type of an async method must be void, Task or Task [AppName]
  • How to insert multiple arrays into a database?
  • multiprocessing freeze computer
  • Is it really posible to close a PhoneGap App?
  • ActionBar three-dot dropdown opens at the wrong place
  • How can you Call a method from a diffrent Project, both in C++?
  • Making mono cross platform support for Task/Intent
  • Error:Execution failed for task ':app:dexDebug'. when importing ParseLoginUI as a module
  • Access 2007 forms with parameterized RecordSource
  • Attemping to change attributes of $(this) inside success function
  • Autohotkey script running program with command line arguments
  • get all processes in parallel
  • Android custom URI scheme incorrectly encoded when type in browser
  • HTML5 video only works in IE. The other browsers shows the black screen
  • what makes a request a new request in asp.net C#
  • Possible to “watch” both HAML and SASS at the same time?
  • Salesforce Different WSDL files and when to use
  • How to specify input and output paths from cmd.exe for a PowerShell script?
  • presentShareDialogWithParams posts to FB wall, but callback handler results say error
  • System.InvalidCastException: Specified cast is not valid
  • Alternative To body {overflow:scroll;} That Will Prevent Page Jostling/Wriggling?
  • Time complexity of a program which involves multiple variables
  • Setting up SourceTree to merge unity3d scenes with UnityYAMLMerge
  • Checking free space on FTP server
  • Read a local file using javascript
  • CSS Linear-gradient formatting issue accross different browsers
  • Using $this when not in object context
  • How do I fake an specific browser client when using Java's Net library?
  • How reduce the height of an mschart by breaking up the y-axis
  • Apache 2.4 and php-fpm does not trigger apache http basic auth for php pages
  • How to redirect a user to a different server and include HTTP basic authentication credentials?
  • Perl system calls when running as another user using sudo
  • Can I make an Android app that runs a web view in Chrome 39?
  • Apache 2.4 - remove | delete | uninstall
  • Run Powershell script from inside other Powershell script with dynamic redirection to file
  • -fvisibility=hidden not passed by compiler for Debug builds
  • LevelDB C iterator
  • Linking SubReports Without LinkChild/LinkMaster