77666

Cannot setup Scrapy on windows

Question:

I tried to setup Scrapy on Windows 7 by steps described at <a href="http://doc.scrapy.org/en/latest/intro/install.html" rel="nofollow">http://doc.scrapy.org/en/latest/intro/install.html</a> . On my PC was installed Python 3.5.1. Although Scrapy not support this python version it was installed successfully with latest Anaconda but fails to run spider script. I find that Scrapy only works with Python 3.3.+ version so uninstall version 3.5.1, uninstall Anaconda, install python 3.3.5, install pywin32 and install pip. pip fails pip install Scrapy, so I install Anaconda and run conda install -c scrapinghub scrapy Scrapy installed, but I saw that libs installed was for python 3.5 like: scrapy: 1.1.0-py35_0

Now I run the

c:\python\olxscrapy>scrapy crawl OlxCatalogSpider

and get error

File "C:\Anaconda3\lib\site-packages\twisted\internet\stdio.py", line 30, in module> from twisted.internet import _win32stdio ImportError: cannot import name '_win32stdio'

How to make Scrapy run with python 3.3.+

Answer1:

On this blog:

<a href="https://blog.scrapinghub.com/2016/05/25/data-extraction-with-scrapy-and-python-3/" rel="nofollow">https://blog.scrapinghub.com/2016/05/25/data-extraction-with-scrapy-and-python-3/</a>

it says Scrapy on Python 3 doesn't work in Windows environments yet

Edit: I recently installed scrapy on Ubuntu for Python 3.5 and received a lot of errors. The errors stopped after: "sudo apt-get install python3.5-dev".

Answer2:

Try to create a virtual env:

pip install virtualenv (instalation) virtualenv -p python3.3.5 envName (creation with specific python version) source ./envName/bin/activate (activate virtual env)

This way you can guarantee that's the right python version. Also scrapy has some requirements that can't be installed via pip and this may cause your pip install scrapy to fail

So install at your computer: python-dev libxslt1-dev libxslt1.1 libxml2-dev libxml2 libssl-dev

After this you finaly be able to install scrapy via pip inside your virtual env (probably)

Sry for my poor English isn't my native lang. Hope this work =]

Answer3:

I add the follow package and it works:

pip install twisted-win==0.5.5

Answer4:

Installation of Scrapy on Windows may facing error while installing Twisted.

<ol><li>Download Twisted according to your Python and windows version on this site <a href="http://www.lfd.uci.edu/~gohlke/pythonlibs/#twisted" rel="nofollow">http://www.lfd.uci.edu/~gohlke/pythonlibs/#twisted</a></li> <li>Turn to your download folder and pip install <downloaded filename></li> <li>pip install scrapy</li> </ol>

Recommend

  • Unable to install mysqlclient using pip3 on MacOS sierra
  • Where should I store M2Crypto files
  • tensorflow import error in anaconda
  • May i install caffe or caffe2 on Google Colaboratory?
  • Django - failed to push some refs to git@heroku.com
  • Spark Installation and Configuration on MacOS ImportError: No module named pyspark
  • Django : Unable to run server
  • How do I manually set the limits of a holoview's colorbar?
  • Unable to import caffe
  • Accessing twitter using tornado.httpclient using proxy [closed]
  • How to keep data_files between package updates?
  • Flask-Bcrypt - AttributeError: 'module' object has no attribute 'ffi' - deployed
  • TypeError : 'NoneType' object not callable when using split in Python with BeautifulSoup
  • Install Poppler for Python on Mac
  • Fabric set environment variable globally
  • distutils ignores changes to setup.py when building an extension?
  • Injecting arguments in scrapy's pipeline
  • tf.assign to variable slice doesn't work inside tf.while_loop
  • Kivy Recycleview. How to use it?
  • Get own applications in Django
  • Anaconda installation of Tensorflow fails with Debian 8 VirtualBox
  • how to convert this date string to “2011-02-15T12:00+00:00” python datetime object
  • I need Anaconda 32bit and 64bit
  • Can't install scipy
  • How to modify search result page given by Solr?
  • Error while installing ggplot in python
  • How to install PyAutoGUI
  • Scrapy recursive link crawler
  • Run Powershell script from inside other Powershell script with dynamic redirection to file
  • Load html files in TinyMce
  • How can I get HTML syntax highlighting in my editor for CakePHP?
  • Hits per day in Google Big Query
  • coudnt use logback because of log4j
  • FormattedException instead of throw new Exception(string.Format(…)) in .NET
  • Linking SubReports Without LinkChild/LinkMaster
  • XCode 8, some methods disappeared ? ex: layoutAttributesClass() -> AnyClass
  • Observable and ngFor in Angular 2
  • How to Embed XSL into XML
  • UserPrincipal.Current returns apppool on IIS
  • Conditional In-Line CSS for IE and Others?