39154

twisted.internet.error.ConnectError when run scrapy spider

I'm using scrapy to run a spider and get the following errors:

DEBUG: Retrying http://xixichengyuanlc.fang.com/esf/> (failed 2 times): An error occurred while connecting: [Failure instance: Traceback (failure with no frames): : Connection to the other side was lost in a non-clean fashion: Connection lost.

I have ever successfully run this spider for several times but I want to use some user agents to run faster and get the errors above. At first I thought there might be something wrong with my user agents, so I checked but still can't figure out.And then I want to try the former spider again but still get the same errors.

below is my <strong>settings.py</strong>

# Scrapy settings for soufang project SPIDER_MODULES = ['soufang.spiders'] NEWSPIDER_MODULE = 'soufang.spiders' DEFAULT_ITEM_CLASS = 'soufang.items.Community_info' ITEM_PIPELINES = ['soufang.pipelines.MySQLStorePipeline'] #DOWNLOADER_MIDDLEWARES={ #'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': None, #'soufang.misc.middlewares.CustomUserAgentMiddleware':400}

Answer1:

The ITEM_PIPELINES setting is not a list, but a dict:

ITEM_PIPELINES = { 'soufang.pipelines.MySQLStorePipeline': 100 }

Other than that, I can't say what's wrong exactly. I don't see you have set USER_AGENT in your settings? Also, paste the full log.

Recommend

  • Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2
  • passing parameter to server in ExtJs
  • Python equivalent of Scala's exists() function?
  • Extjs, handling success or failure when doing a standard submit in a form
  • Is there a equivalent to JSON.Net in Java? [duplicate]
  • ValueError: Found arrays with inconsistent numbers of samples
  • How to implement simple validation in Scala
  • How to change default stop edit behavior in jtable
  • Specify HTTPS for custom WCF Binding
  • end daemon processes with multiprocessing module
  • user data scripts fails without giving reason
  • Cannot find “Grammar.txt” in python-sphinx
  • Rodeo UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 0: ordinal
  • Making Google Visualization - Annotation Chart to work in GWT
  • PayPal API Listener Website Payments Standard URI
  • How can I tell a form not to dispose a particular control when it closes?
  • Hash Code in SQL Server?
  • Python ImageIO Gif Set Delay Between Frames
  • as3-flash: any way to access all the instances placed in different frames from document class?
  • Ember.js model to be organised as a tree structure
  • Intel-64 and ia32 atomic operations acquire-release semantics and GCC 5+
  • Read text file that is not in the main package in a runnable jar
  • Check for zero lines output from command over SSH
  • nonblocking BIO_do_connect blocked when there is no internet connected
  • How can I speed up CURL tasks?
  • How can I set a binding to a Combox in a UserControl?
  • Jackson Parser: ignore deserializing for type mismatch
  • print() is showing quotation marks in results
  • MailKit: The IMAP server replied to the 'EXAMINE' command with a 'BAD' response
  • OpenGL ES texture problem, 4 duplicate columns and horizontal lines (Android)
  • Exception “firebase.functions() takes … no argument …” when specifying a region for a Cloud Function
  • Highlight one bar in a series in highcharts?
  • Update CALayer sublayers immediately
  • Is my CUDA kernel really runs on device or is being mistekenly executed by host in emulation?
  • TFS: Get latest causes slow project reloading
  • ActionScript 2 vs ActionScript 3 performance
  • Display Images one by one with next and previous functionality
  • How can I estimate amount of memory left with calling System.gc()?
  • Calling of Constructors in a Java
  • PHP: When would you need the self:: keyword?