13820

urllib2 gives HTTP Error 400: Bad Request for certain urls, works for others

Question:

I'm trying to do a simple HTTP get request with Python's urllib2 module. It works sometimes, but other times I get HTTP Error 400: Bad Request. I know it's not an issue with the URL, because if I use urllib and simply do urllib.urlopen(url) it works fine - but when I add headers and do urllib2.urlopen() I get Bad Request on certain sites.

Here is the code that's not working:

# -*- coding: utf-8 -*- import re,sys,urllib,urllib2 url = "http://www.gamestop.com/" headers = {'User-Agent:':'Mozilla/5.0'} req = urllib2.Request(url,None,headers) response = urllib2.urlopen(req,None) html1 = response.read()

(gamestop.com is an example of a URL that does not work)

Some different sites work, some don't, so I'm not sure what I'm doing wrong here. Am I missing some important headers? Making the request incorrectly? Using the wrong User-Agent? (I also tried using the exact User-Agent of my browser, and that didn't fix anything)

Thanks!

Answer1:

You've got an extra colon in your headers.

headers = { 'User-Agent:': 'Mozilla/5.0' }

Should be:

headers = { 'User-Agent': 'Mozilla/5.0' }

Recommend

  • Why doesn't my WTForms-JSON form update correctly?
  • stuck with encodings in python with BeautifulSoup
  • Permission to overwrite files
  • Python-3 Request parameter encoding error
  • Removing html tags using python?
  • Scrap the absolute URL instead of a relative path in python
  • Beautiful Soup findAll doesn't find them all
  • Basic defensive programming [duplicate]
  • TFS - how do I sum child task hours to parent
  • How to make R's read_csv2() recognise the text characters properly
  • WPF ICommand CanExecute(): RaiseCanExecuteChanged() or automatic handling via DispatchTimer?
  • How solve “Qt: Untested Windows version 10.0 detected!”
  • Android - Material Design - NavigationView - How to put vertical scroll?
  • Content-Length header not returned from Pylons response
  • Python urlparse: small issue
  • Jquery UI tool tip close icon
  • Sony Xperia Z Tablet not found by adb
  • How to recover from a Spring Social ExpiredAuthorizationException
  • Javascript Callbacks with Object constructor
  • SSO with signing and signature validation doesn't work
  • How to apply VCL Styles to DLL-based forms in Inno Setup?
  • Arrow is showed instead of the material design version hamburger icon. Why doesn't syncState in
  • How can I use Kendo UI with Razor?
  • How can I estimate amount of memory left with calling System.gc()?
  • Warning: Can't call setState (or forceUpdate) on an unmounted component
  • Circular dependency while pushing http interceptor
  • Run Powershell script from inside other Powershell script with dynamic redirection to file
  • Matrix multiplication with MKL
  • AngularJs get employee from factory
  • How to set the response of a form post action to a iframe source?
  • Hits per day in Google Big Query
  • Angular 2 constructor injection vs direct access
  • Java static initializers and reflection
  • Change div Background jquery
  • Android Google Maps API OnLocationChanged only called once
  • File not found error Google Drive API
  • Can't mass-assign protected attributes when import data from csv file
  • costura.fody for a dll that references another dll
  • UserPrincipal.Current returns apppool on IIS
  • Converting MP3 duration time