18207

hadoop /usr/bin/env: python: No such file or directory

Question:

I am trying to run a hadoop streaming server with following command from a shell script

hadoop jar /usr/local/hadoop/contrib/streaming/hadoop-0.19.2-streaming.jar -input $1 -output Twitter/Net.pegasus -mapper 'mapper.py Reverse' -reducer NONE -file mapper.py hadoop jar /usr/local/hadoop/contrib/streaming/hadoop-0.19.2-streaming.jar -input $1 -output Twitter/Net.exclude -mapper 'mapper.py Reverse' -reducer reducer.py -file mapper.py -file reducer.py -file ../twitter/exclude.txt hadoop jar /usr/local/hadoop/contrib/streaming/hadoop-0.19.2-streaming.jar -input $1 -output Twitter/Net.complete -mapper 'mapper.py Reverse' -reducer reducer.py -file mapper.py -file reducer.py

I am getting following error

/usr/bin/env: python2.5: No such file or directory java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 127 at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:362) at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:576) at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:135) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57) at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136) at org.apache.hadoop.mapred.Child.main(Child.java:249)

However the higher version of python is already installed

$ which python /usr/bin/python $ python --version Python 2.7.3

I read on some other post that by "apt-get install python2.5" it will work but that package isnt available and I also tried adding #!/usr/bin/env python to the top of my python script and it didnt work either

<a href="https://stackoverflow.com/questions/4460522/hadoop-streaming-job-failed-error-in-python" rel="nofollow">Hadoop Streaming Job failed error in python</a>

Answer1:

$ which python /usr/bin/python $ python --version Python 2.7.3

Indicate you have a higher version of Python installed than default. You have to make sure this is the same case for every node in the cluster. For example, I work with a cluster Redhat 6.4 version cluster and the default Python is Python 2.6 I think as default, but I have customized the master node a bit and it is different from the datanode. So you need to configure your mapper to call a python that exist on every node.

put

#!/usr/bin/python

at the beginning of the python script always work for me.

Recommend

  • AS3 error #1009
  • Error Generating Certificate SHA-1
  • CCSpriteFrame warning..AnchorPoint won't work as expected.Regenerate the .plist?
  • Not getting each error email alert from logstash 1.5.4
  • True coding convention about usings/try/catches
  • Using a Pipe to Write and Read Array of Int
  • How do i verify form input before displaying record [duplicate]
  • Android : Draw TextEdit(s) or TextView(s) when user enter number of them.[exp: 6 then draw 6 TextVie
  • How to sort list items by their priority in C?
  • How to return exit status and output of system commands in Racket?
  • awslogs-agent-setup.py not working on Ubuntu 17.10 (artful)
  • can I redirect the output of this program to script?
  • Running multiple adb commands with python Popen or os.system
  • Run wine commands from Python 3.6 on Mac OSX
  • Process a set of files from a source directory to a destination directory in Python
  • Open PDF to Bookmark/Named Destination?
  • How to depends of a system command with python/distutils?
  • subprocess ssh command fails for some commands but not others (command works in terminal)
  • Python: ValueError: Mixing iteration and read methods would lose data
  • beginner-opening explorer to show folder contents
  • Are there problems with ActivityInstrumentationTestCase2 in Android 2.1?
  • Prevent OS X from going to sleep with Python?
  • Node.js decipher not works for other ciphers' ciphertext
  • Android Studio 2 error :app:compileDebugJavaWithJack
  • LibGDX GwtApplication Exception (TypeError) in HTML Deployment
  • Python subprocess communicate kills my process
  • Delete specific queue when client's idle time in activemq
  • Open Video file in VLC through Python
  • How to fork and exec a server and wait until it's ready?
  • os.system not working, but typing the same thing into the command prompt works
  • How to start a Mysql server in Java programatically?
  • Programmatically check if PHP is installed using Python
  • Downloading files from Google Storage using Spark (Python) and Dataproc
  • URLConnection doesn't work since API 10 and higher?
  • What does 'Language neutral' mean with regard to MAKELANGID?
  • Problem deserializing objects from cache on MyBatis 3/Java
  • Display issues when we change from one jquery mobile page to another in firefox
  • Python - Map / Reduce - How do I read JSON specific field in using DISCO count words example
  • Hibernate gives error error as “Access to DialectResolutionInfo cannot be null when 'hibernate.
  • coudnt use logback because of log4j