11766

Error while running Map reduce on Hadoop 2.6.0 on Windows

Question:

I've setup a single node Hadoop 2.6.0 cluster on my Windows 8.1 using this tutorial - <a href="https://wiki.apache.org/hadoop/Hadoop2OnWindows" rel="nofollow">https://wiki.apache.org/hadoop/Hadoop2OnWindows</a>.

All daemons are up and running. I'm able to access hdfs using hadoop fs -ls / but I've not loaded anything, so there is nothing to show up as of now.

But when I run a simple map reduce program, I get the below erorr :

log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSumsByteArray(II[BI[BIILjava/lang/String;JZ)V at org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSumsByteArray(Native Method) at org.apache.hadoop.util.NativeCrc32.calculateChunkedSumsByteArray(NativeCrc32.java:86) at org.apache.hadoop.util.DataChecksum.calculateChunkedSums(DataChecksum.java:430) at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunks(FSOutputSummer.java:202) at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:163) at org.apache.hadoop.fs.FSOutputSummer.flushBuffer(FSOutputSummer.java:144) at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.close(ChecksumFileSystem.java:400) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106) at org.apache.hadoop.mapreduce.split.JobSplitWriter.createSplitFiles(JobSplitWriter.java:80) at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:603) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:614) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:492) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314) at wordcount.Wordcount.main(Wordcount.java:62)

Error from hadoop fs -put command :

<img alt="enter image description here" class="b-lazy" data-src="https://i.stack.imgur.com/mCzXc.png" data-original="https://i.stack.imgur.com/mCzXc.png" src="https://etrip.eimg.top/images/2019/05/07/timg.gif" />

Any advise would be of great help.

Answer1:

org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSumsByteArray is part of <a href="https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/NativeCrc32.c" rel="nofollow">hadoop.dll</a>:

JNIEXPORT void JNICALL Java_org_apache_hadoop_util_NativeCrc32_nativeComputeChunkedSumsByteArray (JNIEnv *env, jclass clazz, jint bytes_per_checksum, jint j_crc_type, jarray j_sums, jint sums_offset, jarray j_data, jint data_offset, jint data_len, jstring j_filename, jlong base_pos, jboolean verify) { ...

Unsatisfied link would indicate that you did not deploy Hadoop.dll in %HADOOP_HOME%\bin or the process loaded a wrong dll from somewhere else. Make sure the correct dll is placed in %HADOOP_HOME%\bin, and make sure this is the one loaded (use <a href="https://technet.microsoft.com/en-us/sysinternals/bb896653.aspx" rel="nofollow">process explorer</a>)

You should also see in the log the <a href="https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NativeCodeLoader.java" rel="nofollow">NativeCodeLoader</a> output:

private static boolean nativeCodeLoaded = false; static { // Try to load native hadoop library and set fallback flag appropriately if(LOG.isDebugEnabled()) { LOG.debug("Trying to load the custom-built native-hadoop library..."); } try { System.loadLibrary("hadoop"); LOG.debug("Loaded the native-hadoop library"); nativeCodeLoaded = true; } catch (Throwable t) { // Ignore failure to load if(LOG.isDebugEnabled()) { LOG.debug("Failed to load native-hadoop with error: " + t); LOG.debug("java.library.path=" + System.getProperty("java.library.path")); } } if (!nativeCodeLoaded) { LOG.warn("Unable to load native-hadoop library for your platform... " + "using builtin-java classes where applicable"); }

Enable DEBUG level for this component and you should see "Loaded the native-hadoop library" (since your code acts as if the hadoop.dll was loaded). The most likely problem is that the wrong one is loaded because is found first in the PATH.

Answer2:

hadoop.dll should also be added to C:/Windows/System32. I got it working with the help of this link - <a href="http://cnblogs.com/marost/p/4372778.html" rel="nofollow">http://cnblogs.com/marost/p/4372778.html</a> (Use an online translator to translate it to your native language)

Recommend

  • How to pass a list from view to a MVC controller
  • Calculate VAT helper method [closed]
  • java.lang.NoSuchMethodError: net.jpountz.util.Utils.checkRange
  • knockout Interdependent variables
  • How to count all values of columns where two conditions are satisfied using mysql?
  • How to loop through Java List object in JavaScript?
  • How to set the value for radio button while editing using jQuery?
  • Compile C++ source to a .dll
  • Send XML using XMLHttpRequest using GWT
  • SocketChannel: java.io.IOException: An existing connection was forcibly closed by the remote host
  • How to fix ruby error: sh: 1: make: not found
  • custom_annotation with npc coordinates in ggplot2
  • Java Jackson JSON parse into Map
  • “Permission denied” while connecting to a page, even with INTERNET permission
  • How To Pass Props From Redux Store into Apollo GraphQL Query
  • Heroku not using the specified Ruby 1.9.3
  • No Buffer Space available(maximum connection reached?) Form Postgres EDB Driver
  • GWT: Calling external JavaScript library (sjcl) method
  • React Native - get refs of custom components in listview from renderRow
  • backbone.js collection.get() undefined
  • Using JNA to Get NSWindow from JComponent on mac
  • Linux java wrapper and installer
  • Getting android_key_not_configured facebook error
  • Why do native C++ projects have a TargetFrameworkVersion?
  • When I try to upload and save an image file using Struts in eclipse it shows the file name as null
  • React Native + Redux: What's the best and preferred navigation?
  • Handling exceptions in a class library enveloping a device driver
  • joomla! 1.5.26 and “You are currently logged in to the private area of this site”
  • Print: Entry, “:CFBundleIdentifier”, Does Not Exist have tried most solutions
  • UML diagram generator in Visual Studio 2010
  • Embedded Glassfish JPA Datasource connection fail
  • Custom Tabgroup Appcelerator
  • why do I get the error when installing the gem 'pg'? [duplicate]
  • Get data from AJAX - How to
  • Initializer list vs. initialization method
  • Spring security and special characters
  • Trying to switch camera back to front but getting exception
  • Free memory of cv::Mat loaded using FileStorage API
  • Angular 2 constructor injection vs direct access
  • Programmatically clearing map cache