Is it possible to debug Hadoop's source code in Eclipse?I'm not asking about the map reduce tasks. I want to see which part of the Hadoop source code is responsible for scheduling the map reduce tasks and how it works. Is there any mechanism by which it can be done?
You can download Hadoop project and integrate it to your eclipse, and use F5 or F6 to debug. You have different mode of debugging in eclipse:<ol> <li>F5 : Step by Step debugging</li> <li>F6 : Skips loops and Subroutines</li> <li>F7 : Skips the loop or subroutine and returns to the last cursor point.</li> <li>F8 : Execute and come out of debugging</li> </ol>
Or you can try yourself to understand the workflow by following step by step, you can begin from your
run() method in your main.
To answer your question: who does schedule the map task? <img src="https://i.stack.imgur.com/cXBny.png" alt="">
As you can see in this schema, files are divided by the
InputFormat class into fixed-size pieces called InputSplits. Each split is then given to a mapper, which is a node that was assigned a map task.
InputFormat class also provides a
RecordReader responsible for parsing the split and extracting records.Each record is passed to a map function as a (key, value) pair. So the
Mapper class is the one who call
Here is the workflow of the wordcount example:
<img src="https://i.stack.imgur.com/IssNF.png" alt="enter image description here">
Here are instructions from Apache Hadoop documentation. I haven't tried them out, but the instructions are good enough to get started.