Hadoop is a big data platform that stores and processes data using a network of computers. What makes Hadoop so appealing is that a cluster may be run on a limited number of low-cost dedicated servers. An individual can handle their data using low-cost hardwares as well and offers lots of scalability options. The system can begin with just one machine and can be grown to an endless number of servers.
The MapReduce component of the Hadoop ecosystem improves big data processing by using scattered and parallel algorithms. This programming technique is used to examine large amounts of data generated from internet users in social platforms and e-commerce. This blog explains how to use MapReduce in Hadoop. It will give readers an understanding of how large amounts of data are simplified and how MapReduce is applied in real-world scenarios.
What is Hadoop MapReduce?
MapReduce is a distributed computing processing technology and programming architecture based on Java. Map and Reduce are two fundamental tasks in the MapReduce algorithm. The “map” translates a piece of data into another set of data, breaking down different parts into tuples (key/value pairs). Second, there’s “reduce”, which takes the result of a map as an input and condenses the data tuples into a smaller set.
The ‘reduce’ work is always executed after the ‘map’ function, as the name MapReduce suggests. One of MapReduce’s main advantages is its ability to scale data processing across multiple processors. A MapReduce model uses primitives called mappers and reducers to process data. It’s not always easy to break down a data processing application into mappers and reducers.
Once we build a MapReduce application, we can scale it across hundreds, thousands, or even tens of thousands of machines in a cluster with only a configuration change. Because it is easy to scale, the MapReduce approach has been popular with programmers.
- MapReduce’s speed allows it to process large amounts of unstructured data in a short amount of time.
- The MapReduce framework is capable of dealing with failures with the help of its fault-tolerance feature
- Hadoop includes a scale-out function that allows users to process or store information at a lower cost.
- Hadoop provides a framework that is extremely scalable. Users can use MapReduce to run applications across several nodes.
- Data replicas are transmitted to different nodes within the network. This ensures that in the case of a failure, copies of the data would still be available.
- Multiple job-parts of the same dataset can be processed in parallel using MapReduce. This cuts down on the amount of time it takes to finish a task..
How MapReduce in Hadoop works
Understanding how MapReduce in Hadoop works will require a review of MapReduce Architecture.
1. MapReduce architecture
MapReduce’s architecture is made up of several components. A quick rundown of these components can help us better understand how MapReduce works.
Job: The work which needs to be completed or processed is referred to as a “job”.
Task: A task is a portion of the actual work that must be completed or processed. A MapReduce job is made up of numerous tiny tasks.
Job Tracker: This tracker is responsible for scheduling jobs and keeping track of all tasks allocated to it.
Task Tracker: This tracker plays the role of tracking tasks and reporting the status of tasks to the job tracker.
Input data: This is the data that will be processed during the mapping step.
Output data: The result of mapping and reducing is output data.
Client: A client is a software that submits jobs to MapReduce using an Application Programming Interface (API). MapReduce is capable of accepting jobs from a large number of customers.
Hadoop MapReduce Master: The Hadoop MapReduce Master divides jobs into different components for better feasibility.
Job-parts: They are the sub-jobs that occur as a result of the primary job’s division.
Clients submit jobs to the MapReduce Master in the MapReduce architecture. The job will then be divided into equal sub-parts by the master. The job portions would be utilized for MapReduce’s two primary tasks: mapping and reducing.
The developer has to write logic that meets the organization’s or company’s needs. The data from the input will be divided and mapped. After that, the intermediary data will be organized and combined. The result will be processed by the reducer that will provide a final output that will be placed in the Hadoop Distributed File System (HDFS).
How Do Task And Job Trackers Work?
Every job in MapReduce has two main components: task mapping and task reduction. The map task is responsible for breaking down jobs into sections and mapping input results. The shuffling and reduction of intermediate data into smaller units is the function of the reduced task.
The job tracker serves as a supervisor. It assures that all of our tasks get completed. Client-submitted jobs are scheduled using the job tracker. Task trackers will be assigned jobs. Each task tracker starts with a map task and then reduces it. The status of each assigned job is sent to the job tracker by task trackers.
1. Input Files
Input files hold the data for a MapReduce process, and are commonly kept in HDFS. The format of these files can be whatever you choose, including line-based log files and binary format.
2. Phases of MapReduce
There are three primary steps to the MapReduce programs: mapping, shuffling, and reducing. There is also a combiner phase, which is an optional step.
3. Mapping Phase
It is the system’s first phase. This phase consists of two steps: splitting and mapping. In the splitting process, a dataset is divided into equal parts (input splits). Hadoop includes a RecordReader that transforms input splits into key-value pairs using TextInputFormat.
In the mapping, the key-value pairs are used as inputs and this is the only data format that a mapper can read and understand. A coding mechanism is applied to such data blocks during the mapping stage. The mapper processes the key-value pairs in this phase and produces the outputs in the same format (key-value pairs).
4. Shuffling phase
It is a second phase that begins after the Mapping phase. It is divided into two parts: sorting and merging. The key-value pairs are ordered by using the keys in the sorting step. Merging assures the combination of key-value pairs.
The shuffling phase makes it easier to get rid of identical values and group them together. Values with comparable keys get placed together. This phase’s output, like the Mapping phase’s, will be keys and values.
5. RecordReader
It interacts with Hadoop MapReduce’s InputSplit and turns data into key-value pairs that the mapper can read. TextInputFormat is used by default to turn data into a key-value pair. Until the file reading is finished, the RecordReader talks with the InputSplit. Each line in the file is given a byte offset (a unique integer). These key-value pairs are then passed to the mapper to be processed further.
6. Reducer phase
The result of a shuffling phase is utilised as the inputs in the reduction phase. These inputs are processed further by the reducer, which reduces the intermediate values to smaller values. It gives an overview of the complete dataset. This phase’s output is saved in the HDFS.
7. Combiner phase
This is a phase that can be skipped if you want to optimise the MapReduce process. Duplicate outputs from the map outputs might be consolidated into a single output during this step. The combiner phase improves Job performance, which speeds up the Shuffling phase.
How MapReduce Organizes Work?
Hadoop splits the job into two jobs: Map tasks (Splits & Mapping) and Reduce tasks, (Shuffling, Reducing) as mentioned above. The entire execution process (both Map and Reduce) is governed by two of the following entities:
- Jobtracker: Can be understood as a master responsible for complete execution of the submitted job
- Multiple Task Trackers: Can be understood as slaves, each doing the assigned work.
Only one Jobtracker gets deployed on Namenode for every job submitted for execution in the system. However, there are many task trackers on Datanode for each job submitted for execution.
- A job is subdivided into many tasks, which are then distributed throughout a cluster’s data nodes.
- The job tracker is in charge of coordinating the activity by scheduling tasks to run on separate data nodes.
- Actual work execution is then handled by a task tracker, which is deployed on each data node that is performing a part of the work.
- It is the task tracker’s responsibility to send the job tracker the progress report.
- In addition, the task tracker transmits a ‘heartbeat’ signal to the Jobtracker on a regular basis to inform them about the system’s current situation.
- As a result, the work tracker keeps track of each job’s overall progress. If a task fails, the job tracker can reschedule it for a later date on a different task tracker.
Conclusion
The conventional methods were not designed to deal with the volume and complexity of incoming data, which eventually created a problem when dealing with big data. Hadoop MapReduce was brought in to help with this. Parallel processing, error handling, fault tolerance, logging, and reporting are some of the advantages of using MapReduce.
A MapReduce job consists of two main components: task mapping and task reduction. The map task is responsible for breaking down jobs into sections and mapping input results. As part of the reduced task, the intermediate data are shuffled and reduced in size. The purpose of this blog is to explain MapReduce’s complete working process along with how they plan their work.Hope you find it useful.