YARN for Large Scale Computing: Beginner’s Edition

Introduction

YARN stands for Yet Another Resource Negotiator. It is a powerful resource management system for a horizontal server environment. It is designed to be more flexible and generic than the original Hadoop MapReduce system, making it an attractive choice for companies looking to implement Hadoop. It allows companies to process data types and run various data processing engines on the same cluster, like Spark, Hive, Pig, etc.

One of the critical features of YARN is its ability to handle multiple types of workloads. This is achieved using two key components: the Resource Manager and the Node Manager. The Resource Manager allocates resources to different applications, while the Node Manager manages the resources on individual nodes in the cluster.

YARN has many applications and use cases, including batch processing, interactive SQL, real-time streaming, machine learning, graph processing, long-running services, and big data processing.

Learning Objectives:

1. We will get a basic introduction to YARN.
2. Then you will learn about the pros and cons of using it.
3. Discuss the applications and use cases.
4. Understand the critical difference between YARN and MapReduce.
5. And finally, you will get hands-on with various coded and practical examples.

Table of Contents

1. Advantages and Disadvantages of YARN

2. Applications and Use Cases of YARN

3. Learn the Difference: YARN vs. Map Reduce

4. Understanding the Concept With Basic Code Examples

5. Conclusion

Advantages and Disadvantages of Using YARN

There are benefits to using YARN for extensive systems, but it has many disadvantages too. Below are some of the pros and cons.

Advantages of YARN:

• Improved Isolation and Management of Resources: It provides better isolation and management of resources, which can improve the stability and performance of the cluster.
• Flexibility: It can handle multiple types of workloads, which allows companies to run different types of applications on the same work, leading to improved consumption of resources and cost savings.
• Support for Many Computer Applications: It supports many distributed computing applications, including batch processing, interactive SQL, real-time streaming, and machine learning.
• Scalability: It can dynamically scale up and down the cluster, allowing organizations to adapt to changing workloads.
• Rolling Upgrades: It supports rolling upgrades, which makes companies improve their work without interruption to service.
• Multi-version Support: It allows organizations to run multiple versions of Hadoop on the same cluster.

Disadvantages of YARN:

Applications and Use Cases of YARN

It has many applications and use cases that are highly beneficial for current tech giants. Some of them are discussed below.

• Batch Processing: YARN can enable batch processing jobs, such as data warehousing and ETL (extract, transform, load). This allows organizations to process large amounts of data efficiently and cost-effectively.
• Interactive SQL: YARN can run interactive SQL queries on large data sets. This allows organizations to perform real-time analytics and gain insights from their data.
• Real-time Streaming: YARN can process real-time streaming data, such as log data, sensor data, and social media data. This allows organizations to gain insights and make decisions in near real-time.
• Machine Learning: YARN can be used to allocate machine learning algorithms on large data sets. This allows organizations to build and train models for various applications, such as image and speech recognition, natural language processing, and predictive analytics.
• Graph Processing: YARN can process graph data like social networks and recommendation systems. This allows organizations to analyze relationships and connections within their data.
• Long-Running Services: YARN can allocate long-running services like web services and data pipelines. This allows organizations to deploy and manage these services in a scalable and fault-tolerant manner.
• Big Data Processing: YARN enables different data processing engines to run on the same cluster. This allows organizations to process structured, semi-structured, and unstructured data using tools like Spark, Hive, Pig, etc.

Learn the Difference: YARN vs. Map Reduce

This section will discuss the difference between YARN and Map Reduce.

Understanding the Concept with Basic Code Examples

This section will discuss some basic examples that beginners can easily understand, like running a simple word count job using YARN and MapReduce or running a Pig script using YARN.

Example 1: Running a simple word count job using YARN and MapReduce:

$ hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount /input/text.txt /output/wordcount

In this example, we are using the built-in wordcount example job that comes with Hadoop. We pass two arguments, the input file /input/text.txt and the output directory /output/wordcount, where the job results will be stored. This job will count the number of occurrences of each word in the input text file and write the results to the output directory.

Example 2: Running a Pig script using YARN:

$ pig -x yarn -f /path/to/pig/script.pig

In this example, we use the Pig command-line interface to run a Pig script located at “/path/to/pig/script.pig” on a YARN cluster. The “-x yarn” option tells Pig to use YARN as the execution mode. This can be used to run Pig scripts that perform data processing, data cleaning, and data transformation tasks.

Example 3: Running a Hive query using YARN:

$ hive -e "SELECT * FROM mytable LIMIT 10"

In this example, we use the Hive command-line interface to run a SQL-like query on a table named “mytable.” The query selects all columns and rows from the table and returns only the first ten rows. The results of the question will be printed on the console. This can be used for interactive data querying and exploration.

Example 4: Running a Spark job using YARN:

$ spark-submit --class org.example.MySparkJob --master yarn --deploy-mode client --executor-memory 512m --num-executors 10 /path/to/jar/my-spark-job.jar arg1 arg2

In this example, we are using the Spark command-line interface to submit a Spark job named “MySparkJob” that is packaged in a jar file located at “/path/to/jar/my-spark-job.jar”. The job takes two arguments, “arg1” and “arg2”, passed to the main method. The “–master yarn” option tells Spark to use YARN as the cluster manager. “–deploy-mode client” means spark to run the driver program on the client machine, and “–executor-memory 512m” and “–num-executors 10” are used to configure the amount of memory per executor and the number of executors respectively.

Conclusion

In conclusion, YARN is a powerful resource management system for Hadoop. It allows different types of data processing workloads, such as batch processing, interactive queries, real-time streaming, and machine learning, to run on the same cluster. It gives a centralized management system for a Hadoop cluster, making it easier to manage and monitor resources. However, it’s important to note that it has limitations, such as added complexity, inefficiencies in resource allocation, and limited support by specific frameworks and tools.

Key takeaways of this article:
1. Firstly, we have discussed the YARN framework. It is a resource management system for large-scale distributed computing in Hadoop that allows multiple data processing frameworks to run on the same cluster.
2. Then, we discussed the advantages and disadvantages of using it. We have discussed scalability, the benefits of cluster-making, etc.
3. After that, we discussed its use cases and wrote the code of some of its primary applications, like using a pig script or running a simple command for word count.

The media shown in this article is not owned by Analytics Vidhya and is used at the Author’s discretion.