YARN "Hello World" application on your own Hadoop cluster

Run a cluster with a few mouse clicks and submit/execute YARN application

Hadoop YARN Tutorial Intro

Hadoop YARN Tutorial Part I – run a small Hadoop cluster and execute Yarn "Hello World" application (using Hadjo)

Hadoop YARN Tutorial Part II – Learn the fundamental components of YARN architecture via "Hello World" Yarn application (using Hadjo)

Hadoop YARN Tutorial Part III – "Hello World" application source code and components explanation.

(The screenshots are taken on MacOS but it really does not matter if you run Hadjo on Windows or Linux)
(The example is performed with Hadoop 2.8 and Java 1.8, but it will work with other configurations, ex. Hadoop 3.1 and Java 10)

1. Run Hadjo desktop GUI application. If you have not installed it, please go to quick start Hadjo. If you have no idea what Hadjo is, please go to Hadjo about page

2. Open the workspace "HouseElfs-2.8" (or use your own cluster setup if you prefer)

   

    

3. (optional) Build your image if not yet done by clicking the "Build" button at the bottom right corner. (More info on image build can be found at Build a cluster page)

   

    

4. Start your cluster. (More info on how to start your cluster can be found at Run your cluster)

   

    

5. Great! Your cluster is up. Let's see if HDFS and YARN services are all set. Click on your local HDFS Namenode Web UI and YARN ResourceManager Web UI.
   You should see the slave nodes in both screens. See the screenshots underneath for clarity:

   

    

   

    

6. We shall execute a basic Hadoop command on your cluster through Hadjo. We shall create /apps directory on HDFS if it does not exist.
   From the menu "Cluster Interactions" click on menu item "Execute command (master OS)".
   In the input field type hdfs dfs -mkdir -p /apps Here option "-p" means "Do not fail if the directory already exists":

   

    

7. We shall execute another basic Hadoop command on your cluster through Hadjo in order to list the HDFS root directories (we expect /apps too).
   From the menu "Cluster Interactions" click on menu item "Execute command (master OS)".
   In the input field type hdfs dfs -ls /

   

    

8. Download YARN "Hello World" appplication JAR. Source code is also available at YARN "Hello World" Maven project. If you wish you can prepare your own JAR, but for simplicity we have built it for you.

9. Next we shall upload to HDFS /apps directory the downloaded YARN JAR application.
   From the menu "Cluster Interactions" click on menu item "File Upload (HDFS)".
   Then browse the downloaded JAR and select it. Then you will be prompted to type the HDFS directory. Type "/apps" in the input field:

   

    

10. Next we shall upload the same JAR to master's OS file system. Usually it does not matter where the application will be submitted to Yarn from.
    From the menu "Cluster Interactions" click on menu item "File Upload (master OS)".
    Then browse the downloaded JAR and select it. Then you will be prompted to type the OS directory. Leave the default value of "/home/hadjo" in the input field:

    

     

11. We are almost there! It is time to execute the YARN application. It is available in HDFS and on the master's OS too.
    From the menu "Cluster Interactions" click on menu item "Execute command (master OS)".
    Place the following command in the input field hadoop jar yarn-hello-world.jar com.lazyweaver.yarn.Client:

    

     

12. The application is executed successfully if you see an end line of type Client: Finished "application_" with state "FINISHED"

    

     

Great!!! You have successfully ran a Hadoop cluster and executed a YARN application.
You are ready for the deep dive of what just happened behind the scenes in Part I and explain the fundamental components of YARN architecture.

Let's go to Part II