Docker is a mandatory prerequisite for Hadjo. Behind the scenes Hadjo creates for you Docker images, configures a Docker local network and manages Docker instances in a their own network. To check if Docker is installed and configured well, please issue the following command "docker ps -a" on your command shell.
You should see something similar:
$docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
If there are any created containers they will be listed. An example with one container output is:
$docker ps -a
Note if there are no permission errors on Unix type OS - Docker gives the options to "sudo" or add an user to the Docker group. Hadjo requires the user which runs the application to be able to execute Docker commands without "sudo".
Please, check Docker setup for your OS.
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1df0795984e5 mysql:5.7.17 "docker-entrypoint.s…" 8 days ago Up 2 hours 0.0.0.0:53306->3306/tcp mysql
Although Hadjo supports Docker with Toolbox it is better (if possible) to run Docker natively on the corresponding OS.
Docker Toolbox is for older Mac and Windows systems (ex. Windows 8) that do not meet the requirements of Docker Desktop for Mac and Docker Desktop for Windows.
Double-click on the downloaded jar file or run "java -jar hadjo-version.jar".
Hadjo is a desktop platform for developers, DevOps, BI specialists, students or just anyone that is interested in developing, deploying, running or just studying cluster Hadoop applications. A Hadoop cluster managed by Hadjo is a group of Docker containers that are orchestrated to run together in a private network on your notebook, PC or Mac.
Prior knowledge of Docker to run Hadjo is not required. The nodes are all managed for you behind the scenes. Some understanding of Docker concepts and knowledge of basic commands may be of help but not required.
Welcome! We are thrilled that you want to learn Hadjo. The Get Started section will guide you through step by step:
hadjo@localhost:22 with password hadjo. Note : If you are using Docker with Toolbox, please check the configurations of the VM that runs Docker and find the IP to connect to and map SSH port if neededYour Big Data environment is ready...
hadjo@Dobby:~$
hadjo@Dobby:~$whoami
hadjo
hadjo@Dobby:~$
hadjo@Dobby:~$ssh hadjo@192.168.0.3
Your Big Data environment is ready...
hadjo@Winky:~$ jps
9106 Jps
202 NodeManager
126 DataNode
exit
hadjo@Dobby:~$ssh hadjo@192.168.0.4
Your Big Data environment is ready...
hadjo@Kreacher:~$ jps
9191 Jps
203 NodeManager
127 DataNode
exit
docker ps -a --format "{{.ID}} {{.Names}} {{.Status}} {{.Image}}"
$docker ps -a --format "{{.ID}} {{.Names}} {{.Status}} {{.Image}}"
94d7c2def3e0 Kreacher Up 26 minutes hadjo/houseelfs-2.8_cluster
91f8af5403a4 Winky Up 26 minutes hadjo/houseelfs-2.8_cluster
9b860ee94da0 Dobby Up 26 minutes hadjo/houseelfs-2.8_cluster
1df0795984e5 dtp_mysql Exited (255) 20 hours ago mysql:5.7.17
A workspace is related to a single cluster - its nodes, selected software on them (ex. Hadoop version, Java version, etc.) and cluster environment (ex. network settings). The screens that are strongly related for a workspace are "Cluster Environment" and "Nodes". There is always and active workspace when working with Hadjo.
New workspace - creates a new empty workspace with no nodes. The current active workspace remains unchanged. You can switch to your new workspace any time. "Cluster Environment" section will be populated with default values all of which can be modified.
Open workspace - switches to an existing workspace. The cluster must be stopped in order to open a new workspace. Switching between workspaces does not affect the cluster data and logs. For example : whatever you put on HDFS from one workspace will be still there when opening the same workspace back. A cluster is bound to its workspace. A different workspace means a different cluster. There is a clean separation of data between clusters and they do not know of each other.
Save as new workspace - creates a new workspace with the nodes and settings of the current one. You can switch to your new workspace any time. Data related to the running cluster of the current workspace is not copied.
Delete workspace - deletes all data and nodes from the chosen workspace. The Docker images is also removed behind the scenes. Can not delete and active workspace.
Exit - exits the application. If instances are running, they will not be stopped. On next start running instances will be detected and put in managed context.
Nodes - opens the main screen with the existing nodes
Start (all) - Starts the whole cluster by rising up the "master" node first. If some nodes are already running, the command will start all non-running instances. The already running instances will not be affected.
Stop (all) - Stops the whole cluster by taking down the salve nodes first. The "master" node is put on halt last. Already stopped instances are not affected, the command will ignore them.
CPU/Mem Statistics - Displays the current state of CPU and memory usage of the instances in accordance to the allocated resources on Docker container manager.
Build - Opens the screen for building the main Docker image of the current active workspace
Clean - Cleans the content of the active workspace's instances - applications logs and mounted content. On next instance start the content will be populated as if as new.
Execute command (master OS) - executes a Linux command on "master" node and display an output File upload (master OS) - Browse and choose a file from your own PC, laptop or Mac. You will be prompted to specify a full path on the "master" where the file will be copied.
File upload (HDFS) - Browse and choose a file from your own PC, laptop or Mac. You will be prompted to specify a relative or full path on HDFS where the file will be placed. It will be available on your Hadoop cluster. Note : non-existent HDFS directories of your supplied path will be created!
Language - changes the application language. By default English is loaded.
Cluster Environment - The settings in this area are strongly related to the Docker image creation (done behind the scenes by Hadjo) and Docker network setup (also done by Hadjo through Docker). The image users, groups and passwords are related to the Linux cluster instances (using the built Docker image). The local mount directory should be readable and writable by the Docker containers and for that reason by default it is under the user's home directory! "Image OS user" and "Image OS user password" are important if you plan to use SSH access to "master" node. The container network name and range have a cross-workspace context. In most of the cases it is just fine to use the same name and IP range when switching between workspaces.
Cluster Software - by default the nodes installable software (ex. Hadoop) comes with basic default settings (described in another section). If you need to modify Hadoop configurations, ex. increase the replication factor from 1 to 2, just edit and save hdfs-site.xml. See the screenshot below for an example of changing hdfs-site.xml of Hadoop version 2.9.2!
Section devoted to Hadoop application development - MapReduce, custom YARN applications and more...
How Hadoop YARN works and your first YARN application - use Hadjo to run YARN "Hello World". This article will explain YARN fundamental components in practice and head you start to your own YARN application development... Other topics coming up pretty soon ...The table below gives selectable software combinations that have been tested in several Hadjo clusters.
| Software | Java 8 | Java 9 | Java 10 |
|---|---|---|---|
| Hadoop 2.8 |  |
|
|
| Hadoop 2.9 | |
|
|
| Hadoop 2.10 | |
|
|
| Hadoop 3.0 | |
|
|
| Hadoop 3.1 | |
|
|
| Hadoop 3.2 | |
|
|
| Hadoop 3.3 | |
|
|