Biulding a Single-Node Hadoop Installation

For instructions on setting up a single-node Hadoop installation, please refer to: https://example.com/single-node-hadoop-setup

Creating a Hadoop Cluster

Cloning Virtual Machines

• Right-click on hadoop1 → Manage → Clone
• Click Next
• Select the current state of the virtual machine → Click Next
• Choose Create a full clone → Click Next
• Set the name for the cloned virtual machine → Click Finish
• Repeat for all three virtual machines and start them

Configuring the Three Virtual Machines

• Modify virtual machine names:

vim /etc/hostname

9. Modify domain name mapping:

vim /etc/hosts

11. Modify IP addresses:

node1 192.168.223.200
node2 192.168.223.201
node3 192.168.223.202

vim /etc/sysconfig/network-scripts/ifcfg-ens33

13. Restart network services:

systemctl restart network

15. Perform similar operations on all three virtual machines
16. Use ping command to check communicasion between virtual machines

ping hostname
ping node1
ping node2
ping node3

If all virtual machines can ping each other, the configuration is correct.

Setting Up SSH Passwordless Login

• On each virtual machine, generate RSA asymmetric keys:

ssh-keygen -t rsa
# Press Enter three times consecutively

22. In node1, copy the public key to the authorized_keys file:

cp id_rsa.pub authorized_keys
# This file will be created in the /root/.ssh/ directory

24. In node2, sync the authorized_keys file to node2 and append node2's public key:

# scp allows copying between virtual machines if they can communicate
# Copy the file
scp authorized_keys root@node2:/root/.ssh/
# Append the public key
cat id_rsa.pub >> authorized_keys

26. In node3, sync the authorized_keys file to node3, apend node3's public key, and sync the file back to node1 and node2:

# Copy to node3
scp authorized_keys root@node3:/root/.ssh/
# Append node3's public key
cat id_rsa.pub >> authorized_keys
# Sync back to node1 and node2
scp authorized_keys root@node1:/root/.ssh/
scp authorized_keys root@node2:/root/.ssh/

28. After completion, you can switch between virtual machines using ssh username@hostname/ip without a password:

ssh root@node2

Configuring Hadoop Cluster Files

core-site.xml

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://node1:9000</value>
    </property>
    
    <property>
        <name>hadoop.tmp.dir</name>
        
        <value>file:/opt/softtools/hadoop/tmp</value>
    </property>
</configuration>

yarn-site.xml

<configuration>
    <property>
        
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>node1</value>
    </property>
</configuration>

mapred-site.xml

<configuration>
    <property>
        
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>

hdfs-site.xml

<configuration>
    <property>
        
        
        <name>dfs.replication</name>
        <value>3</value>
    </property>
    <property>
        <name>dfs.namenode.name.dir</name>
        
        <value>file:/opt/softtools/hadoop/tmp/hdfs/name</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        
        <value>file:/opt/softtools/hadoop/tmp/hdfs/data</value>
    </property>
    <property>
        <name>dfs.namenode.http-address</name>
        <value>192.168.223.200:9870</value>
    </property>
</configuration>

Browser Testing

• In a browser, enter node1:9870
• In a browser, enter node1:8088

If the correct web pages are displayed, congratulations! You have successfully set up a Hadoop cluster.

Note: If the pages do not appear correctly, one possible reason is that the namenode was not reformatted.

# Delete the tmp and logs directories on all nodes.
# Then execute the following command on the master node:
hdfs namenode -format