Cluster Nodes and Prerequisites

A fully distributed Hadoop 2.10 setup requires at least four CentOS 7 nodes: one NameNode and three DataNodes. The following table outlines the planned topology using VMware virtual machines.

Begin by creating a base VM with the required software stack: a dedicated hdfs user and group, a configured JDK, and the Hadoop 2.10 binary package. After preparing this template, clone it to produce the remaining VMs and adjust their hostnames and IP addresses accordingly.

Network and Host Configuration

Assign each node a short hostname derived from its IP suffix. Edit the /etc/hostname file on every machine:

vim /etc/hostname

Populate the /etc/hosts file on all nodes so that hostname-based communication is possible:

127.0.0.1   localhost
192.168.30.141 s141
192.168.30.142 s142
192.168.30.143 s143
192.168.30.144 s144

Passwordless SSH Setup

The NameNode (s141) must authenticate to all DataNodes over SSH without a password. Log in as the hdfs user on s141 and generate an RSA key pair:

ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa

Distribute the public key to the authorized_keys file on every node. On s141, append the key locally:

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

For the remote DataNodes, transfer the key using scp. If the remote .ssh directory does not exist, the command will fail; ensure it exists first (e.g., mkdir -p /home/hdfs/.ssh). The scp approach places the key as authorized_keys directly:

scp ~/.ssh/id_rsa.pub hdfs@s142:/home/hdfs/.ssh/authorized_keys
scp ~/.ssh/id_rsa.pub hdfs@s143:/home/hdfs/.ssh/authorized_keys
scp ~/.ssh/id_rsa.pub hdfs@s144:/home/hdfs/.ssh/authorized_keys

Set appropriate permissions, as incorrect permissions will silently break SSH key authentication:

chmod 700 ~/.ssh
chmod 644 ~/.ssh/authorized_keys

Do the same on the DataNodes after transferring the keys.

Hadoop Configuration Files

All XML configuration files reside in $HADOOP_HOME/etc/hadoop/. Modify them on s141 first, then propagate the entire hadoop configuration directory to the DataNodes.

core-site.xml – Set the default file system URI pointing to the NameNode:

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://s141:9000</value>
    </property>
</configuration>

hdfs-site.xml – Define block replication factor (commonly 3 for a three-node DataNode cluster):

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>dfs.replication</name>
        <value>3</value>
    </property>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>/data/hadoop/hdfs/namenode</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>/data/hadoop/hdfs/datanode</value>
    </property>
</configuration>

mapred-site.xml – Configure MapReduce to use YARN:

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>

yarn-site.xml – Define the YARN ResourceManager location and auxiliary service for MapReduce shuffle:

<?xml version="1.0"?>
<configuration>
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>s141</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
</configuration>

slaves (or workers in newer Hadoop releases) lists the hostnames for DataNodes:

s142
s143
s144

hadoop-env.sh – Ensure JAVA_HOME is defined:

export JAVA_HOME=/opt/soft/jdk

Distributing Configuration to DataNodes

Copy the entire $HADOOP_HOME/etc/hadoop directory from s141 to each DataNode:

scp -r $HADOOP_HOME/etc/hadoop hdfs@s142:/opt/soft/hadoop/etc/
scp -r $HADOOP_HOME/etc/hadoop hdfs@s143:/opt/soft/hadoop/etc/
scp -r $HADOOP_HOME/etc/hadoop hdfs@s144:/opt/soft/hadoop/etc/

Formatting the HDFS NameNode

Before the initial start, remove any previous HDFS metadata stored under the configured directories (e.g., /data/hadoop/hdfs or /tmp/hadoop-*). Also clean old log files from $HADOOP_HOME/logs on all nodes. Then, format the NameNode only on s141:

hdfs namenode -format

The command should complete with a success message indicating that the storage directories have been formatted.

Launching and Verifying the Cluster

Start all Hadoop services from the NameNode (s141):

start-dfs.sh
start-yarn.sh

Or, if using the legacy convenience script:

start-all.sh

Check running Java processes on each node with jps. On s141, you should see NameNode, ResourceManager, and possibly SecondaryNameNode. Each DataNode (s142, s143, s144) should show DataNode and NodeManager.

Access the NameNode web UI at http://192.168.30.141:50070 and verify that the cluster summary reports three live DataNodes. The YARN ResourceManager UI will be accessible at http://s141:8088.