Building a Hadoop+Spark+Zookeeper+HBase+Hive Cluster: Complete Setup Guide

Environment Overview

Cluster Architecture

This cluster uses three machines configured as master-slave topology. The following table summarizes the hardware specificasions:

Node	Role	CPU Cores	RAM	Disk Space
master	NameNode + ResourceManager	4	16GB	500GB
slave1	DataNode + NodeManager	4	16GB	500GB
slave2	DataNode + NodeManager	4	16GB	500GB

Component Versions

Component	Version	Purpose
CentOS	6.8	Operating System
JDK	1.8	Hadoop/Spark dependency
Hadoop	2.8.x	Distributed storage and processing
Spark	Pre-built with Hive support	In-memory processing engine
Scala	2.12.x	Spark dependency
Zookeeper	3.4.x	Distributed coordination
HBase	1.2.x	NoSQL database
Hive	2.1.x	SQL data warehouse

Download Links

Hadoop: http://www.apache.org/dyn/closer.cgi/hadoop/common

Spark: http://spark.apache.org/downloads.html

Scala: http://www.scala-lang.org/download

JDK: http://www.oracle.com/technetwork/java/javase/downloads

HBase: http://mirror.bit.edu.cn/apache/hbase/

Zookeeper: http://mirror.bit.edu.cn/apache/zookeeper/

Hive: http://mirror.bit.edu.cn/apache/hive/

Cluster Pre-Configuration

Hostname Configuration

Edit /etc/sysconfig/network on each machine:

vim /etc/sysconfig/network

Modify the HOSTNAME value for each node:

HOSTNAME=master    # on master node
HOSTNAME=slave1    # on slave1 node
HOSTNAME=slave2    # on slave2 node

Reboot each node for changes to take effect.

IP-Hosst Mapping

Edit /etc/hosts on all machines:

vim /etc/hosts

Add the following entries:

192.169.0.23 master
192.169.0.24 slave1
192.169.0.25 slave2

Distribute this file to all cluster nodes:

scp /etc/hosts root@slave1:/etc
scp /etc/hosts root@slave2:/etc

SSH Passwordless Authentication

Generate RSA keys on all machines:

ssh-keygen -t rsa -P ''

Create the authorized keys file:

touch /root/.ssh/authorized_keys

Collect public keys from all nodes and append them to authorized_keys:

cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys

For each additional node, copy its public key content into the authorized_keys file on all other nodes.

Distribute the authorized_keys file:

scp /root/.ssh/authorized_keys root@slave1:/root/.ssh
scp /root/.ssh/authorized_keys root@slave2:/root/.ssh

Verify passwordless login:

ssh slave1
exit
ssh slave2
exit

Firewall Configuration

Disable the firewall on all nodes:

service iptables stop
chkconfig iptables off

Time Synchronization

Configure NTP service for time synchronization across the cluster. Refer to official NTP documentation for detailed setup instructions.

Shell Aliases

Add useful aliases to ~/.bashrc for convenience:

vim ~/.bashrc

# Navigation aliases
alias chd='cd /opt/hadoop/hadoop2.8'
alias chb='cd /opt/hbase/hbase1.2'
alias chi='cd /opt/hive/hive2.1'
alias czk='cd /opt/zookeeper/zookeeper3.4'
alias csp='cd /opt/spark/spark2.0-hadoop2.4-hive'

# Hadoop control aliases
alias fhadoop='/opt/hadoop/hadoop2.8/bin/hdfs namenode -format'
alias starthadoop='/opt/hadoop/hadoop2.8/sbin/start-all.sh'
alias stophadoop='/opt/hadoop/hadoop2.8/sbin/stop-all.sh'

# HBase control aliases
alias starthbase='/opt/hbase/hbase1.2/bin/start-hbase.sh'
alias stophbase='/opt/hbase/hbase1.2/bin/stop-hbase.sh'

# Zookeeper control aliases
alias startzk='/opt/zookeeper/zookeeper3.4/bin/zkServer.sh start'
alias stopzk='/opt/zookeeper/zookeeper3.4/bin/zkServer.sh stop'
alias statuszk='/opt/zookeeper/zookeeper3.4/bin/zkServer.sh status'

# Spark control aliases
alias startsp='/opt/spark/spark1.6-hadoop2.4-hive/sbin/start-all.sh'
alias stopsp='/opt/spark/spark1.6-hadoop2.4-hive/sbin/stop-all.sh'

Activate the changes:

source ~/.bashrc

Environment Variables

Add the following to /etc/profile:

# Java Configuration
export JAVA_HOME=/opt/java/jdk1.8
export JRE_HOME=/opt/java/jdk1.8/jre
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib

# Scala Configuration
export SCALA_HOME=/opt/scala/scala2.12

# Spark Configuration
export SPARK_HOME=/opt/spark/spark1.6-hadoop2.4-hive

# Zookeeper Configuration
export ZK_HOME=/opt/zookeeper/zookeeper3.4

# HBase Configuration
export HBASE_HOME=/opt/hbase/hbase1.2

# Hadoop Configuration
export HADOOP_HOME=/opt/hadoop/hadoop2.8
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"

# Hive Configuration
export HIVE_HOME=/opt/hive/hive2.1
export HIVE_CONF_DIR=${HIVE_HOME}/conf

export PATH=.:${JAVA_HOME}/bin:${SCALA_HOME}/bin:${SPARK_HOME}/bin:${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin:${ZK_HOME}/bin:${HBASE_HOME}/bin:${HIVE_HOME}/bin:$PATH

Hadoop Installation

JDK Setup

Remove the existing OpenJDK if present:

rpm -qa | grep java
rpm -e --nodeps <openjdk-package-name>

Extract and install the Oracle JDK:

tar -xvf jdk-8u144-linux-x64.tar.gz
mv jdk1.8.0_144 /opt/java
mv /opt/java/jdk1.8.0_144 /opt/java/jdk1.8

Add to /etc/profile:

export JAVA_HOME=/opt/java/jdk1.8
export JRE_HOME=/opt/java/jdk1.8/jre
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib
export PATH=.:${JAVA_HOME}/bin:$PATH

Apply and verify:

source /etc/profile
java -version

Hadoop Installation

Extract the Hadoop package:

tar -xvf hadoop-2.8.2.tar.gz
mv hadoop-2.8.2 /opt/hadoop
mv /opt/hadoop/hadoop-2.8.2 /opt/hadoop/hadoop2.8

Add Hadoop environment variables to /etc/profile:

export HADOOP_HOME=/opt/hadoop/hadoop2.8
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"
export PATH=.:${JAVA_HOME}/bin:${HADOOP_HOME}/bin:$PATH

Apply changes:

source /etc/profile

Hadoop Configuration Files

Navigate to the Hadoop configuration directory:

cd /opt/hadoop/hadoop2.8/etc/hadoop

core-site.xml

<configuration>
  <property>
    <name>hadoop.tmp.dir</name>
    <value>file:/root/hadoop/tmp</value>
  </property>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://master:9000</value>
  </property>
  <property>
    <name>hadoop.proxyuser.root.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.root.groups</name>
    <value>*</value>
  </property>
</configuration>

hadoop-env.sh

Set the absolute path for JAVA_HOME:

export JAVA_HOME=/opt/java/jdk1.8

hdfs-site.xml

<configuration>
  <property>
    <name>dfs.replication</name>
    <value>2</value>
  </property>
  <property>
    <name>dfs.namenode.name.dir</name>
    <value>file:/root/hadoop/name</value>
  </property>
  <property>
    <name>dfs.datanode.data.dir</name>
    <value>file:/root/hadoop/data</value>
  </property>
</configuration>

mapred-site.xml

If the file doesn't exist, create it from the template:

cp mapred-site.xml.template mapred-site.xml

Add the following configuration:

<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
</configuration>

yarn-site.xml

<configuration>
  <property>
    <name>yarn.resourcemanager.hostname</name>
    <value>master</value>
  </property>
  <property>
    <name>yarn.resourcemanager.address</name>
    <value>${yarn.resourcemanager.hostname}:8032</value>
  </property>
  <property>
    <name>yarn.resourcemanager.scheduler.address</name>
    <value>${yarn.resourcemanager.hostname}:8030</value>
  </property>
  <property>
    <name>yarn.resourcemanager.webapp.address</name>
    <value>${yarn.resourcemanager.hostname}:8088</value>
  </property>
  <property>
    <name>yarn.resourcemanager.webapp.https.address</name>
    <value>${yarn.resourcemanager.hostname}:8090</value>
  </property>
  <property>
    <name>yarn.resourcemanager.resource-tracker.address</name>
    <value>${yarn.resourcemanager.hostname}:8031</value>
  </property>
  <property>
    <name>yarn.resourcemanager.admin.address</name>
    <value>${yarn.resourcemanager.hostname}:8033</value>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.scheduler.maximum-allocation-mb</name>
    <value>8182</value>
  </property>
  <property>
    <name>yarn.nodemanager.vmem-pmem-ratio</name>
    <value>2.1</value>
  </property>
  <property>
    <name>yarn.nodemanager.resource.memory-mb</name>
    <value>2048</value>
  </property>
  <property>
    <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
  </property>
</configuration>

Note: The yarn.nodemanager.vmem-check-enabled setting disables virtual memory checking, which is useful for virtualized environments.

slaves

slave1
slave2

Distribute and Start Hadoop

Copy the installation to all nodes:

scp -r /opt/java root@slave1:/opt
scp -r /opt/java root@slave2:/opt
scp -r /opt/hadoop root@slave1:/opt
scp -r /opt/hadoop root@slave2:/opt

Initialize the namenode (run only once on master):

cd /opt/hadoop/hadoop2.8/bin
./hdfs namenode -format

Start the cluster:

cd /opt/hadoop/hadoop2.8/sbin
start-dfs.sh
start-yarn.sh

Verify the installation:

jps  # should show NameNode, DataNode, NodeManager, ResourceManager on appropriate nodes

Access the web interfaces:

NameNode: http://master:50070
ResourceManager: http://master:8088

Spark Installation

Scala Setup

Extract and configure Scala:

tar -xvf scala-2.12.2.tgz
mv scala-2.12.2 /opt/scala
mv /opt/scala/scala-2.12.2 /opt/scala/scala2.12

Add to /etc/profile:

export SCALA_HOME=/opt/scala/scala2.12
export PATH=.:${JAVA_HOME}/bin:${SCALA_HOME}/bin:$PATH

source /etc/profile
scala -version

Spark Setup

Extract the pre-built Spark package with Hive support:

tar -xvf spark-1.6.3-bin-hadoop2.4-without-hive.tgz
mv spark-1.6.3-bin-hadoop2.4-without-hive /opt/spark
mv /opt/spark/spark-1.6.3-bin-hadoop2.4-without-hive /opt/spark/spark1.6-hadoop2.4-hive

Add to /etc/profile:

export SPARK_HOME=/opt/spark/spark1.6-hadoop2.4-hive
export PATH=.:${JAVA_HOME}/bin:${SCALA_HOME}/bin:${SPARK_HOME}/bin:$PATH

source /etc/profile

Spark Configuration Files

Navigate to the Spark conf directory:

cd /opt/spark/spark1.6-hadoop2.4-hive/conf

spark-env.sh

Create from template if needed:

cp spark-env.sh.template spark-env.sh

Add the following configuration:

export SCALA_HOME=/opt/scala/scala2.12
export JAVA_HOME=/opt/java/jdk1.8
export HADOOP_HOME=/opt/hadoop/hadoop2.8
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export SPARK_HOME=/opt/spark/spark1.6-hadoop2.4-hive
export SPARK_MASTER_IP=master
export SPARK_EXECUTOR_MEMORY=4G

slaves

slave1
slave2

Distribute and Start Spark

Copy to all nodes:

scp -r /opt/scala root@slave1:/opt
scp -r /opt/scala root@slave2:/opt
scp -r /opt/spark root@slave1:/opt
scp -r /opt/spark root@slave2:/opt

Start Spark (ensure Hadoop is running first):

cd /opt/spark/spark1.6-hadoop2.4-hive/sbin
start-all.sh

Verify at http://master:8080

Zookeeper Installation

Setup

Extract and configure:

tar -xvf zookeeper-3.4.10.tar.gz
mv zookeeper-3.4.10 /opt/zookeeper
mv /opt/zookeeper/zookeeper-3.4.10 /opt/zookeeper/zookeeper3.4

Add to /etc/profile:

export ZK_HOME=/opt/zookeeper/zookeeper3.4
export PATH=.:${JAVA_HOME}/bin:${SCALA_HOME}/bin:${SPARK_HOME}/bin:${ZK_HOME}/bin:$PATH

source /etc/profile

Create Required Directories

On all nodes:

mkdir /opt/zookeeper/data
mkdir /opt/zookeeper/dataLog
touch /opt/zookeeper/data/myid

Assign unique IDs in myid files:

master: 1
slave1: 2
slave2: 3

echo "1" > /opt/zookeeper/data/myid  # on master

zoo.cfg Configuration

Navigate to conf directory:

cd /opt/zookeeper/zookeeper3.4/conf
cp zoo_sample.cfg zoo.cfg

Edit zoo.cfg:

dataDir=/opt/zookeeper/data
dataLogDir=/opt/zookeeper/dataLog
server.1=master:2888:3888
server.2=slave1:2888:3888
server.3=slave2:2888:3888
clientPort=2181
tickTime=2000
initLimit=10
syncLimit=5

Configuration explanation:

clientPort: Client connection port
dataDir: Stores snapshot files
dataLogDir: Stores write-ahead logs
server.N: Server configuration where N matches the myid value
tickTime: Heartbeat interval in milliseconds
initLimit: Maximum heartbeats for initial connection
syncLimit: Maximum heartbeats for request acknowledgment

Distribute and Start Zookeeper

scp -r /opt/zookeeper root@slave1:/opt
scp -r /opt/zookeeper root@slave2:/opt

Important: Update the myid file on each slave node to match its server ID.

Start Zookeeper on all nodes:

cd /opt/zookeeper/zookeeper3.4/bin
zkServer.sh start

Check status:

zkServer.sh status

This will show which node is leader and which are followers.

HBase Installation

Setup

Extract and configure:

tar -xvf hbase-1.2.6-bin.tar.gz
mv hbase-1.2.6 /opt/hbase
mv /opt/hbase/hbase-1.2.6 /opt/hbase/hbase1.2

Add to /etc/profile:

export HBASE_HOME=/opt/hbase/hbase1.2
export PATH=.:${JAVA_HOME}/bin:${SCALA_HOME}/bin:${SPARK_HOME}/bin:$PATH

source /etc/profile
hbase version

HBase Configuration Files

Navigate to the conf directory:

cd /opt/hbase/hbase1.2/conf

hbase-env.sh

Add the following:

export JAVA_HOME=/opt/java/jdk1.8
export HADOOP_HOME=/opt/hadoop/hadoop2.8
export HBASE_HOME=/opt/hbase/hbase1.2
export HBASE_CLASSPATH=/opt/hadoop/hadoop2.8/etc/hadoop
export HBASE_PID_DIR=/root/hbase/pids
export HBASE_MANAGES_ZK=false

Note: HBASE_MANAGES_ZK=false disables the built-in Zookeeper, requiring an external Zookeeper cluster.

hbase-site.xml

<configuration>
  <property>
    <name>hbase.rootdir</name>
    <value>hdfs://master:9000/hbase</value>
  </property>
  <property>
    <name>hbase.zookeeper.property.clientPort</name>
    <value>2181</value>
  </property>
  <property>
    <name>zookeeper.session.timeout</name>
    <value>120000</value>
  </property>
  <property>
    <name>hbase.master.maxclockskew</name>
    <value>150000</value>
  </property>
  <property>
    <name>hbase.zookeeper.quorum</name>
    <value>master,slave1,slave2</value>
  </property>
  <property>
    <name>hbase.tmp.dir</name>
    <value>/root/hbase/tmp</value>
  </property>
  <property>
    <name>hbase.cluster.distributed</name>
    <value>true</value>
  </property>
  <property>
    <name>hbase.master</name>
    <value>master:60000</value>
  </property>
</configuration>

regionservers

slave1
slave2

Distribute and Start HBase

scp -r /opt/hbase root@slave1:/opt
scp -r /opt/hbase root@slave2:/opt

Start HBase (ensure Hadoop and Zookeeper are running):

cd /opt/hbase/hbase1.2/bin
start-hbase.sh

Verify at http://master:16010

Hive Installation

Hive can be installed on a single node as it doesn't require cluster configuration. Refer to the official Hive documentation for detailed installation instructions.

Startup Order

When starting the cluster, follow this sequence:

Zookeeper (all nodes)
Hadoop (HDFS + YARN)
Spark
HBase
Hive

When stopping the cluster, reverse the order.

Tags: Hadoop Spark ZooKeeper HBase Hive

Posted on Mon, 27 Jul 2026 16:51:19 +0000 by qaokpl

Freaks City

Building a Hadoop+Spark+Zookeeper+HBase+Hive Cluster: Complete Setup Guide

Environment Overview

Cluster Architecture

Component Versions

Download Links

Cluster Pre-Configuration

Hostname Configuration

IP-Hosst Mapping

SSH Passwordless Authentication

Firewall Configuration

Time Synchronization

Shell Aliases

Environment Variables

Hadoop Installation

JDK Setup

Hadoop Installation

Hadoop Configuration Files

core-site.xml

hadoop-env.sh

hdfs-site.xml

mapred-site.xml

yarn-site.xml

slaves

Distribute and Start Hadoop

Spark Installation

Scala Setup

Spark Setup

Spark Configuration Files

spark-env.sh

slaves

Distribute and Start Spark

Zookeeper Installation

Setup

Create Required Directories

zoo.cfg Configuration

Distribute and Start Zookeeper

HBase Installation

Setup

HBase Configuration Files

hbase-env.sh

hbase-site.xml

regionservers

Distribute and Start HBase

Hive Installation

Startup Order

Hot Tags