What is Spark?
Spark is a unified analytics engine designed for large-scale data processing. It supports custom computations on various data types, including structured, semi-structured, and unstructured data.
Origin
Spark was developed at UC Berkeley in 2009 and open-sourced in 2010. The original team founded Databricks in 2013 and donated Spark to the Apache Foundation the same year. In 2014, it became an Apache top-level project. Major versions include Spark 2.0 (2016) and Spark 3.0 (2019).
Spark vs Hadoop (MapReduce)
While Spark offers significant advantages over MapReduce, it does not replace Hadoop entirely:
- Spark excels in iterative and complex computations compared to MapReduce.
- Hadoop includes storage (HDFS) and resource management (YARN), which remain critical components in many big data architectures.
Spark Features
- Speed: Spark is up to 100x faster than MapReduce due to in-memory processing and DAG-based iterative computation.
- Usability: Supports multiple programming languages: Python, Java, Scala, and R.
- Universality: Includes modules like Spark SQL, Spark Streaming, MLlib, and GraphX.
- Runtime Modes: Runs on Hadoop, Mesos, Kubernetes, cloud platforms, and standalone clusters.
Spark Framework Modules
Spark consists of several core modules:
- Spark Core: The foundation of Spark, providing RDD-based APIs for batch processing.
- Spark SQL: Handles structured data processing, including streaming with Structured Streaming.
- Spark Streaming: Enables real-time data processing with streaming capabilities.
- MLlib: Offers distributed machine learning algorithms and utilities.
- GraphX: Supports graph-based computations and analysis.
Spark Runtime Modes
Spark can run in multiple modes:
- Local Mode: Single-node execution using internal threads.
- Standalone Cluster: A Spark-specific cluster manager.
- YARN Mode: Runs on Hadoop YARN for resource management.
- Kubernetes Mode: Deploys Spark on Kubernetes clusters.
- Cloud Mode: Executes Spark in cloud environments.
Spark Architecture Roles
YARN Roles
- ResourceManager: Manages cluster resources.
- NodeManager: Manages resources on individual nodes.
- ApplicationMaster: Coordinates application execution.
- Task: Executes individual units of work.
Spark Roles
- Master: Cluster resource manager.
- Worker: Manages resources on each node.
- Driver: Main program execution process.
- Executor: Runs tasks assigned by the Driver.
Spark Environment Setup - Local Mode
System Requirements
Ensure the following components are installed:
- Hadoop 3+
- JDK 1.8+
- CentOS 7
- Anaconda
- Spark
Installation Steps
Install Anaconda
bash Miniconda-latest.sh
source .bashrc
conda create -n spark_env python=3.8 -y
Install Spark
tar -zxvf spark-3.3.0-bin-hadoop3.2.tar.gz -C /usr/local/
ln -s /usr/local/spark-3.3.0-bin-hadoop3.2 /usr/local/spark
Configure Environment Variables
vim /etc/profile
export JAVA_HOME=/usr/local/jdk
export SPARK_HOME=/usr/local/spark
export HADOOP_HOME=/usr/local/hadoop
export PYSPARK_PYTHON=/root/anaconda3/bin/python
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
Start Spark
./bin/pyspark --master local[*]
./bin/spark-shell
Local Mode Summary
Local mode simulates Spark's distributed environment using a single JVM process with multiple threads. The Driver runs in the same process as the Executor, making it suitable for development and testing.
Spark Environment Setup - Standalone Mode
Cluster Configuration
Set up a Spark cluster with:
- Master node (node1)
- Worker nodes (node2, node3)
- History Server (optional)
Configuration Files
workers
node1
node2
node3
spark-env.sh
JAVA_HOME=/usr/local/jdk
HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
YARN_CONF_DIR=/usr/local/hadoop/etc/hadoop
SPARK_MASTER_HOST=node1
SPARK_MASTER_PORT=7077
SPARK_MASTER_WEBUI_PORT=8080
SPARK_WORKER_CORES=2
SPARK_WORKER_MEMORY=2g
SPARK_WORKER_PORT=7078
SPARK_WORKER_WEBUI_PORT=8081
Cluster Operations
./sbin/start-all.sh
./sbin/stop-all.sh
Spark Application Submission
./bin/spark-submit --master spark://node1:7077 /path/to/your_script.py
Spark on YARN Configuration
Environment Setup
Ensure YARN and HDFS are properly configured and accessible from the Spark client.
Spark Submit Command
./bin/spark-submit --master yarn --deploy-mode client --driver-memory 512m --executor-memory 512m --num-executors 3 /path/to/spark_app.py
Deployment Modes
- Client Mode: Driver runs on the client machine. Suitable for development and testing.
- Cluster Mode: Driver runs on a YARN container. Recommendde for production due to better resource utilization.
Spark HA (High Availability) with Zookeeper
Zookeeper Configuration
SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=node1:2181,node2:2181,node3:2181 -Dspark.deploy.zookeeper.dir=/spark-ha"
Spark HA Benefits
- Automatic failover of Master nodes using Zookeeper.
- Cluster information persisted to HDFS for recovery.
- Minimal impact on running jobs during failover.
Spark Application Development
SparkContext Initialization
from pyspark import SparkConf, SparkContext
conf = SparkConf().setAppName('WordCount').setMaster('yarn')
sc = SparkContext(conf=conf)
Distributed Execution Flow
Spark applications follow this execution flow:
- Application submission to the cluster.
- Job scheduling and task distribution by the Driver.
- Task execution across Executors on Worker nodes.
- Result aggregation and output.
Spark Monitoring Ports
- 4040: Driver web UI port for task monitoring.
- 8080: Master web UI port for cluster status.
- 18080: History Server port for reviewing past applications.
Spark Application Architecture
A Spark application consists of:
- Jobs: High-level tasks defined by user actions (e.g., collect(), save()).
- Stages: Divisions of jobs based on dependencies (e.g., shuffle operations). Stages are represented as tasks.
- Tasks: Individual units of work executed by Executors.