Getting PySpark running on Ubuntu involves installing several dependencies and configuring your environment properly. This guide walks through the complete setup process and demonstrates how to build a word counting application.

Prerequisites Installation

Before installing PySpark, you need to set up the Java runtime environment since Spark runs on the JVM. Install OpenJDK with the following commands:

sudo apt update
sudo apt install default-jre default-jdk

Spark also depends on Scala, which can be installed via apt:

sudo apt install scala

Make sure Python 3 is available on your system:

sudo apt install python3

Apache Spark Setup

Download the Apache Spark distribution from the official downloads page. Choose a version compatible with your Hadoop distribution:

wget https://downloads.apache.org/spark/spark-3.5.0/spark-3.5.0-bin-hadoop3.tgz
tar -xzvf spark-3.5.0-bin-hadoop3.tgz

Configure your shell environment by adding the following exports to your ~/.bashrc or ~/.zshrc:

export SPARK_HOME=$HOME/spark-3.5.0-bin-hadoop3
export PATH=$PATH:$SPARK_HOME/bin
export PYSPARK_PYTHON=python3

Apply the changes with source ~/.bashrc and verify the enstallation works by running pyspark.

Installing PySpark via pip

For easier management, install the Python package:

pip install pyspark

Building a Word Count Application

Create a file named text_analyzer.py and implement the distributed word counting logic:

from pyspark import SparkContext

if __name__ == "__main__":
    # Initialize the Spark context
    context = SparkContext(appName="TextAnalyzer", master="local[*]")
    
    # Load the input dataset
    source_data = context.textFile("input/sample_text.txt")
    
    # Split into tokens, count occurrences, and aggregate
    token_frequencies = (
        source_data
        .flatMap(lambda sentence: sentence.strip().split())
        .map(lambda token: (token.lower(), 1))
        .reduceByKey(lambda count1, count2: count1 + count2)
    )
    
    # Persist results to the output directory
    token_frequencies.coalesce(1).saveAsTextFile("output/word_counts")
    
    # Release resources
    context.stop()

Replace input/sample_text.txt with the path to your text file and ensure the output directory does not already exist.

Executing the Application

Submit the job using the spark-submit command:

spark-submit text_analyzer.py

The framework will distribute the processing across available cores and write the word frequency pairs to the specified output directory. Each partition will be processed in parallel, making this approach suitable for large text corpora.

You can customize the script further by adding filtering logic, sorting results by frequency, or processing multiple input files by adjusting the input path pattern.