Freaks City

Working with Spark SQL DataFrames and Datasets Table of Contents- Spark SQL - - - DataFrame - Dataset - Project Setup with IDEA - Custom Functions - User Defined Aggregate Functions (Weakly Typed) - User Defined Aggregate Functions (Strongly Typed) DataFrame Operations Initializing Spark Session scala> spark res0: org.apache.spark.sql.SparkS ...

What is Spark? Spark is a unified analytics engine designed for large-scale data processing. It supports custom computations on various data types, including structured, semi-structured, and unstructured data. Origin Spark was developed at UC Berkeley in 2009 and open-sourced in 2010. The original team founded Databricks in 2013 and donated Spa ...

Environment Overview Cluster Architecture This cluster uses three machines configured as master-slave topology. The following table summarizes the hardware specificasions: Node Role CPU Cores RAM Disk Space master NameNode + ResourceManager 4 16GB 500GB slave1 DataNode + NodeManager 4 16GB 500GB slave2 DataNode + NodeManager 4 16GB 5 ...

Implementing a Data Pipeline for Product Review Analysis This project outlines a data pipeline for analyzing customer reviews from an e-commerce platform. The process involves collecting data via web scraping, processing it using distributed computing frameworks, and visualizing the results. The goal is to extract insights regarding user behavi ...

This approach leverages Spark's implicit conversinos to infer column names from case class attributes. import org.apache.spark.sql.SparkSession val spark = SparkSession.builder() .appName("RDDConversionExample") .master("local[*]") .getOrCreate() import spark.implicits._ case class User(id: Int, username: String, sc ...

Data as the Cornerstone of Modern Recommender Engines At the heart of every effective recommendation system lies a deep understanding of user behavior. Rather than relying on static assumptions, modern systems derive user preferences from observed interactions—clicks, views, likes, purchases, and more. These behavioral signals form the foundati ...

Database and SQL Query Execution Order The order of execution for SQL statements: FROM - identifies the source tables JOIN - combines rows from multiple tables ON - specifies join conditions WHERE - filters rows based on conditions GROUP BY - groups rows by specified columns HAVING - filters groups after aggregation SELECT - selects columns to ...

Hadoop Pseudo-Distributed Mode Issues Configuration Parsing Failure in hdfs-site.xml When you encounter FATAL conf.Configuration: error parsing conf hdfs-site.xml, the root cause is typically an encoding mismatch. Resolve it by opening the file and saving it with a uniform character encoding such as UTF-8. HDFS Command Deprecation Warning The w ...

System Requirements Operating System: CentOS 7 (virtual machine) CPU: 2 cores Memory: 2 GB Disk: 40 GB Software Versions JDK: 1.8 (jdk-8u144-linux-x64.tar.gz) Hadoop: 2.8.2 (hadoop-2.8.2.tar.gz) Scala: 2.12.2 (scala-2.12.2.tgz) Spark: 1.6.3 (spark-1.6.3-bin-hadoop2.4-without-hive.tgz) Initial System Configuration Set Hostname hostnamectl set- ...

Integrating Apache Hudi with Spark This guide covers the essential steps to integrate Apache Hudi with Apache Spark for building data lake solutions. The integration enables ACID transactions, time-travel queries, and efficient upserts on large datasets. Environment Setup Before starting, ensure you have Spark installed and Hadoop services runn ...