Freaks City

System Requirements Operating System: CentOS 7 (virtual machine) CPU: 2 cores Memory: 2 GB Disk: 40 GB Software Versions JDK: 1.8 (jdk-8u144-linux-x64.tar.gz) Hadoop: 2.8.2 (hadoop-2.8.2.tar.gz) Scala: 2.12.2 (scala-2.12.2.tgz) Spark: 1.6.3 (spark-1.6.3-bin-hadoop2.4-without-hive.tgz) Initial System Configuration Set Hostname hostnamectl set- ...

Apache Spark serves as a quasi-real-time big data processing engine that requires resource scheduling and task management. While Spark includes its own standalone resource scheduler, it also supports deployment on external platforms such as Yarn, Mesos, and Kubernetes. This guide covers three deployment modes: Local Mode: Ideal for local devel ...

Environment: Windows 10, CentOS 7.9, Hadoop 3.2, HBase 2.5.3, and Zookeeper 3.8 in fully distributed mode; Environment setup procedures can be found in these articles: CentOS7 Hadoop3.X Fully Distributed Environment Setup Hadoop3.x Fully Distributed Environment Setup with Zookeeper and Hbase 1. Integrating MapReduce and HBase Copy hbase-site.x ...

In the era of big data and artificial intelligence, the demand for distributed computing frameworks is increasingly urgent. Python Ray library was created to meet this demand. It is a high-performance, easy-to-use distributed application framework designed specifically for solving parallel and distributed computing problems in Python. This arti ...

Hadoop Installation on Windows This guide covers the complete process of deploying Hadoop 2.7.1 on a Windows system, configuring HDFS, and performing file operations. Prerequisites Windows operating system JDK 8 or compatible Java version installed Hadoop 2.7.1 binary distribution from Apache archives Windows-specific Hadoop binaries (hadoopon ...

Extracting Stock Market Data with SeleniumTo retrieve financial information from dynamic web pages, Selenium is used to automate browser interactions, specifically targeting elements that load via JavaScript. The target involves extracting data from the Shanghai A-shares, Shenzhen A-shares, and aggregated boards. The data is persisted in a stru ...

ZooKeeper Watcher Mechanism: Principles and Practical Guide This article explores the ZooKeeper Watcher mechanism, a core component for distributed coordination. It covers fundamental concepts, the publish/subscribe model, workflow, application scenarios, and practical command examples. Background We have three cloud servers (2C4G each) running ...

To build a Spark distribution compatible with Hadoop CDH 5.7.0 and Hive support, navigate to the Spark source directory: [hadoop@hadoop001 spark-2.1.0]$ pwd /home/hadoop/source/spark-2.1.0 Compile using Maven with profiles for YARN, Hadoop 2.6, Hive, and Hive Thriftserver: ./build/mvn -Pyarn -Phadoop-2.6 -Phive -Phive-thriftserver \ -Dhadoop ...