Freaks City

To implement LZO compression within a HBase environment running on Hadoop, it is necessary to compile the native LZO libraries and the corresponding Hadoop-LZO Java bridge from source. Older guides often reference the deprecated hadoop-gpl-compression library, which is incompatible with modern Hadoop versions. The following procedure outlines t ...

This guide walks through the steps to set up a Hadoop 2.10 pseudo-distributed cluster on a single CentOS 7 virtual machine. 1. Create a Hadoop User and Group We will create a dedicated user hdfs and configure it with appropriate permissions. As root user: Create the hdfs user and set a password: adduser hdfs passwd hdfs Add the user to the hdf ...

Binary Extraction and Setup Acquire the Apache Hive 2.3.6 binary archive from the official distribution repository. Extract the contents to a standard application directory and establish a symbolic link for simplified version management. tar -xzf apache-hive-2.3.6-bin.tar.gz -C /usr/local/ cd /usr/local sudo ln -s apache-hive-2.3.6-bin hive E ...

System Requirements Operating System: CentOS 7 (virtual machine) CPU: 2 cores Memory: 2 GB Disk: 40 GB Software Versions JDK: 1.8 (jdk-8u144-linux-x64.tar.gz) Hadoop: 2.8.2 (hadoop-2.8.2.tar.gz) Scala: 2.12.2 (scala-2.12.2.tgz) Spark: 1.6.3 (spark-1.6.3-bin-hadoop2.4-without-hive.tgz) Initial System Configuration Set Hostname hostnamectl set- ...

Apache Spark serves as a quasi-real-time big data processing engine that requires resource scheduling and task management. While Spark includes its own standalone resource scheduler, it also supports deployment on external platforms such as Yarn, Mesos, and Kubernetes. This guide covers three deployment modes: Local Mode: Ideal for local devel ...

Environment: Windows 10, CentOS 7.9, Hadoop 3.2, HBase 2.5.3, and Zookeeper 3.8 in fully distributed mode; Environment setup procedures can be found in these articles: CentOS7 Hadoop3.X Fully Distributed Environment Setup Hadoop3.x Fully Distributed Environment Setup with Zookeeper and Hbase 1. Integrating MapReduce and HBase Copy hbase-site.x ...

In the era of big data and artificial intelligence, the demand for distributed computing frameworks is increasingly urgent. Python Ray library was created to meet this demand. It is a high-performance, easy-to-use distributed application framework designed specifically for solving parallel and distributed computing problems in Python. This arti ...

Hadoop Installation on Windows This guide covers the complete process of deploying Hadoop 2.7.1 on a Windows system, configuring HDFS, and performing file operations. Prerequisites Windows operating system JDK 8 or compatible Java version installed Hadoop 2.7.1 binary distribution from Apache archives Windows-specific Hadoop binaries (hadoopon ...

Extracting Stock Market Data with SeleniumTo retrieve financial information from dynamic web pages, Selenium is used to automate browser interactions, specifically targeting elements that load via JavaScript. The target involves extracting data from the Shanghai A-shares, Shenzhen A-shares, and aggregated boards. The data is persisted in a stru ...

ZooKeeper Watcher Mechanism: Principles and Practical Guide This article explores the ZooKeeper Watcher mechanism, a core component for distributed coordination. It covers fundamental concepts, the publish/subscribe model, workflow, application scenarios, and practical command examples. Background We have three cloud servers (2C4G each) running ...