Freaks City

Environment Overview Cluster Architecture This cluster uses three machines configured as master-slave topology. The following table summarizes the hardware specificasions: Node Role CPU Cores RAM Disk Space master NameNode + ResourceManager 4 16GB 500GB slave1 DataNode + NodeManager 4 16GB 500GB slave2 DataNode + NodeManager 4 16GB 5 ...

Environment Choices Server Selection Cloud provider: Alibaba Cloud (pay-as-you-go entry tier) Operating system: Linux CentOS 6.8 CPU: 1 core Memory: 1 GB Disk: 40 GB Public IP: 39.108.77.250 Software Versions JDK: 1.8 (jdk-8u144-linux-x64.tar.gz) Hadoop: 2.8.2 (hadoop-2.8.2.tar.gz) Download Locations Official sources: JDK: http://www.ora ...

Hive, HBase, and HDFS serve distinct but complementary roles in the Hadoop architecture—each addressing different data access patterns and storage requirements. Hive: SQL Abstraction Over Batch Processing Hive is a data warehousing infrastructure built atop Hadoop that translates declarative SQL-like queries (HiveQL) into distributed batch jobs ...

Hive Introduction What is Hive? Hive is an open-source data warehouse solution originally developed by Facebook that operates on Hadoop infrastructure It provides SQL-like query capabilities (HQL) for structured data stored in HDFS Core functionality involves translating SQL queries into MapReduce jobs Primary use case: batch data analytics wi ...

Issue 1: UnsatisfiedLinkError to gplcompression When attempting to read HBase snapshot data, the following error occurs: java.lang.UnsatisfiedLinkError: no gplcompression in java.library.path at com.hadoop.compression.lzo.GPLNativeCodeLoader.<clinit>(GPLNativeCodeLoader.java:31) at com.hadoop.compression.lzo.LzoCodec.<clinit&gt ...

HDFS File System Commands Disk Usage Information Retrieve disk usage statistics for a specific path: hadoop fs -df /home/myfile Merge Files Combine multiple files from HDFS into a single local file: hadoop fs -getmerge /user/hduser0011/test /home/myfile/dir Write Output to HDFS Direct console output to an HDFS file: echo abc | hadoop fs -put ...

HDFS Overview HDFS (Hadoop Distributed File System) is a distributed storage system designed to handle massive datasets, typically in terabytes or petabytes. It forms the storage layer of the Hadoop ecosystem, enabling applications to work with large-scale data using a unified interface similar to a conventional file system. HDFS streams data d ...

Hadoop Distributed Cluster Setup This guide explains how to set up a fully distributed Hadoop cluster using three or more physical or virtual machines. Cluster Architecture Master Node (hadoop0): NameNode, JobTracker, SecondaryNameNode Worker Nodes (hadoop1, hadoop2): DataNode, TaskTracker Virtual Machine Setup Create three virtual machines u ...

Big Data Processing Overview Big data involves analyzing massive datasets to extract valuable insights for organizational decision-making. Core processing stages include: Data acquisition Data processing Result visualization Hadoop Framework Hadoop provides distributed processing capabilities for large datasets across computer clusters. Its a ...

To implement LZO compression within a HBase environment running on Hadoop, it is necessary to compile the native LZO libraries and the corresponding Hadoop-LZO Java bridge from source. Older guides often reference the deprecated hadoop-gpl-compression library, which is incompatible with modern Hadoop versions. The following procedure outlines t ...