Freaks City

DataX Plugin Overview DataX is an open-source data synchronization framework that supports data transfer between dozens of common data systems, with out-of-the-box reader and writer plugins for most integration scenarios. For ingesting file data from FTP servers in to Hive, we use the official ftpreader plugin for source reading and hive3writer ...

Differences Between Internal and External Tables External tables require the keyword EXTERNAL when creating. When creating an internal table, data is moved to the specified path in the data warehouse. When creating an external table, data is not moved; only the path to the data is recorded. Deleting an internal table removes both metadata and ...

System Requirements This guide demonstrates how to install a big data stack on a personal laptop using VirtualBox with CentOS 7.x (kernel 3.10.0-514.el7.x86_64). RHEL 7.3 is also compatible with this setup. Minimum Requirements: Memory: 2.5GB (4GB recommended for better performance) Components to be installed: Hadoop 2.8.0 Apache Hive 2 ...

Apache Flink and Apache Hive are commonly integrated in modern data pipelines to enable efficient batch and stream processing with persistent, partitioned storage. One powerful capability is writing data from Flink SQL into Hive tables using dynamic partitioning—where partition values are derived from the data itself rather than being hardcoded ...

Row to Column Conversion Problem Analysis Given the following employee data: 1,PK,RD,1 2,XIAOAI,RD,1 3,XIAOHONG,RD,2 4,XIAOZHANG,QA,1 5,XIAOLI,QA,2 6,XIAOFANG,QA,2 Group by department and gender to get: QA,1 XIAOZHANG QA,2 XIAOLI|XIAOFANG RD,1 PK|XIAOAI RD,2 XIAOHONG Required Functions concat: String concatenation concat_ws: Stri ...

Environment Overview Cluster Architecture This cluster uses three machines configured as master-slave topology. The following table summarizes the hardware specificasions: Node Role CPU Cores RAM Disk Space master NameNode + ResourceManager 4 16GB 500GB slave1 DataNode + NodeManager 4 16GB 500GB slave2 DataNode + NodeManager 4 16GB 5 ...

This guide walks through setting up MySQL, Hive, and Sqoop on a single Hadoop node (Hadoop01). The steps assume a CentOS 7 environment with Hadoop 2.7.4 already deployed. All operations are performed on Hadoop01 only. 1. MySQL Installation & Configuration 1.1 Remove Existing MariaDB yum remove mysql-libs -y 1.2 Upload and Extract RPM Bund ...

Process Overview To extract keys from JSON data in Hive, follow these steps: Step Action 1 Create a Hive table 2 Load JSON data into the table 3 Extract keys from the JSON Step 1: Create a Hive Table Define a table to store JSON strings. Use the following SQL command: CREATE TABLE IF NOT EXISTS json_data_table ( json_content S ...

Entroduction to Data Processing Optimization In contemporary data engineering workflows, escalating business complexity correlates directly with intricate task logic and growing data volumes. As performance demands intensify, optimization becomes paramount. While common techniques like small file consolidation and skew handling are well-documen ...

Chapter 3: HDFS http://master:50070 1.1 NameNode NameNode is the manager. It stores metadata, which is data about data. Files in HDFS are split into data blocks of 128 MB (originally 64 MB) for storage. Replication strategy: The default replication factor in HDFS is 3. 1.2 Secondary NameNode 1.3 DataNode 1.4 Client 1.5 File Write Process NameN ...