Freaks City

Differences Between Internal and External Tables External tables require the keyword EXTERNAL when creating. When creating an internal table, data is moved to the specified path in the data warehouse. When creating an external table, data is not moved; only the path to the data is recorded. Deleting an internal table removes both metadata and ...

Apache Hive Overview Apache Hive is a SQL abstraction tool that translates standard SQL queries into distributed compute jobs like MapReduce, simplifying big data development for engineers. A basic example query: SELECT COUNT(*) FROM student_table; Simulating Hive Core Functionality Hive maps data stored on HDFS into structured tabular data. A ...

What is Spark? Spark is a unified analytics engine designed for large-scale data processing. It supports custom computations on various data types, including structured, semi-structured, and unstructured data. Origin Spark was developed at UC Berkeley in 2009 and open-sourced in 2010. The original team founded Databricks in 2013 and donated Spa ...

Apache Hive serves as a data warehouse infrastructure built on top of Hadoop, enabling the summarization, querying, and analysis of large datasets. A critical aspect of managing this data involves transferring it efficient between Hive and external systems. This guide outlines the primary methods for importing data into Hive tables and exportin ...

Data governance is an essential component of enterprise data infrastructure. An effective governance framework activates the entire data pipeline, ensuring control and traceability across data collection, storage, processing, and utilization. This process involves building a systematic approach to managing data assets, focusing on quality, secu ...

Introduction HBase is a highly reliable, high-performance, column-oriented distributed storage system suitable for storing massive amounts of data and enabling real-time read/write operations. This guide will walk you through setting up HBase in single-node mode. Process Overview The table below outlines the steps to start HBase on a single nod ...

This guide walks through setting up MySQL, Hive, and Sqoop on a single Hadoop node (Hadoop01). The steps assume a CentOS 7 environment with Hadoop 2.7.4 already deployed. All operations are performed on Hadoop01 only. 1. MySQL Installation & Configuration 1.1 Remove Existing MariaDB yum remove mysql-libs -y 1.2 Upload and Extract RPM Bund ...

Process Overview To extract keys from JSON data in Hive, follow these steps: Step Action 1 Create a Hive table 2 Load JSON data into the table 3 Extract keys from the JSON Step 1: Create a Hive Table Define a table to store JSON strings. Use the following SQL command: CREATE TABLE IF NOT EXISTS json_data_table ( json_content S ...

Chapter 3: HDFS http://master:50070 1.1 NameNode NameNode is the manager. It stores metadata, which is data about data. Files in HDFS are split into data blocks of 128 MB (originally 64 MB) for storage. Replication strategy: The default replication factor in HDFS is 3. 1.2 Secondary NameNode 1.3 DataNode 1.4 Client 1.5 File Write Process NameN ...

Hive Introduction What is Hive? Hive is an open-source data warehouse solution originally developed by Facebook that operates on Hadoop infrastructure It provides SQL-like query capabilities (HQL) for structured data stored in HDFS Core functionality involves translating SQL queries into MapReduce jobs Primary use case: batch data analytics wi ...