Freaks City

Environment Overview Cluster Architecture This cluster uses three machines configured as master-slave topology. The following table summarizes the hardware specificasions: Node Role CPU Cores RAM Disk Space master NameNode + ResourceManager 4 16GB 500GB slave1 DataNode + NodeManager 4 16GB 500GB slave2 DataNode + NodeManager 4 16GB 5 ...

Installation and Configuration Kafka 3.x and latter versions include an embedded Zookeeper, simplifying the setup process. This guide demonstrates how to install and configure a single-node Kafka cluster. Prerequisites Ensure you have a Java Runtime Environment (JRE) installed on you're system. Step 1: Download and Extract Download the Kafka bi ...

Message Queue Fundamentals Definition A message queue serves as an inter-process communication mechanism that enables asynchronous data exchange between applications. Messages represent the data units transmitted between systems, while the queue provides a reliable delivery mechanism ensuring data integrity throughout the transfer process. Core ...

Background Introduction ZooKeeper Overview Apache ZooKeeper is a distributed coordination service that provides a reliable centralized solution for maintaining configuration information, naming, distributed synchronization, and group services. It is commonly used for service and configuration management within data centers. One of ZooKeeper's d ...

Business Requirement Both the source and target schemas on the same MySQL instance contain a table named student. Any INSERT, UPDATE, or DELETE executed against source.student must be reflected in target.student with sub-second latency. Candidate Approaches 1. Shell Script Quick PoC with bash + mysqldump was abandoned because the team lacked sh ...

Chapter 3: HDFS http://master:50070 1.1 NameNode NameNode is the manager. It stores metadata, which is data about data. Files in HDFS are split into data blocks of 128 MB (originally 64 MB) for storage. Replication strategy: The default replication factor in HDFS is 3. 1.2 Secondary NameNode 1.3 DataNode 1.4 Client 1.5 File Write Process NameN ...

Preparing the Flink 1.20.0 Distribution Begin by downloading the official Flink 1.20.0 binary distribution for Scala 2.12: wget https://archive.apache.org/dist/flink/flink-1.20.0/flink-1.20.0-bin-scala_2.12.tgz tar -xzf flink-1.20.0-bin-scala_2.12.tgz tar -czf flink-1.20.0.tar.gz flink-1.20.0 If Hudi support is required, copy the compatible H ...

Setting Up a Zookeeper Ensemble A Zookeeper cluster maintains coordination metadata for distributed systems. For production, deploy an odd number of nodes (minimum three) to achieve quorum. Configuration Essentials On each node, prepare the configuration file conf/zoo.cfg: tickTime=2000 dataDir=/var/lib/zookeeper dataLogDir=/var/log/zookeeper c ...

Message Queue Fundamentals Understanding Message Queues A message represents data transmitted between applications. Messages can range from simple text strings to complex structures containing embedded objects. A Message Queue (MQ) serves as an asynchronous communication mechanism in software systems. It decouples communication between differen ...

Apache ZooKeeper is a centralized, fault-tolerant service for maintaining configuration information, naming, distributed synchronization, and group services in distributed systems. Installation and Setup ZooKeeper binaries can be downloaded from the official Apache releases page. After extraction, configure zoo.cfg in the conf/ directory. A min ...