Command-Line Operations in HDFS

The Hadoop Distributed File System provides a robust shell interface for managing distributed storage. Below are esssential commands for navigating and manipulating the filesystem:

# List contents of the root directory
hadoop fs -ls /

# Create a new directory structure
hadoop fs -mkdir -p /data/logs

# Transfer a local file to the distributed filesystem
hadoop fs -put ./system_metrics.log /data/logs/

# Retrieve a file from HDFS to the local machine
hadoop fs -get /data/logs/system_metrics.log ./downloaded_metrics.log

# Display the contents of a remote file directly in the terminal
hadoop fs -cat /data/logs/system_metrics.log

These commands mirror standard Unix filesystem utilities, allowing developers to manage distributed data with familiar syntax.

Integrating HDFS Client Libraries

Modern applications interact with HDFS through official and community-maintained client libraries. Below are the dependancy configurations for popular package managers:

Maven

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-client</artifactId>
    <version>3.3.6</version>
</dependency>

Gradle

implementation 'org.apache.hadoop:hadoop-hdfs-client:3.3.6'

Node.js (npm)

npm install webhdfs

Python (pip)

pip install hdfs

Programmatic Access via Java API

Java remains the primary language for Hadoop ecosystem development. The following example demonstrates how to establish a connection to a NameNode, locate a target file, and stream its contents efficiently using the Hadoop FileSystem API:

package com.dataops.hdfs;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;

public class HdfsFileReader {

    private static final String NAMENODE_URI = "hdfs://namenode-host:8020";
    private static final String TARGET_FILE = "/data/logs/system_metrics.log";

    public static void main(String[] args) {
        Configuration config = new Configuration();
        config.set("fs.defaultFS", NAMENODE_URI);

        try (FileSystem hdfs = FileSystem.get(config);
             FSDataInputStream inputStream = hdfs.open(new Path(TARGET_FILE));
             BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream))) {

            String line;
            while ((line = reader.readLine()) != null) {
                System.out.println(line);
            }
        } catch (IOException ex) {
            System.err.println("Failed to read from HDFS: " + ex.getMessage());
            ex.printStackTrace();
        }
    }
}