Hadoop Distributed File System (HDFS) can be accessed programmatically through various client libraries. While Java provides native APIs, Python developers often use the PyHDFS library to interact with HDFS efficiently.

Setting Up PyHDFS

To begin, enstall the PyHDFS package using pip:

pip install PyHDFS

Basic Operations with PyHDFS

The following example demonstrates common file system operations such as directory creation, file upload, reading, downloading, and deletion.

# -*- coding: utf-8 -*-
import pyhdfs

# Initialize HDFS client
client = pyhdfs.HdfsClient(hosts='hadoop01:9870', user_name='root')

# Get user's home directory and active NameNode
print("Home Directory:", client.get_home_directory())
print("Active NameNode:", client.get_active_namenode())

# Create a base directory if it doesn't exist
base_path = '/cyw'
if not client.exists(base_path):
    client.mkdirs(base_path)

# Recursively create subdirectories
sub_path = '/cyw/sub1'
if not client.exists(sub_path):
    client.mkdirs(sub_path)

# Upload a local file to HDFS
local_file = 'test.txt'
remote_file = f'{sub_path}/{local_file}'
if not client.exists(remote_file):
    client.copy_from_local(local_file, remote_file)

# Append data to an existing file
append_target = '/cyw/test.txt'
if not client.exists(append_target):
    client.create(append_target, b'initial content\n')
client.append(append_target, b'appended line\n', encoding='utf-8')

# Read file content from HDFS
with client.open('/cyw/test.txt') as f:
    print("File Content:", f.read())

# Download file from HDFS to local filesystem
client.copy_to_local('/cyw/test.txt', 'downloaded_test.txt')

# Concatenate files (source file will be deleted after merge)
source_file = '/java/data.txt'
target_file = '/java/move.txt'
if client.exists(source_file) and client.exists(target_file):
    client.concat(target_file, [source_file])

# Rename a file
if client.exists('/java/move.txt'):
    client.rename('/java/move.txt', '/java/new_move.txt')

# Delete directories (recursively)
delete_path = '/cyw/delete'
if not client.exists(delete_path):
    client.mkdirs(delete_path)
if client.exists(delete_path):
    client.delete(delete_path)

if client.exists(sub_path):
    client.delete(sub_path, recursive=True)

# Retrieve file metadata
file_info = client.get_file_status('/cyw/test.txt')
print("File Status:", file_info)
print("File Type:", file_info.type)

dir_info = client.get_file_status('/cyw')
print("Directory Status:", dir_info)

# Get content summary
summary = client.get_content_summary('/cyw/test.txt')
print("Content Summary:", summary)

# Compute file checksum
checksum = client.get_file_checksum('/cyw/test.txt')
print("File Checksum:", checksum)

# List directory contents
listing = client.listdir('/cyw')
print("Directory Listing:", listing)

This script successfully performs a range of HDFS operations, confirming that the Hadoop cluster is correctly configured when accessed via Python. If similar operations fail in Java but succeed in Python—as observed in some environments—it may indicate misconfiguration in the Java client setup, such as incorrect Hadoop configuration properties or hostname resolution issues.