DataX Plugin Overview

DataX is an open-source data synchronization framework that supports data transfer between dozens of common data systems, with out-of-the-box reader and writer plugins for most integration scenarios. For ingesting file data from FTP servers in to Hive, we use the official ftpreader plugin for source reading and hive3writer for loading data to Hive tables.

Sample DataX Job Configuration

{
    "job": {
        "setting": {
            "speed": {
                "channel": 2
            }
        },
        "content": [
            {
                "reader": {
                    "name": "ftpreader",
                    "parameter": {
                        "protocol": "sftp",
                        "host": "ftp-data.example.net",
                        "port": 22,
                        "username": "ftp_user",
                        "password": "ftp_pass",
                        "path": "/input/source_data.csv",
                        "column": [
                            {"index": 0, "type": "string"},
                            {"index": 1, "type": "string"}
                        ]
                    }
                },
                "writer": {
                    "name": "hive3writer",
                    "parameter": {
                        "jdbcUrl": "jdbc:hive2://hive-edge.example.net:10000/analytics_db",
                        "username": "hive_user",
                        "password": "hive_pass",
                        "defaultFS": "hdfs://hadoop-nn.example.net:8020",
                        "column": [
                            {"name": "record_id", "type": "string"},
                            {"name": "record_name", "type": "string"}
                        ],
                        "table": "ingested_ftp_data",
                        "writeMode": "append"
                    }
                }
            }
        ]
    }
}

This configuration conncets to an SFTP-enabled FTP server, reads the specified source text file, maps the source columns to matching fields in the target Hive table, and apends new data to the existing table. Adjust the number of concurrent channels, connection credentials, file path, field mappings, and write mode to match your specific data schema and cluster environment.