Integrating DataX and DataX-Web with DataSophon (Single Node Setup)

This guide details the process of integrating DataX and DataX-Web in to a DataSophon environment, specifically focusing on a single-node deployment.

DataX Installation

Prerequisites

Java Development Kit (JDK) version 1.8 or higher (1.8 recommended).
Python 2 or 3. Note that Linux systems typically include Python 2. If using Python 3, script modifications may be necessary.
Apache Maven 3.x: Required for compiling DataX from source. If you are using the pre-compiled datax.tar.gz package, Maven is not needed for this step.

Obtaining and Compiling DataX

Method 1: Using Pre-compiled Package

Download the DataX tool package from the official DataX download page. After downloading, extract it to a chosen directory. You can then run synchronization jobs by navigating to the bin directory:

$ cd {YOUR_DATAX_HOME}/bin
$ python datax.py {YOUR_JOB.json}

To perform a self-check, execute:

python {YOUR_DATAX_HOME}/bin/datax.py {YOUR_DATAX_HOME}/job/job.json

Method 2: Compiling from Source

Clone the DataX source code from its GitHub repository. If you haven't configured SSH keys for GitHub, consider downloading the project as a ZIP archive.

# If SSH keys are configured
git clone git@github.com:alibaba/DataX.git
# Alternatively, if SSH is not configured
# git clone https://github.com/alibaba/DataX.git

Navigate to the DataX source code directory and compile using Maven:

$ cd {DataX_source_code_home}
$ mvn -U clean package assembly:assembly -Dmaven.test.skip=true

This compilation process can take 20-30 minutes depending on your network speed. A succesful build will show output similar to this:

[INFO] BUILD SUCCESS
[INFO] -----------------------------------------------------------------
[INFO] Total time: 08:12 min
[INFO] Finished at: 2015-12-13T16:26:48+08:00
[INFO] Final Memory: 133M/960M
[INFO] -----------------------------------------------------------------

The compiled DataX package will be located at {DataX_source_code_home}/target/datax/datax/. The directory structure will include:

$ cd {DataX_source_code_home}
$ ls ./target/datax/datax/
bin		conf		job		lib		log		log_perf	plugin		script		tmp

Uploading DataX Package

Place the datax.tar.gz package in the designated directory and generate its MD5 checksum:

cd /opt/datasophon/DDP/packages/
md5sum datax.tar.gz | awk '{print $1}' > datax.tar.gz.md5

Configuring DataSophon for DataX

Create a service definition file for DataX within the DataSophon menager's configuration directory:

mkdir -p /opt/datasophon/datasophon-manager-1.2.1/conf/meta/DDP-1.2.1/DATAX
vi /opt/datasophon/datasophon-manager-1.2.1/conf/meta/DDP-1.2.1/DATAX/service_ddl.json

Populate the file with the following JSON configuration:

{
  "name": "Datax",
  "label": "Datax",
  "description": "Offline data synchronization tool",
  "version": "1.0.0",
  "sortNum": 6,
  "dependencies": [],
  "packageName": "datax.tar.gz",
  "decompressPackageName": "datax",
  "runAs":"root",
  "roles": [
    {
      "name": "DataxClient",
      "label": "DataxClient",
      "roleType": "client",
      "cardinality": "1+",
      "logFile": ""
    }
  ],
  "configWriter": {
    "generators": []
  },
  "parameters": []
}

Restart DataSophon Manager API

Apply the configuration changes by restarting the DataSophon manager's API service:

sh /opt/datasophon/datasophon-manager-1.2.1/bin/datasophon-api.sh restart api

Installing DataX via DataSophon

Navigate to the DataSophon interface, add the DataX service, and proceed through the installation wizard. Select the target node(s) for deployment. No specific configuration is typically required during this stage, as DataSophon handles the unpacking of the DataX package to the specified nodes.

Note: The DataSophon integration primarily involves distributing and extracting the DataX package to the cluster nodes, facilitating subsequent integration with DataX-Web.

DataX-Web Installation

Prerequisites

MySQL version 5.5 or higher.
JDK version 1.8.0_xxx (e.g., 1.8.0_131).
Maven version 3.6.1 or higher (e.g., 3.6.3).
DataX version 3.0 or compatible.
Python 2.x (for executing DataX scripts). Note: Support for Python 3 requires modifications to specific DataX scripts found in doc/datax-web/datax-python3. Python 2.7.5 was used in this example.

Compiling DataX-Web

Clone the DataX-Web repository:

git clone https://github.com/WeiYe-Jing/datax-web.git

Compile the project from its root directory:

mvn clean install

Upon successful compilation, the distributable package (datax-web-{VERSION}.tar.gz) will be found in the build/ directory.

Installing DataX-Web

Extract the DataX-Web package to your desired installation location:

# Example for version 2.1.2
tar -zxvf datax-web-2.1.2.tar.gz

Execute the one-click installation script located in the bin directory:

# Interactive installation
./bin/install.sh
# Force installation without prompts
./bin/install.sh --force

The installation script will unpack modules and configure the application. Follow the prompts or use the --force option to skip interactive confirmations.

Modifying Status Script

Edit the status-all.sh script to accurately reflect the services being managed:

vi datax-web-2.1.2/bin/status-all.sh

Ensure the script contains logic to check the status of datax-admin and datax-executor:

#!/bin/bash
set -e

DATAX_SERVICES=("datax-admin" "datax-executor")

get_pid() {
    local service_name=$1
    pid=$(ps -ef | grep -v grep | grep "$service_name" | awk '{print $2}')
    echo "$pid"
}

check_status() {
    local service_names=("$@")
    local all_running=true

    for service_name in "${service_names[@]}"; do
        pid=$(get_pid "$service_name")
        if [ -z "$pid" ]; then
            echo "$service_name is NOT running."
            all_running=false
        else
            echo "$service_name is running with PID $pid."
        fi
    done

    if [ "$all_running" = true ]; then
        exit 0
    else
        exit 1
    fi
}

check_status "${DATAX_SERVICES[@]}"

Preparing DataX-Web Package for DataSophon

Copy the DataX-Web installation directory and its SQL initialization script to the DataSophon packages directory:

cp -r datax-web-2.1.2 /opt/datasophon/DDP/packages/
cp -r datax-web-2.1.2/db/datax_web.sql /opt/datasophon/DDP/packages/
cd /opt/datasophon/DDP/packages/
tar -czf datax-web-2.1.2.tar.gz datax-web-2.1.2
md5sum datax-web-2.1.2.tar.gz | awk '{print $1}' > datax-web-2.1.2.tar.gz.md5

Configuring DataSophon for DataX-Web

Create the DataX-Web service definition file within DataSophon's configuration:

mkdir -p /opt/datasophon/datasophon-manager-1.2.1/conf/meta/DDP-1.2.1/DATAXWEB
vi /opt/datasophon/datasophon-manager-1.2.1/conf/meta/DDP-1.2.1/DATAXWEB/service_ddl.json

Paste the following JSON configuration, adjusting parameters like database credentials and paths as necessary:


{
  "name": "DATAXWEB",
  "label": "DataxWeb",
  "description": "DATAX offline data visualization synchronization tool",
  "version": "2.1.2",
  "sortNum": 22,
  "dependencies": [],
  "packageName": "datax-web-2.1.2.tar.gz",
  "decompressPackageName": "datax-web-2.1.2",
  "roles": [{
  	"name": "DataxWeb",
  	"label": "DataxWeb",
  	"roleType": "worker",
  	"runAs": {
  		"user": "root",
  		"group": "root"
  	},
  	"cardinality": "1+",
  	"sortNum": 3,
  	"jmxPort": 2192,
  	"startRunner": {
  		"timeout": "60",
  		"program": "bin/start-all.sh",
  		"args": []
  	},
  	"stopRunner": {
  		"timeout": "600",
  		"program": "bin/stop-all.sh",
  		"args": []
  	},
  	"statusRunner": {
        "timeout": "60",
        "program": "bin/status-all.sh",
        "args": []
    },
	"externalLink": {
		"name": "DataxWebUi",
		"label": "DataxWebUi",
		"url": "http://${host}:9527/index.html"
	}
  }],
  "configWriter": {
  	"generators": [{
  		"filename": "bootstrap.properties",
  		"configFormat": "custom",
  		"outputDirectory": "modules/datax-admin/conf",
  		"templateName": "bootstrap-properties.ftl",
  		"includeParams": ["DB_HOST", "DB_PORT", "DB_USERNAME", "DB_PASSWORD", "DB_DATABASE"]
  	}, {
  		"filename": "datax-admin/bin/env.properties",
  		"configFormat": "custom",
  		"outputDirectory": "modules",
  		"templateName": "datax-admin-env-properties.ftl",
  		"includeParams": ["MAIL_USERNAME", "MAIL_PASSWORD"]
  	},{
  		"filename": "datax-executor/bin/env.properties",
  		"configFormat": "custom",
  		"outputDirectory": "modules",
  		"templateName": "datax-executor-env-properties.ftl",
  		"includeParams": ["PYTHON_PATH"]
  	}]
  },
  "parameters": [{
  	"name": "DB_HOST",
  	"label": "DataX-Web Database Hostname or IP",
  	"description": "Hostname or IP address of the DataX-Web database",
  	"configType": "map",
  	"required": true,
  	"type": "input",
  	"value": "192.168.10.21",
  	"configurableInWizard": true,
  	"hidden": false,
  	"defaultValue": "localhost"
  }, {
  	"name": "DB_PORT",
  	"label": "DataX-Web Database Port",
  	"description": "Port number the DataX-Web database is listening on",
  	"configType": "map",
  	"required": true,
  	"type": "input",
  	"value": "3306",
  	"configurableInWizard": true,
  	"hidden": false,
  	"defaultValue": "3306"
  }, {
  	"name": "DB_USERNAME",
  	"label": "DataX-Web Database Username",
  	"description": "Username for connecting to the DataX-Web database",
  	"configType": "map",
  	"required": true,
  	"type": "input",
  	"value": "root",
  	"configurableInWizard": true,
  	"hidden": false,
  	"defaultValue": "root"
  }, {
  	"name": "DB_PASSWORD",
  	"label": "DataX-Web Database Password",
  	"description": "Password for connecting to the DataX-Web database",
  	"configType": "map",
  	"required": true,
  	"type": "input",
  	"value": "yixiao666",
  	"configurableInWizard": true,
  	"hidden": false,
  	"defaultValue": "123456"
  }, {
  	"name": "DB_DATABASE",
  	"label": "DataX-Web Database Name",
  	"description": "Name of the database for DataX-Web",
  	"configType": "map",
  	"required": true,
  	"type": "input",
  	"value": "dataxweb",
  	"configurableInWizard": true,
  	"hidden": false,
  	"defaultValue": "dataxweb"
  }, {
  	"name": "MAIL_USERNAME",
  	"label": "Alert Email Username",
  	"description": "Username for the alert email account",
  	"configType": "map",
  	"required": true,
  	"type": "input",
  	"value": "",
  	"configurableInWizard": true,
  	"hidden": false,
  	"defaultValue": ""
  }, {
  	"name": "MAIL_PASSWORD",
  	"label": "Alert Email Password",
  	"description": "Password for the alert email account",
  	"configType": "map",
  	"required": true,
  	"type": "input",
  	"value": "",
  	"configurableInWizard": true,
  	"hidden": false,
  	"defaultValue": ""
  }, {
  	"name": "PYTHON_PATH",
  	"label": "DataX Python Interpreter Path",
  	"description": "Path to the DataX Python interpreter",
  	"configType": "map",
  	"required": true,
  	"type": "input",
  	"value": "/opt/datasophon/datax/bin/datax.py",
  	"configurableInWizard": true,
  	"hidden": false,
  	"defaultValue": "/opt/datasophon/datax/bin/datax.py"
  }]
}

Adding Template Files

On all nodes, create or update the following Freemarker template files in the DataSophon worker configuration directory:

`bootstrap-properties.ftl`

vi /opt/datasophon/datasophon-worker/conf/templates/bootstrap-properties.ftl


#Database
DB_HOST=${DB_HOST}
DB_PORT=${DB_PORT}
DB_USERNAME=${DB_USERNAME}
DB_PASSWORD=${DB_PASSWORD}
DB_DATABASE=${DB_DATABASE}

`datax-admin-env-properties.ftl`

vi /opt/datasophon/datasophon-worker/conf/templates/datax-admin-env-properties.ftl


# environment variables

JAVA_HOME="/usr/local/jdk1.8.0_333"

WEB_LOG_PATH=/opt/datasophon/dataxweb/modules/datax-admin/logs
WEB_CONF_PATH=/opt/datasophon/dataxweb/modules/datax-admin/conf

DATA_PATH=/opt/datasophon/dataxweb/modules/datax-admin/data
SERVER_PORT=9527


# mail account
MAIL_USERNAME="${MAIL_USERNAME}"
MAIL_PASSWORD="${MAIL_PASSWORD}"


#debug
#REMOTE_DEBUG_SWITCH=true
#REMOTE_DEBUG_PORT=7003

`datax-executor-env-properties.ftl`

vi /opt/datasophon/datasophon-worker/conf/templates/datax-executor-env-properties.ftl


# environment variables

JAVA_HOME="/usr/local/jdk1.8.0_333"

SERVICE_LOG_PATH=/opt/datasophon/dataxweb/modules/datax-executor/logs
SERVICE_CONF_PATH=/opt/datasophon/dataxweb/modules/datax-executor/conf
DATA_PATH=/opt/datasophon/dataxweb/modules/datax-executor/data


## datax json file storage location
JSON_PATH=/opt/datasophon/dataxweb/modules/datax-executor/json


## executor_port
EXECUTOR_PORT=9999


## Keep consistent with datax-admin port
DATAX_ADMIN_PORT=

## PYTHON script execution location
PYTHON_PATH=${PYTHON_PATH}



## dataxweb service port
SERVER_PORT=9504


#debug remote debug port
#REMOTE_DEBUG_SWITCH=true
#REMOTE_DEBUG_PORT=7004

Restart Services

Restart the worker services on all nodes:

sh /opt/datasophon/datasophon-worker/bin/datasophon-worker.sh restart worker

Restart the DataSophon manager API on the main node:

sh /opt/datasophon/datasophon-manager-1.2.1/bin/datasophon-api.sh restart api

Database Initialization

Create the dataxweb database if it doesn't exist and run the SQL script:

CREATE DATABASE dataxweb DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
USE dataxweb;
SOURCE /opt/datasophon/DDP/packages/datax_web.sql;

Installing DataX-Web via DataSophon

In the DataSophon interface, add the DataX-Web service. Assign roles (Admin, Executor) to the appropriate nodes in your cluster. Review and adjust the configuration parameters as needed during the installation wizard. A successful installation will confirm the DataX-Web service addition.

Tags: DataSophon DataX DataX-Web ETL Data Integration

Posted on Sun, 16 Aug 2026 17:02:57 +0000 by mgmoses

Freaks City

Integrating DataX and DataX-Web with DataSophon (Single Node Setup)

DataX Installation

Prerequisites

Obtaining and Compiling DataX

Method 1: Using Pre-compiled Package

Method 2: Compiling from Source

Uploading DataX Package

Configuring DataSophon for DataX

Restart DataSophon Manager API

Installing DataX via DataSophon

DataX-Web Installation

Prerequisites

Compiling DataX-Web

Installing DataX-Web

Modifying Status Script

Preparing DataX-Web Package for DataSophon

Configuring DataSophon for DataX-Web

Adding Template Files

`bootstrap-properties.ftl`

`datax-admin-env-properties.ftl`

`datax-executor-env-properties.ftl`

Restart Services

Database Initialization

Installing DataX-Web via DataSophon

Hot Tags