目录：
- 一、准备环境
- （1）安装JDK
- （2）检查Python版本
- （3）DataX 解压及测试
- 二、同步测试
- 1、配置清单
- 2、操作示例
- （1）同步测试环境
- （2）准备测试表（postgresql）
- （3）准备测试表（Oracle）
- （4）创建并编辑json文件
- （5）执行json配置文件
一、准备环境
1、创建用户组和目录
groupadd -g 1500 datax
useradd -g datax -u 1500 datax
mkdir /datax
chown datax:datax /datax
安装datax两端服务器都可以，但是要保证网络畅通
2、系统环境
- Linux
- JDK(1.8以上，推荐1.8)
- Python(2或3都可以)
- Apache Maven 3.x (Compile DataX)（本文使用datax工具包直接解压，未采用编译方式）
（1）安装JDK
安装JDK（1.8.0_361）
# rpm -ivh jdk-8u361-linux-x64.rpm
warning: jdk-8u361-linux-x64.rpm: Header V3 RSA/SHA256 Signature, key ID ec551f03: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:jdk1.8-2000:1.8.0_361-fcs ################################# [100%]
Unpacking JAR files...
tools.jar...
plugin.jar...
javaws.jar...
deploy.jar...
rt.jar...
jsse.jar...
charsets.jar...
localedata.jar...
检查JDK版本
# java -version
java version "1.8.0_361"
Java(TM) SE Runtime Environment (build 1.8.0_361-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.361-b09, mixed mode)
（2）检查Python版本
# python -V
Python 2.7.5
我这边使用的是linux7自带的python版本
（3）DataX 解压及测试
Download ：DataX下载地址，上传到/datax目录
解压缩：
# tar -zxvf datax.tar.gz
自检脚本：
python {YOUR_DATAX_HOME}/bin/datax.py {YOUR_DATAX_HOME}/job/job.json
示例：
# python /datax/datax/bin/datax.py /datax/datax/job/job.json
DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.
2023-10-10 11:31:06.527 [main] INFO MessageSource - JVM TimeZone: GMT+08:00, Locale: zh_CN
2023-10-10 11:31:06.531 [main] INFO MessageSource - use Locale: zh_CN timeZone: sun.util.calendar.ZoneInfo[id="GMT+08:00",offset=28800000,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
2023-10-10 11:31:06.571 [main] INFO VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
2023-10-10 11:31:06.577 [main] INFO Engine - the machine info =>
osInfo: Oracle Corporation 1.8 25.361-b09
jvmInfo: Linux amd64 5.4.17-2011.6.2.el7uek.x86_64
cpu num: 4
totalPhysicalMemory: -0.00G
freePhysicalMemory: -0.00G
maxFileDescriptorCount: -1
currentOpenFileDescriptorCount: -1
GC Names [PS MarkSweep, PS Scavenge]
MEMORY_NAME | allocation_size | init_size
PS Eden Space | 256.00MB | 256.00MB
Code Cache | 240.00MB | 2.44MB
Compressed Class Space | 1,024.00MB | 0.00MB
PS Survivor Space | 42.50MB | 42.50MB
PS Old Gen | 683.00MB | 683.00MB
Metaspace | -0.00MB | 0.00MB
2023-10-10 11:31:06.595 [main] INFO Engine -
{
"content":[
{
"reader":{
"name":"streamreader",
"parameter":{
"column":[
{
"type":"string",
"value":"DataX"
},
{
"type":"long",
"value":19890604
},
{
"type":"date",
"value":"1989-06-04 00:00:00"
},
{
"type":"bool",
"value":true
},
{
"type":"bytes",
"value":"test"
}
],
"sliceRecordCount":100000
}
},
"writer":{
"name":"streamwriter",
"parameter":{
"encoding":"UTF-8",
"print":false
}
}
}
],
"setting":{
"errorLimit":{
"percentage":0.02,
"record":0
},
"speed":{
"channel":1
}
}
}
2023-10-10 11:31:06.615 [main] WARN Engine - prioriy set to 0, because NumberFormatException, the value is: null
2023-10-10 11:31:06.617 [main] INFO PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0
2023-10-10 11:31:06.617 [main] INFO JobContainer - DataX jobContainer starts job.
2023-10-10 11:31:06.620 [main] INFO JobContainer - Set jobId = 0
2023-10-10 11:31:06.641 [job-0] INFO JobContainer - jobContainer starts to do prepare ...
2023-10-10 11:31:06.643 [job-0] INFO JobContainer - DataX Reader.Job [streamreader] do prepare work .
2023-10-10 11:31:06.643 [job-0] INFO JobContainer - DataX Writer.Job [streamwriter] do prepare work .
2023-10-10 11:31:06.643 [job-0] INFO JobContainer - jobContainer starts to do split ...
2023-10-10 11:31:06.643 [job-0] INFO JobContainer - Job set Channel-Number to 1 channels.
2023-10-10 11:31:06.644 [job-0] INFO JobContainer - DataX Reader.Job [streamreader] splits to [1] tasks.
2023-10-10 11:31:06.645 [job-0] INFO JobContainer - DataX Writer.Job [streamwriter] splits to [1] tasks.
2023-10-10 11:31:06.667 [job-0] INFO JobContainer - jobContainer starts to do schedule ...
2023-10-10 11:31:06.674 [job-0] INFO JobContainer - Scheduler starts [1] taskGroups.
2023-10-10 11:31:06.677 [job-0] INFO JobContainer - Running by standalone Mode.
2023-10-10 11:31:06.685 [taskGroup-0] INFO TaskGroupContainer - taskGroupId=[0] start [1] channels for [1] tasks.
2023-10-10 11:31:06.688 [taskGroup-0] INFO Channel - Channel set byte_speed_limit to -1, No bps activated.
2023-10-10 11:31:06.688 [taskGroup-0] INFO Channel - Channel set record_speed_limit to -1, No tps activated.
2023-10-10 11:31:06.701 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
2023-10-10 11:31:07.003 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[304]ms
2023-10-10 11:31:07.004 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] completed it's tasks.
2023-10-10 11:31:16.696 [job-0] INFO StandAloneJobContainerCommunicator - Total 100000 records, 2600000 bytes | Speed 253.91KB/s, 10000 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.050s | All Task WaitReaderTime 0.060s | Percentage 100.00%
2023-10-10 11:31:16.696 [job-0] INFO AbstractScheduler - Scheduler accomplished all tasks.
2023-10-10 11:31:16.696 [job-0] INFO JobContainer - DataX Writer.Job [streamwriter] do post work.
2023-10-10 11:31:16.697 [job-0] INFO JobContainer - DataX Reader.Job [streamreader] do post work.
2023-10-10 11:31:16.697 [job-0] INFO JobContainer - DataX jobId [0] completed successfully.
2023-10-10 11:31:16.698 [job-0] INFO HookInvoker - No hook invoked, because base dir not exists or is a file: /datax/datax/hook
2023-10-10 11:31:16.700 [job-0] INFO JobContainer -
[total cpu info] =>
averageCpu | maxDeltaCpu | minDeltaCpu
-1.00% | -1.00% | -1.00%
[total gc info] =>
NAME | totalGCCount | maxDeltaGCCount | minDeltaGCCount | totalGCTime | maxDeltaGCTime | minDeltaGCTime
PS MarkSweep | 0 | 0 | 0 | 0.000s | 0.000s | 0.000s
PS Scavenge | 0 | 0 | 0 | 0.000s | 0.000s | 0.000s
2023-10-10 11:31:16.700 [job-0] INFO JobContainer - PerfTrace not enable!
2023-10-10 11:31:16.701 [job-0] INFO StandAloneJobContainerCommunicator - Total 100000 records, 2600000 bytes | Speed 253.91KB/s, 10000 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.050s | All Task WaitReaderTime 0.060s | Percentage 100.00%
2023-10-10 11:31:16.705 [job-0] INFO JobContainer -
任务启动时刻 : 2023-10-10 11:31:06
任务结束时刻 : 2023-10-10 11:31:16
任务总计耗时 : 10s
任务平均流量 : 253.91KB/s
记录写入速度 : 10000rec/s
读出记录总数 : 100000
读写失败总数 : 0
自检脚本无法通过报错处理：https://developer.aliyun.com/ask/376622?spm=a2c6h.13148508.0.0.1d734f0eKYBCG7
二、同步测试
1、配置清单
在两个不同数据库之间使用datax数据迁移可以通过命令查看配置模板： python datax.py -r {Sourcedb_READER} -w {Targetdb_WRITER}
示例：查看PostgreSQL到Oracle数据传输json文件模板
# python /datax/datax/bin/datax.py -r postgresqlreader -w oraclewriter
DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.
Please refer to the postgresqlreader document:
https://github.com/alibaba/DataX/blob/master/postgresqlreader/doc/postgresqlreader.md
Please refer to the oraclewriter document:
https://github.com/alibaba/DataX/blob/master/oraclewriter/doc/oraclewriter.md
Please save the following configuration as a json file and use
python {DATAX_HOME}/bin/datax.py {JSON_FILE_NAME}.json
to run the job.
{
"job": {
"content": [
{
"reader": {
"name": "postgresqlreader",
"parameter": {
"connection": [
{
"jdbcUrl": [],
"table": []
}
],
"password": "",
"username": ""
}
},
"writer": {
"name": "oraclewriter",
"parameter": {
"column": [],
"connection": [
{
"jdbcUrl": "",
"table": []
}
],
"password": "",
"preSql": [],
"username": ""
}
}
}
],
"setting": {
"speed": {
"channel": ""
}
}
}
}
详情查看：
Please refer to the postgresqlreader document:
https://github.com/alibaba/DataX/blob/master/postgresqlreader/doc/postgresqlreader.mdPlease refer to the oraclewriter document:
https://github.com/alibaba/DataX/blob/master/oraclewriter/doc/oraclewriter.md
2、操作示例
（1）同步测试环境
| OS版本 | CentOS Linux release 7.6.1810 (Core) | Ol7.9 |
|---|---|---|
| DB版本 | PostgreSQL12.3 | Oracle19c (经典模式) |
| 同步方向 | Reader | Writer |
| IP | 192.168.245.9 | 192.168.245.10 |
| 端口 | 5432 | 1521 |
| 数据库名 | htdb | orcl |
| 用户/密码 | htuser/htuser123 | shuaige/shuaige123 |
| 表名 | httab | httab |
两个虚拟机均使用
OracleLinux7.9操作系统最小化安装，关闭防火墙和selinux
postgresql_304">（2）准备测试表（postgresql）
-- 1、登录postgresql
------------------------------------------------------------------------------------
$ psql -h192.168.245.9 -p5432 -Uhtuser htdb
Password for user htuser:
psql (12.3)
Type "help" for help.
htdb=# \c
You are now connected to database "htdb" as user "htuser".
htdb=#
-- 2、创建测试表
------------------------------------------------------------------------------------
htdb=# create table httab(id serial primary key,name text,age int);
CREATE TABLE
-- 3、检查新建表
------------------------------------------------------------------------------------
htdb=# \d httab
Table "public.httab"
Column | Type | Collation | Nullable | Default
--------+---------+-----------+----------+-----------------------------------
id | integer | | not null | nextval('httab_id_seq'::regclass)
name | text | | |
age | integer | | |
Indexes:
"httab_pkey" PRIMARY KEY, btree (id)
htdb=#
-- 4、插入测试数据
------------------------------------------------------------------------------------
htdb=# insert into httab(name,age) select md5(random()::text),rn from generate_series(1,10000) rn;
INSERT 0 10000
-- 5、查询测试数据
------------------------------------------------------------------------------------
htdb=# select * from httab;
id | name | age
-------+----------------------------------+-------
10001 | 4c943d9e2be750b836365166b8bbfdf1 | 1
10002 | ddbad0963e08c9642d1692e9961677f7 | 2
10003 | 40551c9e3f00b4e7ea7c81062a94b20b | 3
...
（3）准备测试表（Oracle）
-- 1、登录oracle
------------------------------------------------------------------------------------
$ sqlplus / as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Tue Oct 10 16:06:40 2023
Version 19.3.0.0.0
Copyright (c) 1982, 2019, Oracle. All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0
SYS@orcl>
-- 1、创建测试表
------------------------------------------------------------------------------------
SYS@orcl> create table shuaige.httab (id number(30) primary key,name VARCHAR2(1000) not null,age int);
Table created.
注意更改用户名和表空间,不指定使用用户默认表空间
（4）创建并编辑json文件
在/datax/datax/job目录下用postgresqlreader to oraclewriter模板配置json文件
vi /datax/datax/job/postgresql2oracle_httab.json
编辑配置模板按照实际环境修改：
{
"job": {
"content": [
{
"reader": {
"name": "postgresqlreader",
"parameter": {
"connection": [
{
"jdbcUrl": ["jdbc:postgresql://192.168.245.9:5432/htdb"],
"table": ["httab"]
}
],
"column": ["*"],
"splitPk": "id",
"password": "htuser123",
"username": "htuser"
}
},
"writer": {
"name": "oraclewriter",
"parameter": {
"column": ["*"],
"connection": [
{
"jdbcUrl": "jdbc:oracle:thin:@192.168.245.10:1521:orcl",
"table": ["httab"]
}
],
"password": "shuaige123",
"preSql": ["truncate table httab"],
"username": "shuaige"
}
}
}
],
"setting": {
"speed": {
"channel": "4"
}
}
}
}
（5）执行json配置文件
# python /datax/datax/bin/datax.py /datax/datax/job/postgresql2oracle_httab.json
DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.
2023-10-10 17:04:26.033 [main] INFO MessageSource - JVM TimeZone: GMT+08:00, Locale: zh_CN
2023-10-10 17:04:26.035 [main] INFO MessageSource - use Locale: zh_CN timeZone: sun.util.calendar.ZoneInfo[id="GMT+08:00",offset=28800000,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
2023-10-10 17:04:26.060 [main] INFO VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
2023-10-10 17:04:26.066 [main] INFO Engine - the machine info =>
osInfo: Oracle Corporation 1.8 25.361-b09
jvmInfo: Linux amd64 5.4.17-2011.6.2.el7uek.x86_64
cpu num: 4
totalPhysicalMemory: -0.00G
freePhysicalMemory: -0.00G
maxFileDescriptorCount: -1
currentOpenFileDescriptorCount: -1
GC Names [PS MarkSweep, PS Scavenge]
MEMORY_NAME | allocation_size | init_size
PS Eden Space | 256.00MB | 256.00MB
Code Cache | 240.00MB | 2.44MB
Compressed Class Space | 1,024.00MB | 0.00MB
PS Survivor Space | 42.50MB | 42.50MB
PS Old Gen | 683.00MB | 683.00MB
Metaspace | -0.00MB | 0.00MB
2023-10-10 17:04:26.080 [main] INFO Engine -
{
"content":[
{
"reader":{
"name":"postgresqlreader",
"parameter":{
"column":[
"*"
],
"connection":[
{
"jdbcUrl":[
"jdbc:postgresql://192.168.245.9:5432/htdb"
],
"table":[
"httab"
]
}
],
"password":"*********",
"splitPk":"id",
"username":"htuser"
}
},
"writer":{
"name":"oraclewriter",
"parameter":{
"column":[
"*"
],
"connection":[
{
"jdbcUrl":"jdbc:oracle:thin:@192.168.245.10:1521:orcl",
"table":[
"httab"
]
}
],
"password":"**********",
"preSql":[
"truncate table httab"
],
"username":"shuaige"
}
}
}
],
"setting":{
"speed":{
"channel":"4"
}
}
}
2023-10-10 17:04:26.094 [main] WARN Engine - prioriy set to 0, because NumberFormatException, the value is: null
2023-10-10 17:04:26.095 [main] INFO PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0
2023-10-10 17:04:26.096 [main] INFO JobContainer - DataX jobContainer starts job.
2023-10-10 17:04:26.097 [main] INFO JobContainer - Set jobId = 0
2023-10-10 17:04:26.372 [job-0] INFO OriginalConfPretreatmentUtil - Available jdbcUrl:jdbc:postgresql://192.168.245.9:5432/htdb.
2023-10-10 17:04:26.373 [job-0] WARN OriginalConfPretreatmentUtil - 您的配置文件中的列配置存在一定的风险. 因为您未配置读取数据库表的列，当您的表字段个数、类型有变动时，可能影响任务正确性甚至会运行出错。请检查您的配置并作出修改.
2023-10-10 17:04:26.576 [job-0] INFO OriginalConfPretreatmentUtil - table:[httab] all columns:[
ID,NAME,AGE
].
2023-10-10 17:04:26.576 [job-0] WARN OriginalConfPretreatmentUtil - 您的配置文件中的列配置信息存在风险. 因为您配置的写入数据库表的列为*，当您的表字段个数、类型有变动时，可能影响任务正确性甚至会运行出错。请检查您的配置并作出修改.
2023-10-10 17:04:26.578 [job-0] INFO OriginalConfPretreatmentUtil - Write data [
INSERT INTO %s (ID,NAME,AGE) VALUES(?,?,?)
], which jdbcUrl like:[jdbc:oracle:thin:@192.168.245.10:1521:orcl]
2023-10-10 17:04:26.578 [job-0] INFO JobContainer - jobContainer starts to do prepare ...
2023-10-10 17:04:26.578 [job-0] INFO JobContainer - DataX Reader.Job [postgresqlreader] do prepare work .
2023-10-10 17:04:26.578 [job-0] INFO JobContainer - DataX Writer.Job [oraclewriter] do prepare work .
2023-10-10 17:04:26.627 [job-0] INFO CommonRdbmsWriter$Job - Begin to execute preSqls:[truncate table httab]. context info:jdbc:oracle:thin:@192.168.245.10:1521:orcl.
2023-10-10 17:04:26.648 [job-0] INFO JobContainer - jobContainer starts to do split ...
2023-10-10 17:04:26.648 [job-0] INFO JobContainer - Job set Channel-Number to 4 channels.
2023-10-10 17:04:26.667 [job-0] INFO SingleTableSplitUtil - split pk [sql=SELECT MIN(id),MAX(id) FROM httab] is running...
2023-10-10 17:04:26.690 [job-0] INFO SingleTableSplitUtil - After split(), allQuerySql=[
select * from httab where (10001 <= id AND id < 10501)
select * from httab where (10501 <= id AND id < 11001)
select * from httab where (11001 <= id AND id < 11501)
select * from httab where (11501 <= id AND id < 12001)
select * from httab where (12001 <= id AND id < 12501)
select * from httab where (12501 <= id AND id < 13001)
select * from httab where (13001 <= id AND id < 13501)
select * from httab where (13501 <= id AND id < 14001)
select * from httab where (14001 <= id AND id < 14501)
select * from httab where (14501 <= id AND id < 15001)
select * from httab where (15001 <= id AND id < 15501)
select * from httab where (15501 <= id AND id < 16001)
select * from httab where (16001 <= id AND id < 16501)
select * from httab where (16501 <= id AND id < 17001)
select * from httab where (17001 <= id AND id < 17501)
select * from httab where (17501 <= id AND id < 18001)
select * from httab where (18001 <= id AND id < 18501)
select * from httab where (18501 <= id AND id < 19001)
select * from httab where (19001 <= id AND id < 19501)
select * from httab where (19501 <= id AND id <= 20000)
select * from httab where id IS NULL
].
2023-10-10 17:04:26.691 [job-0] INFO JobContainer - DataX Reader.Job [postgresqlreader] splits to [21] tasks.
2023-10-10 17:04:26.692 [job-0] INFO JobContainer - DataX Writer.Job [oraclewriter] splits to [21] tasks.
2023-10-10 17:04:26.712 [job-0] INFO JobContainer - jobContainer starts to do schedule ...
2023-10-10 17:04:26.756 [job-0] INFO JobContainer - Scheduler starts [1] taskGroups.
2023-10-10 17:04:26.758 [job-0] INFO JobContainer - Running by standalone Mode.
2023-10-10 17:04:26.771 [taskGroup-0] INFO TaskGroupContainer - taskGroupId=[0] start [4] channels for [21] tasks.
2023-10-10 17:04:26.775 [taskGroup-0] INFO Channel - Channel set byte_speed_limit to -1, No bps activated.
2023-10-10 17:04:26.776 [taskGroup-0] INFO Channel - Channel set record_speed_limit to -1, No tps activated.
2023-10-10 17:04:26.782 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[14] attemptCount[1] is started
2023-10-10 17:04:26.785 [0-0-14-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab where (17001 <= id AND id < 17501)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:26.786 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[3] attemptCount[1] is started
2023-10-10 17:04:26.787 [0-0-3-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab where (11501 <= id AND id < 12001)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:26.801 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[13] attemptCount[1] is started
2023-10-10 17:04:26.802 [0-0-13-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab where (16501 <= id AND id < 17001)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:26.807 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[2] attemptCount[1] is started
2023-10-10 17:04:26.807 [0-0-2-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab where (11001 <= id AND id < 11501)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:26.831 [0-0-14-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab where (17001 <= id AND id < 17501)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:26.836 [0-0-2-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab where (11001 <= id AND id < 11501)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:26.851 [0-0-13-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab where (16501 <= id AND id < 17001)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:26.865 [0-0-3-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab where (11501 <= id AND id < 12001)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.110 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[13] is successed, used[323]ms
2023-10-10 17:04:27.157 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[4] attemptCount[1] is started
2023-10-10 17:04:27.157 [0-0-4-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab where (12001 <= id AND id < 12501)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.180 [0-0-4-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab where (12001 <= id AND id < 12501)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.258 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[2] is successed, used[456]ms
2023-10-10 17:04:27.258 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[14] is successed, used[477]ms
2023-10-10 17:04:27.259 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[18] attemptCount[1] is started
2023-10-10 17:04:27.261 [0-0-18-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab where (19001 <= id AND id < 19501)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.262 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
2023-10-10 17:04:27.263 [0-0-0-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab where (10001 <= id AND id < 10501)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.282 [0-0-18-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab where (19001 <= id AND id < 19501)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.300 [0-0-0-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab where (10001 <= id AND id < 10501)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.363 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[3] is successed, used[578]ms
2023-10-10 17:04:27.363 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[4] is successed, used[252]ms
2023-10-10 17:04:27.365 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[15] attemptCount[1] is started
2023-10-10 17:04:27.366 [0-0-15-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab where (17501 <= id AND id < 18001)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.368 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[6] attemptCount[1] is started
2023-10-10 17:04:27.368 [0-0-6-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab where (13001 <= id AND id < 13501)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.387 [0-0-15-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab where (17501 <= id AND id < 18001)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.402 [0-0-6-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab where (13001 <= id AND id < 13501)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.569 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[309]ms
2023-10-10 17:04:27.571 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[5] attemptCount[1] is started
2023-10-10 17:04:27.571 [0-0-5-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab where (12501 <= id AND id < 13001)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.590 [0-0-5-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab where (12501 <= id AND id < 13001)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.672 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[6] is successed, used[306]ms
2023-10-10 17:04:27.672 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[15] is successed, used[307]ms
2023-10-10 17:04:27.673 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[18] is successed, used[414]ms
2023-10-10 17:04:27.674 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[1] attemptCount[1] is started
2023-10-10 17:04:27.675 [0-0-1-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab where (10501 <= id AND id < 11001)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.677 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[10] attemptCount[1] is started
2023-10-10 17:04:27.677 [0-0-10-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab where (15001 <= id AND id < 15501)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.679 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[17] attemptCount[1] is started
2023-10-10 17:04:27.679 [0-0-17-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab where (18501 <= id AND id < 19001)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.692 [0-0-1-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab where (10501 <= id AND id < 11001)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.709 [0-0-17-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab where (18501 <= id AND id < 19001)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.725 [0-0-10-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab where (15001 <= id AND id < 15501)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.981 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[10] is successed, used[306]ms
2023-10-10 17:04:27.981 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[17] is successed, used[303]ms
2023-10-10 17:04:27.983 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[11] attemptCount[1] is started
2023-10-10 17:04:27.983 [0-0-11-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab where (15501 <= id AND id < 16001)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.985 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[16] attemptCount[1] is started
2023-10-10 17:04:27.985 [0-0-16-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab where (18001 <= id AND id < 18501)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:28.008 [0-0-11-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab where (15501 <= id AND id < 16001)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:28.021 [0-0-16-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab where (18001 <= id AND id < 18501)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:28.086 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[5] is successed, used[516]ms
2023-10-10 17:04:28.090 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[19] attemptCount[1] is started
2023-10-10 17:04:28.090 [0-0-19-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab where (19501 <= id AND id <= 20000)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:28.217 [0-0-19-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab where (19501 <= id AND id <= 20000)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:28.307 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[1] is successed, used[633]ms
2023-10-10 17:04:28.309 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[7] attemptCount[1] is started
2023-10-10 17:04:28.309 [0-0-7-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab where (13501 <= id AND id < 14001)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:28.376 [0-0-7-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab where (13501 <= id AND id < 14001)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:28.511 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[19] is successed, used[423]ms
2023-10-10 17:04:28.513 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[8] attemptCount[1] is started
2023-10-10 17:04:28.514 [0-0-8-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab where (14001 <= id AND id < 14501)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:28.617 [0-0-8-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab where (14001 <= id AND id < 14501)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:28.717 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[11] is successed, used[735]ms
2023-10-10 17:04:28.717 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[16] is successed, used[734]ms
2023-10-10 17:04:28.718 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[9] attemptCount[1] is started
2023-10-10 17:04:28.719 [0-0-9-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab where (14501 <= id AND id < 15001)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:28.720 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[12] attemptCount[1] is started
2023-10-10 17:04:28.720 [0-0-12-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab where (16001 <= id AND id < 16501)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:28.814 [0-0-9-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab where (14501 <= id AND id < 15001)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:28.892 [0-0-12-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab where (16001 <= id AND id < 16501)
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:29.248 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[8] is successed, used[736]ms
2023-10-10 17:04:29.250 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[20] attemptCount[1] is started
2023-10-10 17:04:29.250 [0-0-20-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab where id IS NULL
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:29.269 [0-0-20-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab where id IS NULL
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:29.502 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[12] is successed, used[783]ms
2023-10-10 17:04:29.706 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[7] is successed, used[1398]ms
2023-10-10 17:04:29.806 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[9] is successed, used[1088]ms
2023-10-10 17:04:29.806 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[20] is successed, used[557]ms
2023-10-10 17:04:29.807 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] completed it's tasks.
2023-10-10 17:04:36.773 [job-0] INFO StandAloneJobContainerCommunicator - Total 10000 records, 408894 bytes | Speed 39.93KB/s, 1000 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.010s | All Task WaitReaderTime 0.000s | Percentage 100.00%
2023-10-10 17:04:36.773 [job-0] INFO AbstractScheduler - Scheduler accomplished all tasks.
2023-10-10 17:04:36.774 [job-0] INFO JobContainer - DataX Writer.Job [oraclewriter] do post work.
2023-10-10 17:04:36.774 [job-0] INFO JobContainer - DataX Reader.Job [postgresqlreader] do post work.
2023-10-10 17:04:36.774 [job-0] INFO JobContainer - DataX jobId [0] completed successfully.
2023-10-10 17:04:36.775 [job-0] INFO HookInvoker - No hook invoked, because base dir not exists or is a file: /datax/datax/hook
2023-10-10 17:04:36.775 [job-0] INFO JobContainer -
[total cpu info] =>
averageCpu | maxDeltaCpu | minDeltaCpu
-1.00% | -1.00% | -1.00%
[total gc info] =>
NAME | totalGCCount | maxDeltaGCCount | minDeltaGCCount | totalGCTime | maxDeltaGCTime | minDeltaGCTime
PS MarkSweep | 1 | 1 | 1 | 0.018s | 0.018s | 0.018s
PS Scavenge | 1 | 1 | 1 | 0.009s | 0.009s | 0.009s
2023-10-10 17:04:36.776 [job-0] INFO JobContainer - PerfTrace not enable!
2023-10-10 17:04:36.776 [job-0] INFO StandAloneJobContainerCommunicator - Total 10000 records, 408894 bytes | Speed 39.93KB/s, 1000 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.010s | All Task WaitReaderTime 0.000s | Percentage 100.00%
2023-10-10 17:04:36.777 [job-0] INFO JobContainer -
任务启动时刻 : 2023-10-10 17:04:26
任务结束时刻 : 2023-10-10 17:04:36
任务总计耗时 : 10s
任务平均流量 : 39.93KB/s
记录写入速度 : 1000rec/s
读出记录总数 : 10000
读写失败总数 : 0
执行成功，检查Oracle数据库查看数据是否同步成功
SYS@orcl> col name for a50
SYS@orcl> select * from shuaige.httab where rownum < 10;
ID NAME AGE
---------- -------------------------------------------- ----------
16809 5f19fd4c4f65352c79318089634ee459 6809
16810 039773606c114ae52af27fd6bb2603e7 6810
16811 9dd5a56b63e6192e6f091c3dfd600f50 6811
16812 564147edcc95fe83af7b5bf2cc42268a 6812
16813 1dc07f8e48bd6baf258c0a341ad8753a 6813
16814 7569553c59834aff4e0c78c8a5292a9f 6814
16815 b93d01952acc42d345e06a062166693f 6815
16816 d60332e3fef6292da8c98d963435ec9e 6816
16817 9f3271cd20b8e473ae9e98c4b258f45d 6817
9 rows selected.
数据传输成功，如果在执行json文件报错，一定要先看好json文件格式，用户密码大小写，[ ]，" “等，“channel"参数至少为“1”，“preSql”:[””] SQL条件可以不加，但是有主键冲突的行会报错传输失败。
