前置操作

Apache SeaTunnel詳解與部署（最新版本2.3.11）-CSDN博客

mkdir /usr/local/soft/apache-seatunnel-2.3.11/job/

一、MySQL to HDFS

官方配置參考：

MySQL | Apache SeaTunnel

Hdfs文件 | Apache SeaTunnel

1、配置確認

將mysql-connector-java-8.0.15.jar放入到/usr/local/soft/apache-seatunnel-2.3.11/lib/文件夾中

2、啟動Hadoop

3、MySQL數據庫表創建

確認MySQL數據庫wjobs表user中有數據，沒有則進行創建

4、創建seatunnel配置文件

cd /usr/local/soft/apache-seatunnel-2.3.11/job/
touch mysql2hdfs.conf

內容如下：

env {parallelism = 2job.mode = "BATCH"
}
source {Jdbc {url = "jdbc:mysql://node11:3306/wjobs"driver = "com.mysql.cj.jdbc.Driver"connection_check_timeout_sec = 100user = "root"password = "root123"table_path = "wjobs.user"query = "select  * from wjobs.user"}
}sink {HdfsFile {fs.defaultFS = "hdfs://node11:9000"path = "/seatunnel_data/user"file_format_type = "text"field_delimiter = "\t"row_delimiter = "\n"custom_filename = truefile_name_expression = "${transactionId}_${now}"filename_time_format = "yyyy.MM.dd"sink_columns = ["id","username","password"]is_enable_transaction = true
}
}

或

env {parallelism = 2job.mode = "BATCH"
}
source {Jdbc {url = "jdbc:mysql://node11:3306/wjobs"driver = "com.mysql.cj.jdbc.Driver"connection_check_timeout_sec = 100user = "root"password = "root123"table_path = "wjobs.user"query = "select  * from wjobs.user"}
}sink {HdfsFile {fs.defaultFS = "hdfs://node11:9000"path = "/seatunnel_data/user"file_format_type = "text"field_delimiter = "\t"row_delimiter = "\n"is_enable_transaction = true
}
}

?5、執行seatunnel轉換命令

cd /usr/local/soft/apache-seatunnel-2.3.11/
./bin/seatunnel.sh --config ./job/mysql2hdfs.conf -m local

注意：此為單次啟動，不用啟動seatunnel服務?

二、Hive to MySQL

參考：Hive | Apache SeaTunnel

MySQL | Apache SeaTunnel

1、準備數據

確認hive中表及數據、MySQL對應表已創建

2、創建seatunnel配置文件

cd /usr/local/soft/apache-seatunnel-2.3.11/job/
touch hive2mysql.conf

內容如下：

env {parallelism = 2job.mode = "BATCH"
}
source {Hive {table_name = "uoapdb.ext_startup_logs"metastore_uri = "thrift://node11:9083"}
}sink {jdbc {url = "jdbc:mysql://node11:3306/wjobs?useUnicode=true&characterEncoding=UTF-8&rewriteBatchedStatements=true"driver = "com.mysql.cj.jdbc.Driver"user = "root"password = "root123"query = "insert into logs(userId,platform) values(?,?)"}
}

3、 啟動Hive

nohup hive --service metastore &
nohup hiveserver2 &

4、執行命令

cd /usr/local/soft/apache-seatunnel-2.3.11/
./bin/seatunnel.sh --config ./job/mysql2hdfs.conf -m local