一、HDFS基礎操作
以root用戶登錄,創建如下HDFS目錄:
/dw/yourname/input
hadoop fs -mkdir -p /dw/zhanggengchen/input
/dw/yourname/output
hadoop fs -mkdir -p /dw/zhanggengchen/output
輸出結果:
[root@master hadoop-mapreduce]# hadoop fs -ls /dw/zhanggengchen
Found 2 items
drwxr-xr-x ? - root supergroup ? ? ? ? ?0 2025-07-03 18:14 /dw/zhanggengchen/input
drwxr-xr-x ? - root supergroup ? ? ? ? ?0 2025-07-03 18:15 /dw/zhanggengchen/output
上傳本地文件 /root/dataExercise/people.txt 到 /dw/yourname/input 目錄下。
hadoop fs -put /root/dataExercise/people.txt /dw/zhanggengchen/input/
列出 /dw/yourname/input 目錄內容,并顯示 people.txt 文件前5行內容。
hadoop fs -cat /dw/zhanggengchen/input/people.txt | head -n 5
輸出結果:
[root@master hadoop-mapreduce]# hadoop fs -cat /dw/zhanggengchen/input/people.txt | head -n 5
Michael, 29
Andy, 30
Justin, 19
二、Sqoop數據導入
Mysql數據庫連接信息:
主機:localhost
端口:3306
用戶名:root
密碼:bimao123
需求
使用Sqoop將MySQL數據庫的bigdatafamily中的customers表導入到HDFS目錄 /dw/yourname/customers,要求:
數據格式為文本,字段分隔符為\t
vim TODO3.sh
sqoop import \
--connect jdbc:mysql://localhost:3306/bigdatafamily \
--username root \
--password bimao123 \
--table customers \
--target-dir /dw/zhanggengchen/customers \
--fields-terminated-by '\t' \
--as-textfile
列出 /dw/yourname/customers 目錄內容,并顯示部分導入數據內容。
[root@master hadoop-mapreduce]# hadoop fs -ls /dw/zhanggengchen/customers
輸出結果:
Found 5 items
-rw-r--r-- ? 1 root supergroup ? ? ? ? ?0 2025-07-03 18:32 /dw/zhanggengchen/customers/_SUCCESS
-rw-r--r-- ? 1 root supergroup ? ?3144522 2025-07-03 18:32 /dw/zhanggengchen/customers/part-m-00000
-rw-r--r-- ? 1 root supergroup ? ?3143941 2025-07-03 18:32 /dw/zhanggengchen/customers/part-m-00001
-rw-r--r-- ? 1 root supergroup ? ?3144303 2025-07-03 18:32 /dw/zhanggengchen/customers/part-m-00002
-rw-r--r-- ? 1 root supergroup ? ?3144580 2025-07-03 18:32 /dw/zhanggengchen/customers/part-m-00003
[root@master hadoop-mapreduce]# hadoop fs -cat /dw/zhanggengchen/customers/part-m-00000 | head -n 3
輸出結果:
1000000?? ?Quentin?? ?Shepard?? ?32092 West 10th Street?? ?Prairie City?? ?SD?? ?57649
1000001?? ?Brandon?? ?Louis?? ?1311 North 2nd Street?? ?Clearfield?? ??? ?IA?? ?50840
1000002?? ?Marilyn?? ?Ham?? ?25831 North 25th Street?? ?Concord?? ?CA?? ?94522