Hadoop之HDFS
HDFS的Shell操作
-
啟動Hadoop集群（方便后續測試）
[atguigu@hadoop102 ~]$ sbin/start-dfs.sh [atguigu@hadoop102 ~]$ sbin/start-yarn.sh -
-help：輸出這個命令參數
[atguigu@hadoop102 ~]$ hadoop fs -help rm -
-ls：顯示目錄信息
[atguigu@hadoop102 ~]$ hadoop fs -ls / -
-mkdir：在HDFS上創建目錄
[atguigu@hadoop102 ~]$ hadoop fs -mkdir -p /user/atguigu/input [atguigu@hadoop102 ~]$ hadoop fs -mkdir /bigdata0523 -
-moveFromLocal：從本地剪切粘貼到HDFS
[atguigu@hadoop102 hadoop-3.1.3]$ vim qiangge.txt --- 只年說：流年同學你好，可以加個微信嗎？ --- [atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -moveFromLocal ./qiangge.txt /bigdata0523 -
-appendToFile：追加一個文件到已經存在的文件末尾
[atguigu@hadoop102 hadoop-3.1.3]$ vim pantongxue.txt --- 流年同學說：你說加就加啊。不給！！ --- [atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -appendToFile pantongxue.txt /bigdata0523/qiangge.txt -
-cat：顯示文件內容
[atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -cat /bigdata0523/qiangge.txt 2025-07-05 12:03:57,587 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false 只年說：流年同學你好，可以加個微信嗎？ 流年同學說：你說加就加啊。不給！！ -
-chgrp、-chmod、-chown：Linux文件系統中的用法一樣，修改文件所屬權限
[atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -chmod u+x /bigdata0523/qiangge.txt -
-copyFromLocal：從本地文件系統中拷貝文件到HDFS路徑去
[atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -copyFromLocal pantongxue.txt /bigdata0523-put效果和它相同，一般用put就行
[atguigu@hadoop102 hadoop-3.1.3]$ vim xinge.txt --- 龍哥說：你們倆在干啥，帶我一個唄！！！ --- [atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -put xinge.txt /bigdata0523 -
-copyToLocal：從HDFS拷貝到本地
[atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -copyToLocal /bigdata0523/qiangge.txt .-get效果和它相同，一般用get就行
[atguigu@hadoop102 hadoop-3.1.3]$ vim mengmeng.txt --- 被班主任耽誤的舞蹈選手 --- [atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -moveFromLocal mengmeng.txt /bigdata0523 [atguigu@hadoop102 hadoop-3.1.3]$ ll | grep mengmeng [atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -get /bigdata0523/mengmeng.txt . [atguigu@hadoop102 hadoop-3.1.3]$ ll | grep mengmeng -rw-r--r--. 1 atguigu atguigu 34 7月 5 12:34 mengmeng.txt -
-cp：從HDFS的一個路徑拷貝到HDFS的另一個路徑
[atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -cp /bigdata0523/mengmeng.txt / -
-mv：在HDFS目錄中移動文件
[atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -mv /mengmeng.txt /user/atguigu/input # 同時還具備改名的功能，下方這種寫法 dageoge.txt前方沒有明確路徑，默認是放到 /user/atguigu 下面了。可以理解為類似于linux也有用戶家目錄的概念 [atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -mv /user/atguigu/input/mengmeng.txt dagaoge.txt # 移動的同時還可以更改名稱 [atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -mv /user/atguigu/dagaoge.txt /user/atguigu/input/mengmeng.txt -
-get：等同于copyToLocal，就是從HDFS下載文件到本地
# 上方copyToLocal已經說過-get，此處不再操作 hadoop fs -get /bigdata0523/mengmeng.txt . -
-getmerge：合并下載多個文件，比如HDFS的目錄/user/atguigu/test 下有多個文件：log.1，log.2，log.3…
# 不用看后綴名，后綴名沒啥用，里面是合并后的內容。getmerge用的比較少 [atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -getmerge /bigdata0523 ./teacherAndStudent.avi [atguigu@hadoop102 hadoop-3.1.3]$ cat teacherAndStudent.avi 被班主任耽誤的舞蹈選手 流年同學說：你說加就加啊。不給！！ 只年說：流年同學你好，可以加個微信嗎？ 流年同學說：你說加就加啊。不給！！ 龍哥說：你們倆在干啥，帶我一個唄！！！ -
-put：等同于copyFromLocal
# 上方copyFromLocal已經說過-put，此處不再操作 hadoop fs -put xinge.txt /bigdata0523 -
-tail：顯示一個文件的末尾
[atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -tail /input/README.txt # -n查看指定多少行不能用！實時查看可以跟-f [atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -tail -f /input/README.txt# 當我們把副本數改為5，再去做操作的時候報錯 [atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -appendToFile mengmeng.txt /input/README.txt 2025-07-05 13:11:46,387 WARN hdfs.DataStreamer: DataStreamer Exception java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[DatanodeInfoWithStorage[192.168.1.102:9866,DS-da913c4d-3a02-424d-b964-2a3602c1db98,DISK], DatanodeInfoWithStorage[192.168.1.103:9866,DS-3849d948-2fd7-40c0-925b-c498f6c67f76,DISK], DatanodeInfoWithStorage[192.168.1.104:9866,DS-692fc8b7-3c6d-464e-8d81-674708d0ee44,DISK]], original=[DatanodeInfoWithStorage[192.168.1.104:9866,DS-692fc8b7-3c6d-464e-8d81-674708d0ee44,DISK], DatanodeInfoWithStorage[192.168.1.103:9866,DS-3849d948-2fd7-40c0-925b-c498f6c67f76,DISK], DatanodeInfoWithStorage[192.168.1.102:9866,DS-da913c4d-3a02-424d-b964-2a3602c1db98,DISK]]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration.at org.apache.hadoop.hdfs.DataStreamer.findNewDatanode(DataStreamer.java:1304)at org.apache.hadoop.hdfs.DataStreamer.addDatanode2ExistingPipeline(DataStreamer.java:1372)at org.apache.hadoop.hdfs.DataStreamer.handleDatanodeReplacement(DataStreamer.java:1598)at org.apache.hadoop.hdfs.DataStreamer.setupPipelineInternal(DataStreamer.java:1499)at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1481)at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:720) appendToFile: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[DatanodeInfoWithStorage[192.168.1.102:9866,DS-da913c4d-3a02-424d-b964-2a3602c1db98,DISK], DatanodeInfoWithStorage[192.168.1.103:9866,DS-3849d948-2fd7-40c0-925b-c498f6c67f76,DISK], DatanodeInfoWithStorage[192.168.1.104:9866,DS-692fc8b7-3c6d-464e-8d81-674708d0ee44,DISK]], original=[DatanodeInfoWithStorage[
