1、flume安裝目錄下新建文件夾 example?
2、在example下新建文件?
log-hdfs.conf
內容如下：
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1#exec 指的是命令
# Describe/configure the source
a1.sources.r1.type = exec
#F根據文件名追中, f根據文件的nodeid追中
a1.sources.r1.command = tail -F /home/hadoop/testdata/testflume.log
a1.sources.r1.channels = c1# Describe the sink
#下沉目標
a1.sinks.k1.type = hdfs
a1.sinks.k1.channel = c1
#指定目錄, flum幫做目的替換
a1.sinks.k1.hdfs.path = /flume/events/%y-%m-%d/%H%M/
#文件的命名, 前綴
a1.sinks.k1.hdfs.filePrefix = events-#10 分鐘就改目錄（創建目錄）， （這些參數影響/flume/events/%y-%m-%d/%H%M/）
a1.sinks.k1.hdfs.round = true
a1.sinks.k1.hdfs.roundValue = 10
a1.sinks.k1.hdfs.roundUnit = minute
#目錄里面有文件
#------start----兩個條件，只要符合其中一個就滿足---
#文件滾動之前的等待時間(秒)
a1.sinks.k1.hdfs.rollInterval = 3
#文件滾動的大小限制(bytes)
a1.sinks.k1.hdfs.rollSize = 500
#寫入多少個event數據后滾動文件(事件個數)
a1.sinks.k1.hdfs.rollCount = 20
#-------end-----#5個事件就往里面寫入
a1.sinks.k1.hdfs.batchSize = 5#用本地時間格式化目錄
a1.sinks.k1.hdfs.useLocalTimeStamp = true#下沉后, 生成的文件類型，默認是Sequencefile，可用DataStream，則為普通文本
a1.sinks.k1.hdfs.fileType = DataStream# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c13、shell命令不斷寫數據到文件
[hadoop@nbdo3 testdata]$ while true; do echo "hello ningbo do" >> testflume.log ; sleep 0.5; done
[hadoop@nbdo3 testdata]$ tail -f testflume.log?
hello ningbo do
hello ningbo do
hello ningbo do
hello ningbo do
hello ningbo do
hello ningbo do
hello ningbo do
hello ningbo do
hello ningbo do
hello ningbo do
5、啟動hadoop
6、啟動flume
flume-ng agent -c ../conf -flog-hdfs.conf ?-n a1 -Dflume.root.logger=INFO,console
7、瀏覽器進入hadoop管理界面。
)
