hadoop - 如何使用flume将数据从本地机器加载到hdfs

Question

我是flume的新手，所以请告诉我......如何使用flume将日志文件从我的本地机器存储到本地我的HDFS
我在设置类路径和 flume.conf 文件时遇到问题

谢谢，
杰伊

Answer 1

agent.sources = weblog
agent.channels = memoryChannel
agent.sinks = mycluster

## Sources #########################################################  
agent.sources.weblog.type = exec  
agent.sources.weblog.command = tail -F REPLACE-WITH-PATH2-your.log-FILE  
agent.sources.weblog.batchSize = 1  
agent.sources.weblog.channels =
REPLACE-WITH-
CHANNEL-NAME

## Channels ########################################################  
agent.channels.memoryChannel.type = memory  
agent.channels.memoryChannel.capacity = 100 agent.channels.memoryChannel.transactionCapacity = 100   

## Sinks ###########################################################   
agent.sinks.mycluster.type =REPLACE-WITH-CLUSTER-TYPE  
agent.sinks.mycluster.hdfs.path=/user/root/flumedata  
agent.sinks.mycluster.channel =REPLACE-WITH-CHANNEL-NAME

将此文件保存为 logagent.conf 并使用以下命令运行

# flume-ng agent –n agent –f logagent.conf &