hadoop - 在 Ubuntu 上安装 Flume

Question

我想在 ubuntu 上安装 Apache Flume 1.4.0 以移动数据并将其存储到 HDFS，但我找不到显示如何正确安装 flume 的安装指南。我下载了二进制 zip。有什么帮助吗？

Answer 1

Ubuntu 中的 Apache Flume 安装步骤:

Step 1 : Download the Latest Version of Flume. 

Step 2 : tar –xzvf apache-flume-1.5.0-bin.tar.gz

Step 3 : sudo mv apache-flume-1.5.2-bin /usr/local/flume

Step 4 : sudo nano ~/.bashrc

Step 5 : export FLUME_HOME=/usr/local/flume
export FLUME_CONF_DIR=$FLUME_HOME/conf
export FLUME_CLASS_PATH=$FLUME_CONF_DIR
export PATH=$FLUME_HOME/bin:$PATH

Step 6 : 

cp conf/flume-env.sh.template conf/flume-env.sh
Step 7 : 

sudo nano conf/flume-env.sh
JAVA_HOME=/usr/lib/jvm/jdk1.8.0
JAVA_OPTS="-Xms100m -Xmx200m -Dcom.sun.management.jmxremote"
Step 8: 
sudo nano flume-conf.properties.template 


agent.channels.memoryChannel.type = memory
agent.channels.memoryChannel.capacity = 100

# Define a source on agent and connect to channel memoryChannel.
agent.sources.tail-source.type = exec
agent.sources.tail-source.command = tail -F /opt/hadoop-2.6.0/logs/hadoop-hadoop-datanode-node1.log
agent.sources.tail-source.channels = memoryChannel

# Define a sink that outputs to logger.
agent.sinks.log-sink.channel = memoryChannel
agent.sinks.log-sink.type = logger

agent.sinks.hdfs-sink.channel = memoryChannel
agent.sinks.hdfs-sink.type = hdfs
agent.sinks.hdfs-sink.hdfs.path = hdfs://node1:8020/flumedata/
agent.sinks.hdfs-sink.hdfs.fileType = DataStream

# Activate channel, source and sinks
agent.channels = memoryChannel
agent.sources = tail-source
agent.sinks = log-sink hdfs-sink
Start Flume-ng agent

source ~/.bashrc

flume-ng agent -n agent -f conf/flume.conf -Dflume.root.logger=DEBUG,console
flume-ng --help 

希望这对您有所帮助。