我正在尝试使用 flume 将 Twitter 数据流式传输到 hdfs 中:https://github.com/cloudera/cdh-twitter-example/
无论我在这里尝试什么,它都会不断在 HDFS 中创建大小在 1.5kB 到 15kB 之间的文件,而我希望看到大文件 (64Mb)。 这是代理配置:
TwitterAgent.sources = Twitter
TwitterAgent.channels = MemChannel
TwitterAgent.sinks = HDFS
TwitterAgent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource
TwitterAgent.sources.Twitter.channels = MemChannel
TwitterAgent.sources.Twitter.consumerKey = xxxxxx
TwitterAgent.sources.Twitter.consumerSecret = xxxxxx
TwitterAgent.sources.Twitter.accessToken = xxxxx
TwitterAgent.sources.Twitter.accessTokenSecret = xxxxxxxx
TwitterAgent.sources.Twitter.keywords = test
TwitterAgent.sinks.HDFS.channel = MemChannel
TwitterAgent.sinks.HDFS.type = hdfs
TwitterAgent.sinks.HDFS.hdfs.path = hdfs://localhost.localdomain:8020/user/flume/tweets/%Y/%m/%d/%H/
TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream
TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text
TwitterAgent.sinks.HDFS.hdfs.batchSize = 10000
TwitterAgent.sinks.HDFS.hdfs.rollSize = 67108864
TwitterAgent.sinks.HDFS.hdfs.rollCount = 0
TwitterAgent.sinks.HDFS.hdfs.rollInterval = 0
TwitterAgent.sinks.HDFS.hdfs.idleTimeout = 0
TwitterAgent.channels.MemChannel.type = memory
TwitterAgent.channels.MemChannel.capacity = 10000
TwitterAgent.channels.MemChannel.transactionCapacity = 1000
编辑:我查看了日志文件,发现这种情况一直在发生:
9:11:27.526 AM WARN org.apache.flume.sink.hdfs.BucketWriter Block Under-replication detected. Rotating file. 9:11:37.036 AM ERROR org.apache.flume.sink.hdfs.BucketWriter
Hit max consecutive under-replication rotations (30); will not continue rolling files under this path due to under-replication
最佳答案
这似乎是 HDFS 复制因子的问题。当我在具有 1 个虚拟数据节点的虚拟机上工作时,我必须将复制因子设置为 1 以使其按预期工作。
关于hadoop - Flume HDFS 接收器不断滚动小文件,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/22145899/