Flume向HDFS写数据实例

2019独角兽企业重金招聘Python工程师标准>>>

Goal:

Use Flume to pool a folder on local file system and write it to HDFS.

Version Information:

hadoop-0.22.0

apache-flume-1.3.1

Flume Configuration:

Edit file  flume-env.sh under  /$FLUME_HOME$/conf:

export JAVA_HOME=your jave home export FLUME_CLASSPATH=your flume home export HADOOP_CLASSPATH=your hadoop home

Edit file flume-conf.properties under /$FLUME_HOME$/conf:

# Configure the agent agent.sources = spooldirSource agent.channels = memoryChannel agent.sinks = hdfsSink# Configure the source agent.sources.spooldirSource.type = spooldir agent.sources.spooldirSource.spoolDir = /tmp/flume/ agent.sources.spooldirSource.channels = memoryChannel# Configure the sink agent.sinks.hdfsSink.type = hdfs agent.sinks.hdfsSink.hdfs.path = hdfs://masternode:9000/flume/events agent.sinks.hdfsSink.hdfs.filePrefix = events- agent.sinks.hdfsSink.channel = memoryChannel# Configure the channel agent.channels.memoryChannel.type = memory agent.channels.memoryChannel.capacity = 100

Copy Hadoop Jars to Flume lib directory:

Copy hadoop-hdfs-0.22.0.jar and hadoop-common-0.22.0.jar to /$FLUME_HOME$/lib.

Start Flume Agent:

./bin/flume-ng agent -n agent -c conf -f conf/flume-conf.properties

Write File:

echo "Hello World">>/tmp/flume/test

View Logs:

Under /$FLUME_HOME$/logs

转载于:https://my.oschina.net/xiangchen/blog/101111

《新程序员》：云原生和全面数字化实践50位技术专家共同创作，文字、视频、音频交互阅读

总结

以上是生活随笔为你收集整理的Flume向HDFS写数据实例的全部内容，希望文章能够帮你解决所遇到的问题。

如果觉得生活随笔网站内容还不错，欢迎将生活随笔推荐给好友。