java.net.UnknownHostException: dbmtimehadoop

完整报错如下:

$ spark-submit --class WordCountLocal --master yarn --deploy-mode cluster  --driver-memory 1g --executor-memory 1g --executor-cores 1  /home/appleyuchi/IdeaProjects/scala-learn/target/scala-learn-1.0-SNAPSHOT.jar
Exception in thread "main" java.lang.IllegalArgumentException: java.net.UnknownHostException: dbmtimehadoop
    at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:378)
    at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:310)
    at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:678)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619)
    at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
    at org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$6.apply(Client.scala:528)
    at org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$6.apply(Client.scala:525)
    at scala.collection.mutable.ArraySeq.foreach(ArraySeq.scala:74)
    at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:525)
    at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:869)
    at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:169)
    at org.apache.spark.deploy.yarn.Client.run(Client.scala:1152)
    at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1520)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.net.UnknownHostException: dbmtimehadoop
    ... 24 more
 

摸索的思路:

①搜到[1]中发现这么个信息:

--conf spark.eventLog.dir=hdfs://dbmtimehadoop/tmp/spark2 \

②猜测是spark.eventLog.dir没设置的关系

 

 

最终解决方案:

hdfs dfs -mkdir /logs
编辑conf/spark-defaults.conf内容如下:

spark.master                     spark://Desktop:9000 spark.eventLog.enabled           true spark.eventLog.dir               hdfs://Desktop:9000/logs

Reference:

[1]spark任务提交到yarn上命令总结

总结

以上是生活随笔为你收集整理的java.net.UnknownHostException: dbmtimehadoop的全部内容，希望文章能够帮你解决所遇到的问题。

如果觉得生活随笔网站内容还不错，欢迎将生活随笔推荐给好友。