Trouble mounting external disk for hadoop pod in kubernetes cluster -


background : have kubernetes cluster has spark cluster (with driver outside cluster) , hadoop pod. mounted folder path of hadoop pod 1 of external disk.

issue : i'm getting below exception when spark cluster tries create checkpointing folder in hdfs.

java.io.eofexception: end of file exception between local host is: "jfgubq745vn2ym-tg1isslukll1u3/10.92.0.135"; destination host is: "dev-dev-hadoop":9000; : java.io.eofexception; more details see: http://wiki.apache.org/hadoop/eofexception  @ sun.reflect.nativeconstructoraccessorimpl.newinstance0(native method)  @ sun.reflect.nativeconstructoraccessorimpl.newinstance(nativeconstructoraccessorimpl.java:57)  @ sun.reflect.delegatingconstructoraccessorimpl.newinstance(delegatingconstructoraccessorimpl.java:45)  @ java.lang.reflect.constructor.newinstance(constructor.java:526)  @ org.apache.hadoop.net.netutils.wrapwithmessage(netutils.java:791)  @ org.apache.hadoop.net.netutils.wrapexception(netutils.java:764)  @ org.apache.hadoop.ipc.client.call(client.java:1472)  @ org.apache.hadoop.ipc.client.call(client.java:1399)  @ org.apache.hadoop.ipc.protobufrpcengine$invoker.invoke(protobufrpcengine.java:232)  @ com.sun.proxy.$proxy28.mkdirs(unknown source)  @ org.apache.hadoop.hdfs.protocolpb.clientnamenodeprotocoltranslatorpb.mkdirs(clientnamenodeprotocoltranslatorpb.java:539)  @ sun.reflect.nativemethodaccessorimpl.invoke0(native method)  @ sun.reflect.nativemethodaccessorimpl.invoke(nativemethodaccessorimpl.java:57)  @ sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl.java:43)  @ java.lang.reflect.method.invoke(method.java:606)  @ org.apache.hadoop.io.retry.retryinvocationhandler.invokemethod(retryinvocationhandler.java:187)  @ org.apache.hadoop.io.retry.retryinvocationhandler.invoke(retryinvocationhandler.java:102)  @ com.sun.proxy.$proxy29.mkdirs(unknown source)  @ org.apache.hadoop.hdfs.dfsclient.primitivemkdir(dfsclient.java:2753)  @ org.apache.hadoop.hdfs.dfsclient.mkdirs(dfsclient.java:2724)  @ org.apache.hadoop.hdfs.distributedfilesystem$17.docall(distributedfilesystem.java:870)  @ org.apache.hadoop.hdfs.distributedfilesystem$17.docall(distributedfilesystem.java:866)  @ org.apache.hadoop.fs.filesystemlinkresolver.resolve(filesystemlinkresolver.java:81)  @ org.apache.hadoop.hdfs.distributedfilesystem.mkdirsinternal(distributedfilesystem.java:866)  @ org.apache.hadoop.hdfs.distributedfilesystem.mkdirs(distributedfilesystem.java:859)  @ org.apache.hadoop.fs.filesystem.mkdirs(filesystem.java:1817)  @ org.apache.spark.streaming.streamingcontext.checkpoint(streamingcontext.scala:233)  ... @ scala.util.try$.apply(try.scala:161)  @ falkonry.tercel.process.main$.main(main.scala:54)  @ falkonry.tercel.process.main.main(main.scala)  @ sun.reflect.nativemethodaccessorimpl.invoke0(native method)  @ sun.reflect.nativemethodaccessorimpl.invoke(nativemethodaccessorimpl.java:57)  @ sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl.java:43)  @ java.lang.reflect.method.invoke(method.java:606)  @ org.apache.spark.deploy.sparksubmit$.org$apache$spark$deploy$sparksubmit$runmain(sparksubmit.scala:664)  @ org.apache.spark.deploy.sparksubmit$.dorunmain$1(sparksubmit.scala:169)  @ org.apache.spark.deploy.sparksubmit$.submit(sparksubmit.scala:192)  @ org.apache.spark.deploy.sparksubmit$.main(sparksubmit.scala:111)  @ org.apache.spark.deploy.sparksubmit.main(sparksubmit.scala)   caused by: java.io.eofexception  @ java.io.datainputstream.readint(datainputstream.java:392)  @ org.apache.hadoop.ipc.client$connection.receiverpcresponse(client.java:1071)  @ org.apache.hadoop.ipc.client$connection.run(client.java:966) 

this works if not mount path external disk.

this issue resolved. docker image using hadoop container not format external mounted disks on startup. edited dockerfile , made format external mounted paths. updated dockerfile https://github.com/falkonry/docker-hadoop/blob/master/dockerfile


Comments

Popular posts from this blog

java - UnknownEntityTypeException: Unable to locate persister (Hibernate 5.0) -

python - ValueError: empty vocabulary; perhaps the documents only contain stop words -

ubuntu - collect2: fatal error: ld terminated with signal 9 [Killed] -