hadoop - Job fails to read from one ORC file and write a subset to another -


working in apache pig interactive shell in hdp 2.3 windows, i've got existing orc file in /path/to/file. if load , save using:

a = load '/path/to/file' using orcstorage(''); store '/path/to/second_file' using orcstorage('');

then works. however, if try:

a = load '/path/to/file' using orcstorage(''); b = limit 10; store b '/path/to/third_file' using orcstorage('');

then following error traceback in logs second job (out of 2 schedules):

2015-08-25 16:03:42,161 fatal [main] org.apache.hadoop.mapreduce.v2.app.mrappmaster: error starting mrappmaster java.lang.noclassdeffounderror: org/apache/hadoop/hive/ql/io/orc/orcnewoutputformat     @ java.lang.class.forname0(native method)     @ java.lang.class.forname(class.java:348)     @ org.apache.pig.impl.pigcontext.resolveclassname(pigcontext.java:657)     @ org.apache.pig.impl.pigcontext.instantiatefuncfromspec(pigcontext.java:726)     @ org.apache.pig.backend.hadoop.executionengine.physicallayer.relationaloperators.postore.getstorefunc(postore.java:251)     @ org.apache.pig.backend.hadoop.executionengine.mapreducelayer.pigoutputcommitter.getcommitters(pigoutputcommitter.java:88)     @ org.apache.pig.backend.hadoop.executionengine.mapreducelayer.pigoutputcommitter.<init>(pigoutputcommitter.java:71)     @ org.apache.pig.backend.hadoop.executionengine.mapreducelayer.pigoutputformat.getoutputcommitter(pigoutputformat.java:289)     @ org.apache.hadoop.mapreduce.v2.app.mrappmaster$1.call(mrappmaster.java:476)     @ org.apache.hadoop.mapreduce.v2.app.mrappmaster$1.call(mrappmaster.java:458)     @ org.apache.hadoop.mapreduce.v2.app.mrappmaster.callwithjobclassloader(mrappmaster.java:1560)     @ org.apache.hadoop.mapreduce.v2.app.mrappmaster.createoutputcommitter(mrappmaster.java:458)     @ org.apache.hadoop.mapreduce.v2.app.mrappmaster.serviceinit(mrappmaster.java:377)     @ org.apache.hadoop.service.abstractservice.init(abstractservice.java:163)     @ org.apache.hadoop.mapreduce.v2.app.mrappmaster$4.run(mrappmaster.java:1518)     @ java.security.accesscontroller.doprivileged(native method)     @ javax.security.auth.subject.doas(subject.java:422)     @ org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation.java:1657)     @ org.apache.hadoop.mapreduce.v2.app.mrappmaster.initandstartappmaster(mrappmaster.java:1515)     @ org.apache.hadoop.mapreduce.v2.app.mrappmaster.main(mrappmaster.java:1448) caused by: java.lang.classnotfoundexception: org.apache.hadoop.hive.ql.io.orc.orcnewoutputformat     @ java.net.urlclassloader.findclass(urlclassloader.java:381)     @ java.lang.classloader.loadclass(classloader.java:424)     @ sun.misc.launcher$appclassloader.loadclass(launcher.java:331)     @ java.lang.classloader.loadclass(classloader.java:357)

i suspect classpath 2 jobs different, causing classnotfound. case? if so, how can fix it? (bonus question: why has happened?)

check dependent library orcstorage placed in nodes.

  • the first option spawn single job
  • the second option spawn multiple jobs maybe run in different machine doesnt have dependent library in classpath.

-

Comments

Post a Comment

Popular posts from this blog

java - UnknownEntityTypeException: Unable to locate persister (Hibernate 5.0) -

Image
in code below when try execute main.java getting exception: exception in thread "main" org.hibernate.unknownentitytypeexception: unable locate persister: com.np.vta.test.pojo.users @ org.hibernate.internal.sessionfactoryimpl.locateentitypersister(sessionfactoryimpl.java:792) @ org.hibernate.internal.sessionimpl.locateentitypersister(sessionimpl.java:2637) @ org.hibernate.internal.sessionimpl.access$2500(sessionimpl.java:164) @ org.hibernate.internal.sessionimpl$identifierloadaccessimpl.<init>(sessionimpl.java:2575) @ org.hibernate.internal.sessionimpl$identifierloadaccessimpl.<init>(sessionimpl.java:2562) @ org.hibernate.internal.sessionimpl.byid(sessionimpl.java:1044) @ org.hibernate.internal.sessionimpl.get(sessionimpl.java:955) @ com.app.test.main.main(main.java:20) but if uncomment cfg.addclass( users.class ).addresource( "com/np/vta/test/pojo/users.hbm.xml" ); code works fine. why not reading <mapping...
Read more

python - ValueError: empty vocabulary; perhaps the documents only contain stop words -

i'm using (for first time) scikit library , got error: valueerror: empty vocabulary; perhaps documents contain stop words file "c:\users\a605563\desktop\velibprojetpreso\traitementtwitterdico.py", line 33, in <module> x_train_counts = count_vect.fit_transform(filetweets) file "c:\python27\lib\site-packages\sklearn\feature_extraction\text.py", line 804, in fit_transform self.fixed_vocabulary_) file "c:\python27\lib\site-packages\sklearn\feature_extraction\text.py", line 751, in _count_vocab raise valueerror("empty vocabulary; perhaps documents contain stop words but don't understand why that's happening. import sklearn sklearn.feature_extraction.text import countvectorizer import pandas pd import numpy import unicodedata import nltk tweetsfile = open('tweets2015-08-13.csv', 'r+') f2 = open('analyzer.txt', 'a') print tweetsfile.readline() count_vect = countvectorizer(strip_accents='ascii...
Read more

ubuntu - collect2: fatal error: ld terminated with signal 9 [Killed] -

Image
i'm trying build precimonious on ubuntu 16.04.3 x64. allocated 1gb memory it. file structure looks like ~ |--- llvm/ |--- precimonious/ where llvm on version 3.0 mentioned @ https://github.com/corvette-berkeley/precimonious#requirement . followed steps on readme command make ended with ... llvm[1]: linking debug shared library libllvm-3.0.so collect2: fatal error: ld terminated signal 9 [killed] compilation terminated. ... i went through answers online , might because there's not enough memory perform link. memory usage the gcc version on machine gcc version 5.4.0 20160609 (ubuntu 5.4.0-6ubuntu1~16.04.4) , installed via apt-get install build-essential . fresh droplet created on digitalocean btw. appreciated. it memory exhaustion. sampling interval memory measurement not small enough covers exact point oom killer kicks in. depending on container/vm technology, may able set vm.overcommit_memory=2 before build, process not killed (but doing requires m...
Read more