hadoop - Error processing complex json object of twitter with pig JsonLoader() of elephant-bird Jars -


i wanted process twitter json object pig using elephant-bird jars wrote pig script below.

register '/usr/lib/pig/lib/elephant-bird-hadoop-compat-4.1.jar';  register '/usr/lib/pig/lib/elephant-bird-pig-4.1.jar';    = load '/user/flume/tweets/data.json' using com.twitter.elephantbird.pig.load.jsonloader('-nestedload') mymap;  b = foreach generate mymap#'id' id,mymap#'created_at' createdat;  dump b;

which gave me error below

2015-08-25 11:06:34,295 [main] info  org.apache.pig.backend.hadoop.executionengine.mapreducelayer.mapreducelauncher - hadoopjobid: job_1439883208520_0177  2015-08-25 11:06:34,295 [main] info  org.apache.pig.backend.hadoop.executionengine.mapreducelayer.mapreducelauncher - processing aliases a,b  2015-08-25 11:06:34,295 [main] info  org.apache.pig.backend.hadoop.executionengine.mapreducelayer.mapreducelauncher - detailed locations: m: a[3,4],b[4,4] c:  r:  2015-08-25 11:06:34,303 [main] info  org.apache.pig.backend.hadoop.executionengine.mapreducelayer.mapreducelauncher - 0% complete  2015-08-25 11:06:34,303 [main] info  org.apache.pig.backend.hadoop.executionengine.mapreducelayer.mapreducelauncher - running jobs [job_1439883208520_0177]  2015-08-25 11:07:06,449 [main] info  org.apache.pig.backend.hadoop.executionengine.mapreducelayer.mapreducelauncher - 50% complete  2015-08-25 11:07:06,449 [main] info  org.apache.pig.backend.hadoop.executionengine.mapreducelayer.mapreducelauncher - running jobs [job_1439883208520_0177]  2015-08-25 11:07:09,458 [main] warn  org.apache.pig.backend.hadoop.executionengine.mapreducelayer.mapreducelauncher - ooops! job has failed! specify -stop_on_failure if want pig stop on failure.  2015-08-25 11:07:09,458 [main] info  org.apache.pig.backend.hadoop.executionengine.mapreducelayer.mapreducelauncher - job job_1439883208520_0177 has failed! stop running dependent jobs  2015-08-25 11:07:09,459 [main] info  org.apache.pig.backend.hadoop.executionengine.mapreducelayer.mapreducelauncher - 100% complete  2015-08-25 11:07:09,667 [main] info  org.apache.hadoop.yarn.client.api.impl.timelineclientimpl - timeline service address: http://trinityhadoopmaster.com:8188/ws/v1/timeline/  2015-08-25 11:07:09,668 [main] info  org.apache.hadoop.yarn.client.rmproxy - connecting resourcemanager @ trinityhadoopmaster.com/192.168.1.135:8032  2015-08-25 11:07:09,678 [main] info  org.apache.hadoop.mapred.clientservicedelegate - application state completed. finalapplicationstatus=failed. redirecting job history server  2015-08-25 11:07:09,779 [main] error org.apache.pig.tools.pigstats.pigstats - error 0: java.lang.classnotfoundexception: org.json.simple.parser.parseexception  2015-08-25 11:07:09,779 [main] error org.apache.pig.tools.pigstats.mapreduce.mrpigstatsutil - 1 map reduce job(s) failed!  2015-08-25 11:07:09,780 [main] info  org.apache.pig.tools.pigstats.mapreduce.simplepigstats - script statistics:    hadoopversion   pigversion      userid  startedat       finishedat      features  2.6.0   0.14.0  hdfs    2015-08-25 11:06:33     2015-08-25 11:07:09     unknown    failed!    failed jobs:  jobid   alias   feature message outputs  job_1439883208520_0177  a,b     map_only        message: job failed!    hdfs://trinityhadoopmaster.com:9000/tmp/temp1554332510/tmp835744559,    input(s):  failed read data "hdfs://trinityhadoopmaster.com:9000/user/flume/tweets/data.json"    output(s):  failed produce result in "hdfs://trinityhadoopmaster.com:9000/tmp/temp1554332510/tmp835744559"    counters:  total records written : 0  total bytes written : 0  spillable memory manager spill count : 0  total bags proactively spilled: 0  total records proactively spilled: 0    job dag:  job_1439883208520_0177      2015-08-25 11:07:09,780 [main] info  org.apache.pig.backend.hadoop.executionengine.mapreducelayer.mapreducelauncher - failed!  2015-08-25 11:07:09,787 [main] error org.apache.pig.tools.grunt.grunt - error 1066: unable open iterator alias b. backend error : java.lang.classnotfoundexception: org.json.simple.parser.parseexception  details @ logfile: /tmp/pig-err.log  grunt>

which have no clue on how approach, can 1 me on this. 

register '/tmp/elephant-bird-core-4.1.jar';  register '/tmp/elephant-bird-pig-4.1.jar';  register '/tmp/elephant-bird-hadoop-compat-4.1.jar';  register '/tmp/google-collections-1.0.jar';  register '/tmp/json-simple-1.1.jar';

it works.


-

Comments

Post a Comment

Popular posts from this blog

java - UnknownEntityTypeException: Unable to locate persister (Hibernate 5.0) -

Image
in code below when try execute main.java getting exception: exception in thread "main" org.hibernate.unknownentitytypeexception: unable locate persister: com.np.vta.test.pojo.users @ org.hibernate.internal.sessionfactoryimpl.locateentitypersister(sessionfactoryimpl.java:792) @ org.hibernate.internal.sessionimpl.locateentitypersister(sessionimpl.java:2637) @ org.hibernate.internal.sessionimpl.access$2500(sessionimpl.java:164) @ org.hibernate.internal.sessionimpl$identifierloadaccessimpl.<init>(sessionimpl.java:2575) @ org.hibernate.internal.sessionimpl$identifierloadaccessimpl.<init>(sessionimpl.java:2562) @ org.hibernate.internal.sessionimpl.byid(sessionimpl.java:1044) @ org.hibernate.internal.sessionimpl.get(sessionimpl.java:955) @ com.app.test.main.main(main.java:20) but if uncomment cfg.addclass( users.class ).addresource( "com/np/vta/test/pojo/users.hbm.xml" ); code works fine. why not reading <mapping...
Read more

python - ValueError: empty vocabulary; perhaps the documents only contain stop words -

i'm using (for first time) scikit library , got error: valueerror: empty vocabulary; perhaps documents contain stop words file "c:\users\a605563\desktop\velibprojetpreso\traitementtwitterdico.py", line 33, in <module> x_train_counts = count_vect.fit_transform(filetweets) file "c:\python27\lib\site-packages\sklearn\feature_extraction\text.py", line 804, in fit_transform self.fixed_vocabulary_) file "c:\python27\lib\site-packages\sklearn\feature_extraction\text.py", line 751, in _count_vocab raise valueerror("empty vocabulary; perhaps documents contain stop words but don't understand why that's happening. import sklearn sklearn.feature_extraction.text import countvectorizer import pandas pd import numpy import unicodedata import nltk tweetsfile = open('tweets2015-08-13.csv', 'r+') f2 = open('analyzer.txt', 'a') print tweetsfile.readline() count_vect = countvectorizer(strip_accents='ascii...
Read more

ubuntu - collect2: fatal error: ld terminated with signal 9 [Killed] -

Image
i'm trying build precimonious on ubuntu 16.04.3 x64. allocated 1gb memory it. file structure looks like ~ |--- llvm/ |--- precimonious/ where llvm on version 3.0 mentioned @ https://github.com/corvette-berkeley/precimonious#requirement . followed steps on readme command make ended with ... llvm[1]: linking debug shared library libllvm-3.0.so collect2: fatal error: ld terminated signal 9 [killed] compilation terminated. ... i went through answers online , might because there's not enough memory perform link. memory usage the gcc version on machine gcc version 5.4.0 20160609 (ubuntu 5.4.0-6ubuntu1~16.04.4) , installed via apt-get install build-essential . fresh droplet created on digitalocean btw. appreciated. it memory exhaustion. sampling interval memory measurement not small enough covers exact point oom killer kicks in. depending on container/vm technology, may able set vm.overcommit_memory=2 before build, process not killed (but doing requires m...
Read more