Remove duplicates from a Spark JavaPairDStream / JavaDStream -


i'm building spark streaming application receives data via sockettextstream. problem is, sended data has duplicates. remove them on spark-side (without pre-filtering on sender side). can use javapairrdd's distinct function via dstream's foreach (i can't find way how that)??? need "filtered" java(pair)dstream later actions...

thank you!

the .transform() method can used arbitrary operations on each time slice of rdds. assuming data strings:

somedstream.transform(new function<javardd<string>, javardd<string>>() {         @override         public javardd<string> call(javardd<string> rows) throws exception {             return rows.distinct();         }     });

-

Comments

Post a Comment

Popular posts from this blog

javascript - Trigger mouseenter when an animated element touches mouse -

i'm new web development , i'm building game in object avoid touching animated circles pointer. set mouseenter event (full jsfiddle below): $("#"+circlename).mouseenter(function(){ $(this).attr("class", "redcircle"); }); this turns circles red when user moves pointer them. however, when mouse stationary, circles can pass through safely. there better way this? by changing mouseenter event hover event, event triggered when animated circle touches stationary mouse. however, there seems delay. know why might be? this sounds it's related issue found here, means may have related browser bug: getting mouseenter event still mouse entering animated element fiddle: http://jsfiddle.net/nbsteuv/6yok3s56/4/ i store last mouse position , whenever circle moves, check whether has intersected mouse position. var mousex, mousey; $( "#container-element-id" ).mousemove(function( event ) { mousex = event.pagex; mouse...
Read more

json - Zend error Connection -

i have problem code, first code <?php require_once('zend/json.php'); require_once('zend/db.php'); //require_once 'zend/db/adapter/pdo/pgsql.php'; class jsonvi { protected $_db; public function _koneksi () { try { $this->_db = zend_db::factory('pdo_pgsql',array( 'host'=>'localhost', 'username'=>'stet', 'password'=>'test', 'dbname'=>'test' )); return $this->_db; } catch(zend_db_exception $e) { return $e->getmessage(); } } public function getdata() { $db = $this->_koneksi(); try { $sql = "select * info "; ...
Read more

java - Using Spring @Transactional with a combination of readOnly and write, when does this entity get committed? -

given class callingclass { @autowired someservice someservice; doit() { someservice.readonlycall(); } } @transactional(readonly = true) class someservice { @autowired anotherservice anotherservice; readonlycall() { // logic may do... anotherservice.writecall(); } } class anotherservice { @transactional // not readonly writecall() { // save entity } } will entity committed database when writecall method exits or when readonlycall method exits? and how can modify commits after writecall method? the entity not committed database in both method exits. because your someservice class has read-only (which have specify). set flushmode.never prevent commit transaction your someservice class , anotherservice class has required propagation type(which default , have not specify other type both class). create new transaction if doesn't exist use current transaction if there is. in case someservice class begin transaction read-only , an...
Read more