Confusion of Storm acker and guaranteed message processing -


now learning storm's guaranteeing message processing , confused concepts in part.

to guarantee message emitted spout processed, storm uses acker achieve this. each time spout emits tuple, acker assign "ack val" initialized 0 store status of tuple tree. each time downstream bolts of tuple emit new tuple or ack "old" tuple, tuple id xor "ack val". acker needs check whether "ack val" 0 or not know tuple has been processed. let's see code below:

public class wordreader implements irichspout {     ... ... while((str = reader.readline()) != null){     this.collector.emit(new values(str), str);     ... ... }

the code piece above spout in word count program "getting started storm" tutorial. in emit method, 2nd parameter "str" messageid. confused parameter: 1) understand, each time tuple (i.e., message) emitted no matter in spouts or in bolts, should storm's responsibility assign 64-bit messageid message. correct? or here "str" human-readable alias message? 2) no matter what's answer 1), here "str" same word in 2 different messages because in text file there should many duplicate words. if true, how storm differentiate different messages? , what's meaning of parameter? 3) in code piece, see spouts use following code set message id in spout emit method:

public class randomintegerspout extends baserichspout {     private long msgid = 0;     collector.emit(new values(..., ++msgid), msgid); }

this closer think should be: message id should totally different across different messages. code piece, confusion is: happen private field "msgid" across different executors? because each executor has own msgid initialized 0, messages in different executors named 0, 1, 2, , on. how storm differentiate these messages?

i novice storm, maybe these problems naive. hope me figure out. thanks!

about message id general: internally might 64bit value, 64bit value computed hash msgid object provided in emit() within spout. can hand object message id (the probability 2 objects hash same value close zero).

about using str: think in example, str contains line (and not word) , unlikely document contains exact same line twice (if there no empty lines might many).

about counter message id: absolutely right observation -- if multiple spouts running in parallel, give message id conflict , break fault tolerance.

if want "fix" counter approach, each counter should initialized differently (best, 1...#spouttasks). can use taskid (which unique , can accessed via topologycontext provided in spout.open()). basically, taskids parallel spout tasks, sort them, , assign each spout task ordering number. furthermore, need increment "number of parallel spouts" instead of 1.


-

Comments

Post a Comment

Popular posts from this blog

python - How to insert QWidgets in the middle of a Layout? -

Image
i'm using qt framework build graphical user interface. use qgridlayout position qwidgets neatly. gui looks this: my application regularly adds new widgets gui @ runtime. these new widgets not added @ end of qlayout, somewhere in middle. the procedure bit cumbersome. applied on figure above, need take out widg_c , widg_d , ... qgridlayout. next, add widg_x and widg_y , , put other widgets again. how remove widgets qgridlayout: for in reversed(range(mygridlayout.count())): self.itemat(i).widget().setparent(none) ### as long you're dealing small amount of widgets, procedure not disaster. in application display lot of small widgets - perhaps 50 or more! application freezes second while procedure ongoing, annoying user. there way insert widgets somewhere in qlayout, without need take out other widgets? edit: apparently solution qvboxlayout simple. use function insertwidget(..) instead of addwidget(..) . docs can found link: http://doc.qt.io/qt-5/qboxlayou
Read more

python - serve multiple gunicorn django instances under nginx ubuntu -

i need serve 2 webs apps (django) on gunicorn under different domain names using nginx on ubuntu. started using tutorial: https://www.digitalocean.com/community/tutorials/how-to-set-up-django-with-postgres-nginx-and-gunicorn-on-ubuntu-16-04 it worked fine 1 using that. tried same thing second gives me 502 gateway second domain. tried following tutorial: http://michal.karzynski.pl/blog/2013/10/29/serving-multiple-django-applications-with-nginx-gunicorn-supervisor/ i'm @ part run gunicorn start script: sudo bin/gunicorn_start i response: starting webuildblack root traceback (most recent call last): file "/home/devin/webuildblack/bin/gunicorn", line 7, in <module> gunicorn.app.wsgiapp import run importerror: no module named 'gunicorn.app'; 'gunicorn' not package you have named program gunicorn.py gunicorn -package. rename file , remove gunicorn.pyc file.
Read more

module - Prestashop displayPaymentReturn hook url -

i new prestashop developer , trying create paymentmodule. have got show payment method can not proceed purchase because not know hot works. does know should redirect run hooddisplaypaymentreturn method? i happy if explain me complete navigation map make purchase. anyway, can find relation between hooks , pages? when have new module payment rely on simplest provided prestashop: bankwire. inside can find 3 hooks. hookpayment: public function hookpayment($params) { if (!$this->active) return; if (!$this->checkcurrency($params['cart'])) return; $this->smarty->assign(array( 'this_path' => $this->_path, 'this_path_bw' => $this->_path, 'this_path_ssl' => tools::getshopdomainssl(true, true).__ps_base_uri__.'modules/'.$this->name.'/' )); return $this->display(__file__, 'payment.tpl'); } hookdisplaypaymenteu: public fun
Read more