hadoop - hbase scan with spark : scan.setCaching() method do not found - 


import org.apache.hadoop.hbase.io.immutablebyteswritable import org.apache.hadoop.hbase.mapreduce.tableinputformat import org.apache.hadoop.hbase.protobuf.protobufutil import org.apache.hadoop.hbase.util.{base64, bytes} import org.apache.hadoop.hbase.hbaseconfiguration import org.apache.hadoop.hbase.client._ import org.apache.hadoop.hbase.client.scan import org.apache.spark.sparkcontext import org.apache.spark.sparkconf      def main(args: array[string]) {     val sparkconf = new sparkconf().setappname("simple spark app")     val sc = new sparkcontext(sparkconf)      var input_hbase_table = args(0)     var output = args(1)      val conf = hbaseconfiguration.create()     conf.set(tableinputformat.input_table, input_hbase_table)      var scan = new scan()     scan.setcaching(1000)     scan.setcacheblocks(false)     conf.set(tableinputformat.scan, convertscantostring(scan))      val hbase_rdd = sc.newapihadooprdd(conf, classof[tableinputformat],       classof[immutablebyteswritable],       classof[result])      // todo     hbase_rdd.map(my_map_func).saveastextfile(output)   }


sbt blow: 

name := "sparkonhbase"  version := "1.0"  scalaversion := "2.10.6"  librarydependencies ++= seq(   "org.apache.spark" %% "spark-core" % "2.0.1",   "org.apache.hadoop" % "hadoop-core" % "1.2.1",   "org.apache.hbase" % "hbase" % "1.2.4",   "org.apache.hbase" % "hbase-client" % "1.2.4",   "org.apache.hbase" % "hbase-common" % "1.2.4",   "org.apache.hbase" % "hbase-server" % "1.2.4" )

the exception in log is:

java.lang.nosuchmethoderror:org.apache.hadoop.hbase.client.scan.setcaching(i)

but, if remove scan.setcaching() , scan.setcacheblocks(), ok.


-

Comments

Post a Comment

Popular posts from this blog

python - How to insert QWidgets in the middle of a Layout? -

Image
i'm using qt framework build graphical user interface. use qgridlayout position qwidgets neatly. gui looks this: my application regularly adds new widgets gui @ runtime. these new widgets not added @ end of qlayout, somewhere in middle. the procedure bit cumbersome. applied on figure above, need take out widg_c , widg_d , ... qgridlayout. next, add widg_x and widg_y , , put other widgets again. how remove widgets qgridlayout: for in reversed(range(mygridlayout.count())): self.itemat(i).widget().setparent(none) ### as long you're dealing small amount of widgets, procedure not disaster. in application display lot of small widgets - perhaps 50 or more! application freezes second while procedure ongoing, annoying user. there way insert widgets somewhere in qlayout, without need take out other widgets? edit: apparently solution qvboxlayout simple. use function insertwidget(..) instead of addwidget(..) . docs can found link: http://doc.qt.io/qt-5/qboxlayou
Read more

python - serve multiple gunicorn django instances under nginx ubuntu -

i need serve 2 webs apps (django) on gunicorn under different domain names using nginx on ubuntu. started using tutorial: https://www.digitalocean.com/community/tutorials/how-to-set-up-django-with-postgres-nginx-and-gunicorn-on-ubuntu-16-04 it worked fine 1 using that. tried same thing second gives me 502 gateway second domain. tried following tutorial: http://michal.karzynski.pl/blog/2013/10/29/serving-multiple-django-applications-with-nginx-gunicorn-supervisor/ i'm @ part run gunicorn start script: sudo bin/gunicorn_start i response: starting webuildblack root traceback (most recent call last): file "/home/devin/webuildblack/bin/gunicorn", line 7, in <module> gunicorn.app.wsgiapp import run importerror: no module named 'gunicorn.app'; 'gunicorn' not package you have named program gunicorn.py gunicorn -package. rename file , remove gunicorn.pyc file.
Read more

module - Prestashop displayPaymentReturn hook url -

i new prestashop developer , trying create paymentmodule. have got show payment method can not proceed purchase because not know hot works. does know should redirect run hooddisplaypaymentreturn method? i happy if explain me complete navigation map make purchase. anyway, can find relation between hooks , pages? when have new module payment rely on simplest provided prestashop: bankwire. inside can find 3 hooks. hookpayment: public function hookpayment($params) { if (!$this->active) return; if (!$this->checkcurrency($params['cart'])) return; $this->smarty->assign(array( 'this_path' => $this->_path, 'this_path_bw' => $this->_path, 'this_path_ssl' => tools::getshopdomainssl(true, true).__ps_base_uri__.'modules/'.$this->name.'/' )); return $this->display(__file__, 'payment.tpl'); } hookdisplaypaymenteu: public fun
Read more