java - What is configuartion required to get data from object storage by SWIFT in Spark -


i go through document still confusing how data swift.

i configured swift in 1 linux machine. using below command able container list,

swift -a https://acc.objectstorage.softlayer.net/auth/v1.0/ -u username -k passwordkey list

i seen many blog blumix(https://console.ng.bluemix.net/docs/services/analyticsforapachespark/index-gentopic1.html#gentopprocid2) , written below code

sc.textfile("swift://container.myacct/file.xml")

i looking integrate in java spark. need configure object storage credential in java code. there sample code or blog?

this notebook illustrates number of ways load data using scala language. scala runs on jvm. java , scala classes can freely mixed, no matter whether reside in different projects or in same. looking @ mechanics of how scala code interacts openstack swift object storage should guide craft java equivalent.

from above notebook, here steps illustrating how configure , extract data openstack swift object storage instance using stocator library using scala language. swift url decomposes into:

swift2d :// container . myacct / filename.extension   ^            ^          ^            ^ stocator     name of   namespace    object storage protocol     container               filename

imports

import org.apache.spark.sparkcontext import scala.util.control.nonfatal import play.api.libs.json.json  val sqlctx = new sqlcontext(sc) val scplain = sqlctx.sparkcontext

sample creds

// @hidden_cell var credentials = scala.collection.mutable.hashmap[string, string](   "auth_url"->"https://identity.open.softlayer.com",   "project"->"object_storage_3xxxxxx3_xxxx_xxxx_xxxx_xxxxxxxxxxxx",   "project_id"->"6xxxxxxxxxx04fxxxxxxxxxx6xxxxxx7",   "region"->"dallas",   "user_id"->"cxxxxxxxxxxaxxxxxxxxxx1xxxxxxxxx",   "domain_id"->"cxxxxxxxxxxaxxyyyyyyxx1xxxxxxxxx",   "domain_name"->"853255",   "username"->"admin_cxxxxxxxxxxaxxxxxxxxxx1xxxxxxxxx",   "password"->"""&m7372!fake""",   "container"->"notebooks",   "tenantid"->"undefined",   "filename"->"file.xml" )

helper method

def setremoteobjectstorageconfig(name:string, sc: sparkcontext, dsconfiguration:string) : boolean = {     try {         val result = scala.util.parsing.json.json.parsefull(dsconfiguration)         result match {             case some(e:map[string,string]) => {                 val prefix = "fs.swift2d.service." + name                 val hconf = sc.hadoopconfiguration                 hconf.set("fs.swift2d.impl","com.ibm.stocator.fs.objectstorefilesystem")                 hconf.set(prefix + ".auth.url", e("auth_url") + "/v3/auth/tokens")                 hconf.set(prefix + ".tenant", e("project_id"))                 hconf.set(prefix + ".username", e("user_id"))                 hconf.set(prefix + ".password", e("password"))                 hconf.set(prefix + "auth.method", "keystonev3")                 hconf.set(prefix + ".region", e("region"))                 hconf.setboolean(prefix + ".public", true)                 println("successfully modified sparkcontext object remote object storage credentials using datasource name " + name)                 println("")                 return true             }             case none => println("failed.")                 return false         }     }     catch {        case nonfatal(exc) => println(exc)            return false     } }

load data

val setobjstor = setremoteobjectstorageconfig("sparksql", scplain, json.tojson(credentials.tomap).tostring) val data_rdd = scplain.textfile("swift2d://notebooks.sparksql/" + credentials("filename")) data_rdd.take(5)

-

Comments

Post a Comment

Popular posts from this blog

python - How to insert QWidgets in the middle of a Layout? -

Image
i'm using qt framework build graphical user interface. use qgridlayout position qwidgets neatly. gui looks this: my application regularly adds new widgets gui @ runtime. these new widgets not added @ end of qlayout, somewhere in middle. the procedure bit cumbersome. applied on figure above, need take out widg_c , widg_d , ... qgridlayout. next, add widg_x and widg_y , , put other widgets again. how remove widgets qgridlayout: for in reversed(range(mygridlayout.count())): self.itemat(i).widget().setparent(none) ### as long you're dealing small amount of widgets, procedure not disaster. in application display lot of small widgets - perhaps 50 or more! application freezes second while procedure ongoing, annoying user. there way insert widgets somewhere in qlayout, without need take out other widgets? edit: apparently solution qvboxlayout simple. use function insertwidget(..) instead of addwidget(..) . docs can found link: http://doc.qt.io/qt-5/qboxlayou
Read more

python - serve multiple gunicorn django instances under nginx ubuntu -

i need serve 2 webs apps (django) on gunicorn under different domain names using nginx on ubuntu. started using tutorial: https://www.digitalocean.com/community/tutorials/how-to-set-up-django-with-postgres-nginx-and-gunicorn-on-ubuntu-16-04 it worked fine 1 using that. tried same thing second gives me 502 gateway second domain. tried following tutorial: http://michal.karzynski.pl/blog/2013/10/29/serving-multiple-django-applications-with-nginx-gunicorn-supervisor/ i'm @ part run gunicorn start script: sudo bin/gunicorn_start i response: starting webuildblack root traceback (most recent call last): file "/home/devin/webuildblack/bin/gunicorn", line 7, in <module> gunicorn.app.wsgiapp import run importerror: no module named 'gunicorn.app'; 'gunicorn' not package you have named program gunicorn.py gunicorn -package. rename file , remove gunicorn.pyc file.
Read more

module - Prestashop displayPaymentReturn hook url -

i new prestashop developer , trying create paymentmodule. have got show payment method can not proceed purchase because not know hot works. does know should redirect run hooddisplaypaymentreturn method? i happy if explain me complete navigation map make purchase. anyway, can find relation between hooks , pages? when have new module payment rely on simplest provided prestashop: bankwire. inside can find 3 hooks. hookpayment: public function hookpayment($params) { if (!$this->active) return; if (!$this->checkcurrency($params['cart'])) return; $this->smarty->assign(array( 'this_path' => $this->_path, 'this_path_bw' => $this->_path, 'this_path_ssl' => tools::getshopdomainssl(true, true).__ps_base_uri__.'modules/'.$this->name.'/' )); return $this->display(__file__, 'payment.tpl'); } hookdisplaypaymenteu: public fun
Read more