performance - apache phoenix, slow write speed when change currentSCN -


have old hbase table (billion records) , need use data create new phoenix table. want preserve record modtime old record. based on phoenix suggestion. give property currentscn when connect. seems slow. while doc

keep in mind creating new connection not expensive operation. same underlying hconnection used connections same cluster, it’s more or less instantiating few objects.

i test using simple program.

        class.forname("org.apache.phoenix.jdbc.phoenixdriver");         int x = 0;         for(int i=1000000 ;i< 2000000;i++ ) {             properties prop = new properties();             prop.setproperty("currentscn",string.valueof(i));             connection con = drivermanager.getconnection("jdbc:phoenix:a1.cluster",prop);             con.setautocommit(false);              preparedstatement  stmt2 = con.preparestatement("upsert fff values (?,?,?,?,?)");             stmt2.setstring(1,string.valueof(i));             stmt2.settimestamp(2, timestamp.from(instant.ofepochmilli(system.currenttimemillis())));             stmt2.setstring(3, null);             stmt2.setdouble(4, double.valueof(i));             stmt2.setstring(5, null);             stmt2.execute();              x=x+1;             if(x % 1000==0) system.out.println("-----------"+x);

a profiling suggest 90% time spend on getconnection , 10% spend on execute. on single machine, insert speed around 1000 record/s without commit, think very slow. how improve upsert speed in case?


-

Comments

Post a Comment

Popular posts from this blog

c++ - CPP, 'X' button listener -

i'm writing cpp program, , want execute function when 'x'(shut down) button pressed. for example, have infinite loop prints same thing everytime iterates, until 'x' button pressed, , want execute function before process ends. is there kind of listener matter? or other solution maybe (like delaying process shut down few seconds or something)? thanks! if want execute before process ends, can register std::atexit handler. function called before process exits.
Read more

shared memory - gstreamer shmsrc and shmsink with h264 data -

i trying share h264 encoded data gstreamer 2 processes(both based on gstreamer).after research way found use shm plugin. trying do gstreamer--->h264 encoder--->shmsink shmrc--->process1 shmrc--->process2 i able raw data videotestsrc , webcam working. h264 encoded data doesn't. test pipeline gst-launch-1.0 videotestsrc ! video/x-raw,width=640,height=480,format=yuy2 ! x264enc ! shmsink socket-path=/tmp/foo sync=true wait-for- connection=false shm-size=10000000 gst-launch-1.0 shmsrc socket-path=/tmp/foo ! avdec_h264 ! video/x- raw,width=640,height=480,framerate=25/1,format=yuy2 ! autovideosink have tried shm plugins h264 encoded data, please help iam not aware of capabilities of sink used in "autovideosink", per knowledge either need use videoconvert if format supported sink (like kmssink or ximagesink) different provided source (in case yuy2) or use videoparse if camera format supported sink. may check using gst-inspect-1...
Read more

.net - Bulk insert via Dapper is slower than inserting rows one-by-one -

i'm using dapper insert data realtime feed sql server, care performance. recently, noticed strange. out of box, if give dapper collection , insert query, fires insert statements each element. tests show can insert 1800 objects 12 fields in 1 second way (counting connection.execute(...) running time. now, didn't find batch insert functionality in dapper , implemented own (constructing parameter list , sql query). after that, found out can insert 1 batch in 3 seconds (which limited 1000 rows) (again, counting connection.execute(...) calls. so, makes batching 6 times slower sending each row in separate query. can explain me? thought people use batch operations speed process. insert time 1 second @ most. use sql server 2012 standard on local network. table i'm inserting has clustered index on primary key (which bigint field), no non-clustered indexes , triggers. can post code, there's nothing special i'm not sure why using dapper execute extension ...
Read more