java - Jsoup, HTMLUnit, phantomJS: How can I click on Button OR fill out Formular TO bypass agecheck -
i collected urls mainpage (steam-gamestore) , want call each single site, of them have agecheck. identified 2 different types of agechecks:
only simple button expected click
a whole table has filled (with concrete specifications of age)
here pictures find in html-code
my question is:
how can click on damn button? in way, for-loop (which running through 11 thousands urls) isn't breaking , redirect me website behind agecheck (so can read out data it)?
i tried phantomjs, javascript , dont know how include in jsoup-code. i'm trying htmlunit. ideas?
and have fill whole form (for complex age-check). how can this? possible htmlunit?
ok, solved problem. in short terms: switched selenium webdriver (for javacode) , selenium ide (firefox plugin).
________ elaborately description (step step):
1. install selenium ide firefox-browser:
go to:
!!!fuck: i'm not allowed post links cause of low reputation. want deeds, hindered (fuck world!) !!!
h**ps://addons.mozilla.org/en-us/firefox/addon/selenium-ide/
(note: replace 2 * t)
and click on "+ add firefox"-button. after rebooting firefox, installation done. attention: be, errors occure @ point (the "selenium ide"-entry doesn't appear in menu of firefox. if so, try install selenium ide firefox-> add-ons->plug ins: search selenium , select:
selenium ide 2.9.1.1-signed"
"highlight elements (selenium ide)"
"selenium ide button 1.2.0.1-signed.1-signed"
navigate in firefoxmenu : tools-> web-developmentextras-> add new tools: (don't know exact term, cause i'm using german version of firefox: -> web-entwickler->weitere tools laden) search selenium , choose: "seleniumx"
after installation "selenium ide"-entry appears in firefoxmenu under: tools-> selenium ide (german: extras).
2. install selenium webdriver eclipse / dynamic webprojects:
got to:
!!!fuck: i'm not allowed post links cause of low reputation. want deeds, hindered (fuck world!) !!!
h**p://www.seleniumhq.org/download/#selenium_ide
(note: replace 2 * t)
and download (first section on site): selenium standalone server => version 3.0.1 (date: 11.5.16 [month-day-year]) after downloading .jar-file, copy in eclipse following folder: nameofproject\webcontent\web-inf\lib
note: import "build path-> configure build path", prefer faster way.
note: creating new "dynamic web project" have install new software in eclipse: help-> install new software: in first line "work with" choose: "luna - forbitten link low reputation-people" (for eclipse luna version, modify eclipse version!). wait, til pending... done , choose (last entry): "web, xml, java ee , osgi enterprise development)
3. using selenium ide identify webelement in html-code creating "test cases" , export them java-code eclipse:
detailed tutorial:
!!!fuck: i'm not allowed post links cause of low reputation. want deeds, hindered (fuck world!) !!!
h**p://docs.seleniumhq.org/docs/02_selenium_ide.jsp
(note: replace 2 * t)
3.1. open firefox-browser: go website want inspect / crawl / parse html-code. (after page loaded) open selenium ide (tools-> selenium ide). assure red button (looks record-button in video-tools) on right position in menubar (over "table / source"-tabs) activated (you can read message mouseover). while recording, each click on website want inspect creates automatically entry "table"-tab (a sort of simple script-command). try execute many actions can / need on website want crawl, cause each action gives element in html-code , helps later identify java-code!
3.2. after finishing "inspectation" mouseclicks, have save "test case" created right now. file (f) ->save test case: choose name wish , confirm save- process.
note: default storelocation test cases "mozilla- firefox"-folder on pc (common path: c:\programs\mozilla firefox).
3.3. export test case java-code eclipse:
!!!!! awsome feature of selenium ide !!!!!
now - after saving test case - go again in selenium ide to: file (f)-> export test case as: choose: java/junit 4/webdriver: again filechooser opens (default: firefox-folder) , can save "export-file" java-file. important: file ending has ".java" (e.g.: "ihatelowreputation.java"). copy / import eclipse-project. can open .java-file , inspect outwritten java code rigth webelements want find / choose / manipulate. can use feeling, how selenium webdriver commands in java has coded. copy required code-lines class.
_____________ , here solutioncode problem above:
package fixwrongentries; import java.io.bufferedreader; import java.io.file; import java.io.fileinputstream; import java.io.filewriter; import java.io.ioexception; import java.io.inputstream; import java.io.inputstreamreader; import java.sql.preparedstatement; import java.sql.resultset; import java.sql.sqlexception; import java.sql.statement; import java.util.arraylist; import java.util.concurrent.timeunit; import org.jsoup.jsoup; import org.jsoup.nodes.document; import org.jsoup.nodes.element; import org.jsoup.select.elements; import org.openqa.selenium.by; import org.openqa.selenium.nosuchelementexception; import org.openqa.selenium.webdriver; import org.openqa.selenium.webelement; import org.openqa.selenium.htmlunit.htmlunitdriver; import org.openqa.selenium.phantomjs.phantomjsdriver; import org.openqa.selenium.support.ui.select; import com.gargoylesoftware.htmlunit.scriptresult; import com.gargoylesoftware.htmlunit.webclient; import com.gargoylesoftware.htmlunit.html.htmlbutton; import com.gargoylesoftware.htmlunit.html.htmlform; import com.gargoylesoftware.htmlunit.html.htmlpage; import com.gargoylesoftware.htmlunit.html.htmlselect; import com.gargoylesoftware.htmlunit.html.htmlsubmitinput; import com.gargoylesoftware.htmlunit.html.htmltextinput; import data.db_steam_spiele; import data.spiel; public class solvebuttonclick_formsubmitt { public static void main(string[] args) { string agecheckbutton = "content in product may not appropriate ages, or may not appropriate viewing @ work."; string agecheckkonkret = "please enter birth date continue:"; string noreviews = "there no reviews product"; try { // turn off annoying htmlunit warnings java.util.logging.logger.getlogger("com.gargoylesoftware").setlevel(java.util.logging.level.off); // enabling javascript => true in brackets htmlunitdriver driver = new htmlunitdriver(true); // link agecheck typ 1 (simply button click) string url = "http://store.steampowered.com/app/324800/?snr=1_7_..."; // link agecheck typ 2 (fill out formular , submitt) //stng url = "http://store.steampowered.com/agecheck/app/72850/"; driver.get(url); // system.out.println(driver.findelement(by.cssselector("h2")).gettext()); system.out.println(driver.getcurrenturl()); /********************************************************************* * * agecheck typ 2 * *********************************************************************/ if(driver.findelement(by.cssselector("h2")).gettext().equals(agecheckkonkret)) { system.out.println("achtung: agecheck konkret!"); // fill out form age-specifications: new select(driver.findelement(by.name("ageday"))).selectbyvisibletext("18"); driver.manage().timeouts().implicitlywait(10, timeunit.seconds); new select(driver.findelement(by.name("agemonth"))).selectbyvisibletext("april"); driver.manage().timeouts().implicitlywait(10, timeunit.seconds); new select(driver.findelement(by.id("ageyear"))).selectbyvisibletext("1970"); driver.manage().timeouts().implicitlywait(10, timeunit.seconds); // click agecheck formular button: fortfahren driver.findelement(by.cssselector("a.btnv6_blue_hoverfade.btn_small > span")).click(); if(driver.findelement(by.id("app_reviews_hash")).gettext().contains(noreviews)) { system.out.println("keine reviews vorhanden!"); continue; } else if(!driver.findelement(by.id("app_reviews_hash")).gettext().contains(noreviews)) { string = driver.findelement(by.xpath("//div[@id='app_reviews_hash']/div[3]/div[2]/label")) .gettext(); string steampurchaser = driver.findelement(by .xpath("//div[@id='app_reviews_hash']/div[3]/div[2]/label[2]")).gettext(); string communityurl = driver.findelement(by.cssselector("a.btnv6_blue_hoverfade.btn_medium")) .getattribute("href"); } } /********************************************************************* * * agechecck type 1 * *********************************************************************/ else if(driver.findelement(by.cssselector("h2")).gettext().equals(agecheckbutton)) { system.out.println("achtung: agecheck button!"); driver.findelement(by.cssselector("a.btn_grey_white_innerfade.btn_medium > span")).click(); if(driver.findelement(by.id("app_reviews_hash")).gettext().contains(noreviews)) { system.out.println("keine reviews vorhanden!"); continue; } else if(!driver.findelement(by.id("app_reviews_hash")).gettext().contains(noreviews)) { string = driver.findelement(by.xpath("//div[@id='app_reviews_hash']/div[3]/div[2]/label")) .gettext(); string steampurchaser = driver.findelement(by .xpath("//div[@id='app_reviews_hash']/div[3]/div[2]/label[2]")).gettext(); string communityurl = driver.findelement(by.cssselector("a.btnv6_blue_hoverfade.btn_medium")) .getattribute("href"); } } /********************************************************************* * * no agecheck * *********************************************************************/ else { if(driver.findelement(by.id("app_reviews_hash")).gettext().contains(noreviews)) { system.out.println("keine reviews vorhanden!"); continue; } else if(!driver.findelement(by.id("app_reviews_hash")).gettext().contains(noreviews)) { string = driver.findelement(by.xpath("//div[@id='app_reviews_hash']/div[3]/div[2]/label")) .gettext(); string steampurchaser = driver.findelement(by .xpath("//div[@id='app_reviews_hash']/div[3]/div[2]/label[2]")).gettext(); string communityurl = driver.findelement(by.cssselector("a.btnv6_blue_hoverfade.btn_medium")) .getattribute("href"); } } } catch(throwable t) { system.out.println("fehlermeldung aufgefangen"); t.printstacktrace(); } } private static boolean iselementpresent(webdriver driver, by) { try { driver.findelement(by); return true; } catch(nosuchelementexception e) { return false; } } } i hope people simular problem.
Comments
Post a Comment