Python BeautifulSoup webcrawling: Formatting output -

-

the site trying crawl http://www.boxofficemojo.com/yearly/chart/?yr=2013&p=.htm. specific page i'm focusing on http://www.boxofficemojo.com/movies/?id=catchingfire.htm.

from page, having trouble 2 things. first thing "foreign gross" amount (under total lifetime grosses). got amount function:

def getforeign(item_url):     response = requests.get(item_url)     soup = beautifulsoup(response.content)     print soup.find(text="foreign:").find_parent("td").find_next_sibling("td").get_text(strip = true)

the problem is, can print amount out console, can't append these values list or write them csv file. previous data needed on site, got individual piece of information each movie , appended them 1 list, exported csv file.

how can "foreign gross" amount separate amount each movie? need change?

the second problem related getting list of actors/actresses each movie. have function:

def getactors(item_url):     source_code = requests.get(item_url)     plain_text = source_code.text     soup = beautifulsoup(plain_text)     tempactors = []     print soup.find(text="actors:").find_parent("tr").text[7:]

this prints out list of actors: jennifer lawrencejosh hutchersonliam hemsworthelizabeth banksstanley tucciwoody harrelsonphilip seymour hoffmanjeffrey wrightjena maloneamanda plummersam claflindonald sutherlandlenny kravitz - so.

i having same problem having foreign gross amount. i want each individual actor seperately, append them temporary list, , later append list full list of movies. did list of directors, since directors links, not of actors/actresses have html links, can't same. issue right there no space between each of actors.

why current functions not working, , how can fix them?

more code::

def spider(max_pages): page = 1 while page <= max_pages:     url = 'http://www.boxofficemojo.com/yearly/chart/?page=' + str(page) + '&view=releasedate&view2=domestic&yr=2013&p=.htm'     source_code = requests.get(url)     plain_text = source_code.text     soup = beautifulsoup(plain_text)     link in soup.select('td > b > font > a[href^=/movies/?]'):         href = 'http://www.boxofficemojo.com' + link.get('href')         details(href)          listofforeign.append(getforeign(href))          listofdirectors.append(getdirectors(href))         str(listofdirectors).replace('[','').replace(']','')          getactors(href)          title = link.string         listoftitles.append(title)     page

listofforeign = []

def getforeign(item_url):     s = urlopen(item_url).read()     soup = beautifulsoup(s)     return soup.find(text="foreign:").find_parent("td").find_next_sibling("td").get_text(strip = true)  def spider(max_pages):     page = 1     while page <= max_pages:         url = 'http://www.boxofficemojo.com/yearly/chart/?page=' + str(page) + '&view=releasedate&view2=domestic&yr=2013&p=.htm'         source_code = requests.get(url)         plain_text = source_code.text         soup = beautifulsoup(plain_text)         link in soup.select('td > b > font > a[href^=/movies/?]'):             href = 'http://www.boxofficemojo.com' + link.get('href')             listofforeign.append(getforeign(href))                     page += 1   print listofforeign

returns

traceback (most recent call last): file "c:/users/younjin/pycharmprojects/untitled/movies.py", line 75, in spider(1) file "c:/users/younjin/pycharmprojects/untitled/movies.py", line 29, in spider listofforeign.append(getforeign(href)) file "c:/users/younjin/pycharmprojects/untitled/movies.py", line 73, in getforeign return soup.find(text="foreign:").find_parent("td").find_next_sibling("td").get_text(strip = true) attributeerror: 'nonetype' object has no attribute 'find_parent'


Comments

Post a Comment

Popular posts from this blog

How has firefox/gecko HTML+CSS rendering changed in version 38? -

-
i have a little website header no longer renders correctly, since updating firefox 38.0.5 [the faceplant logo missing top right, , main logo no longer centred]. still renders expected in other browsers. how has engine's rendering changed in update? according mdn , list of changes is: css support ruby-position , ruby-align have been added , available default ( bug 1055676 , bug 1123917 , bug 1039006 ). the :unresolved pseudo-class has been implemented custom elements ( bug 1111633 ). the predefined style ethiopic-numeric uses space, instead of dot, suffix match recent change spec ( bug 1120721 ). css transitions on generated content (with ::before , ::after ) on both inline , block splits them start expected specification ( bug 1110277 ). the implementation of css logical properties made big progress. [...] how css transitions start has been modified match recent change of specifications, aiming @ having interoperable behavio...
Read more

android - CollapsingToolbarLayout: position the ExpandedText programmatically -

-
i using new android support design view collapsingtoolbarlayout. have need position expandedtitle programmatically. actually way set position expandedtitle in .xml file (example here: https://goo.gl/l5h5fc ) there no public api set text position programmatically , fields private . do know if it's possible without reflection or 'hacky' tricks? i have solved problem exposing properties via reflection ( https://gist.github.com/fpezzato/8e7d45112883dbbc5ffb ), works don't use trick in production code. @google: please no more private or final stuff! :)
Read more

Listeners to visualise results of load test in JMeter -

-
i want run load test in non-gui mode , save results in .csv file. (i have not modified .properties files, standard information saved) i plan load file , visualise results jmeter’s listeners. which listeners suited purpose? should avoid @ cost, might cause me misinterpret information? (e.g. in view results tree response data empty have - - not saved this. however, there has in fact been response. if saw listener might think there hasn’t been one. since want avoid this, listeners have gain information needed .csv file.) i’d know both jmeter listeners plugins. if running test in non-ui mode ideally should disable listeners in script. use '-l' option store test results .jtl extension. can later open jtl file jmeter. aggregate report or summary report should summarize test results , response time , throughput. if want plot graphs see trends on test duration can use http://jmeter-plugins.org/#/ standard , lib set. i suggest following listeners: response ...
Read more