Comments on: More on Screen Scraping http://www.yourhtmlsource.com/phdblog/2005/12/14/more-on-screen-scraping/ Researchin' the day away... Mon, 13 Oct 2008 00:12:32 +0000 http://wordpress.org/?v=2.5 By: Yahya Cheema http://www.yourhtmlsource.com/phdblog/2005/12/14/more-on-screen-scraping/#comment-36253 Yahya Cheema Mon, 29 Sep 2008 07:51:29 +0000 http://yourhtmlsource.com/phdblog/2005/12/14/more-on-screen-scraping/#comment-36253 Thanks Ross! I found this very informative and useful. Thanks Ross! I found this very informative and useful.

]]> By: Ross http://www.yourhtmlsource.com/phdblog/2005/12/14/more-on-screen-scraping/#comment-12 Ross Fri, 16 Dec 2005 23:23:19 +0000 http://yourhtmlsource.com/phdblog/2005/12/14/more-on-screen-scraping/#comment-12 Right, Græme recoded his scraper in Python and it ran well, so it looks like the currently crappy port of Beautiful Soup into Ruby is to blame. Right, Græme recoded his scraper in Python and it ran well, so it looks like the currently crappy port of Beautiful Soup into Ruby is to blame.

]]> By: Ross http://www.yourhtmlsource.com/phdblog/2005/12/14/more-on-screen-scraping/#comment-11 Ross Fri, 16 Dec 2005 00:47:26 +0000 http://yourhtmlsource.com/phdblog/2005/12/14/more-on-screen-scraping/#comment-11 I've spent the night putting together my parser, or at least a rough cut of it, for the Dublin Bus site. It's in pretty good shape now. Running on a page like the <a href="http://www.dublinbus.ie/your_journey/viewer.asp?route=46a" rel="nofollow">46A schedule</a>, which has a <em>frightening</em> number of nested tables, it extracts the times from each table and collates them into a Python list in just over one second (including the page fetch). The source page is pretty atrociously coded too, so that's not the problem. I'm not familiar with Ruby, so I don't know what could be slowing your one down so much. :-S I’ve spent the night putting together my parser, or at least a rough cut of it, for the Dublin Bus site. It’s in pretty good shape now. Running on a page like the 46A schedule, which has a frightening number of nested tables, it extracts the times from each table and collates them into a Python list in just over one second (including the page fetch). The source page is pretty atrociously coded too, so that’s not the problem. I’m not familiar with Ruby, so I don’t know what could be slowing your one down so much. :-S

]]> By: Graeme http://www.yourhtmlsource.com/phdblog/2005/12/14/more-on-screen-scraping/#comment-10 Graeme Thu, 15 Dec 2005 17:33:22 +0000 http://yourhtmlsource.com/phdblog/2005/12/14/more-on-screen-scraping/#comment-10 Nice article Ross. I looked into the Ruby version of Beautiful Soup (<a href="http://www.crummy.com/software/RubyfulSoup/">Rubiful Soup</a>) earlier today, but found it to suffer from large processing overheads (not far short of 60 seconds) when running it on a page with a months worth of flight information (presumably because it builds an complete object model of the page). What kind of noticible overheads (if any) did you experience when you were using the python version? What size of pages were you parsing? Back to regular expressions for the moment I think! Nice article Ross. I looked into the Ruby version of Beautiful Soup (Rubiful Soup) earlier today, but found it to suffer from large processing overheads (not far short of 60 seconds) when running it on a page with a months worth of flight information (presumably because it builds an complete object model of the page). What kind of noticible overheads (if any) did you experience when you were using the python version? What size of pages were you parsing?

Back to regular expressions for the moment I think!

]]> By: Ross http://www.yourhtmlsource.com/phdblog/2005/12/14/more-on-screen-scraping/#comment-9 Ross Thu, 15 Dec 2005 13:44:03 +0000 http://yourhtmlsource.com/phdblog/2005/12/14/more-on-screen-scraping/#comment-9 Those requirements seem ridiculously draconian, given that editors at the IMDb aren't even responsible for the majority of the content on the site. Akin to a book publisher stipulating that people may read their books only while patting their heads and saying a Hail Mary. They do say they'll grant <a href="http://www.imdb.com/help/show_article?conditions" rel="nofollow">express written consent</a> for some people to use robots, if asked nicely. Google had to ask for permission to crawl the site, apparently. ^o) Those requirements seem ridiculously draconian, given that editors at the IMDb aren’t even responsible for the majority of the content on the site. Akin to a book publisher stipulating that people may read their books only while patting their heads and saying a Hail Mary.

They do say they’ll grant express written consent for some people to use robots, if asked nicely. Google had to ask for permission to crawl the site, apparently. ^o)

]]> By: Aaron http://www.yourhtmlsource.com/phdblog/2005/12/14/more-on-screen-scraping/#comment-8 Aaron Wed, 14 Dec 2005 23:01:09 +0000 http://yourhtmlsource.com/phdblog/2005/12/14/more-on-screen-scraping/#comment-8 No scraping for me: FTP all the way (maybe?) http://www.imdb.com/help/show_leaf?usedatasoftware Anyone else interested in this should see if similar policies apply before blindly scraping content. No scraping for me: FTP all the way (maybe?)
http://www.imdb.com/help/show_leaf?usedatasoftware

Anyone else interested in this should see if similar policies apply before blindly scraping content.

]]> By: Aaron http://www.yourhtmlsource.com/phdblog/2005/12/14/more-on-screen-scraping/#comment-7 Aaron Wed, 14 Dec 2005 22:55:19 +0000 http://yourhtmlsource.com/phdblog/2005/12/14/more-on-screen-scraping/#comment-7 Nice info, very good starting point. I personally would like to figure out if IMDB will block too many queries from the same IP. Did you ever contact the sites you scrapped or just hoped for the best? I don't think a site like ugc.ie will do any fancy IP logging. Nice info, very good starting point. I personally would like to figure out if IMDB will block too many queries from the same IP. Did you ever contact the sites you scrapped or just hoped for the best? I don’t think a site like ugc.ie will do any fancy IP logging.

]]>