Comments on: Ruby Screen-Scraper in 60 Seconds

By: Daddy Fixes Everything » Blog Archive » Tutorial: Data Updates using YAML Files in Rails

Sat, 10 May 2008 04:10:33 +0000

[...] the parts you want. I tend to use XPaths because it’s so easy to find them with firefox (see: hpricot/firefox/firebug screen scraping tutorial). It finds my target table (by css selector), then parses through each element looking for the data [...]

By: Ilya Grigorik

Ilya Grigorik — Tue, 22 Apr 2008 03:44:26 +0000

Thanks Jimmy, I updated the link.

By: Jimmy Z

Jimmy Z — Mon, 21 Apr 2008 17:59:50 +0000

Harish Mallipeddi’s python scraping url changed to:

http://mallipeddi.tumblr.com/post/29578374

By: Ilya Grigorik

Ilya Grigorik — Mon, 07 Apr 2008 11:19:22 +0000

Thanks for sharing the tip Nick. What you’re seeing is expected behavior - that’s what XPath was designed to do!

By: Nick

Nick — Sun, 06 Apr 2008 22:11:29 +0000

I just started using this technique as part of a scraping/teach myself Ruby project, and I really like it. One issue I’ve come across that hasn’t been addressed in the comments: Firebug won’t specify the first element in a series, even if it’s necessary. So if your HTML is:


  Test 1

  Test 2

(not sure if this will display properly - it’s a div, with 2 nested divs inside, each with an H1 tag inside)
and you ask FireBug for the XPath of the first H1 tag, it will say “html/body/div/div/h1″. Ask hpricot for the inner_html, like this:

puts(doc/”html/body/div/div/h1″).inner_html

and you get “Test 1Test 2″ - the joined contents of both H1 tags. In order to get the first H1, the one you want, you need to change the XPath to “html/body/div/div[1]/h1″.

Hope that’s helpful to someone - thanks for a great technique!

By: Ilya Grigorik

Ilya Grigorik — Thu, 28 Feb 2008 02:54:06 +0000

Tim, it could be due to a broken DOM tree. Firefox, just like any other browser often ‘magically’ rebuilds broken DOM-trees on the fly. This could be the case here - Firebug is picking up the path from a fixed tree, whereas the actual source code does not contain that node. That would be my best guess.

By: TIm

TIm — Fri, 22 Feb 2008 10:05:40 +0000

Hey,
Thanks for the tip. I do have a question though. It looks as the the following code using the full xpath retrieved by firebug returns nil:

puts (doc/”/html/body/div[2]/div/div/blockquote/p”).inner_html

So in this case, firebug gave an xpath that didn’t actually function as predicted.

The code you used in your program (using a less specific xpath and selecting the first element from the elements of the array) returns the text as expected.

puts (doc/”blockquote/p”).first.inner_html

But both should work. Any explanation?
Thanks,
Tim

By: Scraping the SAQ: Hpricot and Xpath gotchas — Some French Guy

Scraping the SAQ: Hpricot and Xpath gotchas — Some French Guy — Wed, 20 Feb 2008 18:53:14 +0000

[...] play. Building a scraper to retrieve data from those files should be just as easy - see e.g. Ruby Screen-Scraper in 60 Seconds. Basically, copy the XPath for the data you want from firebug, and paste inside your script - and [...]

By: Ilya Grigorik

Ilya Grigorik — Sat, 12 Jan 2008 19:05:42 +0000

Jefe, I would start on Wikipedia (Screen Scraping), and follow the links from there.

By: jefe

jefe — Wed, 09 Jan 2008 20:22:25 +0000

OK I am not getting this, I have been trying to teach myself how to parse a table for info I can use or scrape a screen and havn’t grasped it yet. Does anyone know a tutorial that includes every step about parsing or scraping? php would be ok, I dont have any background in ruby or python.