Pastable version of Screen Scraping Google with Hpricot and Watir

<pre class='prettyprint' language='ruby'>require 'rubygems'

require 'hpricot'

require &quot;watir&quot;

#	Navigate to Google in a new IE instance

#	Search for the following:  pirates vs ninjas
  #	Iterate thought the first 10 pages of google

#	On each page collect the following

#		The blue page title

#		The green url

#	Output all the data into a file named test_output.txt
  #		Format  &quot;Title - URL&quot;

# Prepare Watir

url = &quot;http://www.google.com/&quot;

browser = Watir::Browser.new

browser.goto url

# Input our search term: &quot;Gap, Inc.&quot; and click search button

browser.text_field(:name, &quot;q&quot;).set &quot;pirates vs ninjas&quot;

browser.button(:name, &quot;btnG&quot;).click

results = Array.new

# We loop through 10 pages of results

for page in 1..10

# Bring Watir's HTML into Hpricot so we can easily process it

parsed = Hpricot(browser.html)

# Look in the LIs (class g)

results &lt;&lt; (parsed/&quot;li.g&quot;).map do |ele|

{

# Grab the title from the inner_text of A (Blue Text)

:title =&gt; ele.at(&quot;a&quot;).inner_text,

# Grab the URL from the innter_text of Google's citation (Green Text)

:url =&gt; (ele/&quot;//cite&quot;).first.inner_text

}

end

# Use Watir to head over to the next page unless we are done

browser.link(:text, (page+1).to_s).click unless page &gt;= 10

end

# Time to write to file

outfile = File.new(&quot;test_output.txt&quot;, &quot;w&quot;)

# Results are organized by page.

results.each do |page_results|

# Write each individual SERP's title and url

page_results.each do |serp|

# Clean the nasty - ## - stuff from the URL text

outfile.puts serp[:title] + &quot; - &quot; + serp[:url].gsub(/ -.*-/, '')

end

end</pre>
<a href="http://www.refactormycode.com/codes/673-screen-scraping-google-with-hpricot-and-watir" style="color:#fff" title="As seen on RefactorMyCode.com"><img alt="Small_logo" src="http://www.refactormycode.com/images/small_logo.gif" style="border:0" /></a>

Refactor :my => 'code'

Recent

Popular

Pastable version of Screen Scraping Google with Hpricot and Watir